* [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)
@ 2026-10-30 8:10 Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
` (5 more replies)
0 siblings, 6 replies; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
Dear maintainers,
This series continues the work of adding support for the USB device
controller (UDC) integrated into the Axiado AX3000 and AX3005 SoCs.
The controller is based on the Corigine USB 3.1 IP core, which is
dual-role capable (device and host) at the hardware level. Which role
is active depends on which driver is bound to it: this series binds
it in device role, while host role would instead use a standard xHCI
host driver - the hardware itself does not fix the role. Endpoint 0
shares the same command-ring / event-ring / per-endpoint transfer-ring
TRB hardware as every other endpoint, so the driver tracks control
transfer stages (SETUP/DATA/STATUS) explicitly instead of relying on a
dedicated hardware FIFO. Other upstream TRB/ring based UDC drivers,
such as cdns2 (whose file layout this series' structure is now modeled
on), share this same shape for the same reason.
Since v2, the driver has been restructured to follow the file layout
and conventions of drivers/usb/gadget/udc/cdns2, split into
core/gadget/ep0/ring source files with ftrace tracepoint support, and
brought in line with cdns2's static/global function ratio. See the
detailed changelog below.
The devicetree binding also picks up one item left over from the v2
review: the description now notes the underlying core is dual-role
capable (device and host), used here in device role.
Tested on an Axiado AX3005 EVK with a mass-storage + composite HID
(keyboard/mouse) gadget, including repeated bind/unbind/rebind cycles
and host-driven bus resets.
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
Changes in v4:
Ring and transfer management:
- Prevent stale event-ring dequeue updates after concurrent ring
reinitialization by tracking ring generations.
- Continue scheduling pending requests while transfer-ring space is
available.
- Avoid unsafe request-list iteration when completion callbacks drop
udc_lock.
- Clear stale request bookkeeping before rebuilding cancelled transfer
rings.
- Preserve transfer-ring cycle bits and Link TRBs when rewinding the
ring.
Endpoint 0 handling:
- Prevent NULL dereferences and queue corruption when reusing the shared
EP0 status request.
- Remove duplicate status-stage scheduling from GET_STATUS completion.
- Serialize shared status-request buffer access and completion
callbacks.
- Correct SET_FEATURE and CLEAR_FEATURE validation, recipient handling,
and function suspend options.
- Accept SET_ADDRESS(0) in the Default state and fix error handling.
- Correct SET_SEL payload parsing using struct usb_set_sel_req.
- Set the chain bit on non-final control data-stage TRBs.
- Stop EP0 and wait for hardware completion before cancelling requests
and reinitializing its transfer ring.
Gadget and endpoint operations:
- Prevent TRB interleaving between requests, premature ZLP generation,
and stalled transfers spanning multiple TDs.
- Fix transfer-ring allocation, resizing, and error cleanup to avoid
dangling pointers and concurrent DMA buffer access.
- Synchronize reset work with gadget callback disabling and unbinding.
- Correct last_trb tracking across ring wraparound and multi-TD
transfers.
- Use the correct dequeue cycle state when updating endpoint contexts.
- Wait for hardware endpoint disable and finalize endpoint state before
completing pending requests.
- Correct High-Speed interrupt and isochronous endpoint maxburst
handling.
- Fix port link-state resume handling and disconnect notifications.
- Revalidate endpoint state and EP0 setup identity after DMA mapping.
- Protect gadget start/stop state and register updates with udc_lock.
Core driver:
- Notify gadget drivers of disconnect before clearing endpoint state.
- Initialize endpoint structures, event rings, and EP0 before enabling
IRQs.
- Move gadget removal ahead of controller state changes during teardown.
- Serialize shutdown register updates and cancel pending reset work.
- Fix 32-bit build compatibility when reconstructing 64-bit DMA
addresses.
- Prevent overwriting pending controller commands while RUN is cleared.
- Replace SIMPLE_DEV_PM_OPS with DEFINE_SIMPLE_DEV_PM_OPS and
pm_sleep_ptr().
- Link to v3: https://patch.msgid.link/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com
Changes in v3:
Devicetree binding:
- Note that the underlying Corigine core is dual-role capable (device
and host), used here in device role.
- Drop the explicit maximum-speed property declaration. Krzysztof noted
in the v2 review that compatible already implies SuperSpeed
capability,
and testing confirms the property is not required.
- Reference the generic usb.yaml schema and replace additionalProperties
with unevaluatedProperties to permit standard USB properties while
rejecting unsupported properties.
- Validated the updated binding with dt_binding_check.
Driver structure (matching drivers/usb/gadget/udc/cdns2):
- Split crg_udc.c/crg_udc.h into crg-udc-core.c, crg-udc-gadget.c,
crg-udc-ep0.c, crg-udc-ring.c and crg-udc.h, one concern per file.
- Add ftrace tracepoint support (crg-udc-trace.h/.c) covering pullup,
doorbell, TRB queueing, event processing, control requests, and
request alloc/free/giveback, plus a decode-helper header
(crg-udc-debug.h) for the PORTSC/completion-code/TRB-type strings
used by the trace TP_printk()s.
- Make internal-only functions static and normalize the remaining
exported ones to a consistent crg_udc_ prefix.
- Reorganize #include lists so each file pulls in only the headers it
actually uses, instead of funneling every kernel header the driver
needs through the shared crg-udc.h.
- Match the SPDX/license comment style used elsewhere in the tree:
a line comment for the SPDX tag, block comment for the description
and copyright.
Bug fixes:
- Restore the VBUS-detect vendor-register write on udc_start(),
udc_stop(), remove() and shutdown(). Without it the controller never
considers itself powered and enumeration is unreliable.
- Restore the DATA_STAGE_XFER/DATA_STAGE_RECV cases dropped from the
EP0 status-stage completion handler. Without them, any control read
(e.g. GET_DESCRIPTOR) never arms its STATUS stage and the host times
out waiting for it.
- Fix the transfer-ring TRB-deferral check to defer building a TRB
only when the ring is actually full, not merely because another
request is already queued on the same endpoint. The stricter check
silently broke back-to-back completions on one endpoint, such as
mass storage's CSW immediately following its data stage.
- Restore the RUN-gated completion check in the hardware command
helper, so a command issued after the controller has already been
stopped (e.g. during udc_stop()) does not spin waiting on a
completion the hardware will never post.
- Call usb_gadget_udc_reset() on a bus reset so the composite/function
drivers correctly re-run set_alt() and re-enable their endpoints
after a reset, instead of composite silently treating a repeated
SET_CONFIGURATION as a no-op. This is deferred to a workqueue rather
than called directly from the port-status interrupt path, since the
synchronous call re-enters the function drivers and would otherwise
require dropping the driver lock in the middle of event processing.
- Clear the generic usb_ep.enabled flag and endpoint descriptors during
internal endpoint teardown and controller reset, preventing stale
endpoint state across sessions and allowing endpoints to be correctly
re-enabled after a new bind or reset.
Additional fixes from Sashiko review:
- Use dma_addr_t for DMA offset calculations and validate DMA address
translation failures before programming hardware registers.
- Reserve transfer-ring capacity and publish multi-TRB descriptors
only after all TRBs have been initialized.
- Track remaining TRBs and buffer length for requests spanning
multiple transfer descriptors.
- Advance the hardware dequeue pointer before completing canceled
requests, and restore the producer cycle state when rewinding rings.
- Detach pending requests before completing them during endpoint
teardown, and complete requests that fail during deferred queueing.
- Reduce the hardware reset timeout from 500 ms to 10 ms and release
the spinlock while polling for completion.
- Protect pullup register updates with the controller spinlock.
- Update the event-ring dequeue pointer before yielding the CPU.
- Unregister the interrupt handler before freeing DMA resources.
- Link to v2: https://patch.msgid.link/20260817-axiado-ax3000-usb-device-controller-v2-0-0ef033dd0a68@axiado.com
Changes in v2:
- Drop the dual (GPL-2.0+ OR MIT) license; the driver and header are now
GPL-2.0+ only.
- Register via module_platform_driver() and add MODULE_AUTHOR/MODULE_ALIAS
instead of the hand-written module_init/module_exit pair.
- Acquire clock, reset and PHY through the driver model
(devm_clk_get_optional, devm_reset_control_get_optional_exclusive,
devm_phy_optional_get), with teardown wired up via
devm_add_action_or_reset.
- Switch to devm_request_irq() and drop the manual free_irq() paths.
- Support multiple instances using an IDA (DEFINE_IDA/ida_alloc/ida_free)
instead of a file-static device counter.
- Read match data with device_get_match_data() and move the former
compile-time feature switches to DT properties (dis-u1u2-entry-quirk,
dis-u1u2-initiation-quirk, reinit-ep0-on-bus-reset).
- Set the DMA mask via dma_set_mask_and_coherent() with a 32-bit fallback,
and convert PM to SIMPLE_DEV_PM_OPS.
- Fix endianness handling for TRBs, events and endpoint contexts using
cpu_to_le32()/le32_to_cpu(), and decode setup packets without casting.
- Locking: pass the caller's IRQ flags through the completion paths so the
interrupt state is correctly restored across the unlock/lock window around
gadget callbacks.
- Properly free all coherent DMA allocations and add
dma_mapping_error() checks after dma_map_single()
- Fix DMA directions on the GET_STATUS/SET_SEL control transfers.
- Replace open-coded register busy-wait loops with
readl_poll_timeout_atomic() so they time out instead of spinning forever,
and return proper errnos (-EBUSY/-ETIMEDOUT/-EIO) instead of -1.
- Replace the custom SETF_VAR/GETF/MAKEF_VAR shift-and-mask macros with
FIELD_PREP/FIELD_GET (include linux/bitfield.h), and use BIT()/GENMASK()
and dma_wmb()/dma_rmb().
- Use DMA_MAPPING_ERROR instead of a private DMA_ADDR_INVALID.
- Remove the U-Boot-derived VBUS cable-detect routine, the VBUS-detect
kthread and its wait queue, dead defines and struct fields, the unused
interrupt-register-array macros, the mdelay()-based reset delays and the
bulk of the dev_dbg() tracing.
- Implement the pullup(is_on == 0) path and route suspend/shutdown through
pullup(0); mark crg_udc_ep_ops const and return IRQ_NONE from the shared
IRQ handler when the interrupt is not ours.
- Use dev_err_probe() with structured goto-based unwinding in probe.
- Reword the Kconfig help text for USB_CRG_UDC to name the crg_udc module.
- Link to v1: https://lore.kernel.org/r/20260202-axiado-ax3000-usb-device-controller-v1-0-45ce0a8b014f@axiado.com
To: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
To: Prasad Bolisetty <pbolisetty@axiado.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
To: Tzu-Hao Wei <twei@axiado.com>
To: Karthikeyan Mitran <kmitran@axiado.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Petar Stepanovic <pstepanovic@axiado.com>
Cc: linux-usb@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
---
Petar Stepanovic (6):
dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
usb: gadget: udc: crg: Add hardware definitions and structs
usb: gadget: udc: crg: Add ring and transfer management
usb: gadget: udc: crg: Add endpoint 0 handling
usb: gadget: udc: crg: Add gadget and endpoint operations
usb: gadget: udc: crg: Add core driver and build integration
.../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 +
MAINTAINERS | 9 +
drivers/usb/gadget/udc/Kconfig | 2 +
drivers/usb/gadget/udc/Makefile | 1 +
drivers/usb/gadget/udc/crg/Kconfig | 16 +
drivers/usb/gadget/udc/crg/Makefile | 9 +
drivers/usb/gadget/udc/crg/crg-udc-core.c | 723 ++++++
drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 ++
drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1133 ++++++++++
drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 2377 ++++++++++++++++++++
drivers/usb/gadget/udc/crg/crg-udc-ring.c | 668 ++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.c | 9 +
drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 ++
drivers/usb/gadget/udc/crg/crg-udc.h | 729 ++++++
14 files changed, 6052 insertions(+)
---
base-commit: 870dd89bff3d3b706bfce9ff708b826408b789a9
change-id: 20260128-axiado-ax3000-usb-device-controller-156d3f1840c9
Best regards,
--
Petar Stepanovic <pstepanovic@axiado.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:14 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
` (4 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.
The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
.../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
MAINTAINERS | 8 +++
2 files changed, 77 insertions(+)
diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..b9f7f4879ee0
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+ - Petar Stepanovic <pstepanovic@axiado.com>
+ - Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+ - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+ Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+ Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+ with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+ - $ref: usb.yaml#
+
+properties:
+ compatible:
+ oneOf:
+ - const: axiado,ax3000-udc
+ - items:
+ - const: axiado,ax3005-udc
+ - const: axiado,ax3000-udc
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ phys:
+ maxItems: 1
+
+required:
+ - compatible
+ - reg
+ - interrupts
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ usb@81200000 {
+ compatible = "axiado,ax3000-udc";
+ reg = <0x0 0x81200000 0x0 0x100000>;
+ interrupt-parent = <&gic500>;
+ interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&reset 0>;
+ phys = <&usb_phy>;
+ };
+ };
diff --git a/MAINTAINERS b/MAINTAINERS
index 65e8a4b5c90b..58c5c8e00870 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4508,6 +4508,14 @@ F: Documentation/devicetree/bindings/spi/axiado,ax3000-spi.yaml
F: drivers/spi/spi-axiado.c
F: drivers/spi/spi-axiado.h
+AXIADO USB UDC DRIVER
+M: Petar Stepanovic <pstepanovic@axiado.com>
+M: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+M: Prasad Bolisetty <pbolisetty@axiado.com>
+L: linux-usb@vger.kernel.org
+S: Supported
+F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
L: platform-driver-x86@vger.kernel.org
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:21 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
` (3 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
MAINTAINERS | 1 +
drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 ++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 +++++++
drivers/usb/gadget/udc/crg/crg-udc.h | 729 +++++++++++++++++++++++++++++
4 files changed, 1037 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 58c5c8e00870..d5b6a04a95ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4515,6 +4515,7 @@ M: Prasad Bolisetty <pbolisetty@axiado.com>
L: linux-usb@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F: drivers/usb/gadget/udc/crg/
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+ switch (type) {
+ case TRB_TYPE_XFER_NORMAL:
+ return "Normal";
+ case TRB_TYPE_XFER_DATA_STAGE:
+ return "Data Stage";
+ case TRB_TYPE_XFER_STATUS_STAGE:
+ return "Status Stage";
+ case TRB_TYPE_XFER_DATA_ISOCH:
+ return "Isoch";
+ case TRB_TYPE_LINK:
+ return "Link";
+ case TRB_TYPE_NO_OP:
+ return "No-Op";
+ case TRB_TYPE_EVT_TRANSFER:
+ return "Transfer Event";
+ case TRB_TYPE_EVT_CMD_COMPLETION:
+ return "Command Completion Event";
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ return "Port Status Change Event";
+ case TRB_TYPE_EVT_MFINDEX_WRAP:
+ return "MFINDEX Wrap Event";
+ case TRB_TYPE_EVT_SETUP_PKT:
+ return "Setup Packet Event";
+ default:
+ return "Reserved";
+ }
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+ switch (code) {
+ case CMPL_CODE_INVALID:
+ return "Invalid";
+ case CMPL_CODE_SUCCESS:
+ return "Success";
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ return "Data Buffer Error";
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ return "Babble Detected";
+ case CMPL_CODE_USB_TRANS_ERR:
+ return "USB Transaction Error";
+ case CMPL_CODE_TRB_ERR:
+ return "TRB Error";
+ case CMPL_CODE_TRB_STALL:
+ return "Stall";
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ return "Invalid Stream Type";
+ case CMPL_CODE_SHORT_PKT:
+ return "Short Packet";
+ case CMPL_CODE_RING_UNDERRUN:
+ return "Ring Underrun";
+ case CMPL_CODE_RING_OVERRUN:
+ return "Ring Overrun";
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ return "Event Ring Full";
+ case CMPL_CODE_STOPPED:
+ return "Stopped";
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ return "Stopped - Length Invalid";
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ return "Isoch Buffer Overrun";
+ case CMPL_CODE_PROTOCOL_STALL:
+ return "Protocol Stall";
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ return "Setup Tag Mismatch";
+ case CMPL_CODE_HALTED:
+ return "Halted";
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ return "Halted - Length Invalid";
+ case CMPL_CODE_DISABLED:
+ return "Disabled";
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ return "Disabled - Length Invalid";
+ default:
+ return "Unknown";
+ }
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+ u32 portsc)
+{
+ scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+ portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+ FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+ (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+ (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+ (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+ (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+ struct crg_transfer_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size,
+ "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+ crg_udc_trb_type_string(type),
+ FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+ dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+ struct crg_event_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+ crg_udc_trb_type_string(type),
+ crg_udc_cmpl_code_string(
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+ FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+ FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+ return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+ TP_STRUCT__entry(__field(int, set)),
+ TP_fast_assign(__entry->set = set;),
+ TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+ TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+ TP_STRUCT__entry(__string(text, msg)),
+ TP_fast_assign(__assign_str(text);),
+ TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+ TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+ TP_ARGS(udc_ep, halt, wedge),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(u8, halt) __field(u8, wedge)),
+ TP_fast_assign(__assign_str(name); __entry->halt = halt;
+ __entry->wedge = wedge;),
+ TP_printk("%s: %s%s", __get_str(name),
+ __entry->halt ? "halted" : "cleared",
+ __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+ TP_STRUCT__entry(__field(u32, portsc)),
+ TP_fast_assign(__entry->portsc = portsc;),
+ TP_printk("%s",
+ crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+ TP_ARGS(crg_udc, dci),
+ TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+ __field(int, dci)),
+ TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+ TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+ TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+ __field(u16, wValue) __field(u16, wIndex)
+ __field(u16, wLength)),
+ TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+ __entry->bRequest = ctrl->bRequest;
+ __entry->wValue = le16_to_cpu(ctrl->wValue);
+ __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+ __entry->wLength = le16_to_cpu(ctrl->wLength);),
+ TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->bRequestType,
+ __entry->bRequest, __entry->wValue,
+ __entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+ TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(struct crg_transfer_trb *, trb)
+ __dynamic_array(char, decoded,
+ CRG_MSG_MAX)),
+ TP_fast_assign(__assign_str(name); __entry->trb = trb;
+ crg_udc_decode_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+ __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+ crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+ TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+ __dynamic_array(char, decoded, CRG_MSG_MAX)),
+ TP_fast_assign(__entry->trb = trb;
+ crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req),
+ TP_STRUCT__entry(
+ __string(name, udc_ep->usb_ep.name) __field(
+ struct usb_request *, request) __field(void *, buf)
+ __field(unsigned int, actual) __field(unsigned int,
+ length)
+ __field(int, status) __field(dma_addr_t, dma)
+ __field(int, zero) __field(int,
+ short_not_ok)
+ __field(int, no_interrupt)),
+ TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+ __entry->buf = udc_req->usb_req.buf;
+ __entry->actual = udc_req->usb_req.actual;
+ __entry->length = udc_req->usb_req.length;
+ __entry->status = udc_req->usb_req.status;
+ __entry->dma = udc_req->usb_req.dma;
+ __entry->zero = udc_req->usb_req.zero;
+ __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+ __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+ TP_printk(
+ "%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+ __get_str(name), __entry->request, __entry->buf,
+ __entry->actual, __entry->length, __entry->status,
+ &__entry->dma, __entry->zero ? "Z" : "z",
+ __entry->short_not_ok ? "S" : "s",
+ __entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..246822219a8a
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,729 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+ u32 capability;
+ u32 resv0[3];
+
+ u32 config0;
+ u32 config1;
+ u32 resv1[2];
+
+ u32 control;
+ u32 status;
+ u32 dcbaplo;
+ u32 dcbaphi;
+ u32 portsc;
+ u32 u3portpmsc;
+ u32 u2portpmsc;
+ u32 u3portli;
+
+ u32 doorbell;
+ u32 mfindex;
+ u32 ptmctrl;
+ u32 resv3[5];
+
+ u32 ep_enable;
+ u32 ep_running;
+ u32 resv4[2];
+
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 cmd_control;
+ u32 resv5[1];
+
+ u32 odb_capability;
+ u32 resv6[3];
+ u32 odb_config[8];
+
+ u32 debug0;
+};
+
+struct crg_uicr {
+ u32 iman;
+ u32 imod;
+ u32 erstsz;
+ u32 resv0;
+
+ u32 erstbalo;
+ u32 erstbahi;
+ u32 erdplo;
+ u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK \
+ (CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+ CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+ TRB_TYPE_RSVD = 0,
+ TRB_TYPE_XFER_NORMAL,
+ TRB_TYPE_RSVD2,
+ TRB_TYPE_XFER_DATA_STAGE,
+ TRB_TYPE_XFER_STATUS_STAGE,
+ TRB_TYPE_XFER_DATA_ISOCH,
+ TRB_TYPE_LINK,
+ TRB_TYPE_RSVD7,
+ TRB_TYPE_NO_OP,
+
+ TRB_TYPE_EVT_TRANSFER = 32,
+ TRB_TYPE_EVT_CMD_COMPLETION = 33,
+ TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+ TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+ TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+ CMPL_CODE_INVALID = 0,
+ CMPL_CODE_SUCCESS,
+ CMPL_CODE_DATA_BUFFER_ERR,
+ CMPL_CODE_BABBLE_DETECTED_ERR,
+ CMPL_CODE_USB_TRANS_ERR,
+ CMPL_CODE_TRB_ERR,
+ CMPL_CODE_TRB_STALL,
+ CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+ CMPL_CODE_SHORT_PKT = 13,
+ CMPL_CODE_RING_UNDERRUN,
+ CMPL_CODE_RING_OVERRUN,
+ CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+ CMPL_CODE_STOPPED = 26,
+ CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+ CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+ CMPL_CODE_PROTOCOL_STALL = 192,
+ CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+ CMPL_CODE_HALTED = 194,
+ CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+ CMPL_CODE_DISABLED = 196,
+ CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+ EP_TYPE_INVALID = 0,
+ EP_TYPE_ISOCH_OUTBOUND,
+ EP_TYPE_BULK_OUTBOUND,
+ EP_TYPE_INTR_OUTBOUND,
+ EP_TYPE_INVALID2,
+ EP_TYPE_ISOCH_INBOUND,
+ EP_TYPE_BULK_INBOUND,
+ EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+struct crg_erst {
+ /* 64-bit event ring segment address */
+ __le32 seg_addr_lo;
+ __le32 seg_addr_hi;
+ __le32 seg_size;
+ /* Set to zero */
+ __le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+ CRG_CMD_INIT_EP0 = 0,
+ CRG_CMD_UPDATE_EP0_CFG = 1,
+ CRG_CMD_SET_ADDR = 2,
+ CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+ CRG_CMD_CONFIG_EP = 4,
+ CRG_CMD_SET_HALT = 5,
+ CRG_CMD_CLEAR_HALT = 6,
+ CRG_CMD_RESET_SEQNUM = 7,
+ CRG_CMD_STOP_EP = 8,
+ CRG_CMD_SET_TR_DQPTR = 9,
+ CRG_CMD_FORCE_FLOW_CONTROL = 10,
+ CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+ EP_STATE_DISABLED = 0,
+ EP_STATE_RUNNING = 1,
+ EP_STATE_HALTED = 2,
+ EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+ WAIT_FOR_SETUP = 0,
+ SETUP_PKT_PROCESS_IN_PROGRESS,
+ DATA_STAGE_XFER,
+ DATA_STAGE_RECV,
+ STATUS_STAGE_XFER,
+ STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+ CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+ CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+ void *cpu_addr;
+ dma_addr_t dma_addr;
+ u32 len;
+};
+
+struct crg_sel_value {
+ u16 u2_pel_value;
+ u16 u2_sel_value;
+ u8 u1_pel_value;
+ u8 u1_sel_value;
+};
+
+struct crg_udc_request {
+ struct usb_request usb_req;
+ struct list_head queue;
+ bool mapped;
+ /*
+ * Direction used for the DMA mapping call that produced the mapping
+ * tracked above. ep0 is bidirectional and its
+ * descriptor address is always 0, so usb_endpoint_dir_in() on the
+ * endpoint cannot tell an unmap apart from a control IN or OUT
+ * transfer; the direction actually used to map is recorded here
+ * instead and reused as-is at unmap time.
+ */
+ enum dma_data_direction map_dir;
+ struct scatterlist *sg;
+ unsigned int sg_index;
+ u32 sg_offset;
+ u64 buff_len_left;
+ u32 trbs_needed;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+ bool all_trbs_queued;
+ /* A TD has been published and its completion event is outstanding. */
+ bool td_pending;
+};
+
+struct crg_udc_ep {
+ struct usb_ep usb_ep;
+
+ struct crg_buffer_info tran_ring_info;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+
+ struct crg_transfer_trb *enq_pt;
+ struct crg_transfer_trb *deq_pt;
+ u8 pcs;
+
+ char name[10];
+ u8 dci;
+ struct list_head queue;
+ const struct usb_endpoint_descriptor *desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc;
+ bool tran_ring_full;
+ bool configuring;
+ u32 generation;
+ struct crg_gadget_dev *crg_udc;
+
+ enum crg_ep_state ep_state;
+
+ bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+ struct crg_buffer_info erst;
+ struct crg_erst *p_erst;
+ struct crg_buffer_info event_ring;
+ struct crg_event_trb *evt_dq_pt;
+ /* Changed on every ring reinitialization under udc_lock. */
+ u32 generation;
+ u8 ccs;
+ struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+ EP0_STATE_IDLE = 0,
+ EP0_STATE_ABORTING,
+ EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+ void __iomem *mmio_virt_base;
+ struct crg_uccr __iomem *uccr;
+ struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+ /* udc_lock device lock */
+ spinlock_t udc_lock;
+
+ struct device *dev;
+ struct clk *clk;
+ struct phy *phy;
+
+ struct usb_gadget gadget;
+ struct usb_gadget_driver *gadget_driver;
+ /* Speed selected for the currently bound gadget driver. */
+ enum usb_device_speed max_speed;
+
+ int irq;
+
+ struct usb_endpoint_descriptor ep0_desc;
+
+ struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+ struct crg_buffer_info ep_cx;
+ struct crg_ep_cx *p_epcx;
+
+ struct crg_udc_event udc_event[CRG_RING_NUM];
+
+ struct crg_udc_request *status_req;
+ /* Protected by udc_lock, remains set during a data-stage giveback. */
+ bool status_req_callback_pending;
+ struct crg_sel_value sel_value;
+ void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+ enum crg_setup_status setup_status;
+ /* EP0 rejects new requests while the previous control transfer is
+ * being aborted and its completion callbacks are being given back.
+ */
+ enum crg_ep0_state ep0_state;
+
+ u8 device_state;
+ u16 dev_addr;
+ u8 setup_tag;
+ u8 set_tm;
+
+ u32 num_enabled_eps;
+
+ bool connected;
+ bool pullup_on;
+ bool async_callbacks;
+ /*
+ * Set while crg_udc_reset() has udc_lock dropped and is polling for
+ * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+ * on another endpoint during that window is rejected instead of
+ * being built into a ring and doorbell-rung while the controller is
+ * still mid-reset.
+ */
+ bool resetting;
+
+ bool u2_rwe;
+ bool feature_u1_enable;
+ bool feature_u2_enable;
+
+ bool portsc_on_reconnecting;
+ bool u12_forbidden;
+ bool u12_initiate_forbidden;
+ bool reinit_ep0_on_bus_reset;
+
+ /*
+ * usb_gadget_udc_reset() re-enters the gadget/function drivers
+ * synchronously. Calling it directly from the port-status IRQ path
+ * would require dropping udc_lock mid-event-processing, opening a
+ * window for a second port-status event to run crg_udc_reinit()
+ * again while the first event's SET_CONFIGURATION/set_alt() work is
+ * still landing on another thread, tearing the just-enabled
+ * endpoints back down. Defer the notification to this work item so
+ * the IRQ-thread's event processing never has to drop the lock.
+ */
+ struct work_struct udc_reset_work;
+};
+
+/* Shared prototypes */
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1);
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT \
+ (CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+ CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+ u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:29 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
` (2 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ring.c | 668 ++++++++++++++++++++++++++++++
1 file changed, 668 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..f1de8893b790
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,668 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+ struct crg_udc_request *udc_req;
+
+ if (!list_empty(&udc_ep->queue)) {
+ udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
+ queue);
+
+ if (udc_req->first_trb)
+ udc_ep->deq_pt = udc_req->first_trb;
+ else
+ udc_ep->deq_pt = udc_ep->enq_pt;
+ } else {
+ udc_ep->deq_pt = udc_ep->enq_pt;
+ }
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+ unsigned int num_trbs = 0;
+
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ }
+
+ return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ u64 remaining = udc_req->usb_req.length;
+ unsigned int i;
+ unsigned int num_trbs = 0;
+
+ for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+ dma_addr_t addr;
+ u64 len;
+
+ if (!sg_dma_len(sg))
+ continue;
+ if (!remaining)
+ break;
+
+ len = min_t(u64, sg_dma_len(sg), remaining);
+ addr = sg_dma_address(sg);
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ remaining -= chunk;
+ }
+ }
+
+ return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+ struct crg_event_trb *event)
+{
+ dma_addr_t dma_addr = 0;
+ unsigned long seg_offset;
+
+ if (!udc_event || !event)
+ return DMA_MAPPING_ERROR;
+
+ if ((void *)event < udc_event->event_ring.cpu_addr ||
+ (void *)event >=
+ udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+ return DMA_MAPPING_ERROR;
+
+ seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+ dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+ return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+ bool erst_allocated = false;
+
+ buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+ if (!udc_event->erst.cpu_addr) {
+ udc_event->erst.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->erst.cpu_addr)
+ return -ENOMEM;
+ erst_allocated = true;
+ } else {
+ mapping = udc_event->erst.dma_addr;
+ }
+
+ udc_event->erst.len = buff_length;
+ udc_event->erst.dma_addr = mapping;
+ udc_event->p_erst = udc_event->erst.cpu_addr;
+
+ buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+ if (!udc_event->event_ring.cpu_addr) {
+ udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->event_ring.cpu_addr) {
+ ret = -ENOMEM;
+ goto err_free_erst;
+ }
+ } else {
+ mapping = udc_event->event_ring.dma_addr;
+ }
+
+ udc_event->event_ring.len = buff_length;
+ udc_event->event_ring.dma_addr = mapping;
+ /* Invalidate event handlers which dropped udc_lock during reset. */
+ udc_event->generation++;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+ udc_event->evt_seg0_last_trb =
+ (struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+ (CRG_EVENT_RING_SIZE - 1);
+
+ udc_event->ccs = 1;
+
+ udc_event->p_erst->seg_addr_lo =
+ cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_addr_hi =
+ cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+ udc_event->p_erst->rsvd = cpu_to_le32(0);
+ /* Ensure that event ring is updated */
+ dma_wmb();
+ /* clear the event ring, to avoid hw unexpected ops
+ * because of dirty data
+ */
+ memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+ /* Ensure that event ring is clear before programming
+ * ERSTBA/ERSTSZ/ERDP
+ */
+ dma_wmb();
+
+ writel(CRG_ERST_SIZE, &uicr->erstsz);
+ writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+ writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+ writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+ writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+ /* Interrupt moderation: 4000 * 250ns = 1ms */
+ writel(4000, &uicr->imod);
+ return 0;
+
+err_free_erst:
+ /* Only free what this call allocated; a pre-existing erst is owned
+ * by an earlier successful init and released by crg_udc_free_resources().
+ */
+ if (erst_allocated) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ }
+ return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+
+ udc_event = &crg_udc->udc_event[index];
+ if (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ return true;
+ }
+
+ return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ if (!is_event_ring_x_empty(crg_udc, i)) {
+ dev_err(crg_udc->dev, "event ring not empty\n");
+ return false;
+ }
+ }
+ return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep)
+{
+ if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+ return false;
+ if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+ return true;
+ if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+ return true;
+ return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ u32 val;
+ dma_addr_t erdp;
+ int ret;
+ u32 generation;
+ unsigned int processed = 0;
+
+ if (!uicr)
+ return -ENODEV;
+
+ val = readl(&uicr->iman);
+ val |= CRG_U3DC_IMAN_INT_PEND;
+ writel(val, &uicr->iman);
+
+ udc_event = &crg_udc->udc_event[index];
+ generation = udc_event->generation;
+ while (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ break;
+ /* Read the payload only after observing device ownership. */
+ dma_rmb();
+
+ ret = crg_udc_handle_event(crg_udc, event, flags);
+ /* The giveback callback may release the lock and reset the ring. */
+ if (unlikely(generation != udc_event->generation))
+ return 0;
+ if (ret == -ECONNRESET) {
+ /* The disconnect path rebuilt and reprogrammed the ring. */
+ return 0;
+ }
+ if (ret < 0)
+ dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+ ret);
+
+ /* Consume malformed events so they cannot pin the ring head. */
+ if (event == udc_event->evt_seg0_last_trb) {
+ udc_event->ccs = udc_event->ccs ? 0 : 1;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+ } else {
+ udc_event->evt_dq_pt++;
+ }
+
+ /*
+ * udc_event state is only re-read from crg_udc, never cached
+ * across this drop, so releasing and reacquiring the lock
+ * here is safe and lets other udc_lock users, and this CPU's
+ * other interrupts, make progress during a busy event ring.
+ */
+ if (++processed % CRG_EVENT_RING_BATCH == 0) {
+ erdp = event_trb_virt_to_dma(udc_event,
+ udc_event->evt_dq_pt);
+ if (!dma_mapping_error(crg_udc->dev, erdp)) {
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ cond_resched();
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (unlikely(generation != udc_event->generation))
+ return 0;
+ }
+ }
+
+ /* update dequeue pointer */
+ erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+ if (dma_mapping_error(crg_udc->dev, erdp))
+ return -EINVAL;
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+ return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+ struct crg_udc_request *req;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 generation = crg_udc->udc_event[0].generation;
+ u32 ep_generation = udc_ep_ptr->generation;
+ int ret;
+
+restart:
+ /* A giveback can reset/disable the endpoint while udc_lock is dropped. */
+ if (crg_udc->resetting || !udc_ep_ptr->desc ||
+ udc_ep_ptr->ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
+ if (req->all_trbs_queued)
+ continue;
+
+ /* A large request must finish its current TD before the next
+ * TD (or the next USB request) can be submitted: the event
+ * handler uses all_trbs_queued to complete the USB request.
+ */
+ if (req->td_pending)
+ return 0;
+
+ ret = crg_udc_build_td(udc_ep_ptr, req);
+ if (ret == -ENOSPC || ret == -EBUSY)
+ return ret;
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to build pending transfer: %d\n", ret);
+ /* Giveback removes req and drops udc_lock: callbacks can
+ * remove or free every saved request-list iterator.
+ */
+ crg_udc_req_done(udc_ep_ptr, req, ret);
+ if (generation != crg_udc->udc_event[0].generation ||
+ ep_generation != udc_ep_ptr->generation)
+ return 0;
+ goto restart;
+ }
+
+ if (udc_ep_ptr->tran_ring_full || !req->all_trbs_queued)
+ return 0;
+ }
+ return 0;
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt)
+{
+ u32 i = 0;
+
+ if (enq_pt == dq_pt) {
+ /* ring is empty */
+ return num_trbs - 2;
+ }
+
+ while (enq_pt != dq_pt) {
+ i++;
+
+ enq_pt++;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK)
+ enq_pt = p_ring;
+
+ if (i > num_trbs)
+ break;
+ }
+
+ return i - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+ /* TRB_Transfer_Length
+ * For USB_DIR_OUT, this field is the number of data bytes expected
+ * from xhc. For USB_DIR_IN, this field is the number of data bytes
+ * the device will send.
+ */
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb)
+{
+ u32 dw = 0;
+
+ link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+ link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+ link_trb->dw2 = cpu_to_le32(0);
+
+ FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+ if (toggle)
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+ else
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+ /* Publish ownership only after the link address is visible. */
+ dma_wmb();
+ link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall)
+{
+ u32 tmp, dir = 0;
+
+ /* There are some cases where crg_udc_setup_status_trb() is called with
+ * usb_req set to NULL.
+ */
+ p_trb->dw0 = cpu_to_le32(0);
+ p_trb->dw1 = cpu_to_le32(0);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+ dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+ FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+ FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+ if (b_setup_stage)
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+ if (b_isoc) {
+ FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+ FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+ }
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+ struct crg_udc_request *next_req;
+ u8 new_pcs, clear_pcs;
+
+ if (!trb)
+ return;
+
+ /* Rewind the producer to the cancelled request. These TRBs and
+ * everything queued after them will be discarded and rebuilt.
+ */
+ new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+ clear_pcs = new_pcs;
+ while (trb != udc_ep_ptr->enq_pt) {
+ /* Zero is not always invalid: after a wrap the consumer
+ * expects cycle 0. Invalidate each slot for its cycle pass.
+ */
+ le32p_replace_bits(&trb->dw3, clear_pcs ^ 1,
+ TRB_CYCLE_BIT_MASK);
+ trb++;
+ if (trb == udc_ep_ptr->last_trb) {
+ trb = udc_ep_ptr->first_trb;
+ clear_pcs ^= 1;
+ }
+ }
+
+ udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+ udc_ep_ptr->pcs = new_pcs;
+ udc_ep_ptr->tran_ring_full = false;
+ /* Do not overwrite the Link TRB: a previous active TD may still
+ * require its cycle/chain bits to traverse the ring.
+ */
+ dma_wmb();
+
+ /* Invalidate the bookkeeping of ALL subsequent requests before
+ * attempting to rebuild any: rebuilding may stop when space runs out.
+ * The caller gives back the cancelled request before rebuilding.
+ */
+ next_req = udc_req_ptr;
+ list_for_each_entry_continue(next_req, &udc_ep_ptr->queue, queue) {
+ next_req->usb_req.status = -EINPROGRESS;
+ next_req->usb_req.actual = 0;
+ crg_udc_clear_req_container(next_req);
+ }
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address)
+{
+ dma_addr_t offset;
+ struct crg_transfer_trb *trb_virt;
+
+ if (lower_32_bits(address) & 0xf)
+ return NULL;
+
+ offset = address - udc_ep->tran_ring_info.dma_addr;
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return NULL;
+ offset = offset / sizeof(struct crg_transfer_trb);
+ trb_virt = udc_ep->first_trb + offset;
+ return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb)
+{
+ unsigned long offset;
+ int trb_idx;
+ dma_addr_t dma_addr = 0;
+
+ trb_idx = trb - udc_ep->first_trb;
+ if (unlikely(trb_idx < 0))
+ return DMA_MAPPING_ERROR;
+
+ offset = trb_idx * sizeof(*trb);
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return DMA_MAPPING_ERROR;
+ dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+ return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep)
+{
+ u32 deq_pt_lo = le32_to_cpu(event->dw0);
+ u32 deq_pt_hi = le32_to_cpu(event->dw1);
+ u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ struct crg_transfer_trb *deq_pt;
+
+ deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+ if (!deq_pt)
+ return;
+ deq_pt++;
+ if (deq_pt == udc_ep->last_trb)
+ deq_pt = udc_ep->first_trb;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+ deq_pt = udc_ep->first_trb;
+
+ udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+ u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+ (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+ return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause)
+{
+ struct crg_transfer_trb *trb = udc_req->first_trb;
+ u32 actual = 0;
+ u32 guard = 0;
+ u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
+
+ if (!trb || !pause)
+ return 0;
+
+ while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+ u32 len;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_LINK) {
+ trb = udc_ep->first_trb;
+ continue;
+ }
+ len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+ actual += len;
+ trb++;
+ if (trb == udc_ep->last_trb)
+ trb = udc_ep->first_trb;
+ }
+
+ return min(actual, udc_req->usb_req.length);
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (2 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:40 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1133 ++++++++++++++++++++++++++++++
1 file changed, 1133 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..073d9090dd07
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1133 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+ if (default_value) {
+ udc_req_ptr = crg_udc->status_req;
+ /* The shared request may still be in its data-stage callback. */
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue)) {
+ dev_err(crg_udc->dev, "EP0 status request unavailable\n");
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return;
+ }
+ udc_req_ptr->usb_req.length = 0;
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = NULL;
+ } else {
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ }
+
+ crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+ udc_ep_ptr->pcs, set_addr, stall);
+
+ enq_pt++;
+
+ /* check if we are at end of trb segment. If so, update
+ * pcs and enq for next segment
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ udc_ep_ptr->enq_pt = enq_pt;
+
+ crg_udc_knock_doorbell(crg_udc, 0);
+
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 tmp;
+ int ret;
+
+ /*
+ * A new SETUP is allowed to abort the current control transfer.
+ * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+ * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+ * callback, so the callback may re-enter the UDC during the nuke.
+ */
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+ /*
+ * Stop EP0 before touching the transfer ring. The controller may still
+ * own the old TRBs even when no EP0 request is running from the gadget's
+ * point of view. Do not clear the DMA-visible ring until the hardware
+ * reports that EP0 is no longer running.
+ */
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to stop EP0 for SETUP abort: %d\n", ret);
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+ 10, 10000);
+ if (ret) {
+ dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ udc_ep0->ep_state = EP_STATE_STOPPED;
+ /* Complete requests belonging to the aborted control transfer. */
+ crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+
+ if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -ENODEV;
+ }
+
+ /*
+ * The controller has stopped consuming EP0 TRBs. It is now safe to
+ * discard the old transfer descriptors and restart the ring from its
+ * beginning. No ring memory is freed here.
+ */
+ memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+ udc_ep0->enq_pt = udc_ep0->first_trb;
+ udc_ep0->deq_pt = udc_ep0->first_trb;
+ udc_ep0->pcs = 1;
+ udc_ep0->tran_ring_full = false;
+ crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+ udc_ep0->tran_ring_info.dma_addr);
+
+ /* Make the new ring contents visible before handing the ring to HW. */
+ dma_wmb();
+
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -EIO;
+ }
+
+ cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to reinitialize EP0 for SETUP: %d\n", ret);
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ udc_ep0->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ return 0;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+ enable_setup_event(crg_udc);
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+
+ return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ val = readl(&uccr->config1);
+ val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+ writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+ switch (crg_udc->setup_status) {
+ case DATA_STAGE_XFER:
+ crg_udc->setup_status = STATUS_STAGE_RECV;
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+ 0, 0);
+ break;
+ case DATA_STAGE_RECV:
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+ 0, 0);
+ break;
+ default:
+ if (crg_udc->setup_fn_call_back) {
+ crg_udc->setup_fn_call_back(crg_udc);
+ crg_udc->setup_fn_call_back = NULL;
+ }
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+ }
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ unsigned long flags;
+ void *buf;
+
+ /* The data-stage completion handler queues the status stage. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u16 value, u16 index, u16 length)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ u16 status_val = 0;
+ u32 dci;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) || value ||
+ length != sizeof(u16)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ switch (request_type & USB_RECIP_MASK) {
+ case USB_RECIP_DEVICE:
+ if (index)
+ goto invalid;
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (crg_udc->u2_rwe)
+ status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+ } else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (crg_udc->feature_u1_enable)
+ status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+ if (crg_udc->feature_u2_enable)
+ status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+ }
+ if (crg_udc->gadget.is_selfpowered)
+ status_val |= BIT(USB_DEVICE_SELF_POWERED);
+ break;
+
+ case USB_RECIP_INTERFACE:
+ if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+ (index & 0xff00))
+ goto invalid;
+ break;
+
+ case USB_RECIP_ENDPOINT:
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto invalid;
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto invalid;
+ if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto invalid;
+ if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+ status_val = BIT(USB_ENDPOINT_HALT);
+ break;
+
+ default:
+ goto invalid;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf)
+ goto invalid;
+
+ *(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+ udc_req_ptr->usb_req.length = sizeof(status_val);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = get_status_cmpl;
+ crg_udc->status_req_callback_pending = true;
+
+ udc_req_ptr->map_dir = DMA_TO_DEVICE;
+ ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+ true);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ goto invalid;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_XFER;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ return;
+
+invalid:
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0_ptr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ int ret;
+
+ if (!crg_udc_is_event_rings_empty(crg_udc) ||
+ crg_udc->portsc_on_reconnecting == 1)
+ return -EBUSY;
+
+ udc_ep0_ptr = &crg_udc->udc_ep[0];
+ /* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+ * align with enqueue pointer, all remaining xfer trbs became dumb once
+ * which will not produce xfer event anymore.
+ *
+ * If we considering the opposite solution, we should wait all ep0 xfer
+ * trbs be completed(with some err complete code)
+ */
+ if (crg_udc->reinit_ep0_on_bus_reset) {
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 running;
+
+ /* Block requeue from giveback callbacks, then quiesce DMA. */
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+ if (crg_udc_get_ep_state(crg_udc, 0) == EP_STATE_RUNNING) {
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(0), 0);
+ if (ret)
+ goto reset_error;
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+ !(running & BIT(0)), 10, 10000);
+ if (ret)
+ goto reset_error;
+ }
+ udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
+ crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+
+ /* context related ops */
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+ udc_ep0_ptr->enq_pt);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ ret = -EIO;
+ goto reset_error;
+ }
+ cmd_param0 =
+ (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0,
+ cmd_param0, cmd_param1);
+ if (ret)
+ goto reset_error;
+
+ udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+ udc_ep0_ptr->tran_ring_full = false;
+ udc_ep0_ptr->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ } else {
+ if (!list_empty(&udc_ep0_ptr->queue)) {
+ dev_err(crg_udc->dev,
+ "remaining ep0 xfer trbs on reset\n");
+ return -EBUSY;
+ }
+ }
+
+ crg_udc_enable_setup(crg_udc);
+
+ return 0;
+
+reset_error:
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ crg_udc->dev_addr != 0) {
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+ } else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (crg_udc->dev_addr == 0) {
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_DEFAULT);
+ }
+
+ }
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ enum crg_ep_state ep_state;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+ if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
+
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+ return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_set_sel_req *sel;
+ unsigned long flags;
+ void *buf;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ if (buf && !req->status && req->actual >= sizeof(*sel)) {
+ sel = buf;
+ crg_udc->sel_value.u1_sel_value = sel->u1_sel;
+ crg_udc->sel_value.u1_pel_value = sel->u1_pel;
+ crg_udc->sel_value.u2_sel_value = le16_to_cpu(sel->u2_sel);
+ crg_udc->sel_value.u2_pel_value = le16_to_cpu(sel->u2_pel);
+ }
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->set_tm != 0) {
+ u32 tmp;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ tmp = readl(&uccr->u2portpmsc);
+ FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+ writel(tmp, &uccr->u2portpmsc);
+
+ crg_udc->set_tm = 0;
+ }
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+ u16 index, u16 length)
+{
+ u32 param0;
+
+ if (value > 127 || index || length ||
+ (crg_udc->device_state != USB_STATE_DEFAULT &&
+ crg_udc->device_state != USB_STATE_ADDRESS))
+ goto invalid;
+
+ /* SET_ADDRESS(0) is legal in the Default state. */
+ param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+ goto invalid;
+
+ crg_udc->dev_addr = value;
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc->setup_fn_call_back = set_address_cmpl;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+ -EINPROGRESS, NULL, 1, 0);
+ return;
+
+invalid:
+ crg_udc->setup_fn_call_back = NULL;
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u8 b_request, u16 value, u16 index, u16 length,
+ unsigned long *flags)
+{
+ int status = -EINPROGRESS;
+ u8 dci;
+ struct crg_udc_ep *udc_ep_ptr;
+ u32 tmp;
+ u32 set_feat = 0;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ if (length || (request_type & USB_DIR_IN))
+ goto set_feature_error;
+
+ /* TEST_MODE is permitted in the Default state; other features aren't. */
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ !((request_type & USB_RECIP_MASK) == USB_RECIP_DEVICE &&
+ value == USB_DEVICE_TEST_MODE &&
+ b_request == USB_REQ_SET_FEATURE))
+ goto set_feature_error;
+
+ set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+ if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+ dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+ if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (index != 0) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+ }
+
+ if (value != USB_ENDPOINT_HALT)
+ goto set_feature_error;
+
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto set_feature_error;
+
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto set_feature_error;
+ if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto set_feature_error;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ status = crg_udc_ep_halt(
+ udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+ 0, flags);
+
+ if (status < 0)
+ goto set_feature_error;
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+ switch (value) {
+ case USB_DEVICE_REMOTE_WAKEUP:
+ /* REMOTE_WAKEUP selector is not used by USB3.0 */
+ if (index ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ crg_udc->u2_rwe = set_feat;
+ crg_udc->gadget.wakeup_armed = set_feat;
+
+ break;
+ case USB_DEVICE_U1_ENABLE:
+ case USB_DEVICE_U2_ENABLE: {
+ u32 timeout_val;
+
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (index != 0 ||
+ crg_udc->gadget.speed < USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (set_feat == 1 && crg_udc->u12_forbidden) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ tmp = readl(&uccr->u3portpmsc);
+ if (value == USB_DEVICE_U1_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U1IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u1_enable = set_feat;
+ }
+
+ if (value == USB_DEVICE_U2_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U2IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u2_enable = set_feat;
+ }
+ writel(tmp, &uccr->u3portpmsc);
+
+ break;
+ }
+ case USB_DEVICE_TEST_MODE: {
+ u32 u_pattern;
+
+ if (b_request != USB_REQ_SET_FEATURE ||
+ crg_udc->gadget.speed != USB_SPEED_HIGH ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ (index & 0xff) || (index >> 8) == 0 ||
+ (index >> 8) > 6)
+ goto set_feature_error;
+ u_pattern = index >> 8;
+ crg_udc->set_tm = u_pattern;
+ crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+ break;
+ }
+
+ default:
+ goto set_feature_error;
+ }
+
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "interface u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Suspend Option */
+ if (value != USB_INTRF_FUNC_SUSPEND) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Both cleared LP/RW bits and either suspend option are valid. */
+ if (index & ~(0xff | USB_INTRF_FUNC_SUSPEND_LP |
+ USB_INTRF_FUNC_SUSPEND_RW))
+ goto set_feature_error;
+
+ if (index & USB_INTRF_FUNC_SUSPEND_RW)
+ dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+ else
+ dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+ /* Do not need to return status stage here
+ * Pass to composite gadget driver to process
+ * the request
+ */
+ return false;
+ } else {
+ /* Unknown recipient must not receive a successful status stage. */
+ goto set_feature_error;
+ }
+
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL, 0, 0);
+ return true;
+
+set_feature_error:
+ crg_udc_set_ep0_halt(crg_udc);
+ return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+ u16 length)
+{
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct usb_set_sel_req *sel;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) ||
+ crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+ length != sizeof(struct usb_set_sel_req)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ sel = udc_req_ptr->usb_req.buf;
+ udc_req_ptr->usb_req.length = sizeof(*sel);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = set_sel_cmpl;
+ crg_udc->status_req_callback_pending = true;
+ udc_req_ptr->map_dir = DMA_FROM_DEVICE;
+ ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+ false);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_RECV;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+ u16 maxpacketsize = 0;
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ u32 param0;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ maxpacketsize = 512;
+ else
+ maxpacketsize = 64;
+
+ param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+ return -EIO;
+
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+ udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+ return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags)
+{
+ u16 w_value = le16_to_cpu(setup_pkt->wValue);
+ u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+ u16 w_length = le16_to_cpu(setup_pkt->wLength);
+ struct usb_gadget_driver *driver;
+ u32 tmp;
+
+ /* No SETUP (including SET_ADDRESS) may overtake a pending reset. */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ flush_work(&crg_udc->udc_reset_work);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (crg_udc->resetting || !crg_udc->gadget_driver ||
+ crg_udc->ep0_state == EP0_STATE_ABORTING)
+ return;
+
+ trace_crg_udc_ctrl_req(setup_pkt);
+
+ /* EP0 come backs to running when new setup packet comes */
+ crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+ crg_udc->setup_tag = setup_tag;
+ crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+ crg_udc->setup_fn_call_back = NULL;
+
+ if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+ switch (setup_pkt->bRequest) {
+ case USB_REQ_GET_STATUS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_INTERFACE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_ENDPOINT |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ getstatusrequest(crg_udc, setup_pkt->bRequestType,
+ w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ADDRESS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ setaddressrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_SEL:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ crg_udc->setup_status = DATA_STAGE_RECV;
+ setselrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ISOCH_DELAY:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_index || w_length ||
+ w_value >
+ FIELD_MAX(
+ CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ tmp = readl(&crg_udc->uccr->ptmctrl);
+ FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+ w_value);
+ writel(tmp, &crg_udc->uccr->ptmctrl);
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+ -EINPROGRESS, NULL, 0, 0);
+ return;
+
+ case USB_REQ_CLEAR_FEATURE:
+ case USB_REQ_SET_FEATURE:
+ /* Need composite gadget driver
+ * to process the function remote wakeup request
+ */
+ if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+ setup_pkt->bRequest, w_value,
+ w_index, w_length, flags)) {
+ /* Get here if request has been processed.*/
+ return;
+ }
+ break;
+ case USB_REQ_SET_CONFIGURATION:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_length || w_value > 0xff) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ /* In theory we need to clear RUN bit before
+ * status stage of deconfig request sent.
+ * But seeing problem if we do it before all the
+ * endpoints belonging to the configuration
+ * disabled.
+ */
+ if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ break;
+ default:
+ dev_dbg(crg_udc->dev,
+ "USB_REQ default bRequest=%d, bRequestType=%d\n",
+ setup_pkt->bRequest, setup_pkt->bRequestType);
+ }
+ }
+
+ if (w_length) {
+ /* data phase from gadget like GET_CONFIGURATION
+ * call the setup routine of gadget driver.
+ * remember the request direction.
+ */
+ crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+ DATA_STAGE_XFER :
+ DATA_STAGE_RECV;
+ }
+ driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+ /* Recheck callback enablement after the locked snapshot. */
+ if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
+ READ_ONCE(crg_udc->gadget_driver) != driver ||
+ driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+ (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+ (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+ if (w_value) {
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_CONFIGURED);
+ } else {
+ /* Endpoint disable may already have updated device_state. */
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_ADDRESS);
+ }
+ }
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ u32 cmd_param0;
+ u32 cmd_param1;
+
+ /* setup transfer ring */
+ if (!udc_ep_ptr->tran_ring_info.cpu_addr) {
+ dma_addr_t dma;
+ u32 ring_size = CRG_UDC_CONTROL_EP_TD_RING_SIZE;
+ void *vaddr;
+ size_t len;
+
+ len = ring_size * sizeof(struct crg_transfer_trb);
+ vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma, GFP_KERNEL);
+ if (!vaddr)
+ return -ENOMEM;
+
+ udc_ep_ptr->tran_ring_info.cpu_addr = vaddr;
+ udc_ep_ptr->tran_ring_info.dma_addr = dma;
+ udc_ep_ptr->tran_ring_info.len = len;
+ udc_ep_ptr->first_trb = vaddr;
+ udc_ep_ptr->last_trb = udc_ep_ptr->first_trb + ring_size - 1;
+ }
+
+ memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+ /* Ensure that transfer ring is updated */
+ dma_wmb();
+ udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->pcs = 1;
+ udc_ep_ptr->tran_ring_full = false;
+
+ crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+ udc_ep_ptr->tran_ring_info.dma_addr);
+
+ /* context related ops */
+ cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+ CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+ cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1))
+ return -EIO;
+
+ udc_ep_ptr->desc = &crg_udc->ep0_desc;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->pcs = 1;
+
+ return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ enum crg_ep_state ep_state;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ struct crg_transfer_trb *p_trb;
+ u32 transfer_length;
+ bool ioc;
+ bool azp;
+ u8 dir = 0;
+ u8 setup_tag = crg_udc->setup_tag;
+
+ /* A new SETUP aborts the previous EP0 transaction. The completion
+ * callback of the aborted request is allowed to run, but it must not
+ * requeue the old request before the new SETUP has been handed to the
+ * gadget driver.
+ */
+ if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+ return -ECONNRESET;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, 0);
+ /* Need to queue the request even ep is paused or halted */
+ if (ep_state != EP_STATE_RUNNING) {
+ dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+ return -EINVAL;
+ }
+ if (list_empty(&udc_ep_ptr->queue)) {
+ /* For control endpoint, we can handle one setup request at a
+ * time. so if there are TD pending in the transfer ring.
+ * wait for the sequence number error event. Then put the new
+ * request to transfer ring
+ */
+ if (enq_pt == dq_pt) {
+ u32 i;
+ bool need_zlp = false;
+ dma_addr_t trb_dma = usb_req->dma;
+ u64 remaining = usb_req->length;
+ /* Transfer ring is empty
+ * setup data stage TRBs
+ */
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+ if (crg_udc->setup_status == DATA_STAGE_XFER)
+ dir = 0;
+ else if (crg_udc->setup_status == DATA_STAGE_RECV)
+ dir = 1;
+ else
+ dev_dbg(crg_udc->dev,
+ "unexpected setup_status!%d\n",
+ crg_udc->setup_status);
+ if (udc_req_ptr->usb_req.zero == 1 &&
+ udc_req_ptr->usb_req.length != 0 &&
+ ((udc_req_ptr->usb_req.length %
+ udc_ep_ptr->usb_ep.maxpacket) == 0))
+ need_zlp = true; /* zero length packet */
+ if (num_of_trbs_needed > 32)
+ return -EMSGSIZE;
+
+ if (num_of_trbs_needed >
+ crg_udc_room_on_ring(
+ crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+ udc_ep_ptr->first_trb, enq_pt, dq_pt))
+ return -ENOSPC;
+
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ for (i = 0; i < num_of_trbs_needed; i++) {
+ p_trb = enq_pt;
+ transfer_length = crg_udc_trb_chunk_len(
+ trb_dma, remaining);
+ ioc = (i == num_of_trbs_needed - 1);
+ azp = ioc && need_zlp;
+ crg_udc_setup_datastage_trb(
+ p_trb, trb_dma,
+ i == 0 ? (original_pcs ^ 1) :
+ udc_ep_ptr->pcs,
+ transfer_length,
+ num_of_trbs_needed - i - 1, ioc,
+ !ioc, azp, dir, setup_tag);
+ udc_req_ptr->all_trbs_queued = 1;
+ trb_dma += transfer_length;
+ remaining -= transfer_length;
+ enq_pt++;
+ if (FIELD_GET(TRB_TYPE_MASK,
+ le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(
+ TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(
+ &enq_pt->dw3,
+ udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+ dma_wmb();
+ le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+ original_pcs, TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->enq_pt = enq_pt;
+ crg_udc_knock_doorbell(crg_udc, 0);
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb =
+ udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+ } else {
+ /*
+ * The software queue is empty but the transfer ring
+ * still holds pending TDs from a previous control
+ * transfer (the host issued a new setup before the
+ * old one drained). We process one setup at a time,
+ * so we intentionally do not queue TRBs here.
+ * Returning success leaves this request on the
+ * endpoint queue; the stale TDs and this request are
+ * flushed and completed (with -EINVAL) by the
+ * sequence-number-error / setup-tag-mismatch event in
+ * crg_udc_handle_xfer_event().
+ */
+ dev_err(crg_udc->dev,
+ "ep0 ring not idle on new setup\n");
+ }
+ } else {
+ dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+ /* New setup packet came
+ * Drop this request.
+ */
+ return -EINVAL;
+ }
+ return 0;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (3 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:50 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 2377 +++++++++++++++++++++++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.c | 9 +
2 files changed, 2386 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..8c52f8856490
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,2377 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event);
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+ unsigned long *flags);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+
+ return FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+ readl(&crg_udc->uccr->mfindex));
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (is_on && crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ crg_udc->pullup_on = !!is_on;
+ crg_udc_set_pullup(crg_udc, crg_udc->pullup_on);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ trace_crg_udc_pullup(crg_udc->pullup_on);
+ return 0;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->u2_rwe = !!set;
+ g->wakeup_armed = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ g->is_selfpowered = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+ enum usb_device_speed speed)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+ crg_udc->max_speed = speed;
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+ struct usb_gadget_driver *driver)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /* Do not reopen a controller whose previous reset failed. */
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EBUSY;
+ }
+ /* The UDC core calls udc_set_speed() before udc_start(). */
+ ret = crg_udc_program_speed(crg_udc);
+ if (!ret) {
+ crg_udc->gadget_driver = driver;
+ crg_udc_set_vbus_detect(crg_udc, true);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc;
+ unsigned long flags = 0;
+
+ crg_udc = gadget_to_udc(g);
+
+ /*
+ * Clear gadget_driver first so crg_udc_handle_port_status() can no
+ * longer schedule udc_reset_work (it only does so while
+ * gadget_driver is set), then cancel_work_sync() is guaranteed to
+ * be the last word: no new reset notification can be queued after
+ * it returns.
+ */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->gadget_driver = NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ cancel_work_sync(&crg_udc->udc_reset_work);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc_set_vbus_detect(crg_udc, false);
+
+ /*
+ * crg_udc_reset() completes any queued requests via
+ * crg_udc_ep_nuke() -> crg_udc_req_done(). crg_udc_req_done() drops udc_lock
+ * (IRQ-safe) around each ->complete() callback and re-acquires it
+ * afterwards, so gadget completion callbacks are never invoked with
+ * the spinlock held. flags is threaded through for that purpose.
+ */
+ if (crg_udc_reset(crg_udc, &flags)) {
+ dev_err(crg_udc->dev, "controller reset failed during stop\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+ if (crg_udc_reset_data_struct(crg_udc)) {
+ dev_err(crg_udc->dev,
+ "controller data reset failed during stop\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ crg_udc->connected = 0;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+ if (crg_udc_init_ep0(crg_udc)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ /* The hardware and EP0 have been reinitialized successfully. */
+ crg_udc->resetting = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags;
+ u32 portsc;
+ int ret = 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+ if (!crg_udc->u2_rwe) {
+ ret = -EACCES;
+ goto out;
+ }
+
+ portsc = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+ CRG_U3DC_PORTSC_PLS_U3) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+ portsc |= CRG_U3DC_PORTSC_LWS;
+ writel(portsc, &uccr->portsc);
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->async_callbacks = enable;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* Drain any in-flight notification before the gadget is unbound. */
+ if (!enable)
+ cancel_work_sync(&crg_udc->udc_reset_work);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+ if (!udc_req_ptr)
+ return NULL;
+
+ udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+ INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+ trace_crg_udc_alloc_request(
+ container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+ return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ int status = 0;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 num_trbs_needed;
+ u64 buffer_length;
+
+ if (udc_req_ptr->trbs_needed) {
+ /* If this is called to complete pending TRB transfers
+ * of previous Request
+ */
+ buffer_length = udc_req_ptr->buff_len_left;
+ num_trbs_needed = udc_req_ptr->trbs_needed;
+ } else {
+ buffer_length = udc_req_ptr->usb_req.length;
+ if (udc_req_ptr->usb_req.num_mapped_sgs)
+ num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+ else
+ num_trbs_needed = crg_udc_count_trbs_linear(
+ udc_req_ptr->usb_req.dma, buffer_length);
+ if (!num_trbs_needed && buffer_length)
+ return -EINVAL;
+ if (!num_trbs_needed)
+ num_trbs_needed = 1;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+ num_trbs_needed);
+ else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+ CRG_UDC_ISOC_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_BULK_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else
+ /* interrupt endpoint */
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_INT_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+
+ if (!status) {
+ udc_req_ptr->td_pending = true;
+ if (!usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ }
+
+ return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 tmp;
+
+ tmp = readl(&uccr->u3portpmsc);
+
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+ writel(tmp, &uccr->u3portpmsc);
+
+ crg_udc->feature_u1_enable = 0;
+ crg_udc->feature_u2_enable = 0;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_uccr __iomem *uccr;
+ struct crg_udc_request *udc_req = NULL;
+ struct crg_udc_request *iter;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_transfer_trb *pause_pt;
+ enum crg_ep_state old_ep_state;
+ u32 tmp, deq_pt_lo, deq_pt_hi;
+ dma_addr_t dq_pt_addr;
+ u8 dci;
+ bool found = false;
+ u32 generation;
+ unsigned long flags;
+ const struct usb_endpoint_descriptor *old_desc;
+ u32 event_generation;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ uccr = crg_udc->uccr;
+ dci = udc_ep_ptr->dci;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ old_ep_state = udc_ep_ptr->ep_state;
+ old_desc = udc_ep_ptr->desc;
+ event_generation = crg_udc->udc_event[0].generation;
+ generation = udc_ep_ptr->generation;
+
+ list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+ if (&iter->usb_req == _req) {
+ udc_req = iter;
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ if (!udc_req->first_trb) {
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ if (dci == 0) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ if (old_ep_state == EP_STATE_RUNNING) {
+ int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(dci), 0);
+
+ if (ret) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+ }
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & BIT(dci)), 10, 10000)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ETIMEDOUT;
+ }
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ }
+
+ p_ep_cx = &crg_udc->p_epcx[dci - 2];
+ deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+ deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+ dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+ if (!pause_pt) {
+ if (old_ep_state == EP_STATE_RUNNING) {
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ crg_udc_knock_doorbell(crg_udc, dci);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+ udc_ep_ptr)) {
+ crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+ /* Give back the cancelled request BEFORE scheduling the tail:
+ * its callback may add/remove requests, and dropping udc_lock
+ * while traversing the old list is unsafe.
+ */
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else if (udc_req->last_trb &&
+ crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+ udc_ep_ptr)) {
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else {
+ struct crg_udc_request *next_req = NULL;
+
+ udc_req->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+ if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+ next_req = list_entry(udc_req->queue.next,
+ struct crg_udc_request, queue);
+
+ if (next_req && next_req->first_trb)
+ udc_ep_ptr->deq_pt = next_req->first_trb;
+ else
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+ if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+ udc_ep_ptr->tran_ring_full = false;
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ }
+
+ if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
+ udc_ep_ptr->desc == old_desc &&
+ event_generation == crg_udc->udc_event[0].generation &&
+ udc_ep_ptr->generation == generation &&
+ udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->tran_ring_full = false;
+ if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ crg_udc_knock_doorbell(crg_udc, dci);
+ }
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_uccr __iomem *uccr;
+ unsigned long flags;
+ u32 running;
+ int ret;
+
+ if (!ep)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep->crg_udc;
+ if (udc_ep->dci == 0)
+ return 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (udc_ep->configuring) {
+ ret = -EBUSY;
+ goto out;
+ }
+ if (udc_ep->ep_state == EP_STATE_DISABLED) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ uccr = crg_udc->uccr;
+ p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+ writel(BIT(udc_ep->dci), &uccr->ep_enable);
+ /* Do not unmap request buffers while the controller still owns them. */
+ ret = readl_poll_timeout_atomic(&uccr->ep_enable, running,
+ !(running & BIT(udc_ep->dci)), 10, 10000);
+ if (ret)
+ goto out;
+
+ udc_ep->generation++;
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ udc_ep->usb_ep.enabled = 0;
+ udc_ep->tran_ring_full = false;
+ if (crg_udc->num_enabled_eps)
+ crg_udc->num_enabled_eps--;
+ memset(p_ep_cx, 0, sizeof(*p_ep_cx));
+
+ if (!crg_udc->num_enabled_eps &&
+ crg_udc->device_state == USB_STATE_CONFIGURED)
+ crg_udc->device_state = USB_STATE_ADDRESS;
+
+ /* crg_udc_req_done() drops the lock during giveback. All endpoint
+ * state has to be finalized before that can happen.
+ */
+ crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+ ret = 0;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_transfer_trb *new_ring = NULL;
+ void *old_ring = NULL;
+ dma_addr_t new_dma = 0, old_dma = 0;
+ size_t len, old_len = 0;
+ unsigned long flags;
+ u32 ring_size, param0;
+ int ret;
+
+ if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ if (udc_ep->dci == 0)
+ return 0;
+ crg_udc = udc_ep->crg_udc;
+
+ if (usb_endpoint_xfer_bulk(desc))
+ ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_isoc(desc))
+ ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_int(desc))
+ ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+ else
+ return -EINVAL;
+ len = ring_size * sizeof(struct crg_transfer_trb);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->gadget_driver || crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (udc_ep->configuring || udc_ep->ep_state != EP_STATE_DISABLED) {
+ ret = -EBUSY;
+ goto out;
+ }
+ udc_ep->configuring = true;
+ udc_ep->desc = desc;
+ udc_ep->comp_desc = ep->comp_desc;
+ ret = crg_udc_validate_ep_desc(udc_ep, desc);
+ if (ret)
+ goto err;
+
+ /* Leave the old allocation valid until its replacement is ready.
+ * Other ep operations reject the configuring endpoint.
+ */
+ if (!udc_ep->tran_ring_info.cpu_addr ||
+ udc_ep->tran_ring_info.len != len) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
+ GFP_KERNEL);
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!new_ring) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ if (!crg_udc->gadget_driver || crg_udc->resetting ||
+ udc_ep->ep_state != EP_STATE_DISABLED ||
+ udc_ep->desc != desc) {
+ ret = -ESHUTDOWN;
+ goto err;
+ }
+ old_ring = udc_ep->tran_ring_info.cpu_addr;
+ old_dma = udc_ep->tran_ring_info.dma_addr;
+ old_len = udc_ep->tran_ring_info.len;
+ udc_ep->tran_ring_info.cpu_addr = new_ring;
+ udc_ep->tran_ring_info.dma_addr = new_dma;
+ udc_ep->tran_ring_info.len = len;
+ new_ring = NULL;
+ }
+
+ udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
+ udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+ memset(udc_ep->first_trb, 0, len);
+ dma_wmb();
+ crg_udc_setup_link_trb(udc_ep->last_trb, true,
+ udc_ep->tran_ring_info.dma_addr);
+ udc_ep->enq_pt = udc_ep->first_trb;
+ udc_ep->deq_pt = udc_ep->first_trb;
+ udc_ep->pcs = 1;
+ udc_ep->tran_ring_full = false;
+ ret = crg_udc_epcx_setup(udc_ep);
+ if (ret)
+ goto err;
+ param0 = BIT(udc_ep->dci);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+ if (ret)
+ goto err;
+
+ udc_ep->generation++;
+ udc_ep->wedge = false;
+ udc_ep->usb_ep.enabled = 1;
+ udc_ep->ep_state = EP_STATE_RUNNING;
+ udc_ep->configuring = false;
+ crg_udc->num_enabled_eps++;
+ if (crg_udc->device_state == USB_STATE_ADDRESS)
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (old_ring)
+ dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
+ return 0;
+
+err:
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->configuring = false;
+ /* Keep a valid ring allocation for a later retry. */
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (new_ring)
+ dma_free_coherent(crg_udc->dev, len, new_ring, new_dma);
+ if (old_ring)
+ dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
+ return ret;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+ struct crg_udc_request *req;
+ struct list_head nuke_list;
+
+ list_replace_init(&udc_ep->queue, &nuke_list);
+
+ while (!list_empty(&nuke_list)) {
+ req = list_first_entry(&nuke_list, struct crg_udc_request,
+ queue);
+
+ crg_udc_req_done(udc_ep, req, status);
+ }
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+ gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ enum dma_data_direction dma_data_dir;
+ const struct usb_endpoint_descriptor *desc;
+ unsigned long flags;
+ int status;
+ bool is_sg;
+ u8 setup_tag = 0;
+ u32 generation = 0;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+ if (is_sg && !_req->sg)
+ return -EINVAL;
+ if (_req->stream_id)
+ return -EOPNOTSUPP;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /*
+ * A completion callback for an aborted EP0 transfer may requeue the
+ * same request. Reject EP0 requests while the previous control
+ * transfer is being aborted or EP0 is in an error state. EP0 becomes
+ * available again only after the new SETUP has been accepted and
+ * the EP0 transfer ring has been successfully reinitialized.
+ */
+ if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ECONNRESET;
+ }
+ if (!udc_ep_ptr->first_trb || !_req->complete ||
+ (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ desc = udc_ep_ptr->desc;
+ if (udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+ udc_ep_ptr->configuring) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ setup_tag = crg_udc->setup_tag;
+ generation = udc_ep_ptr->generation;
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EOPNOTSUPP;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+ if (crg_udc->setup_status == DATA_STAGE_XFER)
+ crg_udc->setup_status = STATUS_STAGE_RECV;
+ else
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
+ -EINPROGRESS, udc_req_ptr, 0, 0);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+ if (crg_udc->setup_status == DATA_STAGE_XFER ||
+ crg_udc->setup_status == STATUS_STAGE_XFER) {
+ dma_data_dir = DMA_TO_DEVICE;
+ } else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+ crg_udc->setup_status == STATUS_STAGE_RECV) {
+ dma_data_dir = DMA_FROM_DEVICE;
+ } else {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ } else {
+ dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+ DMA_TO_DEVICE :
+ DMA_FROM_DEVICE;
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ crg_udc_clear_req_container(udc_req_ptr);
+ udc_req_ptr->mapped = false;
+
+ if (_req->length) {
+ udc_req_ptr->map_dir = dma_data_dir;
+ status = usb_gadget_map_request_by_dev(
+ crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
+ if (status)
+ goto err_unmap;
+
+ if (is_sg) {
+ if (!_req->num_mapped_sgs ||
+ crg_udc_sg_init(udc_req_ptr)) {
+ status = -EIO;
+ goto err_unmap;
+ }
+ } else if (_req->dma_mapped) {
+ udc_req_ptr->mapped = true;
+ }
+ }
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+ !list_empty(&udc_req_ptr->queue) || crg_udc->resetting ||
+ udc_ep_ptr->configuring ||
+ udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+ udc_ep_ptr->generation != generation ||
+ (udc_ep_ptr->dci == 0 &&
+ (crg_udc->ep0_state != EP0_STATE_IDLE ||
+ crg_udc->setup_tag != setup_tag))) {
+ status = -ESHUTDOWN;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+ }
+
+ _req->status = -EINPROGRESS;
+ _req->actual = 0;
+
+ /* Do not queue a newer request ahead of a partially queued one. */
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ struct crg_udc_request *tail;
+
+ tail = list_last_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ status = (!tail->all_trbs_queued ||
+ udc_ep_ptr->tran_ring_full) ? 0 :
+ crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ } else if (udc_ep_ptr->tran_ring_full) {
+ status = 0;
+ } else {
+ status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ }
+
+ if (!status)
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ if (status)
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+
+err_unmap:
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ if (value && usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+ !list_empty(&udc_ep_ptr->queue)) {
+ dev_err(crg_udc->dev, "set_halt: list not empty\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EAGAIN;
+ }
+
+ status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ udc_ep_ptr->wedge = 1;
+
+ status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+ enum crg_ep_type ep_type;
+ u16 maxburst = 0;
+ u8 maxstreams = 0;
+ u16 maxsize;
+ u32 dw;
+
+ /* corigine gadget dir should be opposite to host dir */
+ if (usb_endpoint_dir_out(desc))
+ ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+ else
+ ep_type = usb_endpoint_type(desc);
+
+ maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp_desc)
+ return -EINVAL;
+ maxburst = comp_desc->bMaxBurst;
+
+ if (usb_endpoint_xfer_bulk(udc_ep->desc))
+ maxstreams = comp_desc->bmAttributes & 0x1f;
+
+ } else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) &&
+ (usb_endpoint_xfer_int(udc_ep->desc) ||
+ usb_endpoint_xfer_isoc(udc_ep->desc))) {
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+ maxburst = usb_endpoint_maxp_mult(desc) - 1;
+ if (maxburst == 0x3) {
+ dev_err(crg_udc->dev, "invalid maxburst\n");
+ maxburst = 0x2;
+ }
+ }
+
+ /* fill ep_dw0 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+ FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+ if (maxstreams) {
+ dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+ maxstreams);
+ }
+ epcx->dw0 = cpu_to_le32(dw);
+
+ /* fill ep_dw1 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+ FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+ FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+ epcx->dw1 = cpu_to_le32(dw);
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+ EP_CX_TR_DQPT_LO_MASK;
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+ epcx->dw3 = cpu_to_le32(dw);
+ /* Ensure that epcx is updated */
+ dma_wmb();
+ return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx;
+ u32 dw;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+
+ if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+ dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+ return -EINVAL;
+ }
+
+ epcx = &crg_udc->p_epcx[dci - 2];
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+ if (dqptaddr == DMA_MAPPING_ERROR)
+ return -EINVAL;
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+ /* The consumer cycle can lag the producer by one ring wrap. */
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw,
+ udc_ep->deq_pt > udc_ep->enq_pt ?
+ udc_ep->pcs ^ 1 : udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(dqptaddr);
+ epcx->dw3 = cpu_to_le32(dw);
+
+ cmd_param0 = BIT(udc_ep->dci);
+ /* Ensure that dqptr is updated */
+ dma_wmb();
+
+ return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ if (!_ep || !_req)
+ return;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+ trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+ udc_req_ptr);
+
+ if (udc_req_ptr->mapped || udc_req_ptr->usb_req.num_mapped_sgs)
+ crg_udc_unmap_request(container_of(_ep, struct crg_udc_ep,
+ usb_ep),
+ udc_req_ptr);
+ kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ u32 num_trbs_ava, count, i;
+ u32 td_trbs;
+ u32 td_size;
+ u32 j = 1;
+ u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+ CRG_UDC_BULK_EP_INTERRUPT_RATE;
+ bool full_td = true;
+ bool need_zlp = false;
+ bool is_sg = usb_req->num_mapped_sgs;
+
+ if (!b_isoc && usb_req->zero && usb_req->length &&
+ !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+ need_zlp = true;
+
+ if (!num_trbs_needed)
+ return -EINVAL;
+
+ /* TD_SIZE is five bits and encodes the number of TRBs remaining
+ * in this TD minus one. A request may contain multiple TDs.
+ */
+ td_trbs = min_t(u32, num_trbs_needed, 32);
+
+ num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+ p_xfer_ring, udc_ep_ptr->enq_pt,
+ udc_ep_ptr->deq_pt);
+
+ if (num_trbs_ava < td_trbs) {
+ if (b_isoc)
+ return -EBUSY;
+ trace_crg_udc_no_room_on_ring(
+ "deferring TRBs to next doorbell");
+ count = num_trbs_ava;
+ full_td = false;
+ udc_ep_ptr->tran_ring_full = true;
+ need_zlp = false;
+ } else {
+ count = td_trbs;
+ }
+
+ if (!count)
+ return -ENOSPC;
+
+ td_size = count - 1;
+
+ if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
+ return -EINVAL;
+
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ for (i = 0; i < count; i++) {
+ dma_addr_t trb_buf_addr;
+ u32 trb_len, sg_len = 0;
+ bool chain_bit = true;
+ bool ioc = false;
+ bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+ if (is_sg) {
+ if (!crg_udc_sg_current(udc_req_ptr, &trb_buf_addr,
+ &sg_len))
+ return -EINVAL;
+ if (!sg_len ||
+ dma_mapping_error(crg_udc->dev, trb_buf_addr))
+ return -EINVAL;
+ trb_len = min_t(
+ u64, buffer_length,
+ min_t(u32, sg_len,
+ crg_udc_trb_chunk_len(trb_buf_addr,
+ buffer_length)));
+ } else {
+ if (buffer_length)
+ trb_buf_addr = usb_req->dma + (usb_req->length -
+ buffer_length);
+ else
+ trb_buf_addr = 0;
+ trb_len = crg_udc_trb_chunk_len(trb_buf_addr,
+ buffer_length);
+ }
+
+ if (!trb_len && buffer_length)
+ return -EINVAL;
+
+ buffer_length -= trb_len;
+
+ if (i == count - 1) {
+ chain_bit = false;
+ ioc = true;
+ if (!buffer_length)
+ udc_req_ptr->all_trbs_queued = true;
+ }
+
+ if (!full_td && j == intr_rate) {
+ ioc = true;
+ j = 0;
+ }
+
+ if (b_isoc) {
+ crg_udc_setup_trb(
+ enq_pt, trb_len, trb_buf_addr, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ TRB_TYPE_XFER_DATA_ISOCH, short_pkt, chain_bit,
+ ioc, false, 0, true, 0, 1, 0);
+ } else {
+ bool azp = ioc && !chain_bit && !buffer_length && need_zlp;
+
+ crg_udc_setup_trb(
+ enq_pt, trb_len, trb_buf_addr, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+ false, 0, false, 0, 0, azp);
+ }
+
+ trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+
+ if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
+ return -EINVAL;
+
+ td_size--;
+ enq_pt++;
+ j++;
+
+ if (enq_pt == udc_ep_ptr->last_trb) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 1;
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+
+ dma_wmb();
+ le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+ TRB_CYCLE_BIT_MASK);
+
+ if (!udc_req_ptr->trbs_needed)
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+
+ udc_ep_ptr->enq_pt = enq_pt;
+ if (count == num_trbs_ava)
+ udc_ep_ptr->tran_ring_full = true;
+ udc_req_ptr->buff_len_left = buffer_length;
+ udc_req_ptr->td_pending = true;
+ if (buffer_length)
+ udc_req_ptr->trbs_needed = num_trbs_needed - count;
+ else
+ udc_req_ptr->trbs_needed = 0;
+
+ /* Track the last TRB of the current TD, even for multi-TD requests. */
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+
+ return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 i, tmp;
+ struct crg_udc_ep *udc_ep_ptr;
+
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ /* Base on Figure 9-1, default USB_STATE is attached */
+ crg_udc->device_state = USB_STATE_RECONNECTING;
+
+ /* halt all the endpoints */
+
+ /* disable all the endpoints */
+ tmp = readl(&uccr->ep_enable);
+ writel(tmp, &uccr->ep_enable);
+ if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
+ dev_dbg(crg_udc->dev,
+ "timeout waiting for all endpoints to disable\n");
+ return -ETIMEDOUT;
+ }
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ udc_ep_ptr = &crg_udc->udc_ep[i];
+ udc_ep_ptr->usb_ep.enabled = 0;
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ if (udc_ep_ptr->desc) {
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+ }
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ }
+ crg_udc->num_enabled_eps = 0;
+
+ /* we don't handle ep0 here, we crg_udc_init_ep0 when event ring is empty */
+
+ if (crg_udc->dev_addr != 0) {
+ u32 param0;
+
+ param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+ return -EIO;
+ crg_udc->dev_addr = 0;
+ }
+
+ crg_udc_clear_portpm(crg_udc);
+
+ return 0;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ if (is_on) {
+ /* Enable event reporting and interrupts, then assert RUN */
+ val = readl(&uccr->config1);
+ val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+ CRG_U3DC_CFG1_PEC_EVENT_EN |
+ CRG_U3DC_CFG1_PPC_EVENT_EN |
+ CRG_U3DC_CFG1_PRC_EVENT_EN |
+ CRG_U3DC_CFG1_PLC_EVENT_EN |
+ CRG_U3DC_CFG1_CEC_EVENT_EN);
+ writel(val, &uccr->config1);
+
+ val = readl(&uccr->control);
+ val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+ CRG_U3DC_CTRL_RUN);
+ writel(val, &uccr->control);
+ } else {
+ val = readl(&uccr->control);
+ val &= ~CRG_U3DC_CTRL_RUN;
+ writel(val, &uccr->control);
+ }
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+ u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+ if (enable)
+ val |= CRG_UDC_VBUS_DETECT;
+ else
+ val &= ~CRG_UDC_VBUS_DETECT;
+
+ writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+ struct scatterlist *sg = udc_req->sg;
+ unsigned int i = udc_req->sg_index;
+
+ while (bytes) {
+ u32 len;
+
+ if (!sg || i >= udc_req->usb_req.num_sgs)
+ return false;
+
+ len = sg_dma_len(sg);
+ if (!len || udc_req->sg_offset == len) {
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ continue;
+ }
+
+ len -= udc_req->sg_offset;
+ if (bytes < len) {
+ udc_req->sg_offset += bytes;
+ bytes = 0;
+ break;
+ }
+
+ bytes -= len;
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ }
+
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len)
+{
+ struct scatterlist *sg = udc_req->sg;
+
+ while (sg && udc_req->sg_index < udc_req->usb_req.num_sgs &&
+ !sg_dma_len(sg)) {
+ sg = sg_next(sg);
+ udc_req->sg_index++;
+ udc_req->sg_offset = 0;
+ }
+ udc_req->sg = sg;
+
+ if (!sg || udc_req->sg_index >= udc_req->usb_req.num_sgs)
+ return false;
+
+ if (udc_req->sg_offset >= sg_dma_len(sg))
+ return false;
+
+ *dma = sg_dma_address(sg) + udc_req->sg_offset;
+ *len = sg_dma_len(sg) - udc_req->sg_offset;
+ return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ unsigned int i;
+
+ for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+ if (sg_dma_len(sg)) {
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ udc_req->sg_offset = 0;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
+
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ struct usb_request *req = &udc_req->usb_req;
+ bool is_in = udc_req->map_dir == DMA_TO_DEVICE;
+
+ if (req->num_mapped_sgs || req->dma_mapped)
+ usb_gadget_unmap_request_by_dev(crg_udc->dev, req, is_in);
+
+ if (udc_req->mapped) {
+ req->dma = DMA_MAPPING_ERROR;
+ udc_req->mapped = false;
+ }
+
+ udc_req->sg = NULL;
+ udc_req->sg_index = 0;
+ udc_req->sg_offset = 0;
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+ u16 maxpacket = usb_endpoint_maxp(desc);
+
+ if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+ maxpacket > CRG_MAX_PACKET_SIZE)
+ return -EINVAL;
+
+ if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+ return -EINVAL;
+
+ if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+ return -EOPNOTSUPP;
+
+ if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+ return -EINVAL;
+
+ if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp || comp->bMaxBurst > 15)
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ ((comp->bmAttributes & 0x3) > 2))
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+ if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+ usb_endpoint_xfer_isoc(desc)))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ usb_endpoint_maxp_mult(desc) > 3)
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+ return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+ const char *name)
+{
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+ ep->dci = dci;
+
+ if (ep->dci > 1) {
+ strscpy_pad(ep->name, name, sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+
+ ep->usb_ep.caps.type_iso = 1;
+ ep->usb_ep.caps.type_bulk = 1;
+ ep->usb_ep.caps.type_int = 1;
+ if (ep->dci % 2)
+ ep->usb_ep.caps.dir_out = 1;
+ else
+ ep->usb_ep.caps.dir_in = 1;
+ usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+ } else {
+ strscpy(ep->name, "ep0", sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+ ep->usb_ep.maxpacket = 512;
+ ep->usb_ep.caps.type_control = 1;
+ ep->usb_ep.caps.dir_in = 1;
+ ep->usb_ep.caps.dir_out = 1;
+ }
+
+ ep->usb_ep.ops = &crg_udc_ep_ops;
+ ep->crg_udc = crg_udc;
+
+ INIT_LIST_HEAD(&ep->queue);
+ if (ep->dci > 1)
+ list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+ struct crg_gadget_dev *crg_udc =
+ container_of(work, struct crg_gadget_dev, udc_reset_work);
+
+ struct usb_gadget_driver *driver;
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* Disabling async callbacks drains this work before driver unbind. */
+ if (driver)
+ usb_gadget_udc_reset(&crg_udc->gadget, driver);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ bool was_connected = crg_udc->connected;
+ u32 portsc_val;
+ u32 tmp;
+ u32 speed;
+ int ret;
+
+ /* handle Port Reset */
+ portsc_val = readl(&uccr->portsc);
+ trace_crg_udc_portsc(portsc_val);
+ writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+ tmp = readl(&uccr->portsc);
+
+ if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+ tmp = readl(&uccr->portsc);
+ if (tmp & CRG_U3DC_PORTSC_PRC) {
+ dev_dbg(crg_udc->dev, "PRC is still set\n");
+ } else if (tmp & CRG_U3DC_PORTSC_PR) {
+ /* first port status change event for port reset */
+ dev_dbg(crg_udc->dev,
+ "PRC is not set, but PR is set\n");
+ } else {
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ (!(tmp & CRG_U3DC_PORTSC_PED))) {
+ dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+ tmp);
+ return 0;
+ }
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ return 0;
+ }
+
+ if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+ /*
+ * A single enumeration commonly involves two
+ * bus resets (an initial one, then another
+ * after SET_ADDRESS before SET_CONFIGURATION).
+ * By the second reset device_state is already
+ * USB_STATE_ADDRESS, which is >= DEFAULT but
+ * has no active configuration to tear down.
+ * Only notify composite when a configuration
+ * was actually active: otherwise this races
+ * with the SET_CONFIGURATION that's about to
+ * arrive, since the deferred work below can
+ * run after fsg_set_alt() has already raised
+ * its own FSG_STATE_CONFIG_CHANGE exception
+ * and reset_config()'s ->disable() would
+ * overwrite that exception's argument back to
+ * NULL before the function's thread consumes
+ * it, silently skipping the real enable.
+ */
+ bool was_configured = crg_udc->device_state ==
+ USB_STATE_CONFIGURED;
+ int reinit_ret;
+
+ reinit_ret = crg_udc_reinit(crg_udc);
+ if (reinit_ret) {
+ dev_dbg(crg_udc->dev,
+ "bus reset reinit failed: %d\n",
+ reinit_ret);
+ return reinit_ret;
+ }
+
+ /*
+ * Notify the function/composite driver of the
+ * bus reset so it tears down the active
+ * configuration (usb_gadget_udc_reset() ->
+ * ->reset()). Without this, a host that
+ * re-sends the same SET_CONFIGURATION value
+ * after a reset can see composite treat it as
+ * a no-op and skip re-running set_alt(), even
+ * though every endpoint was just disabled in
+ * software above.
+ *
+ * This must not be called synchronously here:
+ * usb_gadget_udc_reset() re-enters the
+ * function drivers, which requires dropping
+ * udc_lock. Doing that mid-event-processing
+ * would let a second port-status event run
+ * crg_udc_reinit() again while this event's
+ * SET_CONFIGURATION/set_alt() work is still
+ * landing on another thread, tearing the
+ * just-enabled endpoints back down before
+ * they can be used. Defer it to a work item
+ * so this function never has to drop the
+ * lock.
+ */
+ if (was_configured && crg_udc->async_callbacks &&
+ crg_udc->gadget_driver)
+ schedule_work(&crg_udc->udc_reset_work);
+ }
+
+ crg_udc->gadget.speed = speed;
+
+ crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+ }
+ }
+ /* handle Port Reset end */
+
+ /* Connection changes must not mask independent link-state events. */
+ if (portsc_val & CRG_U3DC_PORTSC_CSC) {
+ tmp = readl(&uccr->portsc);
+ if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+ (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ (!(tmp & CRG_U3DC_PORTSC_PED)))
+ return 0;
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ return 0;
+ }
+ crg_udc->gadget.speed = speed;
+
+ crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+
+ } else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
+ bool cable_connected;
+ int ccs_drop_ignore = 0;
+
+ if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+ (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+ CRG_U3DC_PORTSC_SPEED_SS)) {
+ ccs_drop_ignore = 1;
+ dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+ }
+
+ if (!ccs_drop_ignore)
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+ cable_connected = crg_udc_vbus_connected(crg_udc);
+ if (cable_connected && !ccs_drop_ignore) {
+ int ret;
+
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, flags);
+ ret = crg_udc_reinit(crg_udc);
+ if (ret)
+ return ret;
+ crg_udc->connected = 0;
+ } else if (!cable_connected) {
+ dev_dbg(crg_udc->dev,
+ "cable disconnected, rst controller\n");
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, flags);
+ ret = crg_udc_reset(crg_udc, flags);
+ if (!ret)
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "controller reset failed on disconnect: %d\n",
+ ret);
+ else
+ crg_udc->resetting = false;
+ crg_udc->connected = 0;
+
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ return -ECONNRESET;
+ }
+ }
+ }
+ /* handle Port Connection Change end */
+
+ if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+ tmp = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+ tmp &= ~CRG_U3DC_PORTSC_W1C_MASK;
+ FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+ tmp |= CRG_U3DC_PORTSC_LWS;
+ writel(tmp, &uccr->portsc);
+ }
+ }
+
+ return 0;
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ u32 dw3 = le32_to_cpu(event->dw3);
+ u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+ u16 comp_code =
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_transfer_trb *trb;
+ bool trbs_dequeued = false;
+ u32 residual;
+
+ if (dci >= CRG_NUM_EP_CX || dci == 1)
+ return -EINVAL;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ if (!udc_ep_ptr->first_trb ||
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ return -ENODEV;
+
+ if (comp_code != CMPL_CODE_STOPPED &&
+ comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_DISABLED &&
+ comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_HALTED &&
+ comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
+ crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+
+ if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
+ trbs_dequeued = true;
+ goto queue_more_trbs;
+ }
+
+ switch (comp_code) {
+ case CMPL_CODE_SUCCESS:
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb || list_empty(&udc_ep_ptr->queue))
+ break;
+
+ /*
+ * A control transfer has separate data-stage and status-stage
+ * TRBs. The status request is an internal zero-length request
+ * and its successful completion must not go through
+ * crg_udc_build_td(). Doing that attempts to queue another
+ * control TD while the status request is already on the EP0
+ * software queue, leaving it stuck in -EINPROGRESS. The next
+ * SETUP then sees a non-empty queue and enumeration fails.
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_XFER_STATUS_STAGE) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ if (dci == 0) {
+ udc_req_ptr->usb_req.actual = 0;
+ trbs_dequeued = true;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+ } else {
+ dev_err(crg_udc->dev,
+ "status-stage TRB on endpoint %u\n",
+ dci);
+ }
+ break;
+ }
+
+ if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ trbs_dequeued = true;
+ udc_req_ptr->td_pending = false;
+
+ if (udc_req_ptr->all_trbs_queued) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ if (residual > udc_req_ptr->usb_req.actual)
+ residual = 0;
+ udc_req_ptr->usb_req.actual -= residual;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+ if (dci == 0 && udc_ep_ptr->desc &&
+ usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_ep0_req_complete(udc_ep_ptr,
+ flags);
+ } else {
+ int ret;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret == -ENOSPC || ret == -EBUSY) {
+ /* A previous TD completed, but this next TD
+ * could not yet be published. Retry later.
+ */
+ udc_ep_ptr->tran_ring_full = true;
+ } else if (ret) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+ ret);
+ }
+ }
+ }
+ break;
+
+ case CMPL_CODE_SHORT_PKT:
+ if (list_empty(&udc_ep_ptr->queue))
+ break;
+
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb)
+ break;
+
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ udc_req_ptr->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)) -
+ min_t(u32, residual,
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)));
+ udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
+ udc_req_ptr->usb_req.length);
+
+ /* Update the ring before dropping udc_lock for giveback. */
+ trbs_dequeued = true;
+ crg_udc_advance_dequeue_pt(udc_ep_ptr);
+ if (udc_req_ptr->usb_req.short_not_ok &&
+ usb_endpoint_dir_out(udc_ep_ptr->desc))
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+ else
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ if (dci == 0 && udc_ep_ptr->desc &&
+ crg_udc->ep0_state == EP0_STATE_IDLE &&
+ usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+ break;
+
+ case CMPL_CODE_PROTOCOL_STALL:
+ case CMPL_CODE_TRB_STALL:
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+ trbs_dequeued = true;
+ }
+ if (dci == 0)
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ /*
+ * EP0 SETUP changes are serialized by the SETUP event handler.
+ * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+ * EP0, gives back the old request, resets the EP0 ring, and then
+ * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+ * afterwards therefore belongs to the aborted control transfer.
+ *
+ * Do not complete the current EP0 request or change setup_status
+ * here: doing so could operate on the new control transfer and
+ * recreate the EP0 queue/state race this path is intended to avoid.
+ */
+ if (dci == 0) {
+ dev_dbg(crg_udc->dev,
+ "ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+ break;
+ }
+
+ /* This completion code is meaningful only for EP0. */
+ dev_warn(crg_udc->dev,
+ "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+ break;
+
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ case CMPL_CODE_RING_UNDERRUN:
+ case CMPL_CODE_RING_OVERRUN:
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ case CMPL_CODE_USB_TRANS_ERR:
+ case CMPL_CODE_TRB_ERR:
+ dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+ comp_code);
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+ trbs_dequeued = true;
+ }
+ if (dci)
+ set_ep_halt(crg_udc, dci, flags);
+ break;
+
+ case CMPL_CODE_STOPPED:
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ case CMPL_CODE_HALTED:
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ case CMPL_CODE_DISABLED:
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ break;
+
+ default:
+ dev_err(crg_udc->dev,
+ "unsupported transfer completion code 0x%x\n",
+ comp_code);
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+ trbs_dequeued = true;
+ }
+ break;
+ }
+
+queue_more_trbs:
+ /* Even if the ring was not full, a later request may be waiting
+ * behind a TD from an earlier request (32 TRBs per TD).
+ */
+ if (trbs_dequeued && dci != 0 &&
+ udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
+ !crg_udc->resetting) {
+ udc_ep_ptr->tran_ring_full = false;
+ if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ }
+
+ return 0;
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ int ret;
+
+ trace_crg_udc_event_trb(event);
+
+ switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+ crg_udc->portsc_on_reconnecting = 1;
+ break;
+ }
+ /* 1.Port Reset 2.Port Connection Change 3.Port Link Change */
+ ret = crg_udc_handle_port_status(crg_udc, flags);
+ if (ret)
+ return ret;
+
+ break;
+ case TRB_TYPE_EVT_TRANSFER:
+ if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+ dev_err(crg_udc->dev,
+ "Xfer compl event rcved when dev state=%d !\n",
+ crg_udc->device_state);
+ break;
+ }
+
+ ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+ if (ret)
+ return ret;
+ break;
+ case TRB_TYPE_EVT_SETUP_PKT: {
+ struct usb_ctrlrequest setup;
+ u32 dw0 = le32_to_cpu(event->dw0);
+ u32 dw1 = le32_to_cpu(event->dw1);
+ u8 setup_tag;
+
+ BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+ setup.bRequestType = dw0 & 0xff;
+ setup.bRequest = (dw0 >> 8) & 0xff;
+ setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+ setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+ setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+ setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+ le32_to_cpu(event->dw3));
+ if (crg_udc->setup_status != WAIT_FOR_SETUP) {
+ /* A new SETUP aborts the previous control transfer. The old
+ * EP0 request must be completed and its TRBs discarded before
+ * the new setup is handed to the gadget driver.
+ */
+ if (crg_udc_abort_ep0_for_setup(crg_udc)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ break;
+ }
+ }
+
+ crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
+
+ break;
+ }
+ default:
+ dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+ FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+ break;
+ }
+
+ return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ /* Report link loss even when there is no gadget callback. */
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+ if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+ !crg_udc->gadget_driver->disconnect)
+ return;
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ crg_udc->gadget_driver->disconnect(&crg_udc->gadget);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ enum crg_ep_state ep_state;
+ bool reset_seq_only = false;
+ int do_halt;
+ u32 param0;
+ u32 tmp;
+
+ if (crg_udc->resetting)
+ return -ESHUTDOWN;
+ if (!udc_ep_ptr->desc) {
+ dev_err(crg_udc->dev, "NULL desc\n");
+ return -EINVAL;
+ }
+
+ if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+ dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (udc_ep_ptr->dci == 0)
+ return 0;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+ if (ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ if (ep_state == EP_STATE_HALTED) {
+ if (halt != 0)
+ return 0;
+
+ /* want unhalt an halted ep */
+ if (udc_ep_ptr->wedge && !ignore_wedge) {
+ do_halt = -1;
+ reset_seq_only = true;
+ } else {
+ do_halt = 0;
+ }
+
+ } else {
+ /* ep state == running or stopped */
+ if (halt != 0) {
+ /* want halt a running ep */
+ do_halt = 1;
+ } else {
+ /* reset a running ep */
+ do_halt = 0;
+ reset_seq_only = true;
+ }
+ }
+
+ trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+ param0 = BIT(udc_ep_ptr->dci);
+ if (do_halt == 1) {
+ /* setting ep to halt */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+ return -EIO;
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & param0), 10, 10000))
+ dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
+ udc_ep_ptr->dci);
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ }
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+ } else if (do_halt == 0) {
+ /* clearing ep halt state */
+ /* reset sequence number */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+
+ if (!reset_seq_only) {
+ /* Clear halt for a halted EP. */
+ /* NOTE: we must CLEAR_HALT first,
+ * then SET_TR_DQPTR
+ */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_CLEAR_HALT,
+ param0, 0))
+ return -EIO;
+ if (crg_udc_epcx_update_dqptr(udc_ep_ptr))
+ return -EIO;
+ }
+
+ udc_ep_ptr->wedge = 0;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ /* set endpoint to running state */
+ /* clear pause for the endpoint */
+ if (!list_empty(&udc_ep_ptr->queue))
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ } else {
+ /* wedged EP deny CLEAR HALT */
+ dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+ udc_ep_ptr->dci);
+ /* reset sequence number */
+ if (reset_seq_only &&
+ crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+ }
+
+ return 0;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+
+ /*
+ * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+ * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+ * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+ */
+ if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+ return EP_STATE_DISABLED;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+ return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+ if (index == 0)
+ return 0;
+
+ return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+ ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+
+ /* ep0 is not included in ep contexts in crg udc */
+ buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+ if (!crg_udc->ep_cx.cpu_addr) {
+ crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+ if (!crg_udc->ep_cx.cpu_addr) {
+ ret = -ENOMEM;
+ return ret;
+ }
+ } else {
+ mapping = crg_udc->ep_cx.dma_addr;
+ }
+
+ crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+ crg_udc->ep_cx.len = buff_length;
+ crg_udc->ep_cx.dma_addr = mapping;
+ memset(crg_udc->p_epcx, 0, buff_length);
+ /*
+ * Ensure the endpoint contexts are initialized before making their
+ * DMA address visible to the controller.
+ */
+ dma_wmb();
+ /* hw ops DCBAPLO DCBAPHI */
+ writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+ writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+ return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ /* udc_ep[0] is reserved */
+ crg_ep_struct_setup(crg_udc, 0, NULL);
+
+ for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+ char name[14];
+
+ snprintf(name, sizeof(name), "ep%din", i);
+ crg_ep_struct_setup(crg_udc, i * 2, name);
+ snprintf(name, sizeof(name), "ep%dout", i);
+ crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+ }
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event)
+{
+ struct crg_udc_request *udc_req;
+ u32 trb_pt_lo = le32_to_cpu(event->dw0);
+ u32 trb_pt_hi = le32_to_cpu(event->dw1);
+ u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+ struct crg_transfer_trb *trb_pt;
+ bool status = true;
+
+ if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
+ return false;
+
+ trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+ if (!trb_pt)
+ return false;
+ list_for_each_entry(udc_req, &udc_ep->queue, queue) {
+ if (trb_pt == udc_req->last_trb ||
+ trb_pt == udc_req->first_trb) {
+ status = false;
+ break;
+ }
+
+ if (crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+ udc_ep) &&
+ crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+ udc_ep)) {
+ status = false;
+ break;
+ }
+ }
+
+ return status;
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+ u32 tmp;
+ struct crg_uccr __iomem *uccr;
+
+ uccr = crg_udc->uccr;
+ /* Ensure everything is written before notifying the HW */
+ dma_wmb();
+
+ tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+ writel(tmp, &uccr->doorbell);
+
+ trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+ if (likely(udc_req->usb_req.status == -EINPROGRESS))
+ udc_req->usb_req.status = status;
+
+ trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+ list_del_init(&udc_req->queue);
+
+ crg_udc_unmap_request(udc_ep, udc_req);
+
+ if (udc_req->usb_req.complete) {
+ spin_unlock(&crg_udc->udc_lock);
+ usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+ spin_lock(&crg_udc->udc_lock);
+ }
+}
+
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+ unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+ u32 param0;
+ u32 tmp;
+
+ if (dci == 0)
+ return 0;
+
+ if (udc_ep_ptr->ep_state == EP_STATE_DISABLED ||
+ udc_ep_ptr->ep_state == EP_STATE_HALTED)
+ return 0;
+ if (udc_ep_ptr->desc && usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+ return -EOPNOTSUPP;
+
+ param0 = BIT(dci);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+ return -EIO;
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & param0),
+ 10, 10000))
+ return -ETIMEDOUT;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+ return 0;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+ udc_req_ptr->buff_len_left = 0;
+ udc_req_ptr->trbs_needed = 0;
+ udc_req_ptr->all_trbs_queued = 0;
+ udc_req_ptr->td_pending = false;
+ udc_req_ptr->first_trb = NULL;
+ udc_req_ptr->last_trb = NULL;
+ udc_req_ptr->sg = NULL;
+ udc_req_ptr->sg_index = 0;
+ udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+ .enable = crg_udc_ep_enable,
+ .disable = crg_udc_ep_disable,
+ .alloc_request = crg_udc_alloc_request,
+ .free_request = crg_udc_free_request,
+ .queue = crg_udc_ep_queue,
+ .dequeue = crg_udc_ep_dequeue,
+ .set_halt = crg_udc_ep_set_halt,
+ .set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+ .get_frame = crg_gadget_get_frame,
+ .wakeup = crg_gadget_wakeup,
+ .set_remote_wakeup = crg_gadget_set_remote_wakeup,
+ .set_selfpowered = crg_gadget_set_selfpowered,
+ .pullup = crg_gadget_pullup,
+ .udc_start = crg_gadget_start,
+ .udc_stop = crg_gadget_stop,
+ .udc_set_speed = crg_udc_set_speed,
+ .udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (4 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:46 ` sashiko-bot
2026-10-30 9:34 ` Philipp Zabel
5 siblings, 2 replies; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/Kconfig | 2 +
drivers/usb/gadget/udc/Makefile | 1 +
drivers/usb/gadget/udc/crg/Kconfig | 16 +
drivers/usb/gadget/udc/crg/Makefile | 9 +
drivers/usb/gadget/udc/crg/crg-udc-core.c | 723 ++++++++++++++++++++++++++++++
5 files changed, 751 insertions(+)
diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
source "drivers/usb/gadget/udc/cdns2/Kconfig"
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
#
# LAST -- dummy/emulated controller
#
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC) += aspeed_udc.o
obj-$(CONFIG_USB_BDC_UDC) += bdc/
obj-$(CONFIG_USB_MAX3420_UDC) += max3420_udc.o
obj-$(CONFIG_USB_CDNS2_UDC) += cdns2/
+obj-$(CONFIG_USB_CRG_UDC) += crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+ tristate "Corigine / Axiado USB Device Controller"
+ depends on USB_GADGET
+ depends on HAS_DMA
+ depends on ARCH_AXIADO || COMPILE_TEST
+ help
+ Say Y here to support the USB device controller found on Axiado AX3000
+ and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+ This controller supports High-Speed and SuperSpeed.
+
+ Say "y" to link the driver statically, or "m" to build a
+ dynamically linked module called "crg-udc.ko".
+
+ If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o := -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC) += crg-udc.o
+
+crg-udc-y := crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING) += crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..33be5c920ebe
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,723 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc;
+
+ crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+ kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status;
+ u32 tmp;
+ bool running;
+
+ running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
+ /* Only INIT_EP0 may be staged while the controller is stopped. */
+ if (!running && type != CRG_CMD_INIT_EP0)
+ return -ESHUTDOWN;
+
+ /*
+ * A staged command must complete before the command registers can
+ * be reused. Hardware does not consume commands while RUN is clear.
+ */
+ tmp = readl(&uccr->cmd_control);
+ if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+ if (!running)
+ return -EBUSY;
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE),
+ 1, 10000))
+ return -ETIMEDOUT;
+ }
+ /* Ensure that everything is written before issuing new command */
+ dma_wmb();
+
+ writel(param0, &uccr->cmd_param0);
+ writel(param1, &uccr->cmd_param1);
+
+ tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+ FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+ writel(tmp, &uccr->cmd_control);
+
+ if (!running)
+ return 0;
+
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+ 1000)) {
+ dev_dbg(crg_udc->dev, "command timed out\n");
+ return -ETIMEDOUT;
+ }
+
+ status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+ if (status) {
+ dev_dbg(crg_udc->dev, "command failed with status %u\n",
+ status);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+ clk_disable_unprepare(data);
+}
+
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ if (crg_udc->uccr) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ if (crg_udc->irq >= 0) {
+ devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+ crg_udc->irq = -1;
+ crg_udc->gadget.irq = 0;
+ }
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+ if (udc_event->event_ring.cpu_addr) {
+ dma_free_coherent(crg_udc->dev,
+ udc_event->event_ring.len,
+ udc_event->event_ring.cpu_addr,
+ udc_event->event_ring.dma_addr);
+ udc_event->event_ring.cpu_addr = NULL;
+ udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->event_ring.len = 0;
+ udc_event->evt_dq_pt = NULL;
+ }
+ if (udc_event->erst.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->erst.len = 0;
+ udc_event->p_erst = NULL;
+ }
+ }
+
+ if (crg_udc->ep_cx.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+ crg_udc->ep_cx.cpu_addr,
+ crg_udc->ep_cx.dma_addr);
+ crg_udc->ep_cx.cpu_addr = NULL;
+ crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+ crg_udc->ep_cx.len = 0;
+ crg_udc->p_epcx = NULL;
+ }
+
+ for (i = 0; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+
+ if (udc_ep->tran_ring_info.cpu_addr) {
+ dma_free_coherent(crg_udc->dev,
+ udc_ep->tran_ring_info.len,
+ udc_ep->tran_ring_info.cpu_addr,
+ udc_ep->tran_ring_info.dma_addr);
+ udc_ep->tran_ring_info.cpu_addr = NULL;
+ udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+ udc_ep->tran_ring_info.len = 0;
+ udc_ep->first_trb = NULL;
+ udc_ep->last_trb = NULL;
+ udc_ep->enq_pt = NULL;
+ udc_ep->deq_pt = NULL;
+ }
+ }
+
+ if (crg_udc->status_req) {
+ crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+ &crg_udc->status_req->usb_req);
+ crg_udc->status_req = NULL;
+ }
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+ struct device *dev = crg_udc->dev;
+ struct reset_control *rst;
+ int ret;
+
+ crg_udc->clk = devm_clk_get_optional(dev, NULL);
+ if (IS_ERR(crg_udc->clk))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+ "failed to get clock\n");
+
+ ret = clk_prepare_enable(crg_udc->clk);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+ ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+ if (ret)
+ return ret;
+
+ rst =
+ devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+ if (IS_ERR(rst))
+ return dev_err_probe(dev, PTR_ERR(rst),
+ "failed to get reset\n");
+
+ crg_udc->phy = devm_phy_optional_get(dev, NULL);
+ if (IS_ERR(crg_udc->phy))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+ "failed to get phy\n");
+
+ ret = phy_init(crg_udc->phy);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to init phy\n");
+
+ ret = phy_power_on(crg_udc->phy);
+ if (ret) {
+ phy_exit(crg_udc->phy);
+ return dev_err_probe(dev, ret, "failed to power on phy\n");
+ }
+
+ return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+ struct phy *phy = data;
+
+ phy_power_off(phy);
+ phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+ unsigned long flags = 0;
+ int ret;
+ int i;
+ struct crg_gadget_dev *crg_udc;
+ const struct crg_udc_priv *priv;
+
+ priv = device_get_match_data(&pdev->dev);
+ if (!priv)
+ return -ENODEV;
+
+ crg_udc = kzalloc_obj(*crg_udc, GFP_KERNEL);
+ if (!crg_udc)
+ return -ENOMEM;
+ usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+ crg_udc_gadget_release);
+
+ crg_udc->dev = &pdev->dev;
+ crg_udc->irq = -1;
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (ret) {
+ dev_err(&pdev->dev, "no usable DMA mask\n");
+ goto err_free_resources;
+ }
+ }
+
+ ret = crg_udc_init_hw(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+ crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+ crg_udc->ep0_desc.bEndpointAddress = 0;
+ crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+ crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+ crg_udc->u12_initiate_forbidden = priv->quirks &
+ CRG_QUIRK_DIS_U1U2_INITIATION;
+ crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+ CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+ spin_lock_init(&crg_udc->udc_lock);
+ INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+ platform_set_drvdata(pdev, crg_udc);
+
+ crg_udc->gadget.ops = &crg_gadget_ops;
+ crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+ INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+ /*
+ * The controller supports up to SuperSpeed. Honour the standard
+ * "maximum-speed" property to cap it; default to the hardware
+ * maximum when the property is absent.
+ */
+ crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+ if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
+ crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+ crg_udc->max_speed = crg_udc->gadget.max_speed;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ crg_udc->gadget.name = KBUILD_MODNAME;
+ crg_udc->gadget.sg_supported = true;
+ crg_udc->gadget.wakeup_capable = true;
+ crg_udc->connected = 0;
+ crg_udc->dev_addr = 0;
+
+ crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(crg_udc->mmio_virt_base)) {
+ ret = dev_err_probe(&pdev->dev,
+ PTR_ERR(crg_udc->mmio_virt_base),
+ "failed to map registers\n");
+ goto err_put_gadget;
+ }
+
+ /* set controller device role */
+ writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+ CRG_UDC_ROLE_DEVICE),
+ crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+ i * CRG_UICR_STRIDE;
+ }
+ crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+ /* Initialize endpoint objects before reset, DMA setup or any IRQ. */
+ crg_udc_init_ep_info(crg_udc);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ ret = crg_udc_reset(crg_udc, &flags);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc_clear_portpm(crg_udc);
+
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+ goto err_free_resources;
+ }
+
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+ /* Release the reset gate only after EP0 and all event rings exist. */
+ crg_udc->resetting = false;
+
+ ret = crg_gadget_irq_init(pdev, crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ ret = usb_add_gadget(&crg_udc->gadget);
+ if (ret)
+ goto err_free_resources;
+
+ return 0;
+
+err_free_resources:
+ crg_udc_free_resources(crg_udc);
+err_put_gadget:
+ usb_put_gadget(&crg_udc->gadget);
+ return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ u32 tmp;
+
+ /*
+ * The gadget core must notify/unbind the function driver before we
+ * change the controller's visible state or disable VBUS detection.
+ */
+ usb_del_gadget(&crg_udc->gadget);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+ crg_udc_free_resources(crg_udc);
+
+ /* set controller host role */
+ tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+ writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+
+ platform_set_drvdata(pdev, NULL);
+ usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->pullup_on = false;
+ crg_udc->resetting = true;
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc_set_pullup(crg_udc, false);
+ crg_udc_set_vbus_detect(crg_udc, false);
+ if (crg_udc->uccr) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+}
+
+static int crg_udc_suspend(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+
+ /* Stop the pullup while the controller is suspended. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->resetting)
+ crg_udc_set_pullup(crg_udc, false);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+}
+
+static int crg_udc_resume(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->resetting && crg_udc->gadget_driver && crg_udc->pullup_on)
+ crg_udc_set_pullup(crg_udc, true);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+ u32 cfg, speed;
+
+ switch (crg_udc->max_speed) {
+ case USB_SPEED_FULL:
+ speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+ break;
+ case USB_SPEED_HIGH:
+ speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+ break;
+ case USB_SPEED_SUPER:
+ speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+ break;
+ case USB_SPEED_SUPER_PLUS:
+ speed = CRG_U3DC_CFG0_MAXSPEED_SSP;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ cfg = readl(&crg_udc->uccr->config0);
+ cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+ cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+ writel(cfg, &crg_udc->uccr->config0);
+
+ return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+ int ret;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+
+ val = readl(&uccr->control);
+ val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+ writel(val, &uccr->control);
+
+ ret = crg_udc_program_speed(crg_udc);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ ret = crg_udc_init_event_ring(crg_udc, i);
+ if (ret)
+ return ret;
+ }
+
+ ret = crg_udc_init_device_context(crg_udc);
+ if (ret)
+ return ret;
+
+ if (!crg_udc->status_req) {
+ struct usb_request *req;
+
+ req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+ GFP_ATOMIC);
+ if (!req)
+ return -ENOMEM;
+
+ crg_udc->status_req =
+ container_of(req, struct crg_udc_request, usb_req);
+ }
+
+ /* other hw ops */
+ return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags = 0;
+ u32 tmp_status;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ tmp_status = readl(&uccr->status);
+
+ if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+ bool was_connected = crg_udc->connected;
+
+ dev_err(crg_udc->dev, "controller system error\n");
+ writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+ /* Function drivers must see enabled endpoints on disconnect. */
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, &flags);
+ crg_udc->connected = 0;
+ ret = crg_udc_reset(crg_udc, &flags);
+ if (!ret)
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "failed to recover from controller error: %d\n",
+ ret);
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ if (!ret)
+ crg_udc->resetting = false;
+ if (!ret && crg_udc->pullup_on)
+ crg_udc_set_pullup(crg_udc, true);
+
+ /*
+ * tmp_status was captured before the reset above, which tore
+ * down and rebuilt the event rings. It no longer reflects
+ * the controller's current state, and if recovery failed the
+ * rings may not be consistent at all. Skip the rest of this
+ * interrupt's processing; any event still pending will raise
+ * a fresh interrupt once the controller is running again.
+ */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+
+ if (tmp_status & CRG_U3DC_STATUS_EINT) {
+ int i;
+
+ writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "process event ring error: %d\n", ret);
+ }
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->portsc_on_reconnecting == 1 &&
+ crg_udc_is_event_rings_empty(crg_udc)) {
+ crg_udc->portsc_on_reconnecting = 0;
+ crg_udc_handle_port_status(crg_udc, &flags);
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->connected == 1)
+ crg_udc_prepare_for_setup(crg_udc);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status = readl(&uccr->status);
+
+ if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+ return IRQ_NONE;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+
+ crg_gadget_handle_interrupt(crg_udc);
+
+ return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc)
+{
+ int irq;
+ int ret;
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ /*
+ * IRQF_ONESHOT keeps this (level-triggered) line masked until
+ * crg_udc_irq_thread() has cleared the status bits, which is the
+ * correct pairing for IRQF_SHARED with a threaded handler.
+ */
+ ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+ crg_udc_irq_thread,
+ IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+ crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+ } else {
+ crg_udc->irq = irq;
+ crg_udc->gadget.irq = irq;
+ }
+
+ return ret;
+}
+
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 i, tmp;
+ struct crg_udc_ep *udc_ep_ptr;
+ u32 val;
+
+ /*
+ * Keep all endpoint operations gated until the caller has recreated
+ * the event rings, device context and EP0 after this reset.
+ */
+ crg_udc->resetting = true;
+ val = readl(&uccr->control);
+ val |= CRG_U3DC_CTRL_SWRST;
+ writel(val, &uccr->control);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ /* Poll until SWRST clears; hardware reset takes up to 10 ms */
+ if (readl_poll_timeout_atomic(&uccr->control, tmp,
+ !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ dev_err(crg_udc->dev, "reset error\n");
+ return -ETIMEDOUT;
+ }
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+
+ crg_udc_clear_portpm(crg_udc);
+
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ /* Base on Figure 9-1, default USB_STATE is attached */
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc->dev_addr = 0;
+
+ /*
+ * Mark all endpoints unavailable before request callbacks drop the
+ * lock. A callback must not enqueue on a later, not-yet-nuked EP.
+ */
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ udc_ep_ptr = &crg_udc->udc_ep[i];
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ /*
+ * This bypasses the generic usb_ep_disable() API, so the
+ * framework's own usb_ep.enabled bookkeeping needs to be
+ * cleared directly here too (crg_udc_reinit() already does
+ * this for the live bus-reset case). Otherwise a later
+ * usb_ep_enable() call for this endpoint short-circuits on
+ * the stale "already enabled" flag without ever calling our
+ * .enable callback, leaving desc/the ring never set up for
+ * the new session.
+ */
+ udc_ep_ptr->usb_ep.enabled = 0;
+ }
+ crg_udc->num_enabled_eps = 0;
+
+ udc_ep_ptr = &crg_udc->udc_ep[0];
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+ crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+
+ return 0;
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+ .quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+ { .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+ { },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static DEFINE_SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend,
+ crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+ .probe = crg_udc_probe,
+ .remove = crg_udc_remove,
+ .shutdown = crg_udc_shutdown,
+ .driver = {
+ .name = KBUILD_MODNAME,
+ .of_match_table = of_crg_udc_match,
+ .pm = pm_sleep_ptr(&crg_udc_pm_ops),
+ },
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-10-30 8:14 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:14 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb
> Add a Devicetree binding for the USB device controller integrated into
> the Axiado AX3000 and AX3005 SoCs.
>
> The controller is based on a Corigine USB 3.1 IP core that is
> dual-role capable (device and host), used here in device role. It
> provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
> supports control, bulk, interrupt, and isochronous transfer types
> across multiple configurable endpoints.
>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-10-30 8:21 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:21 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb
> This patch introduces the hardware definitions, register offsets, and
> fundamental data structures for the Axiado AX3000/AX3005 USB Device
> Controller driver.
>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=2
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-10-30 8:29 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:29 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] usb: gadget: crg: out-of-bounds TRB read in crg_udc_update_dequeue_pt()
- [High] usb: gadget: crg: premature wrap skips pause in crg_udc_actual_before()
- [High] usb: gadget: crg: crg_udc_process_event_ring() misses in-progress reset after lock drop
--
commit 050a1d6b53db454c24d8712a72246d62e4e494d9
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add ring and transfer management
This commit introduces memory management and ring buffer logic for TRBs and
event rings required by the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
> new file mode 100644
> index 0000000000000..f1de8893b790f
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
[ ... ]
> +int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
> + unsigned long *flags)
> +{
> + struct crg_event_trb *event;
> + struct crg_udc_event *udc_event;
> + struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
> + u32 val;
> + dma_addr_t erdp;
> + int ret;
> + u32 generation;
> + unsigned int processed = 0;
> +
> + if (!uicr)
> + return -ENODEV;
> +
> + val = readl(&uicr->iman);
> + val |= CRG_U3DC_IMAN_INT_PEND;
> + writel(val, &uicr->iman);
> +
> + udc_event = &crg_udc->udc_event[index];
> + generation = udc_event->generation;
> + while (udc_event->evt_dq_pt) {
> + event = (struct crg_event_trb *)udc_event->evt_dq_pt;
> +
> + if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
> + le32_to_cpu(event->dw3)) != udc_event->ccs)
> + break;
> + /* Read the payload only after observing device ownership. */
> + dma_rmb();
> +
> + ret = crg_udc_handle_event(crg_udc, event, flags);
> + /* The giveback callback may release the lock and reset the ring. */
> + if (unlikely(generation != udc_event->generation))
> + return 0;
> + if (ret == -ECONNRESET) {
> + /* The disconnect path rebuilt and reprogrammed the ring. */
> + return 0;
> + }
> + if (ret < 0)
> + dev_dbg(crg_udc->dev, "failed to process event: %d\n",
> + ret);
> +
> + /* Consume malformed events so they cannot pin the ring head. */
> + if (event == udc_event->evt_seg0_last_trb) {
> + udc_event->ccs = udc_event->ccs ? 0 : 1;
> + udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
> + } else {
> + udc_event->evt_dq_pt++;
> + }
> +
> + /*
> + * udc_event state is only re-read from crg_udc, never cached
> + * across this drop, so releasing and reacquiring the lock
> + * here is safe and lets other udc_lock users, and this CPU's
> + * other interrupts, make progress during a busy event ring.
> + */
> + if (++processed % CRG_EVENT_RING_BATCH == 0) {
> + erdp = event_trb_virt_to_dma(udc_event,
> + udc_event->evt_dq_pt);
> + if (!dma_mapping_error(crg_udc->dev, erdp)) {
> + writel(upper_32_bits(erdp), &uicr->erdphi);
> + writel(lower_32_bits(erdp) |
> + CRG_U3DC_ERDPLO_EHB,
> + &uicr->erdplo);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + cond_resched();
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (unlikely(generation != udc_event->generation))
> + return 0;
> + }
> + }
[Severity: High]
As reported by Sashiko on v3 [1], can crg_udc_process_event_ring() miss an
in-progress reset when crg_udc->udc_lock is dropped inside
crg_udc_handle_event() or across the cond_resched() batch boundary?
When crg_gadget_stop() -> crg_udc_reset() runs on another CPU while
crg_udc->udc_lock is released, it sets crg_udc->resetting = true, asserts
CRG_U3DC_CTRL_SWRST, and drops crg_udc->udc_lock while polling for SWRST to
clear and again during crg_udc_ep_nuke() -> crg_udc_req_done() givebacks.
Because udc_event->generation is only incremented later when
crg_udc_reset_data_struct() -> crg_udc_init_event_ring() runs (and is not
incremented at all if crg_udc_reset() times out),
crg_udc_process_event_ring() can re-acquire crg_udc->udc_lock during those
windows with generation == udc_event->generation still true while
crg_udc->resetting is set.
Would this allow crg_udc_process_event_ring() to continue advancing
udc_event->evt_dq_pt, processing stale pre-reset events, and writing
uicr->erdphi and uicr->erdplo while the controller is mid-reset? Should the
post-lock checks in crg_udc_process_event_ring() also test
crg_udc->resetting, similar to crg_udc_queue_pending_trbs() and
crg_udc_handle_setup_pkt()?
[ ... ]
> +struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
> + dma_addr_t address)
> +{
> + dma_addr_t offset;
> + struct crg_transfer_trb *trb_virt;
> +
> + if (lower_32_bits(address) & 0xf)
> + return NULL;
> +
> + offset = address - udc_ep->tran_ring_info.dma_addr;
> + if (unlikely(offset >= udc_ep->tran_ring_info.len))
> + return NULL;
> + offset = offset / sizeof(struct crg_transfer_trb);
> + trb_virt = udc_ep->first_trb + offset;
> + return trb_virt;
> +}
> +
> +dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
> + struct crg_transfer_trb *trb)
> +{
> + unsigned long offset;
> + int trb_idx;
> + dma_addr_t dma_addr = 0;
> +
> + trb_idx = trb - udc_ep->first_trb;
> + if (unlikely(trb_idx < 0))
> + return DMA_MAPPING_ERROR;
> +
> + offset = trb_idx * sizeof(*trb);
> + if (unlikely(offset >= udc_ep->tran_ring_info.len))
> + return DMA_MAPPING_ERROR;
> + dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
> + return dma_addr;
> +}
> +
> +void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
> + struct crg_udc_ep *udc_ep)
> +{
> + u32 deq_pt_lo = le32_to_cpu(event->dw0);
> + u32 deq_pt_hi = le32_to_cpu(event->dw1);
> + u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> + struct crg_transfer_trb *deq_pt;
> +
> + deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
> + if (!deq_pt)
> + return;
> + deq_pt++;
> + if (deq_pt == udc_ep->last_trb)
> + deq_pt = udc_ep->first_trb;
> +
> + if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
> + deq_pt = udc_ep->first_trb;
> +
> + udc_ep->deq_pt = deq_pt;
> +}
[Severity: High]
What happens in crg_udc_update_dequeue_pt() if the event TRB's DMA address
points to the Link TRB at udc_ep->last_trb (for example, on a
CMPL_CODE_TRB_ERR event)?
Because udc_ep->last_trb sits at &udc_ep->first_trb[ring_size - 1],
crg_udc_tran_trb_dma_to_virt() accepts its DMA address and returns
deq_pt == udc_ep->last_trb.
Advancing deq_pt++ before checking against udc_ep->last_trb moves deq_pt to
udc_ep->last_trb + 1, which is one element past the end of the
udc_ep->tran_ring_info coherent DMA buffer.
Doesn't that cause the subsequent read of deq_pt->dw3 to access memory out of
bounds and potentially store udc_ep->last_trb + 1 into udc_ep->deq_pt, which
in turn causes crg_udc_room_on_ring() to hit its loop guard and treat the
entire ring as free?
> +
> +u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
> +{
> + u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
> + (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
> +
> + return min_t(u64, remaining, boundary);
> +}
> +
> +u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
> + struct crg_udc_request *udc_req,
> + struct crg_transfer_trb *pause)
> +{
> + struct crg_transfer_trb *trb = udc_req->first_trb;
> + u32 actual = 0;
> + u32 guard = 0;
> + u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
> +
> + if (!trb || !pause)
> + return 0;
> +
> + while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
> + u32 len;
> +
> + if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
> + TRB_TYPE_LINK) {
> + trb = udc_ep->first_trb;
> + continue;
> + }
> + len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
> + actual += len;
> + trb++;
> + if (trb == udc_ep->last_trb)
> + trb = udc_ep->first_trb;
> + }
> +
> + return min(actual, udc_req->usb_req.length);
> +}
[Severity: High]
In crg_udc_actual_before(), if crg_udc_ep_dequeue() stops an active endpoint
while a multi-TRB request wraps around the transfer ring and the hardware
dequeue pointer (pause) is at the Link TRB (udc_ep->last_trb), will this loop
skip right past pause?
At the end of the iteration for udc_ep->last_trb - 1, trb++ reaches
udc_ep->last_trb and is immediately wrapped to udc_ep->first_trb before the
while (trb != pause && ...) condition is evaluated:
trb++;
if (trb == udc_ep->last_trb)
trb = udc_ep->first_trb;
Because trb jumps from udc_ep->last_trb - 1 directly to udc_ep->first_trb, it
never compares equal to pause == udc_ep->last_trb (and the TRB_TYPE_LINK
check at the top of the loop is also bypassed), causing
crg_udc_actual_before() to sum all remaining queued TRBs up to
udc_ep->enq_pt into udc_req->usb_req.actual.
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-3-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=3
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-10-30 8:40 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:40 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] usb: gadget: udc: crg: stale setup_status on EP0 halt in crg_udc_set_ep0_halt()
- [High] usb: gadget: udc: crg: missing state checks after relocking in crg_udc_handle_setup_pkt()
- [High] usb: gadget: crg: missing reset check after ep0 nuke in setup paths
- [Medium] usb: gadget: crg: crg_udc_prepare_for_setup() skips stopping halted EP0
--
commit 0dcda6f8ceb35327942fb48b7002136244fccc5c
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add endpoint 0 handling
This patch adds endpoint 0 setup packet, status stage, and control transfer
handling for the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> new file mode 100644
> index 0000000000000..073d9090dd07c
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
[ ... ]
> +int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + dma_addr_t dqptaddr;
> + u32 cmd_param0;
> + u32 cmd_param1;
> + u32 tmp;
> + int ret;
[ ... ]
> + udc_ep0->ep_state = EP_STATE_STOPPED;
> + /* Complete requests belonging to the aborted control transfer. */
> + crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
> +
> + if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
> + crg_udc->ep0_state = EP0_STATE_ERROR;
> + return -ENODEV;
> + }
[Severity: High]
When crg_udc_ep_nuke() runs here in crg_udc_abort_ep0_for_setup(), and
similarly in crg_udc_prepare_for_setup(), crg_udc_req_done() drops
crg_udc->udc_lock to invoke the request completion callback.
While udc_lock is dropped, another CPU running crg_gadget_stop() or
crg_udc_shutdown() can acquire udc_lock and call crg_udc_reset(), setting
crg_udc->resetting = true, incrementing udc_ep[0].generation, and dropping
udc_lock while polling for CRG_U3DC_CTRL_SWRST to clear, followed by
crg_udc_reset_data_struct() and crg_udc_init_ep0().
After crg_udc_ep_nuke() returns, neither crg_udc_abort_ep0_for_setup() nor
crg_udc_prepare_for_setup() checks crg_udc->resetting, udc_ep[0].generation,
or udc_event[0].generation before touching controller registers and issuing
CRG_CMD_INIT_EP0. If crg_udc_reset() is still polling SWRST, this issues
MMIO commands mid-reset; if crg_gadget_stop() has already staged
CRG_CMD_INIT_EP0 with CRG_U3DC_CMD_CTRL_ACTIVE set while RUN is clear, the
subsequent CRG_CMD_INIT_EP0 fails with -EBUSY, leaving ep0_state in
EP0_STATE_ERROR and causing crg_udc_handle_event() to call
crg_udc_set_ep0_halt() and queue status_req after the gadget has stopped.
Should both functions snapshot the generation counter before
crg_udc_ep_nuke() and abort if crg_udc->resetting is set or the generation
changed while udc_lock was dropped?
[ ... ]
> +int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep0_ptr;
> + dma_addr_t dqptaddr;
> + u32 cmd_param0;
> + u32 cmd_param1;
> + int ret;
> +
> + if (!crg_udc_is_event_rings_empty(crg_udc) ||
> + crg_udc->portsc_on_reconnecting == 1)
> + return -EBUSY;
> +
> + udc_ep0_ptr = &crg_udc->udc_ep[0];
> + /* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
> + * align with enqueue pointer, all remaining xfer trbs became dumb once
> + * which will not produce xfer event anymore.
> + *
> + * If we considering the opposite solution, we should wait all ep0 xfer
> + * trbs be completed(with some err complete code)
> + */
> + if (crg_udc->reinit_ep0_on_bus_reset) {
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 running;
> +
> + /* Block requeue from giveback callbacks, then quiesce DMA. */
> + crg_udc->ep0_state = EP0_STATE_ABORTING;
> + if (crg_udc_get_ep_state(crg_udc, 0) == EP_STATE_RUNNING) {
> + ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> + BIT(0), 0);
> + if (ret)
> + goto reset_error;
> +
> + ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
> + !(running & BIT(0)), 10, 10000);
> + if (ret)
> + goto reset_error;
> + }
> + udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
> + crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
[Severity: Medium]
As reported by Sashiko on v3 [1], what happens if a USB bus reset occurs
after a control transfer has stalled?
When crg_udc_set_ep0_halt() stalls EP0, it calls crg_udc_build_ep0_status()
to queue a status-stage stall TRB, rings doorbell 0, places status_req on
udc_ep[0].queue, and immediately sets udc_ep[0].ep_state = EP_STATE_HALTED
in software while the hardware is still processing the TRB.
If a bus reset then triggers crg_udc_prepare_for_setup() while ep_state is
EP_STATE_HALTED, checking only for EP_STATE_RUNNING skips issuing
CRG_CMD_STOP_EP and polling uccr->ep_running before crg_udc_ep_nuke() and
CRG_CMD_INIT_EP0. Should crg_udc_prepare_for_setup() stop EP0 whenever it
is not disabled or stopped, matching crg_udc_abort_ep0_for_setup()?
[ ... ]
> +int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
> + enum crg_ep_state ep_state;
> +
> + ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
> + if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
> + return 0;
> +
> + crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
> +
> + udc_ep_ptr->ep_state = EP_STATE_HALTED;
> +
> + return 0;
> +}
[Severity: High]
As reported by Sashiko on v3 [2], does crg_udc_set_ep0_halt() leave
crg_udc->setup_status in a stale data-stage state when stalling a control
transfer with wLength > 0?
In crg_udc_handle_setup_pkt() and getstatusrequest(), crg_udc->setup_status
is set to DATA_STAGE_XFER or DATA_STAGE_RECV before calling setselrequest(),
crg_udc_build_td(), or driver->setup():
crg_udc_handle_setup_pkt()
crg_udc->setup_status = DATA_STAGE_XFER / DATA_STAGE_RECV;
driver->setup() -> fails
crg_udc_set_ep0_halt()
crg_udc_build_ep0_status(..., stall = 1)
Because crg_udc_set_ep0_halt() does not update crg_udc->setup_status to
STATUS_STAGE_XFER (or clear crg_udc->setup_fn_call_back), two problems
follow:
1. crg_udc_setup_status_trb() computes dir as
(crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1, which evaluates
to 1 (OUT) instead of 0 (IN).
2. When the stall status TRB completes and invokes
crg_udc_ep0_req_complete(), switch (crg_udc->setup_status) matches
DATA_STAGE_XFER or DATA_STAGE_RECV instead of the default branch:
crg_udc_ep0_req_complete() {
switch (crg_udc->setup_status) {
case DATA_STAGE_XFER:
crg_udc->setup_status = STATUS_STAGE_RECV;
crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
0, 0);
break;
case DATA_STAGE_RECV:
crg_udc->setup_status = STATUS_STAGE_XFER;
crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
0, 0);
break;
This queues a second, unrequested, non-stalled status-stage TRB on EP0 and
rings doorbell 0 on the halted endpoint. Should crg_udc_set_ep0_halt() set
crg_udc->setup_status = STATUS_STAGE_XFER and clear
crg_udc->setup_fn_call_back before calling crg_udc_build_ep0_status()?
[ ... ]
> +void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
> + struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
> + unsigned long *flags)
> +{
> + u16 w_value = le16_to_cpu(setup_pkt->wValue);
> + u16 w_index = le16_to_cpu(setup_pkt->wIndex);
> + u16 w_length = le16_to_cpu(setup_pkt->wLength);
> + struct usb_gadget_driver *driver;
> + u32 tmp;
> +
> + /* No SETUP (including SET_ADDRESS) may overtake a pending reset. */
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + flush_work(&crg_udc->udc_reset_work);
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (crg_udc->resetting || !crg_udc->gadget_driver ||
> + crg_udc->ep0_state == EP0_STATE_ABORTING)
> + return;
[ ... ]
> + if (w_length) {
> + /* data phase from gadget like GET_CONFIGURATION
> + * call the setup routine of gadget driver.
> + * remember the request direction.
> + */
> + crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
> + DATA_STAGE_XFER :
> + DATA_STAGE_RECV;
> + }
> + driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> +
> + /* Recheck callback enablement after the locked snapshot. */
> + if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
> + READ_ONCE(crg_udc->gadget_driver) != driver ||
> + driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + crg_udc_set_ep0_halt(crg_udc);
> + return;
> + }
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
> + (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
> + (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
> + if (w_value) {
> + crg_udc->device_state = USB_STATE_CONFIGURED;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_CONFIGURED);
> + } else {
> + /* Endpoint disable may already have updated device_state. */
> + crg_udc->device_state = USB_STATE_ADDRESS;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_ADDRESS);
> + }
> + }
> +}
[Severity: High]
Can dropping crg_udc->udc_lock in crg_udc_handle_setup_pkt() race with a
concurrent reset, unbind, or new setup packet?
First, around flush_work(&crg_udc->udc_reset_work), if a new setup packet
aborted an in-flight control transfer, crg_udc_abort_ep0_for_setup() has
already set ep0_state = EP0_STATE_IDLE before udc_lock is dropped for
flush_work(), allowing a concurrent usb_ep_queue() on EP0 to bypass the
EP0_STATE_ABORTING check in crg_udc_queue_ctrl(). In addition, if a
controller reset (crg_udc_reset() and crg_udc_reset_data_struct())
completes while udc_lock is dropped, crg_udc->resetting is false again and
udc_event[0].generation has been incremented, but crg_udc_handle_setup_pkt()
does not verify udc_event[0].generation and continues processing the stale
pre-reset setup packet.
Second, after udc_lock is dropped for driver->setup() and reacquired,
neither the error branch nor the USB_REQ_SET_CONFIGURATION branch rechecks
crg_udc->resetting, crg_udc->gadget_driver, crg_udc->async_callbacks,
crg_udc->ep0_state, crg_udc->setup_tag, or udc_event[0].generation:
1. If async_callbacks was cleared during gadget stop/unbind, or if
crg_udc_reset() is currently polling SWRST with udc_lock dropped (where
crg_udc->resetting is true while udc_ep[0].ep_state is still
EP_STATE_RUNNING), the error path unconditionally calls
crg_udc_set_ep0_halt(crg_udc), queueing a status TRB and ringing
doorbell 0 during reset or teardown.
2. If driver->setup() succeeded for USB_REQ_SET_CONFIGURATION while a
concurrent reset or disconnect occurred, the post-lock block overwrites
crg_udc->device_state and the gadget state with USB_STATE_CONFIGURED or
USB_STATE_ADDRESS.
Should crg_udc_handle_setup_pkt() keep ep0_state in EP0_STATE_ABORTING
until flush_work() finishes, and validate the event generation,
crg_udc->resetting, crg_udc->async_callbacks, and crg_udc->setup_tag after
reacquiring udc_lock?
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-4-3012ecfc0a93@axiado.com
[2] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-4-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=4
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
@ 2026-10-30 8:46 ` sashiko-bot
2026-10-30 9:34 ` Philipp Zabel
1 sibling, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:46 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] usb: udc: crg: dequeue fails in crg_udc_issue_command() when stopped
- [High] usb: gadget: crg: controller left stopped after disconnect reset
- [High] usb: gadget: crg: deferred crg_udc_prepare_for_setup() races with SETUP
- [High] usb: gadget: crg: GFP_KERNEL allocation in crg_udc_ep_enable()
- [High] usb: gadget: crg: race conditions when crg_udc_reset() drops udc_lock
- [Medium] usb: udc: crg: stale reconnect and ep0 state across crg_udc_reset()
- [Medium] usb: gadget: udc: crg: state inconsistency across crg_udc_suspend() and resume
- [Medium] usb: gadget: udc: crg: race in crg_udc_suspend() re-enables pullup
- [Medium] usb: gadget: crg: race with IRQ thread in crg_udc_shutdown() and teardown
- [Low] usb: gadget: crg: uncapped and unvalidated max_speed in crg_udc_probe()
--
commit ff1f5c0791ff275876830e2f2710870a97b34e7c
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add core driver and build integration
This commit introduces the core probe, remove, power management, interrupt
handling, and controller reset functions for the Axiado AX3000/AX3005 USB
Device Controller driver, and integrates it into Kconfig and Makefile.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 0000000000000..33be5c920ebee
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> @@ -0,0 +1,723 @@
[ ... ]
> +int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
> + enum crg_cmd_type type, u32 param0, u32 param1)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 status;
> + u32 tmp;
> + bool running;
> +
> + running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
> + /* Only INIT_EP0 may be staged while the controller is stopped. */
> + if (!running && type != CRG_CMD_INIT_EP0)
> + return -ESHUTDOWN;
[Severity: High]
As reported by Sashiko on v3 [1], rejecting all commands other than
CRG_CMD_INIT_EP0 when CRG_U3DC_CTRL_RUN is clear causes crg_udc_ep_dequeue()
to fail with -ESHUTDOWN whenever the pullup has been disabled.
During gadget unbind (gadget_unbind_driver() -> usb_gadget_disconnect_locked()
-> crg_gadget_pullup()), soft_connect disable, or crg_udc_suspend(),
crg_udc_set_pullup(crg_udc, false) clears CRG_U3DC_CTRL_RUN in uccr->control
while enabled endpoints remain in EP_STATE_RUNNING with active requests on
udc_ep->queue:
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_dequeue() {
...
if (old_ep_state == EP_STATE_RUNNING) {
int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
BIT(dci), 0);
if (ret) {
spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
return ret;
}
...
}
When a function driver's disconnect() or unbind() callback (or a FunctionFS
endpoint close) calls usb_ep_dequeue() -> crg_udc_ep_dequeue() before
crg_gadget_stop() runs, crg_udc_issue_command() returns -ESHUTDOWN for
CRG_CMD_STOP_EP (and would also reject CRG_CMD_SET_TR_DQPTR in
crg_udc_epcx_update_dqptr()). Because crg_udc_ep_dequeue() aborts without
unlinking or giving back the request, a subsequent usb_ep_free_request()
frees the request while it is still linked on udc_ep->queue, leading to a
use-after-free when crg_gadget_stop() -> crg_udc_reset() -> crg_udc_ep_nuke()
later traverses udc_ep->queue.
How should crg_udc_ep_dequeue() unlink and give back queued requests when
CRG_U3DC_CTRL_RUN is already clear?
[ ... ]
> +static int crg_udc_probe(struct platform_device *pdev)
> +{
[ ... ]
> + crg_udc->gadget.ops = &crg_gadget_ops;
> + crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
> + INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
> + /*
> + * The controller supports up to SuperSpeed. Honour the standard
> + * "maximum-speed" property to cap it; default to the hardware
> + * maximum when the property is absent.
> + */
> + crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
> + if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
> + crg_udc->gadget.max_speed = USB_SPEED_SUPER;
> + crg_udc->max_speed = crg_udc->gadget.max_speed;
[Severity: Low]
The comment, Kconfig, and DT binding state that the controller supports
High-Speed and SuperSpeed and that the maximum-speed property caps the speed
up to SuperSpeed, but crg_udc_probe() only checks for USB_SPEED_UNKNOWN.
If the DT maximum-speed property is set to super-speed-plus
(USB_SPEED_SUPER_PLUS), crg_udc->gadget.max_speed is not capped to
USB_SPEED_SUPER and crg_udc_program_speed() writes CRG_U3DC_CFG0_MAXSPEED_SSP.
Conversely, if maximum-speed (or a gadget driver's max_speed passed to
crg_udc_set_speed()) is USB_SPEED_LOW, crg_udc_program_speed() returns
-EINVAL and aborts crg_udc_reset_data_struct().
Should crg_udc_probe() and crg_udc_set_speed() clamp speeds above
USB_SPEED_SUPER (or outside the supported range) to USB_SPEED_SUPER?
[ ... ]
> +static void crg_udc_shutdown(struct platform_device *pdev)
> +{
> + struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + crg_udc->pullup_on = false;
> + crg_udc->resetting = true;
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + crg_udc_set_pullup(crg_udc, false);
> + crg_udc_set_vbus_detect(crg_udc, false);
> + if (crg_udc->uccr) {
> + u32 val = readl(&crg_udc->uccr->control);
> +
> + val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
> + writel(val, &crg_udc->uccr->control);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (crg_udc->irq >= 0)
> + synchronize_irq(crg_udc->irq);
> + cancel_work_sync(&crg_udc->udc_reset_work);
> +}
[Severity: Medium]
As reported by Sashiko on v3 [2], what happens if crg_udc_irq_thread() is
already mid-recovery in crg_gadget_handle_interrupt() (or in the disconnect
reset path of crg_udc_handle_port_status()) and has dropped udc_lock inside
crg_udc_notify_disconnect() or crg_udc_reset() when crg_udc_shutdown() runs?
Once crg_udc_shutdown() releases udc_lock and waits in synchronize_irq(), the
in-flight IRQ thread reacquires udc_lock, runs crg_udc_reset_data_struct() and
crg_udc_init_ep0(), and sets crg_udc->resetting = false, undoing the shutdown
gate.
Similarly, in crg_udc_free_resources(), uccr->control is modified without
holding udc_lock before devm_free_irq() is called, while crg_udc_irq_thread()
can still be running after crg_gadget_stop() dropped udc_lock during
crg_udc_reset(). Couldn't that race on uccr->control or with resource
teardown?
> +
> +static int crg_udc_suspend(struct device *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> + unsigned long flags;
> +
> + /* Stop the pullup while the controller is suspended. */
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!crg_udc->resetting)
> + crg_udc_set_pullup(crg_udc, false);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> +}
[Severity: Medium]
Can an in-flight CRG_U3DC_STATUS_SYS_ERR recovery in
crg_gadget_handle_interrupt() re-enable the pullup after crg_udc_suspend()
returns?
During system suspend, crg_udc_suspend() runs before device IRQs are disabled.
If crg_gadget_handle_interrupt() is handling CRG_U3DC_STATUS_SYS_ERR and has
dropped udc_lock in crg_udc_notify_disconnect() or crg_udc_reset(),
crg_udc_suspend() either sees crg_udc->resetting == true and skips
crg_udc_set_pullup(crg_udc, false), or clears CRG_U3DC_CTRL_RUN and returns
while crg_udc->pullup_on remains true. When the IRQ thread finishes reset
recovery, it checks:
if (!ret && crg_udc->pullup_on)
crg_udc_set_pullup(crg_udc, true);
which re-asserts CRG_U3DC_CTRL_RUN while the controller is suspended. Should
crg_udc_suspend() synchronize with the IRQ/reset path and track a suspended
state so recovery does not re-assert CRG_U3DC_CTRL_RUN?
> +
> +static int crg_udc_resume(struct device *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!crg_udc->resetting && crg_udc->gadget_driver && crg_udc->pullup_on)
> + crg_udc_set_pullup(crg_udc, true);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + return 0;
> +}
[Severity: Medium]
When crg_udc_suspend() clears CRG_U3DC_CTRL_RUN via
crg_udc_set_pullup(crg_udc, false) while the gadget is enumerated, the
controller halts and drops bus terminations so the host sees a disconnect,
but crg_udc_suspend() does not notify the gadget driver of disconnect or
reset crg_udc->connected, crg_udc->device_state, crg_udc->dev_addr, or
enabled endpoints.
When crg_udc_resume() re-asserts CRG_U3DC_CTRL_RUN:
1. If the cable was unplugged while suspended, CCS is already 0 when
CRG_U3DC_CTRL_RUN is set, so no CSC event is generated and the gadget
driver remains stuck in a connected/configured state with queued requests
never completing.
2. If the cable remains connected to a SuperSpeed host, link training to U0
raises CRG_U3DC_PORTSC_CSC without CRG_U3DC_PORTSC_PRC. In
crg_udc_handle_port_status(), the CSC connect branch does not call
crg_udc_reinit(), and because crg_udc->device_state is still
USB_STATE_CONFIGURED or USB_STATE_ADDRESS (>= USB_STATE_RECONNECTING),
neither crg_udc_enable_setup() nor crg_udc_prepare_for_setup() runs, the
stale dev_addr and endpoints remain active, and setaddressrequest() stalls
SET_ADDRESS in USB_STATE_CONFIGURED.
Should crg_udc_suspend() disconnect the gadget session and reset controller
state before suspending?
[ ... ]
> +static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + unsigned long flags = 0;
> + u32 tmp_status;
> + int ret;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> + tmp_status = readl(&uccr->status);
> +
> + if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
> + bool was_connected = crg_udc->connected;
> +
> + dev_err(crg_udc->dev, "controller system error\n");
> + writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> + /* Function drivers must see enabled endpoints on disconnect. */
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, &flags);
> + crg_udc->connected = 0;
> + ret = crg_udc_reset(crg_udc, &flags);
> + if (!ret)
> + ret = crg_udc_reset_data_struct(crg_udc);
> + if (!ret)
> + ret = crg_udc_init_ep0(crg_udc);
> + if (ret)
> + dev_err(crg_udc->dev,
> + "failed to recover from controller error: %d\n",
> + ret);
> + crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> + if (!ret)
> + crg_udc->resetting = false;
> + if (!ret && crg_udc->pullup_on)
> + crg_udc_set_pullup(crg_udc, true);
[Severity: High]
Here in the CRG_U3DC_STATUS_SYS_ERR recovery path, after crg_udc_reset(),
crg_udc_reset_data_struct(), and crg_udc_init_ep0() complete, the driver
restores the controller's running state via crg_udc_set_pullup(crg_udc, true)
when crg_udc->pullup_on is set.
However, in the cable disconnect (!cable_connected) branch of
crg_udc_handle_port_status(), the driver runs the same crg_udc_reset(),
crg_udc_reset_data_struct(), and crg_udc_init_ep0() sequence (which clears
uccr->config1 as well as CRG_U3DC_CTRL_INT_EN and CRG_U3DC_CTRL_RUN in
uccr->control, and stages CRG_CMD_INIT_EP0 while stopped) and then returns
-ECONNRESET without calling crg_udc_set_pullup(crg_udc, true):
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_port_status() {
...
} else if (!cable_connected) {
dev_dbg(crg_udc->dev,
"cable disconnected, rst controller\n");
if (was_connected)
crg_udc_notify_disconnect(crg_udc, flags);
ret = crg_udc_reset(crg_udc, flags);
if (!ret)
ret = crg_udc_reset_data_struct(crg_udc);
if (!ret)
ret = crg_udc_init_ep0(crg_udc);
if (ret)
dev_err(crg_udc->dev,
"controller reset failed on disconnect: %d\n",
ret);
else
crg_udc->resetting = false;
crg_udc->connected = 0;
crg_udc->device_state = USB_STATE_ATTACHED;
return -ECONNRESET;
}
}
Because gadget disconnect callbacks do not re-invoke the pullup callback,
won't unplugging the USB cable leave CRG_U3DC_CTRL_RUN, CRG_U3DC_CTRL_INT_EN,
and uccr->config1 cleared so the controller never consumes CRG_CMD_INIT_EP0
or detects a subsequent cable reconnection?
[ ... ]
> + if (tmp_status & CRG_U3DC_STATUS_EINT) {
> + int i;
> +
> + writel(CRG_U3DC_STATUS_EINT, &uccr->status);
> +
> + for (i = 0; i < CRG_RING_NUM; i++) {
> + ret = crg_udc_process_event_ring(crg_udc, i, &flags);
[Severity: High]
When crg_udc_process_event_ring() dispatches a SET_CONFIGURATION or
SET_INTERFACE control transfer via crg_udc_handle_event() ->
crg_udc_handle_setup_pkt(), it drops crg_udc->udc_lock and calls
driver->setup(&crg_udc->gadget, setup_pkt).
In composite_setup(), set_config() and f->set_alt() run under
spin_lock(&cdev->lock) and call usb_ep_enable() -> crg_udc_ep_enable().
However, crg_udc_ep_enable() drops udc_lock and allocates the transfer ring
with GFP_KERNEL:
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_enable() {
...
spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
GFP_KERNEL);
spin_lock_irqsave(&crg_udc->udc_lock, flags);
...
}
Since usb_ep_enable() is called while composite holds cdev->lock (and the
gadget API requires ep->ops->enable() to be callable in atomic context),
won't dma_alloc_coherent(..., GFP_KERNEL) sleep in atomic context?
> + if (ret)
> + dev_err(crg_udc->dev,
> + "process event ring error: %d\n", ret);
> + }
> + }
> +
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->portsc_on_reconnecting == 1 &&
> + crg_udc_is_event_rings_empty(crg_udc)) {
> + crg_udc->portsc_on_reconnecting = 0;
> + crg_udc_handle_port_status(crg_udc, &flags);
> + }
> +
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->connected == 1)
> + crg_udc_prepare_for_setup(crg_udc);
[Severity: High]
When a bus reset (CRG_U3DC_PORTSC_PRC) completes while crg_udc->device_state
is at least USB_STATE_DEFAULT, crg_udc_handle_port_status() calls
crg_udc_reinit(), which sets crg_udc->device_state = USB_STATE_RECONNECTING,
and sets crg_udc->connected = 1 while CRG_U3DC_CFG1_SETUP_EVENT_EN remains
enabled in uccr->config1 from the prior connection.
Because crg_udc_prepare_for_setup() is only called here after
crg_udc_process_event_ring() finishes (and returns -EBUSY if the event ring
is not empty), what happens if the host's first post-reset SETUP packet is
already in the event ring behind the port status event, or arrives before
crg_udc_prepare_for_setup() checks crg_udc_is_event_rings_empty()?
In that case, crg_udc_process_event_ring() processes TRB_TYPE_EVT_SETUP_PKT
while crg_udc->device_state is still USB_STATE_RECONNECTING:
setaddressrequest() rejects USB_REQ_SET_ADDRESS and stalls EP0 because
device_state is neither USB_STATE_DEFAULT nor USB_STATE_ADDRESS, or another
control request queues a transfer on EP0 only for crg_udc_prepare_for_setup()
to immediately stop EP0 and nuke the newly queued request with -ESHUTDOWN.
[ ... ]
> +int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 i, tmp;
> + struct crg_udc_ep *udc_ep_ptr;
> + u32 val;
> +
> + /*
> + * Keep all endpoint operations gated until the caller has recreated
> + * the event rings, device context and EP0 after this reset.
> + */
> + crg_udc->resetting = true;
> + val = readl(&uccr->control);
> + val |= CRG_U3DC_CTRL_SWRST;
> + writel(val, &uccr->control);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + /* Poll until SWRST clears; hardware reset takes up to 10 ms */
> + if (readl_poll_timeout_atomic(&uccr->control, tmp,
> + !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + dev_err(crg_udc->dev, "reset error\n");
> + return -ETIMEDOUT;
> + }
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> +
> + crg_udc_clear_portpm(crg_udc);
> +
> + crg_udc->setup_status = WAIT_FOR_SETUP;
> + /* Base on Figure 9-1, default USB_STATE is attached */
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + crg_udc->dev_addr = 0;
[Severity: Medium]
When crg_udc_reset() resets crg_udc->device_state to USB_STATE_ATTACHED, it
does not clear crg_udc->portsc_on_reconnecting, crg_udc->setup_fn_call_back,
or crg_udc->set_tm (nor does crg_udc_reset_data_struct()).
If a port status event sets crg_udc->portsc_on_reconnecting = 1 while
crg_udc->device_state == USB_STATE_RECONNECTING and crg_udc_reset() runs
before that flag is consumed (for example due to a cable disconnect,
CRG_U3DC_STATUS_SYS_ERR, or crg_gadget_stop()), won't the stale
portsc_on_reconnecting == 1 persist until the next connection's bus reset
sets device_state = USB_STATE_RECONNECTING, causing crg_udc_prepare_for_setup()
to fail with -EBUSY or crg_gadget_handle_interrupt() to call
crg_udc_handle_port_status() spuriously?
> +
> + /*
> + * Mark all endpoints unavailable before request callbacks drop the
> + * lock. A callback must not enqueue on a later, not-yet-nuked EP.
> + */
> + for (i = 2; i < CRG_NUM_EP_CX; i++) {
> + udc_ep_ptr = &crg_udc->udc_ep[i];
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + udc_ep_ptr->tran_ring_full = false;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + /*
> + * This bypasses the generic usb_ep_disable() API, so the
> + * framework's own usb_ep.enabled bookkeeping needs to be
> + * cleared directly here too (crg_udc_reinit() already does
> + * this for the live bus-reset case). Otherwise a later
> + * usb_ep_enable() call for this endpoint short-circuits on
> + * the stale "already enabled" flag without ever calling our
> + * .enable callback, leaving desc/the ring never set up for
> + * the new session.
> + */
> + udc_ep_ptr->usb_ep.enabled = 0;
> + }
> + crg_udc->num_enabled_eps = 0;
> +
> + udc_ep_ptr = &crg_udc->udc_ep[0];
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> +
> + for (i = 2; i < CRG_NUM_EP_CX; i++)
> + crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
> + crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
[Severity: High]
As reported by Sashiko on v3 [3], dropping udc_lock inside crg_udc_reset()
(both during the 10 ms CRG_U3DC_CTRL_SWRST poll and inside crg_udc_ep_nuke()
-> crg_udc_req_done()) introduces several races:
1. Neither crg_gadget_handle_interrupt() nor crg_udc_process_event_ring()
checks crg_udc->resetting after acquiring or reacquiring udc_lock, and
udc_event->generation is not incremented until crg_udc_reset_data_struct()
-> crg_udc_init_event_ring() runs after crg_udc_reset() returns. If
crg_udc_irq_thread() acquires udc_lock while crg_udc_reset() is polling
CRG_U3DC_CTRL_SWRST or giving back requests, it can access controller
registers and process stale event ring entries mid-reset.
2. Callers of crg_udc_reset() (crg_gadget_handle_interrupt(),
crg_udc_handle_port_status(), and crg_gadget_stop()) do not serialize
against an already-running reset. If two reset paths overlap across the
udc_lock drops, one can run crg_udc_reset_data_struct() and
crg_udc_init_ep0() while the other's CRG_U3DC_CTRL_SWRST is active, or
the second crg_udc_init_ep0() can see CRG_U3DC_CMD_CTRL_ACTIVE still set
from the first staged CRG_CMD_INIT_EP0 while CRG_U3DC_CTRL_RUN is 0 and
fail with -EBUSY.
3. During the 10 ms SWRST poll window, crg_udc->resetting is true while
usb_ep.enabled is still 1 and requests remain on udc_ep[i].queue; then
crg_udc_reset() clears udc_ep_ptr->usb_ep.enabled = 0 for all endpoints
before calling crg_udc_ep_nuke() (which drops udc_lock per request). If a
function driver concurrently calls usb_ep_disable() on a higher-numbered
endpoint while an earlier endpoint is being nuked (or during the SWRST
poll), usb_ep_disable() returns immediately without draining that
endpoint's queue; when the function driver then calls
usb_ep_free_request(), the freed request remains on udc_ep[i].queue and is
dereferenced by crg_udc_ep_nuke().
Can reset execution and endpoint teardown be serialized so concurrent callers
wait for the reset and request givebacks to complete?
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
[2] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
[3] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=6
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-10-30 8:50 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:50 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 22 potential issue(s) to consider:
- [High] usb: gadget: crg: premature dequeue pointer update in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: sleeping DMA allocation and free in crg_udc_ep_enable()
- [High] usb: gadget: crg: stale first_trb on multi-TD requests in crg_udc_queue_trbs()
- [High] usb: gadget: crg: broken zero-length EP0 data stage and dequeue handling
- [High] usb: gadget: crg: crg_udc_ep_nuke() hides requests on stack list across unlock
- [High] usb: gadget: crg: corrupt deq_pt on stale events in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: premature giveback before halt in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: silent no-op when stalling EP0 in crg_udc_ep_halt()
- [High] usb: gadget: crg: crg_udc_reinit() clears usb_ep.enabled before reset work
- [High] usb: gadget: crg: lock drop mid-loop in crg_udc_reinit() leaves EPs running
- [High] usb: gadget: crg: DMA unmap on ep_running poll timeout in crg_udc_ep_halt()
- [High] usb: gadget: udc: crg: lost port events in crg_udc_handle_port_status()
- [High] usb: gadget: udc: crg: premature SETUP handling in crg_udc_handle_event()
- [High] usb: gadget: udc: crg: stuck EP0_STATE_ERROR in crg_udc_handle_event()
- [High] usb: gadget: udc: crg: stuck EP_STATE_STOPPED in crg_udc_ep_dequeue()
- [High] usb: gadget: udc: crg: rejected halted/stopped queue in crg_udc_ep_queue()
- [Medium] usb: gadget: crg: crg_udc_ep_queue() rejects 0-length requests with NULL buf
- [Medium] usb: gadget: crg: missing IN queue check in crg_udc_ep_set_wedge()
- [Medium] usb: gadget: crg: missing usb_ep.maxpacket update in crg_udc_ep_enable()
- [Medium] usb: gadget: udc: crg: unrolled ring state on error in crg_udc_queue_trbs()
- [Medium] usb: gadget: udc: crg: clobbered tran_ring_full in crg_udc_handle_xfer_event()
- [Medium] usb: gadget: crg: missing EP0 state revalidation in crg_udc_ep_queue()
--
commit c0163bd83b42f6c6c4258f527a05306a12934416
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add gadget and endpoint operations
This patch implements the usb_gadget_ops and usb_ep_ops callbacks along with
transfer ring, endpoint state, event, and port status handling for the Axiado
AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
> new file mode 100644
> index 0000000000000..8c52f8856490a
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
[ ... ]
> +static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
> +{
> + struct crg_udc_ep *udc_ep_ptr;
> + struct crg_gadget_dev *crg_udc;
> + struct crg_uccr __iomem *uccr;
> + struct crg_udc_request *udc_req = NULL;
> + struct crg_udc_request *iter;
> + struct crg_ep_cx *p_ep_cx;
> + struct crg_transfer_trb *pause_pt;
> + enum crg_ep_state old_ep_state;
> + u32 tmp, deq_pt_lo, deq_pt_hi;
> + dma_addr_t dq_pt_addr;
> + u8 dci;
> + bool found = false;
> + u32 generation;
> + unsigned long flags;
> + const struct usb_endpoint_descriptor *old_desc;
> + u32 event_generation;
> +
> + if (!_ep || !_req)
> + return -EINVAL;
> +
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> + uccr = crg_udc->uccr;
> + dci = udc_ep_ptr->dci;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (crg_udc->resetting) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
> + old_ep_state = udc_ep_ptr->ep_state;
> + old_desc = udc_ep_ptr->desc;
> + event_generation = crg_udc->udc_event[0].generation;
> + generation = udc_ep_ptr->generation;
> +
> + list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
> + if (&iter->usb_req == _req) {
> + udc_req = iter;
> + found = true;
> + break;
> + }
> + }
> +
> + if (!found) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> +
> + if (!udc_req->first_trb) {
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> + }
> +
> + if (dci == 0) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
[Severity: High]
What happens when a gadget driver dequeues its EP0 request in
crg_udc_ep_dequeue(), for instance during __composite_unbind()?
If the request was queued through crg_udc_build_ep0_status(), first_trb is
never assigned. For a newly allocated request where first_trb is NULL,
crg_udc_ep_dequeue() completes and unlinks the request while its status TRB
is still live on the EP0 hardware ring without stopping EP0 DMA.
Conversely, if the EP0 request is a data-stage request (or a reused request
with a stale non-NULL first_trb), crg_udc_ep_dequeue() hits the dci == 0
check and returns -EINVAL without stopping EP0 or removing the request from
udc_ep[0].queue. Doesn't that lead to a use-after-free when the caller
immediately frees the usb_request on unbind?
> + if (old_ep_state == EP_STATE_RUNNING) {
> + int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> + BIT(dci), 0);
> +
> + if (ret) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> + }
> + if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> + !(tmp & BIT(dci)), 10, 10000)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ETIMEDOUT;
> + }
> + udc_ep_ptr->ep_state = EP_STATE_STOPPED;
> + }
> +
> + p_ep_cx = &crg_udc->p_epcx[dci - 2];
> + deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
> + deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
> + dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> + pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
> + if (!pause_pt) {
> + if (old_ep_state == EP_STATE_RUNNING) {
> + udc_ep_ptr->ep_state = EP_STATE_RUNNING;
> + crg_udc_knock_doorbell(crg_udc, dci);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EIO;
> + }
> +
> + if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
> + udc_ep_ptr)) {
> + crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
> + /* Give back the cancelled request BEFORE scheduling the tail:
> + * its callback may add/remove requests, and dropping udc_lock
> + * while traversing the old list is unsafe.
> + */
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else if (udc_req->last_trb &&
> + crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
> + udc_ep_ptr)) {
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else {
> + struct crg_udc_request *next_req = NULL;
> +
> + udc_req->usb_req.actual =
> + crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
> +
> + if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
> + next_req = list_entry(udc_req->queue.next,
> + struct crg_udc_request, queue);
> +
> + if (next_req && next_req->first_trb)
> + udc_ep_ptr->deq_pt = next_req->first_trb;
> + else
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> +
> + if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EIO;
> + }
> + udc_ep_ptr->tran_ring_full = false;
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + }
> +
> + if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
> + udc_ep_ptr->desc == old_desc &&
> + event_generation == crg_udc->udc_event[0].generation &&
> + udc_ep_ptr->generation == generation &&
> + udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
> + udc_ep_ptr->ep_state = EP_STATE_RUNNING;
> + udc_ep_ptr->tran_ring_full = false;
> + if (crg_udc_queue_pending_trbs(udc_ep_ptr))
> + udc_ep_ptr->tran_ring_full = true;
> + crg_udc_knock_doorbell(crg_udc, dci);
> + }
[Severity: High]
Doesn't this leave udc_ep_ptr->ep_state stuck in EP_STATE_STOPPED if
crg_udc_epcx_update_dqptr() fails, and also reject any requests queued from
the completion callback during crg_udc_req_done()?
First, if crg_udc_epcx_update_dqptr(udc_ep_ptr) fails, crg_udc_ep_dequeue()
unlocks and returns -EIO without restoring udc_ep_ptr->ep_state to
EP_STATE_RUNNING (unlike the !pause_pt error path above), leaving the
endpoint permanently wedged in EP_STATE_STOPPED.
Second, crg_udc_req_done() is called while udc_ep_ptr->ep_state is still
EP_STATE_STOPPED, before EP_STATE_RUNNING is restored. The comment notes that
the completion callback may add requests while udc_lock is dropped, but
crg_udc_ep_queue() rejects any request when ep_state != EP_STATE_RUNNING with
-ESHUTDOWN.
[ ... ]
> +static int crg_udc_ep_enable(struct usb_ep *ep,
> + const struct usb_endpoint_descriptor *desc)
> +{
[ ... ]
> + /* Leave the old allocation valid until its replacement is ready.
> + * Other ep operations reject the configuring endpoint.
> + */
> + if (!udc_ep->tran_ring_info.cpu_addr ||
> + udc_ep->tran_ring_info.len != len) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
> + GFP_KERNEL);
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!new_ring) {
> + ret = -ENOMEM;
> + goto err;
> + }
> + if (!crg_udc->gadget_driver || crg_udc->resetting ||
> + udc_ep->ep_state != EP_STATE_DISABLED ||
> + udc_ep->desc != desc) {
> + ret = -ESHUTDOWN;
> + goto err;
> + }
> + old_ring = udc_ep->tran_ring_info.cpu_addr;
> + old_dma = udc_ep->tran_ring_info.dma_addr;
> + old_len = udc_ep->tran_ring_info.len;
> + udc_ep->tran_ring_info.cpu_addr = new_ring;
> + udc_ep->tran_ring_info.dma_addr = new_dma;
> + udc_ep->tran_ring_info.len = len;
> + new_ring = NULL;
> + }
> +
> + udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
> + udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
> + memset(udc_ep->first_trb, 0, len);
> + dma_wmb();
> + crg_udc_setup_link_trb(udc_ep->last_trb, true,
> + udc_ep->tran_ring_info.dma_addr);
> + udc_ep->enq_pt = udc_ep->first_trb;
> + udc_ep->deq_pt = udc_ep->first_trb;
> + udc_ep->pcs = 1;
> + udc_ep->tran_ring_full = false;
> + ret = crg_udc_epcx_setup(udc_ep);
> + if (ret)
> + goto err;
> + param0 = BIT(udc_ep->dci);
> + ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
> + if (ret)
> + goto err;
> +
> + udc_ep->generation++;
> + udc_ep->wedge = false;
> + udc_ep->usb_ep.enabled = 1;
> + udc_ep->ep_state = EP_STATE_RUNNING;
> + udc_ep->configuring = false;
> + crg_udc->num_enabled_eps++;
> + if (crg_udc->device_state == USB_STATE_ADDRESS)
> + crg_udc->device_state = USB_STATE_CONFIGURED;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (old_ring)
> + dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
> + return 0;
> +
> +err:
> + udc_ep->desc = NULL;
> + udc_ep->comp_desc = NULL;
> + udc_ep->configuring = false;
> + /* Keep a valid ring allocation for a later retry. */
> + udc_ep->ep_state = EP_STATE_DISABLED;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (new_ring)
> + dma_free_coherent(crg_udc->dev, len, new_ring, new_dma);
> + if (old_ring)
> + dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
> + return ret;
[Severity: High]
Can crg_udc_ep_enable() sleep or call dma_free_coherent() here?
The USB gadget API allows usb_ep_enable() to be called in atomic context. For
example:
composite_setup() [holds spin_lock(&cdev->lock)]
-> set_config() / f->set_alt()
-> gserial_connect() / ffs_func_eps_enable() [holds spin_lock_irqsave()]
-> usb_ep_enable()
-> crg_udc_ep_enable()
Dropping crg_udc->udc_lock with spin_unlock_irqrestore() only restores the
interrupt state at entry to crg_udc_ep_enable() and does not release spinlocks
held by the caller. Calling dma_alloc_coherent(..., GFP_KERNEL) sleeps while
caller spinlocks are held, and calling dma_free_coherent() when interrupts
were disabled by the caller triggers WARN_ON(irqs_disabled()) in
dma_free_attrs().
[Severity: Medium]
Should crg_udc_ep_enable() also update udc_ep->usb_ep.maxpacket to
usb_endpoint_maxp(desc)?
crg_ep_struct_setup() initializes ep->usb_ep.maxpacket to CRG_MAX_PACKET_SIZE
(1024), and usb_ep_enable() in udc/core.c does not update ep->maxpacket. When
a gadget driver enables an endpoint without calling config_ep_by_speed() (such
as gadgetfs, raw_gadget, or f_printer), udc_ep->usb_ep.maxpacket stays at 1024
even for 512-byte HS or 64-byte FS endpoints, causing crg_udc_queue_trbs() to
miscalculate need_zlp via !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket).
> +out:
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> +}
> +
> +void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
> +{
> + struct crg_udc_request *req;
> + struct list_head nuke_list;
> +
> + list_replace_init(&udc_ep->queue, &nuke_list);
> +
> + while (!list_empty(&nuke_list)) {
> + req = list_first_entry(&nuke_list, struct crg_udc_request,
> + queue);
> +
> + crg_udc_req_done(udc_ep, req, status);
> + }
> +}
[Severity: High]
Can moving all requests onto the stack-local nuke_list before dropping
udc_lock in crg_udc_req_done() cause a use-after-free?
While crg_udc->udc_lock is dropped during the first request's completion
callback, udc_ep->queue is already empty even though the remaining requests on
nuke_list have not been given back yet. If that callback or another thread
calls usb_ep_dequeue() and usb_ep_free_request() on one of those remaining
requests, crg_udc_ep_dequeue() will not find it on udc_ep->queue and will
return -EINVAL immediately, allowing the caller to free the request while
crg_udc_ep_nuke() still holds it on nuke_list and dereferences it on the next
iteration.
> +static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
> + gfp_t gfp_flags)
> +{
> + struct crg_udc_request *udc_req_ptr;
> + struct crg_udc_ep *udc_ep_ptr;
> + struct crg_gadget_dev *crg_udc;
> + enum dma_data_direction dma_data_dir;
> + const struct usb_endpoint_descriptor *desc;
> + unsigned long flags;
> + int status;
> + bool is_sg;
> + u8 setup_tag = 0;
> + u32 generation = 0;
> +
> + if (!_ep || !_req)
> + return -EINVAL;
> +
> + udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> + is_sg = _req->num_sgs != 0 && _req->length != 0;
> +
> + if (is_sg && !_req->sg)
> + return -EINVAL;
> + if (_req->stream_id)
> + return -EOPNOTSUPP;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + /*
> + * A completion callback for an aborted EP0 transfer may requeue the
> + * same request. Reject EP0 requests while the previous control
> + * transfer is being aborted or EP0 is in an error state. EP0 becomes
> + * available again only after the new SETUP has been accepted and
> + * the EP0 transfer ring has been successfully reinitialized.
> + */
> + if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ECONNRESET;
> + }
> + if (!udc_ep_ptr->first_trb || !_req->complete ||
> + (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
[Severity: Medium]
Does checking (!is_sg && !_req->buf) reject valid zero-length requests when
_req->buf is NULL?
Because is_sg is false whenever _req->length == 0, any 0-length request with
_req->buf == NULL (such as scatter-gather requests recycled for 0-length
submissions by uvcg_video_hw_submit()) will fail here with -EINVAL even though
_req->length == 0 does not dereference or DMA-map _req->buf.
> + if (crg_udc->resetting) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
> + if (!udc_ep_ptr->desc) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> + desc = udc_ep_ptr->desc;
> + if (udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
> + udc_ep_ptr->configuring) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
[Severity: High]
Should crg_udc_ep_queue() reject requests when udc_ep_ptr->ep_state is
EP_STATE_HALTED or EP_STATE_STOPPED?
Standard USB gadget function drivers (such as f_mass_storage) halt the bulk-in
endpoint and immediately call usb_ep_queue() to queue the CSW status packet
while the endpoint is still halted, waiting for the host to issue
CLEAR_FEATURE(ENDPOINT_HALT). In fact, crg_udc_ep_halt() checks
!list_empty(&udc_ep_ptr->queue) when clearing a halt to ring the doorbell for
queued requests, and crg_udc_ep_dequeue() temporarily places the endpoint in
EP_STATE_STOPPED during giveback. Returning -ESHUTDOWN whenever ep_state !=
EP_STATE_RUNNING (here and in the post-mapping check below) prevents queuing
requests onto halted or paused endpoints.
> + setup_tag = crg_udc->setup_tag;
> + generation = udc_ep_ptr->generation;
> +
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EOPNOTSUPP;
> + }
> +
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
> + if (crg_udc->setup_status == DATA_STAGE_XFER)
> + crg_udc->setup_status = STATUS_STAGE_RECV;
> + else
> + crg_udc->setup_status = STATUS_STAGE_XFER;
> + crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
> + -EINPROGRESS, udc_req_ptr, 0, 0);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> + }
[Severity: High]
Doesn't this treat a 0-length EP0 Data Stage response (when the host's SETUP
had wLength > 0) as a Status Stage TRB?
When wLength > 0, crg_udc_handle_setup_pkt() sets crg_udc->setup_status to
DATA_STAGE_XFER or DATA_STAGE_RECV, and crg_udc_ep0_req_complete()
automatically queues the Status Stage after the Data Stage completes. If a
gadget driver queues a 0-byte response for the Data Stage, this branch
overwrites setup_status to STATUS_STAGE_RECV / STATUS_STAGE_XFER and calls
crg_udc_build_ep0_status(), queuing a TRB_TYPE_XFER_STATUS_STAGE TRB instead
of a 0-length TRB_TYPE_XFER_DATA_STAGE TRB.
Returning early here also skips crg_udc_clear_req_container(udc_req_ptr),
leaving stale first_trb/last_trb/all_trbs_queued fields if udc_req_ptr was
reused from a previous transfer.
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
> + if (crg_udc->setup_status == DATA_STAGE_XFER ||
> + crg_udc->setup_status == STATUS_STAGE_XFER) {
> + dma_data_dir = DMA_TO_DEVICE;
> + } else if (crg_udc->setup_status == DATA_STAGE_RECV ||
> + crg_udc->setup_status == STATUS_STAGE_RECV) {
> + dma_data_dir = DMA_FROM_DEVICE;
> + } else {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> + } else {
> + dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
> + DMA_TO_DEVICE :
> + DMA_FROM_DEVICE;
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + crg_udc_clear_req_container(udc_req_ptr);
> + udc_req_ptr->mapped = false;
> +
> + if (_req->length) {
> + udc_req_ptr->map_dir = dma_data_dir;
> + status = usb_gadget_map_request_by_dev(
> + crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
> + if (status)
> + goto err_unmap;
> +
> + if (is_sg) {
> + if (!_req->num_mapped_sgs ||
> + crg_udc_sg_init(udc_req_ptr)) {
> + status = -EIO;
> + goto err_unmap;
> + }
> + } else if (_req->dma_mapped) {
> + udc_req_ptr->mapped = true;
> + }
> + }
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
> + !list_empty(&udc_req_ptr->queue) || crg_udc->resetting ||
> + udc_ep_ptr->configuring ||
> + udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
> + udc_ep_ptr->generation != generation ||
> + (udc_ep_ptr->dci == 0 &&
> + (crg_udc->ep0_state != EP0_STATE_IDLE ||
> + crg_udc->setup_tag != setup_tag))) {
[Severity: Medium]
As reported by Sashiko on v3 [1], what happens if a USB bus reset occurs while
udc_lock is dropped for DMA mapping of an EP0 request?
During a bus reset, crg_udc_reinit() and crg_udc_prepare_for_setup() reset
crg_udc->setup_status to WAIT_FOR_SETUP and reinitialize the EP0 ring, but
crg_udc_reinit() only increments udc_ep[i].generation for i >= 2 and leaves
udc_ep[0].generation and crg_udc->setup_tag unchanged. When crg_udc_ep_queue()
reacquires udc_lock, these checks pass because setup_status is not
revalidated and udc_ep[0].generation was not bumped, allowing the stale EP0
request to be queued onto the EP0 ring while setup_status is WAIT_FOR_SETUP.
[ ... ]
> +static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
> +{
> + struct crg_udc_ep *udc_ep_ptr;
> + int status;
> + unsigned long flags = 0;
> + struct crg_gadget_dev *crg_udc;
> +
> + if (!_ep)
> + return -EINVAL;
> +
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> + if (!udc_ep_ptr->desc) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> +
> + udc_ep_ptr->wedge = 1;
> +
> + status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
[Severity: Medium]
Should crg_udc_ep_set_wedge() check for a non-empty queue on IN endpoints and
return -EAGAIN, just like crg_udc_ep_set_halt() does?
Without that check, calling usb_ep_set_wedge() on an IN endpoint with pending
requests sets udc_ep_ptr->wedge = 1 and calls crg_udc_ep_halt(), which halts
the endpoint and (at SuperSpeed) aborts queued IN transfers via
crg_udc_ep_nuke(). Also, if crg_udc_ep_halt() returns an error,
udc_ep_ptr->wedge remains set to 1.
[ ... ]
> +static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
> + struct crg_udc_request *udc_req_ptr, bool b_isoc,
> + u32 xfer_ring_size, u32 num_trbs_needed,
> + u64 buffer_length)
> +{
[ ... ]
> + if (num_trbs_ava < td_trbs) {
> + if (b_isoc)
> + return -EBUSY;
> + trace_crg_udc_no_room_on_ring(
> + "deferring TRBs to next doorbell");
> + count = num_trbs_ava;
> + full_td = false;
> + udc_ep_ptr->tran_ring_full = true;
> + need_zlp = false;
> + } else {
> + count = td_trbs;
> + }
> +
> + if (!count)
> + return -ENOSPC;
> +
> + td_size = count - 1;
> +
> + if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
> + return -EINVAL;
> +
> + u8 original_pcs = udc_ep_ptr->pcs;
> +
> + for (i = 0; i < count; i++) {
[ ... ]
> + trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
> +
> + if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
> + return -EINVAL;
> +
> + td_size--;
> + enq_pt++;
> + j++;
> +
> + if (enq_pt == udc_ep_ptr->last_trb) {
> + dma_wmb();
> + le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
> + TRB_CYCLE_BIT_MASK);
> + udc_ep_ptr->pcs ^= 1;
> + enq_pt = udc_ep_ptr->first_trb;
> + }
> + }
[Severity: Medium]
If crg_udc_queue_trbs() returns an error after setting
udc_ep_ptr->tran_ring_full = true or partway through the TRB loop, doesn't
that leave the ring state inconsistent?
When 0 < num_trbs_ava < td_trbs, udc_ep_ptr->tran_ring_full is set to true
before crg_udc_sg_init() and before the loop; if an error occurs while the
ring was previously empty, tran_ring_full stays true with no TRBs queued, so
subsequent crg_udc_ep_queue() calls will never build TDs or ring the doorbell.
Similarly, if a later loop iteration fails after writing TRBs with active
cycle bits or toggling the Link TRB cycle bit and udc_ep_ptr->pcs,
udc_ep_ptr->enq_pt is not updated and those changes are not rolled back.
> + dma_wmb();
> + le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
> + TRB_CYCLE_BIT_MASK);
> +
> + if (!udc_req_ptr->trbs_needed)
> + udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
> +
> + udc_ep_ptr->enq_pt = enq_pt;
> + if (count == num_trbs_ava)
> + udc_ep_ptr->tran_ring_full = true;
> + udc_req_ptr->buff_len_left = buffer_length;
> + udc_req_ptr->td_pending = true;
> + if (buffer_length)
> + udc_req_ptr->trbs_needed = num_trbs_needed - count;
> + else
> + udc_req_ptr->trbs_needed = 0;
> +
> + /* Track the last TRB of the current TD, even for multi-TD requests. */
> + if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
> + udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
> + else
> + udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
[Severity: High]
For a multi-TD request, why is udc_req_ptr->last_trb updated on every TD while
udc_req_ptr->first_trb is only set when !udc_req_ptr->trbs_needed?
When the first TD of a multi-TD request completes in crg_udc_handle_xfer_event()
and crg_udc_build_td() queues the next TD, udc_req_ptr->trbs_needed is
non-zero, so udc_req_ptr->first_trb remains pointing at the retired first TD
while udc_req_ptr->last_trb advances to the new TD.
Once udc_ep_ptr->enq_pt wraps around the transfer ring to or past that stale
first_trb, crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt, udc_ep) in
is_request_dequeued() returns false for active TRBs, causing valid transfer
completion events to be treated as dequeued and dropped. It also causes
crg_udc_ep_dequeue() and crg_udc_actual_before() to misclassify the active TD
range.
> + return 0;
> +}
> +
> +static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 i, tmp;
> + struct crg_udc_ep *udc_ep_ptr;
> +
> + crg_udc->setup_status = WAIT_FOR_SETUP;
> + /* Base on Figure 9-1, default USB_STATE is attached */
> + crg_udc->device_state = USB_STATE_RECONNECTING;
> +
> + /* halt all the endpoints */
> +
> + /* disable all the endpoints */
> + tmp = readl(&uccr->ep_enable);
> + writel(tmp, &uccr->ep_enable);
> + if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
> + dev_dbg(crg_udc->dev,
> + "timeout waiting for all endpoints to disable\n");
> + return -ETIMEDOUT;
> + }
> +
> + for (i = 2; i < CRG_NUM_EP_CX; i++) {
> + udc_ep_ptr = &crg_udc->udc_ep[i];
> + udc_ep_ptr->usb_ep.enabled = 0;
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + if (udc_ep_ptr->desc) {
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
> + }
> + udc_ep_ptr->tran_ring_full = false;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + }
> + crg_udc->num_enabled_eps = 0;
[Severity: High]
Doesn't clearing udc_ep_ptr->usb_ep.enabled = 0 and setting ep_state =
EP_STATE_DISABLED here in crg_udc_reinit() break gadget function teardown when
udc_reset_work runs on bus reset?
When a bus reset occurs while configured, crg_udc_handle_port_status() calls
crg_udc_reinit() first and then schedules udc_reset_work, which invokes
usb_gadget_udc_reset() -> composite_reset() -> reset_config() -> f->disable().
Standard USB function drivers (such as f_ecm, f_eem, f_rndis, f_ncm, f_acm,
and f_serial) check if (ep->enabled) in their ->disable() callbacks before
tearing down state and freeing requests, and crg_udc_ep_disable() also returns
-EINVAL if ep_state is already EP_STATE_DISABLED. Because usb_ep.enabled was
already cleared in crg_udc_reinit(), those function drivers will skip teardown
and leak requests across bus resets.
[Severity: High]
Also in this loop, crg_udc_ep_nuke() drops crg_udc->udc_lock around each
request's completion callback for endpoint i before endpoints i + 1 ..
CRG_NUM_EP_CX - 1 have been marked disabled in software.
While udc_lock is dropped during crg_udc_ep_nuke() on endpoint i, higher-
numbered endpoints still have ep_state == EP_STATE_RUNNING and desc != NULL
even though their hardware enable bits in uccr->ep_enable were just cleared,
allowing callbacks or concurrent threads to queue TRBs and ring doorbells on
disabled hardware endpoints. And if endpoint i is re-enabled while udc_lock is
dropped, the assignments after crg_udc_ep_nuke() will overwrite its state back
to EP_STATE_DISABLED and reset crg_udc->num_enabled_eps to 0.
[ ... ]
> +int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
> + unsigned long *flags)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + bool was_connected = crg_udc->connected;
> + u32 portsc_val;
> + u32 tmp;
> + u32 speed;
> + int ret;
> +
> + /* handle Port Reset */
> + portsc_val = readl(&uccr->portsc);
> + trace_crg_udc_portsc(portsc_val);
> + writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
> +
> + tmp = readl(&uccr->portsc);
> +
> + if (portsc_val & CRG_U3DC_PORTSC_PRC) {
> + tmp = readl(&uccr->portsc);
> + if (tmp & CRG_U3DC_PORTSC_PRC) {
> + dev_dbg(crg_udc->dev, "PRC is still set\n");
> + } else if (tmp & CRG_U3DC_PORTSC_PR) {
> + /* first port status change event for port reset */
> + dev_dbg(crg_udc->dev,
> + "PRC is not set, but PR is set\n");
> + } else {
> + if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
> + (!(tmp & CRG_U3DC_PORTSC_PED))) {
> + dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
> + tmp);
> + return 0;
> + }
[ ... ]
> + /* Connection changes must not mask independent link-state events. */
> + if (portsc_val & CRG_U3DC_PORTSC_CSC) {
> + tmp = readl(&uccr->portsc);
> + if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
> + (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
> + if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
> + (!(tmp & CRG_U3DC_PORTSC_PED)))
> + return 0;
> +
> + switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
> + case CRG_U3DC_PORTSC_SPEED_SSP:
> + speed = USB_SPEED_SUPER_PLUS;
> + break;
> + case CRG_U3DC_PORTSC_SPEED_SS:
> + speed = USB_SPEED_SUPER;
> + break;
> + case CRG_U3DC_PORTSC_SPEED_HS:
> + speed = USB_SPEED_HIGH;
> + break;
> +
> + case CRG_U3DC_PORTSC_SPEED_FS:
> + speed = USB_SPEED_FULL;
> + break;
> +
> + case CRG_U3DC_PORTSC_SPEED_LS:
> + speed = USB_SPEED_LOW;
> + break;
> + default:
> + speed = USB_SPEED_UNKNOWN;
> + return 0;
> + }
> + crg_udc->gadget.speed = speed;
> +
> + crg_udc_update_ep0_maxpacketsize(crg_udc);
> +
> + crg_udc->connected = 1;
> +
> + if (crg_udc->device_state < USB_STATE_RECONNECTING)
> + crg_udc_enable_setup(crg_udc);
> +
> + } else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
> + bool cable_connected;
> + int ccs_drop_ignore = 0;
> +
> + if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
> + (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
> + CRG_U3DC_PORTSC_SPEED_SS)) {
> + ccs_drop_ignore = 1;
> + dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
> + }
> +
> + if (!ccs_drop_ignore)
> + crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> +
> + cable_connected = crg_udc_vbus_connected(crg_udc);
> + if (cable_connected && !ccs_drop_ignore) {
> + int ret;
> +
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, flags);
> + ret = crg_udc_reinit(crg_udc);
> + if (ret)
> + return ret;
> + crg_udc->connected = 0;
> + } else if (!cable_connected) {
> + dev_dbg(crg_udc->dev,
> + "cable disconnected, rst controller\n");
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, flags);
> + ret = crg_udc_reset(crg_udc, flags);
> + if (!ret)
> + ret = crg_udc_reset_data_struct(crg_udc);
> + if (!ret)
> + ret = crg_udc_init_ep0(crg_udc);
> + if (ret)
> + dev_err(crg_udc->dev,
> + "controller reset failed on disconnect: %d\n",
> + ret);
> + else
> + crg_udc->resetting = false;
> + crg_udc->connected = 0;
> +
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + return -ECONNRESET;
> + }
> + }
> + }
> + /* handle Port Connection Change end */
> +
> + if (portsc_val & CRG_U3DC_PORTSC_PLC) {
[Severity: High]
As reported by Sashiko on v3 [1], can early returns in the PRC and CSC blocks
drop concurrent CSC or PLC events that were already cleared in uccr->portsc?
At the top of crg_udc_handle_port_status(), writel(portsc_val &
CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc) clears all W1C bits (including CSC
and PLC) in hardware. However, if FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0
or !(tmp & CRG_U3DC_PORTSC_PED) (or speed is unknown) in the PRC block or the
CSC connect block, the function executes return 0 instead of falling through
to the remaining status-change checks. For example, if a disconnect or link
error aborts a port reset so both PRC and CSC/PLC are set and PED is 0, or if
both CSC and PLC (with PLS == 0xf) are set, returning 0 skips the CSC and PLC
handlers after their hardware status bits have already been cleared.
Also, in the CSC disconnect branches, crg_udc_notify_disconnect() drops
udc_lock while crg_udc->connected is still 1 and before crg_udc_reinit() or
crg_udc_reset() runs, without revalidating controller state after reacquiring
udc_lock.
[ ... ]
> +int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
> + struct crg_event_trb *event, unsigned long *flags)
> +{
[ ... ]
> + if (comp_code != CMPL_CODE_STOPPED &&
> + comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
> + comp_code != CMPL_CODE_DISABLED &&
> + comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
> + comp_code != CMPL_CODE_HALTED &&
> + comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
> + crg_udc_update_dequeue_pt(event, udc_ep_ptr);
> +
> + if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
> + trbs_dequeued = true;
> + goto queue_more_trbs;\n> + }
[Severity: High]
Doesn't calling crg_udc_update_dequeue_pt() before is_request_dequeued()
corrupt udc_ep_ptr->deq_pt when a stale event arrives for an already-dequeued
request or an aborted EP0 transfer?
1. When crg_udc_ep_dequeue() cancels an active request, it advances
udc_ep_ptr->deq_pt to next_req->first_trb or udc_ep_ptr->enq_pt. If the
hardware had already posted a transfer event for the cancelled request before
CRG_CMD_STOP_EP completed, calling crg_udc_update_dequeue_pt() before
is_request_dequeued() rewinds udc_ep_ptr->deq_pt back to the stale event's
TRB + 1.
2. In is_request_dequeued(), if udc_ep->queue is empty (because the cancelled
request was the only request on the endpoint), is_request_dequeued() returns
false instead of true:
if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
return false;
That leaves udc_ep_ptr->deq_pt pointing into the middle of an empty ring and
allows a stale error event to fall through and halt the endpoint via
set_ep_halt().
3. On EP0, CMPL_CODE_SETUP_TAG_MISMATCH is not excluded from
crg_udc_update_dequeue_pt(), so a stale mismatch event arriving after
crg_udc_abort_ep0_for_setup() resets the EP0 ring will overwrite
udc_ep[0].deq_pt and break the enq_pt == dq_pt check in crg_udc_queue_ctrl().
> + switch (comp_code) {
> + case CMPL_CODE_SUCCESS:
[ ... ]
> + if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + trbs_dequeued = true;
> + udc_req_ptr->td_pending = false;
> +
> + if (udc_req_ptr->all_trbs_queued) {
> + udc_req_ptr->usb_req.actual =
> + udc_req_ptr->usb_req.length -
> + udc_req_ptr->buff_len_left;
> + residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> + le32_to_cpu(event->dw2));
> + if (residual > udc_req_ptr->usb_req.actual)
> + residual = 0;
> + udc_req_ptr->usb_req.actual -= residual;
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
> +
> + if (dci == 0 && udc_ep_ptr->desc &&
> + usb_endpoint_xfer_control(udc_ep_ptr->desc))
> + crg_udc_ep0_req_complete(udc_ep_ptr,
> + flags);
> + } else {
> + int ret;
> +
> + ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> + if (ret == -ENOSPC || ret == -EBUSY) {
> + /* A previous TD completed, but this next TD
> + * could not yet be published. Retry later.
> + */
> + udc_ep_ptr->tran_ring_full = true;
> + } else if (ret) {
> + udc_req_ptr->usb_req.actual =
> + udc_req_ptr->usb_req.length -
> + udc_req_ptr->buff_len_left;
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
> + ret);
> + }
> + }
> + }
> + break;
> +
> + case CMPL_CODE_SHORT_PKT:
> + if (list_empty(&udc_ep_ptr->queue))
> + break;
> +
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request, queue);
> + trb = crg_udc_tran_trb_dma_to_virt(
> + udc_ep_ptr,
> + (u64)le32_to_cpu(event->dw0) |
> + ((u64)le32_to_cpu(event->dw1) << 32));
> + if (!trb)
> + break;
> +
> + residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> + le32_to_cpu(event->dw2));
> + udc_req_ptr->usb_req.actual =
> + crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
> + FIELD_GET(TRB_TRANSFER_LEN_MASK,
> + le32_to_cpu(trb->dw2)) -
> + min_t(u32, residual,
> + FIELD_GET(TRB_TRANSFER_LEN_MASK,
> + le32_to_cpu(trb->dw2)));
> + udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
> + udc_req_ptr->usb_req.length);
> +
> + /* Update the ring before dropping udc_lock for giveback. */
> + trbs_dequeued = true;
> + crg_udc_advance_dequeue_pt(udc_ep_ptr);
> + if (udc_req_ptr->usb_req.short_not_ok &&
> + usb_endpoint_dir_out(udc_ep_ptr->desc))
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
> + else
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
[Severity: High]
Isn't crg_udc_advance_dequeue_pt(udc_ep_ptr) called here while udc_req_ptr is
still at the head of udc_ep_ptr->queue?
In drivers/usb/gadget/udc/crg/crg-udc-ring.c, crg_udc_advance_dequeue_pt()
sets udc_ep->deq_pt from udc_ep->queue.next:
if (!list_empty(&udc_ep->queue)) {
udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
queue);
if (udc_req->first_trb)
udc_ep->deq_pt = udc_req->first_trb;
Because udc_req_ptr is not unlinked from udc_ep_ptr->queue until
list_del_init(&udc_req->queue) inside crg_udc_req_done(), calling
crg_udc_advance_dequeue_pt() before udc_req_ptr is removed rewinds
udc_ep_ptr->deq_pt back to the completing request's own first_trb instead of
advancing it to the next request or udc_ep_ptr->enq_pt. That leaks TRB slots
in crg_udc_room_on_ring() on every short packet until the ring wedges with
-ENOSPC, and breaks subsequent EP0 transfers that require enq_pt == dq_pt in
crg_udc_queue_ctrl().
> + if (dci == 0 && udc_ep_ptr->desc &&
> + crg_udc->ep0_state == EP0_STATE_IDLE &&
> + usb_endpoint_xfer_control(udc_ep_ptr->desc))
> + crg_udc_ep0_req_complete(udc_ep_ptr, flags);
> + break;
[ ... ]
> + case CMPL_CODE_BABBLE_DETECTED_ERR:
> + case CMPL_CODE_DATA_BUFFER_ERR:
> + case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
> + case CMPL_CODE_RING_UNDERRUN:
> + case CMPL_CODE_RING_OVERRUN:
> + case CMPL_CODE_EVENT_RING_FULL_ERR:
> + case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
> + case CMPL_CODE_USB_TRANS_ERR:
> + case CMPL_CODE_TRB_ERR:
> + dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
> + comp_code);
> + if (!list_empty(&udc_ep_ptr->queue)) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
> + trbs_dequeued = true;
> + }
> + if (dci)
> + set_ep_halt(crg_udc, dci, flags);
> + break;
[Severity: High]
Doesn't calling crg_udc_req_done() before set_ep_halt() unmap the request's
DMA buffer and drop udc_lock while the endpoint is still running?
If the completion callback (or another CPU while udc_lock is dropped in
crg_udc_req_done()) queues a new request on udc_ep_ptr, crg_udc_ep_queue()
sees ep_state == EP_STATE_RUNNING and rings the doorbell; then when
crg_udc_req_done() returns, set_ep_halt() immediately halts the endpoint and
nukes the newly queued request with -ECONNRESET. Furthermore, set_ep_halt()
does not check whether crg_udc->resetting or udc_ep_ptr->generation changed
while udc_lock was dropped.
> +
> + case CMPL_CODE_STOPPED:
> + case CMPL_CODE_STOPPED_LENGTH_INVALID:
> + case CMPL_CODE_HALTED:
> + case CMPL_CODE_HALTED_LENGTH_INVALID:
> + case CMPL_CODE_DISABLED:
> + case CMPL_CODE_DISABLED_LENGTH_INVALID:
> + break;
> +
> + default:
> + dev_err(crg_udc->dev,
> + "unsupported transfer completion code 0x%x\n",
> + comp_code);
> + if (!list_empty(&udc_ep_ptr->queue)) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
> + trbs_dequeued = true;
> + }
> + break;
> + }
> +
> +queue_more_trbs:
> + /* Even if the ring was not full, a later request may be waiting
> + * behind a TD from an earlier request (32 TRBs per TD).
> + */
> + if (trbs_dequeued && dci != 0 &&
> + udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
> + !crg_udc->resetting) {
> + udc_ep_ptr->tran_ring_full = false;
> + if (crg_udc_queue_pending_trbs(udc_ep_ptr))
> + udc_ep_ptr->tran_ring_full = true;
> + }
[Severity: Medium]
As reported by Sashiko on v3 [1], doesn't queue_more_trbs clobber
udc_ep_ptr->tran_ring_full when the !udc_req_ptr->all_trbs_queued branch under
CMPL_CODE_SUCCESS just filled the ring?
When an intermediate TD of a multi-TD request completes, that branch sets
trbs_dequeued = true and calls crg_udc_build_td(udc_ep_ptr, udc_req_ptr). If
that TD consumes the remaining space on the ring (or returns -ENOSPC/-EBUSY
when td_pending is already set), udc_ep_ptr->tran_ring_full is set to true.
Execution then reaches queue_more_trbs, which sets udc_ep_ptr->tran_ring_full
= false and calls crg_udc_queue_pending_trbs(udc_ep_ptr). Because udc_req_ptr
now has td_pending == true or all_trbs_queued == true,
crg_udc_queue_pending_trbs() returns 0 without re-detecting the full ring,
leaving tran_ring_full false so the next crg_udc_ep_queue() call attempts
crg_udc_build_td() and fails with -ENOSPC.
> +
> + return 0;
> +}
> +
> +int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
> + struct crg_event_trb *event, unsigned long *flags)
> +{
> + int ret;
> +
> + trace_crg_udc_event_trb(event);
> +
> + switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
> + case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
> + if (crg_udc->device_state == USB_STATE_RECONNECTING) {
> + crg_udc->portsc_on_reconnecting = 1;
> + break;
> + }
> + /* 1.Port Reset 2.Port Connection Change 3.Port Link Change */
> + ret = crg_udc_handle_port_status(crg_udc, flags);
> + if (ret)
> + return ret;
> +
> + break;
> + case TRB_TYPE_EVT_TRANSFER:
> + if (crg_udc->device_state < USB_STATE_RECONNECTING) {
> + dev_err(crg_udc->dev,
> + "Xfer compl event rcved when dev state=%d !\n",
> + crg_udc->device_state);
> + break;
> + }
> +
> + ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
> + if (ret)
> + return ret;
> + break;
> + case TRB_TYPE_EVT_SETUP_PKT: {
> + struct usb_ctrlrequest setup;
> + u32 dw0 = le32_to_cpu(event->dw0);
> + u32 dw1 = le32_to_cpu(event->dw1);
> + u8 setup_tag;
> +
> + BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
> + setup.bRequestType = dw0 & 0xff;
> + setup.bRequest = (dw0 >> 8) & 0xff;
> + setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
> + setup.wIndex = cpu_to_le16(dw1 & 0xffff);
> + setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
> +
> + setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
> + le32_to_cpu(event->dw3));
> + if (crg_udc->setup_status != WAIT_FOR_SETUP) {
> + /* A new SETUP aborts the previous control transfer. The old
> + * EP0 request must be completed and its TRBs discarded before
> + * the new setup is handed to the gadget driver.
> + */
> + if (crg_udc_abort_ep0_for_setup(crg_udc)) {
> + crg_udc_set_ep0_halt(crg_udc);
> + break;
> + }
> + }
> +
> + crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
> +
> + break;
> + }
[Severity: High]
What happens if a TRB_TYPE_EVT_SETUP_PKT event is processed here while
crg_udc->device_state == USB_STATE_RECONNECTING?
When a bus reset occurs after CRG_U3DC_CFG1_SETUP_EVENT_EN has been enabled,
crg_udc_reinit() sets crg_udc->device_state = USB_STATE_RECONNECTING and
defers EP0 reinitialization (crg_udc_prepare_for_setup()) until the event ring
becomes empty. If a SETUP packet is already in the event ring during the same
drain loop, crg_udc_handle_setup_pkt() runs while device_state is still
USB_STATE_RECONNECTING: setaddressrequest() rejects USB_REQ_SET_ADDRESS and
stalls EP0 because device_state is neither USB_STATE_DEFAULT nor
USB_STATE_ADDRESS, and any control transfer queued for GET_DESCRIPTOR will be
immediately stopped and nuked with -ESHUTDOWN when crg_udc_prepare_for_setup()
runs at the end of the interrupt.
[Severity: High]
In addition, shouldn't the abort condition here also check
crg_udc->ep0_state != EP0_STATE_IDLE (not just
crg_udc->setup_status != WAIT_FOR_SETUP)?
crg_udc_abort_ep0_for_setup() sets crg_udc->setup_status = WAIT_FOR_SETUP at
the very beginning of the function. If crg_udc_abort_ep0_for_setup(),
crg_udc_prepare_for_setup(), or crg_udc_build_ep0_status() fails,
crg_udc->ep0_state is set to EP0_STATE_ERROR while setup_status is
WAIT_FOR_SETUP. On the next SETUP packet, checking only
setup_status != WAIT_FOR_SETUP skips crg_udc_abort_ep0_for_setup(), leaving
ep0_state permanently stuck in EP0_STATE_ERROR and causing crg_udc_ep_queue()
to reject all future EP0 requests with -ECONNRESET.
[ ... ]
> +int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
> + unsigned long *flags)
> +{
> + struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + enum crg_ep_state ep_state;
> + bool reset_seq_only = false;
> + int do_halt;
> + u32 param0;
> + u32 tmp;
> +
> + if (crg_udc->resetting)
> + return -ESHUTDOWN;
> + if (!udc_ep_ptr->desc) {
> + dev_err(crg_udc->dev, "NULL desc\n");
> + return -EINVAL;
> + }
> +
> + if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
> + dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
> + return -EOPNOTSUPP;
> + }
> +
> + if (udc_ep_ptr->dci == 0)
> + return 0;
[Severity: High]
When a gadget driver calls usb_ep_set_halt(gadget->ep0, 1) to stall EP0 (for
example from __ffs_ep0_stall(), uvc_send_response(), or a Control OUT
data-stage completion callback), doesn't returning 0 for dci == 0 turn the
EP0 stall into a silent no-op instead of calling crg_udc_set_ep0_halt()?
Furthermore, in crg_udc_handle_xfer_event() under CMPL_CODE_SUCCESS, after
crg_udc_req_done() runs the EP0 data-stage completion callback, it calls
crg_udc_ep0_req_complete(udc_ep_ptr, flags) without checking that
crg_udc->ep0_state == EP0_STATE_IDLE and udc_ep_ptr->ep_state ==
EP_STATE_RUNNING, which queues an ACK status stage even if the callback
stalled or aborted EP0.
> + ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
> +
> + if (ep_state == EP_STATE_DISABLED)
> + return 0;
> +
> + if (ep_state == EP_STATE_HALTED) {
> + if (halt != 0)
> + return 0;
> +
> + /* want unhalt an halted ep */
> + if (udc_ep_ptr->wedge && !ignore_wedge) {
> + do_halt = -1;
> + reset_seq_only = true;
> + } else {
> + do_halt = 0;
> + }
> +
> + } else {
> + /* ep state == running or stopped */
> + if (halt != 0) {
> + /* want halt a running ep */
> + do_halt = 1;
> + } else {
> + /* reset a running ep */
> + do_halt = 0;
> + reset_seq_only = true;
> + }
> + }
> +
> + trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
> +
> + param0 = BIT(udc_ep_ptr->dci);
> + if (do_halt == 1) {
> + /* setting ep to halt */
> + if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
> + return -EIO;
> + if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> + !(tmp & param0), 10, 10000))
> + dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
> + udc_ep_ptr->dci);
> +
> + if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> + udc_ep_ptr->tran_ring_full = false;
> + }
> + udc_ep_ptr->ep_state = EP_STATE_HALTED;
> + if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
> + crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
[Severity: High]
If readl_poll_timeout_atomic(&uccr->ep_running, ...) times out here, shouldn't
crg_udc_ep_halt() return -ETIMEDOUT (just like set_ep_halt() and
crg_udc_ep_dequeue() do) instead of proceeding to crg_udc_ep_nuke()?
Unmapping request DMA buffers and giving requests back to the gadget driver
via crg_udc_ep_nuke() while the endpoint bit in uccr->ep_running is still set
risks DMA use-after-free while the controller still owns the buffers.
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-5-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=5
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
2026-10-30 8:46 ` sashiko-bot
@ 2026-10-30 9:34 ` Philipp Zabel
1 sibling, 0 replies; 14+ messages in thread
From: Philipp Zabel @ 2026-10-30 9:34 UTC (permalink / raw)
To: Petar Stepanovic, Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Krutik Shah
On Fr, 2026-10-30 at 01:10 -0700, Petar Stepanovic wrote:
> This patch introduces the core probe/remove functions for the Axiado
> AX3000/AX3005 UDC driver, and wires up the directory into the build
> system via Kconfig and Makefile.
>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
> ---
> drivers/usb/gadget/udc/Kconfig | 2 +
> drivers/usb/gadget/udc/Makefile | 1 +
> drivers/usb/gadget/udc/crg/Kconfig | 16 +
> drivers/usb/gadget/udc/crg/Makefile | 9 +
> drivers/usb/gadget/udc/crg/crg-udc-core.c | 723 ++++++++++++++++++++++++++++++
> 5 files changed, 751 insertions(+)
>
[...]
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 000000000000..33be5c920ebe
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> @@ -0,0 +1,723 @@
[...]
> +static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
> +{
> + struct device *dev = crg_udc->dev;
> + struct reset_control *rst;
> + int ret;
> +
> + crg_udc->clk = devm_clk_get_optional(dev, NULL);
> + if (IS_ERR(crg_udc->clk))
> + return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
> + "failed to get clock\n");
> +
> + ret = clk_prepare_enable(crg_udc->clk);
> + if (ret)
> + return dev_err_probe(dev, ret, "failed to enable clock\n");
> +
> + ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
> + if (ret)
> + return ret;
> +
> + rst =
> + devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
This can be put on one line.
regards
Philipp
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-10-30 9:34 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
2026-10-30 8:14 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
2026-10-30 8:21 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
2026-10-30 8:29 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
2026-10-30 8:40 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
2026-10-30 8:50 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
2026-10-30 8:46 ` sashiko-bot
2026-10-30 9:34 ` Philipp Zabel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox