* [PATCH v10 2/4] media: dt-bindings: Add CSI Pixel Formatter DT bindings
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
The i.MX95 CSI pixel formatting module uses packet info, pixel and
non-pixel data from the CSI-2 host controller and reformat them to
match Pixel Link(PL) definition.
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Drop syscon parent node from example
- Drop Reviewed-by tags from Frank and Krzysztof due to binding changes
- Add description for reg property
- Add space after formatter@20 before opening brace in example
- Enhance the port description with more detailed information
- Delete the blank line immediately following the endpoint in example
Changes in v9:
- Use direct node instead of syscon wrapper in example
Changes in v8:
- Use standard port reference instead of video-interfaces.yaml
- Add parent syscon node in example to show device integration
- Add required constraints for port@0 and port@1 in ports node
Changes in v7:
- Change compatible to imx95-csi-formatter as IP is i.MX95 specific per Marco's suggestion
Link: https://lore.kernel.org/linux-media/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com/T/#mcd135b3de179b3cb69daa1fd6e0e8e27c85b3332
---
.../bindings/media/fsl,imx95-csi-formatter.yaml | 88 ++++++++++++++++++++++
1 file changed, 88 insertions(+)
diff --git a/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml b/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml
new file mode 100644
index 000000000000..58c4e1cc056b
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/fsl,imx95-csi-formatter.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: i.MX95 CSI Pixel Formatter
+
+maintainers:
+ - Guoniu Zhou <guoniu.zhou@nxp.com>
+
+description:
+ The CSI pixel formatting module found on i.MX95 uses packet info, pixel
+ and non-pixel data from the CSI-2 host controller and reformat them to
+ match Pixel Link(PL) definition.
+
+properties:
+ compatible:
+ const: fsl,imx95-csi-formatter
+
+ reg:
+ maxItems: 1
+ description: Register offset and size within the parent syscon
+
+ clocks:
+ maxItems: 1
+
+ power-domains:
+ maxItems: 1
+
+ ports:
+ $ref: /schemas/graph.yaml#/properties/ports
+
+ properties:
+ port@0:
+ $ref: /schemas/graph.yaml#/properties/port
+ description:
+ Input port, connects to MIPI CSI-2 receiver output (IDI interface)
+
+ port@1:
+ $ref: /schemas/graph.yaml#/properties/port
+ description:
+ Output port, connects to ISI input via Pixel Link (PL)
+
+ required:
+ - port@0
+ - port@1
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - power-domains
+ - ports
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/nxp,imx95-clock.h>
+
+ formatter@20 {
+ compatible = "fsl,imx95-csi-formatter";
+ reg = <0x20 0x100>;
+ clocks = <&cameramix_csr IMX95_CLK_CAMBLK_CSI2_FOR0>;
+ power-domains = <&scmi_devpd 3>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 {
+ reg = <0>;
+
+ endpoint {
+ remote-endpoint = <&mipi_csi_0_out>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ endpoint {
+ remote-endpoint = <&isi_in_2>;
+ };
+ };
+ };
+ };
--
2.34.1
^ permalink raw reply related
* [PATCH v10 1/4] dt-bindings: clock: imx95-blk-ctl: Use single quotes consistently
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
Change "clocks" to 'clocks' in the description to match the quote style
used for property names like '#clock-cells' throughout the file.
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- New patch to fix inconsistent quote usage (Krzysztof Kozlowski)
---
Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
index 27403b4c52d6..534fa219d9f9 100644
--- a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
+++ b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
@@ -36,7 +36,7 @@ properties:
const: 1
description:
The clock consumer should specify the desired clock by having the clock
- ID in its "clocks" phandle cell. See
+ ID in its 'clocks' phandle cell. See
include/dt-bindings/clock/nxp,imx95-clock.h
required:
--
2.34.1
^ permalink raw reply related
* [PATCH v10 0/4] media: nxp: Add CSI Pixel Formatter support
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou, Frank Li
CSI Pixel Formatter is a module found on i.MX95. It could unpack the
pixels received by the formatter and reformat them to meet the pixel
link format requirement.
This patch series adds a new V4L2 driver for CSI Pixel Formatter.
Background
----------
The Camera CSR binding was upstreamed during the pre-silicon phase when
the hardware integration details were not fully finalized. At that time,
the syscon binding focused on the CSR's primary role as a shared register
provider for clock gating, QoS, and error handling across Camera domain IPs.
However, the Camera CSR hardware is actually a mixed-function register
container with varying control levels over different IP blocks:
- ISP/CSI: Clock gating only
- ISI: QoS configuration
- CSI: Buffer overflow handling
- LPCAC: Error handling only
- ISP: Pixel link selection
- Formatter: Complete IP control (full register set)
Why Child Nodes Now
-------------------
The CSI formatter is unique among these IPs - it's an independent hardware
block whose complete control registers reside within the CSR address
space, not just a few configuration bits. This architectural detail was
not reflected in the original binding because:
1. The pre-silicon binding focused on simple register-level controls
2. The syscon pattern for complete IP control was not initially considered
Discussion with original CSR author(Peng Fan) confirmed this reflects the
intended hardware design that wasn't fully captured during pre-silicon
upstreaming.
v4l2-compliance 1.28.1-5233, 64 bits, 64-bit time_t
v4l2-compliance SHA: fc15e229d9d3 2024-07-23 19:22:15
Compliance test for device /dev/v4l-subdev9:
Driver Info:
Driver version : 7.1.0
Capabilities : 0x00000002
Streams Support
Client Capabilities: 0x0000000000000003
streams interval-uses-which
Required ioctls:
test VIDIOC_SUDBEV_QUERYCAP: OK
test invalid ioctls: OK
Allow for multiple opens:
test second /dev/v4l-subdev9 open: OK
test VIDIOC_SUBDEV_QUERYCAP: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Sub-Device routing ioctls:
test Try VIDIOC_SUBDEV_G_ROUTING/VIDIOC_SUBDEV_S_ROUTING: OK
test Active VIDIOC_SUBDEV_G_ROUTING/VIDIOC_SUBDEV_S_ROUTING: OK
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK (Not Supported)
test VIDIOC_TRY_FMT: OK (Not Supported)
test VIDIOC_S_FMT: OK (Not Supported)
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
test CREATE_BUFS maximum buffers: OK
test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK (Not Supported)
test Requests: OK (Not Supported)
Total for device /dev/v4l-subdev9: 47, Succeeded: 47, Failed: 0, Warnings: 0
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Rebase to latest media/next
- [NEW PATCH] Use single quotes consistently (Krzysztof Kozlowski)
- Drop syscon parent node from example
- Drop Reviewed-by tags from Frank and Krzysztof due to binding changes,
requesting re-review
- Add description for reg property
- Add space after formatter@20 before opening brace in example
- Enhance the port description with more detailed information
- Delete the blank line immediately following the endpoint in example
- Use single quotes for regex pattern to be consistent (Krzysztof Kozlowski)
- Add formatter subnode binding and camera-csr syscon example
- Update commit title and message
- Use u8 for vc in csi_formatter_get_vc() and drop vc < 0 check
- Add MFD_SYSCON dependency to Kconfig
- Fix stream/VC mapping potential mismatch in start/stop_stream functions
- Link to v9: https://lore.kernel.org/r/20260526-csi_formatter-v9-0-ca3d8c334c39@oss.nxp.com
Changes in v9:
- [NEW PATCH] Fix formatter as syscon child node issue
- Link to v8: https://lore.kernel.org/r/20260525-csi_formatter-v8-0-6b646231224b@oss.nxp.com
Changes in v8:
- Rebase to latest media/next
- Use standard port reference instead of video-interfaces.yaml
- Add parent syscon node in example to show device integration
- Remove fmt field and look up format from subdev state instead
- Unify function and structure naming to use csi_formatter_ prefix
- Remove misleading alignment comment from set_fmt function
- Optimize get_frame_desc to call once per start_stream
- Replace V4L2_FRAME_DESC_ENTRY_MAX with CSI_FORMATTER_VC_NUM in loops
- Remove redundant debug message in enable_streams
- Use MEDIA_PAD_FL_MUST_CONNECT flag instead of manual link check
- Link to v7: https://lore.kernel.org/r/20260518-csi_formatter-v7-0-562b750557e3@oss.nxp.com
Changes in v7:
- Change compatible to imx95-csi-formatter as IP is i.MX95 specific per Marco's suggestion
Link: https://lore.kernel.org/linux-media/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com/T/#mcd135b3de179b3cb69daa1fd6e0e8e27c85b3332
- Update references from imx9 to imx95 for consistency with dt-bindings
- Enable PM runtime before async registration
- Link to v6: https://lore.kernel.org/r/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com
Changes in v6:
- Rebase to latest media/next
- Update v4l2-compliace test
- Remove unused header includes
- Unify macro naming: VCx/VCX -> VC and parameter x -> vc
- Remove unused format field from csi_formatter struct
- Use compact initialization for formats array
- Make find_csi_format() return NULL instead of default format
- Use unsigned int for array index in find_csi_format()
- Add err_ prefix to error handling labels
- Add v4l2_subdev_cleanup() and reorder cleanup sequence
- Update enable_streams debug output format
- Rename VC_MAX to VC_NUM and fix boundary check
- Update CSI formatter Kconfig description
- Use v4l2_subdev_get_frame_desc_passthrough() helper
- Fix error paths in async registration and probe
- Add mutex to protect enabled_streams
- Switch to devm_pm_runtime_enable()
- Remove redundant num_routes check in set_routing
- Optimize get_index_by_dt() and add warning for unsupported type
- csi_formatter_start/stop_stream: Process all streams in mask
- Link to v5: https://lore.kernel.org/r/20260123-csi_formatter-v5-0-d5b803f867bf@nxp.com
Changes in v5:
- Remove CSI_FORMATTER_DRV_NAME macro since only use once.
- Remove sd->owner = THIS_MODULE;
- Simplify code by using DEFINE_RUNTIME_DEV_PM_OPS macro.
- Link to v4: https://lore.kernel.org/r/20260122-csi_formatter-v4-0-6f6fcad1c33a@nxp.com
Changes in v4:
- Rebase to latest media/next.
- Add comments to describe the index field in formatter_dt_to_index_map array.
- Link to v3: https://lore.kernel.org/r/20251219-csi_formatter-v3-0-8680d6d87091@nxp.com
Changes in v3:
- Rename nxp,imx9-csi-formatter.yaml to fsl,imx9-csi-formatter.yaml.
- Drop clock-names property.
- Drop macro IMX95_PD_CAMERA definition and use a constant directly.
[PATCH 1/2] media: dt-bindings: Add CSI Pixel Formatter DT bindings
- Remove the assignment driver.owner = THIS_MODULE.
- Assign struct fwnode_handle *ep __free(fwnode_handle) when definition.
- Update yaml file name for csi formatter in MAINTAINERS.
[PATCH 2/2] media: nxp: Add i.MX9 CSI pixel formatter v4l2 driver
- Link to v2: https://lore.kernel.org/r/20251217-csi_formatter-v2-0-62168af80210@nxp.com
Changes in v2:
- Delete "|" for description key.
- Add empty line between child node and property.
- Delete labels for endpoint of child nodes.
[PATCH 1/2] media: dt-bindings: Add CSI Pixel Formatter DT bindings
- Update commit message.
- Use the value defined by bellow macros directly since they are used only once.
#define CSI_FORMATTER_DEF_MBUS_CODE MEDIA_BUS_FMT_UYVY8_1X16
#define CSI_FORMATTER_DEF_PIX_WIDTH 1920U
#define CSI_FORMATTER_DEF_PIX_HEIGHT 1080U
#define CSI_FORMATTER_MAX_PIX_WIDTH 0xffff
#define CSI_FORMATTER_MAX_PIX_HEIGHT 0xffff
- Use macro pm_ptr() to fix build warning when CONFIG_PM is disabled.
- Finish route loop by break statement, instead of goto.
- Return dev_err_probe() when meet errors in probe() function instead of dev_err().
- Remove MODULE_ALIAS().
- Refine .enable(.dsable)_stream callback implementation, include bellow changes:
Add stream checking.
Fix potential pm runtime count unbalance issue.
Add stop stream error handling when enabling remote subdev stream.
- Use __free(fwnode_handle) to drop reference to a device node automatically.
[PATCH 2/2] media: nxp: Add i.MX9 CSI pixel formatter v4l2 driver
- Link to v1: https://lore.kernel.org/r/20251203-csi_formatter-v1-0-eb9e1147b49e@nxp.com
---
Guoniu Zhou (4):
dt-bindings: clock: imx95-blk-ctl: Use single quotes consistently
media: dt-bindings: Add CSI Pixel Formatter DT bindings
dt-bindings: clock: imx95-blk-ctl: Define formatter child node schema
media: nxp: Add i.MX95 CSI pixel formatter v4l2 driver
.../bindings/clock/nxp,imx95-blk-ctl.yaml | 66 +-
.../bindings/media/fsl,imx95-csi-formatter.yaml | 88 +++
MAINTAINERS | 8 +
drivers/media/platform/nxp/Kconfig | 15 +
drivers/media/platform/nxp/Makefile | 1 +
drivers/media/platform/nxp/imx95-csi-formatter.c | 775 +++++++++++++++++++++
6 files changed, 951 insertions(+), 2 deletions(-)
---
base-commit: 06cb687a5132fcffe624c0070576ab852ac6b568
change-id: 20251125-csi_formatter-e6d29316dce6
Best regards,
--
Guoniu Zhou <guoniu.zhou@oss.nxp.com>
^ permalink raw reply
* [PATCH net v2] net: ethernet: ti: icssg: guard PA stat lookups
From: Philippe Schenker @ 2026-06-18 9:30 UTC (permalink / raw)
To: netdev
Cc: Philippe Schenker, Simon Horman, danishanwar, rogerq,
linux-arm-kernel, stable, Andrew Lunn, David Carlier,
David S. Miller, Eric Dumazet, Jacob Keller, Jakub Kicinski,
Kevin Hao, Meghana Malladi, Paolo Abeni, Vadim Fedorenko,
linux-kernel
From: Philippe Schenker <philippe.schenker@impulsing.ch>
icssg_ndo_get_stats64() unconditionally calls emac_get_stat_by_name()
with FW PA stat names regardless of whether the PA stats block is
present on the hardware. emac_get_stat_by_name() already guards the
PA stats lookup with `if (emac->prueth->pa_stats)`; when that pointer
is NULL the lookup falls through to netdev_err() and returns -EINVAL.
Because ndo_get_stats64 is polled regularly by the networking stack
this produces thousands of log entries of the form:
icssg-prueth icssg1-eth end0: Invalid stats FW_RX_ERROR
A secondary consequence is that the int(-EINVAL) return value is
implicitly widened to a near-ULLONG_MAX unsigned value when accumulated
into the __u64 fields of rtnl_link_stats64, silently corrupting the
rx_errors, rx_dropped and tx_dropped counters reported by `ip -s link`.
Every other PA-aware code path in the driver is already guarded with
the same `if (emac->prueth->pa_stats)` check. Apply the same guard
here.
Fixes: 0d15a26b247d ("net: ti: icssg-prueth: Add ICSSG FW Stats")
Signed-off-by: Philippe Schenker <philippe.schenker@impulsing.ch>
Reviewed-by: Simon Horman <horms@kernel.org>
Cc: danishanwar@ti.com
Cc: rogerq@kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: stable@vger.kernel.org
---
Changes in v2:
- Removed newline between Fixes tag and Signed-off-by
- Use return in if statement to guard so we get rid
of the 80 char warnings.
- Added Simon's Reviewed-by. Thanks!
drivers/net/ethernet/ti/icssg/icssg_common.c | 49 +++++++++++---------
1 file changed, 28 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ethernet/ti/icssg/icssg_common.c b/drivers/net/ethernet/ti/icssg/icssg_common.c
index a28a608f9bf4..d9af6419e032 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_common.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_common.c
@@ -1628,28 +1628,35 @@ void icssg_ndo_get_stats64(struct net_device *ndev,
stats->rx_over_errors = emac_get_stat_by_name(emac, "rx_over_errors");
stats->multicast = emac_get_stat_by_name(emac, "rx_multicast_frames");
- stats->rx_errors = ndev->stats.rx_errors +
- emac_get_stat_by_name(emac, "FW_RX_ERROR") +
- emac_get_stat_by_name(emac, "FW_RX_EOF_SHORT_FRMERR") +
- emac_get_stat_by_name(emac, "FW_RX_B0_DROP_EARLY_EOF") +
- emac_get_stat_by_name(emac, "FW_RX_EXP_FRAG_Q_DROP") +
- emac_get_stat_by_name(emac, "FW_RX_FIFO_OVERRUN");
- stats->rx_dropped = ndev->stats.rx_dropped +
- emac_get_stat_by_name(emac, "FW_DROPPED_PKT") +
- emac_get_stat_by_name(emac, "FW_INF_PORT_DISABLED") +
- emac_get_stat_by_name(emac, "FW_INF_SAV") +
- emac_get_stat_by_name(emac, "FW_INF_SA_DL") +
- emac_get_stat_by_name(emac, "FW_INF_PORT_BLOCKED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_TAGGED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_PRIOTAGGED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_NOTAG") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_NOTMEMBER");
+ stats->rx_errors = ndev->stats.rx_errors;
+ stats->rx_dropped = ndev->stats.rx_dropped;
stats->tx_errors = ndev->stats.tx_errors;
- stats->tx_dropped = ndev->stats.tx_dropped +
- emac_get_stat_by_name(emac, "FW_RTU_PKT_DROP") +
- emac_get_stat_by_name(emac, "FW_TX_DROPPED_PACKET") +
- emac_get_stat_by_name(emac, "FW_TX_TS_DROPPED_PACKET") +
- emac_get_stat_by_name(emac, "FW_TX_JUMBO_FRM_CUTOFF");
+ stats->tx_dropped = ndev->stats.tx_dropped;
+
+ if (!emac->prueth->pa_stats)
+ return;
+
+ stats->rx_errors +=
+ emac_get_stat_by_name(emac, "FW_RX_ERROR") +
+ emac_get_stat_by_name(emac, "FW_RX_EOF_SHORT_FRMERR") +
+ emac_get_stat_by_name(emac, "FW_RX_B0_DROP_EARLY_EOF") +
+ emac_get_stat_by_name(emac, "FW_RX_EXP_FRAG_Q_DROP") +
+ emac_get_stat_by_name(emac, "FW_RX_FIFO_OVERRUN");
+ stats->rx_dropped +=
+ emac_get_stat_by_name(emac, "FW_DROPPED_PKT") +
+ emac_get_stat_by_name(emac, "FW_INF_PORT_DISABLED") +
+ emac_get_stat_by_name(emac, "FW_INF_SAV") +
+ emac_get_stat_by_name(emac, "FW_INF_SA_DL") +
+ emac_get_stat_by_name(emac, "FW_INF_PORT_BLOCKED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_TAGGED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_PRIOTAGGED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_NOTAG") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_NOTMEMBER");
+ stats->tx_dropped +=
+ emac_get_stat_by_name(emac, "FW_RTU_PKT_DROP") +
+ emac_get_stat_by_name(emac, "FW_TX_DROPPED_PACKET") +
+ emac_get_stat_by_name(emac, "FW_TX_TS_DROPPED_PACKET") +
+ emac_get_stat_by_name(emac, "FW_TX_JUMBO_FRM_CUTOFF");
}
EXPORT_SYMBOL_GPL(icssg_ndo_get_stats64);
--
2.54.0
base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
branch: fix-icssg_common-pa-stats-errors__master-7-1
^ permalink raw reply related
* Re: [PATCH net] net: ethernet: ti: icssg: guard PA stat lookups
From: Philippe Schenker @ 2026-06-18 9:29 UTC (permalink / raw)
To: Simon Horman
Cc: netdev, danishanwar, rogerq, linux-arm-kernel, stable,
Andrew Lunn, David Carlier, David S. Miller, Eric Dumazet,
Jacob Keller, Jakub Kicinski, Kevin Hao, Meghana Malladi,
Paolo Abeni, Vadim Fedorenko, linux-kernel
In-Reply-To: <20260618091004.GG827683@horms.kernel.org>
[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]
Hi Simon
Thanks for the review and I'll send a v2 with that blank line removed.
Saw it right after sending the patch.
Philippe
On Thu, 2026-06-18 at 10:10 +0100, Simon Horman wrote:
> On Tue, Jun 16, 2026 at 04:35:34PM +0200, Philippe Schenker wrote:
> > From: Philippe Schenker <philippe.schenker@impulsing.ch>
> >
> > icssg_ndo_get_stats64() unconditionally calls
> > emac_get_stat_by_name()
> > with FW PA stat names regardless of whether the PA stats block is
> > present on the hardware. emac_get_stat_by_name() already guards
> > the
> > PA stats lookup with `if (emac->prueth->pa_stats)`; when that
> > pointer
> > is NULL the lookup falls through to netdev_err() and returns -
> > EINVAL.
> > Because ndo_get_stats64 is polled regularly by the networking stack
> > this produces thousands of log entries of the form:
> >
> > icssg-prueth icssg1-eth end0: Invalid stats FW_RX_ERROR
> >
> > A secondary consequence is that the int(-EINVAL) return value is
> > implicitly widened to a near-ULLONG_MAX unsigned value when
> > accumulated
> > into the __u64 fields of rtnl_link_stats64, silently corrupting the
> > rx_errors, rx_dropped and tx_dropped counters reported by `ip -s
> > link`.
> >
> > Every other PA-aware code path in the driver is already guarded
> > with
> > the same `if (emac->prueth->pa_stats)` check. Apply the same guard
> > here.
> >
> > Fixes: 0d15a26b247d ("net: ti: icssg-prueth: Add ICSSG FW Stats")
>
> nit: no blank line between tags
>
> >
> > Signed-off-by: Philippe Schenker <philippe.schenker@impulsing.ch>
> >
> > Cc: danishanwar@ti.com
> > Cc: rogerq@kernel.org
> > Cc: linux-arm-kernel@lists.infradead.org
> > Cc: stable@vger.kernel.org
>
> Reviewed-by: Simon Horman <horms@kernel.org>
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply
* Re: [PATCH v2 2/6] iommu/arm-smmu: Add interconnect bandwidth voting support
From: Konrad Dybcio @ 2026-06-18 9:28 UTC (permalink / raw)
To: Bibek Kumar Patro, Dmitry Baryshkov
Cc: Will Deacon, Robin Murphy, Joerg Roedel, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
linux-arm-kernel, iommu, devicetree, linux-kernel, linux-arm-msm
In-Reply-To: <ea345832-711d-4e32-af0f-bc74d8646aef@oss.qualcomm.com>
On 6/17/26 4:26 PM, Bibek Kumar Patro wrote:
>
>
> On 6/16/2026 5:51 AM, Dmitry Baryshkov wrote:
>> On Mon, Jun 15, 2026 at 06:36:51PM +0530, Bibek Kumar Patro wrote:
>>>
>>>
>>> On 6/8/2026 7:25 PM, Dmitry Baryshkov wrote:
>>>> On Tue, May 26, 2026 at 08:12:03PM +0530, Bibek Kumar Patro wrote:
>>>>> On some SoCs the SMMU registers require an active interconnect
>>>>> bandwidth vote to be accessible. While other clients typically
>>>>> satisfy this requirement implicitly, certain corner cases (e.g.
>>>>> during sleep/wakeup transitions) can leave the SMMU without a
>>>>> vote, causing intermittent register access failures.
>>>>>
>>>>> Add support for an optional interconnect path to the arm-smmu
>>>>> driver and vote for bandwidth while the SMMU is active. The path
>>>>> is acquired from DT if present and ignored otherwise.
>>>>>
>>>>> The bandwidth vote is enabled before accessing SMMU registers
>>>>> during probe and runtime resume, and released during runtime
>>>>> suspend and on error paths.
>>>>>
>>>>> Generally, from an architectural perspective, GEM_NOC and DDR are
>>>>> expected to have an active vote whenever the adreno_smmu block is
>>>>> powered on. In most common use cases, this requirement is implicitly
>>>>> satisfied because other GPU-related clients (for example, the GMU
>>>>> device) already hold a GEM_NOC vote when adreno_smmu is enabled.
>>>>>
>>>>> However, there are certain corner cases, such as during sleep/wakeup
>>>>> transitions, where the GEM_NOC vote can be removed before adreno_smmu
>>>>> is powered down. If adreno_smmu is then accessed while the interconnect
>>>>> vote is missing, it can lead to the observed failures. Because of the
>>>>> precise ordering involved, this scenario is difficult to reproduce
>>>>> consistently.
>>>>> (also GDSC is involved in adreno usecases can have an independent vote)
>>>>>
>>>>> Signed-off-by: Bibek Kumar Patro <bibek.patro@oss.qualcomm.com>
>>>>> ---
>>>>> drivers/iommu/arm/arm-smmu/arm-smmu.c | 57 +++++++++++++++++++++++++++++++++--
>>>>> drivers/iommu/arm/arm-smmu/arm-smmu.h | 2 ++
>>>>> 2 files changed, 57 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> index 0bd21d206eb3e75c3b9fb1364cdc92e82c5aa499..07c7e44ec6a5bd1488f00f87d859a20495e46601 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> @@ -53,6 +53,11 @@
>>>>> #define MSI_IOVA_BASE 0x8000000
>>>>> #define MSI_IOVA_LENGTH 0x100000
>>>>> +/* Interconnect bandwidth vote values for the SMMU register access path */
>>>>> +#define ARM_SMMU_ICC_AVG_BW 0
>>>>> +#define ARM_SMMU_ICC_PEAK_BW_HIGH 1000
>>>>
>>>> totally random numbers, which might be different for non-Qualcomm platform.
>>>>
>>>
>>> Ideally, any non-zero value would be enough to keep the path active.
>>
>> This is true for Qualcomm devices. However, you are adding this to a
>> generic code.
>>
>>> Here 1 Would be enough to keep the path active, but might be too small to
>>> reliably keep the bus active.
>>> Other is UINT_MAX, which will reliably keep the bus active but might cause a
>>> power penalty.
>>>
>>> #define ARM_SMMU_ICC_PEAK_BW_HIGH UINT_MAX
>>>
>>> seems to be suitable here to reliably keep the bus active by BCM
>>> for both Qualcomm and non-Qualcomm platforms (with some power penalty).
>>>
>>> LMK, if you feel otherwise.
>>
>> Shift it to the qcom instance or provide platform-specific values? (My
>> preference would be towards the first solution).
>>
>
>
> To support platform-specific values, we may need to introduce a LUT-based approach in the driver. (Bandwidth voting values cannot be placed in device-tree property IIRC ?)
>
> Currently, all Qualcomm platforms use 0x1000 for SMMU ICC voting. I
(you used decimal 1000)
> can evaluate if this could be moved to a Qualcomm-specific
> implementation.
Add a vendor hook to arm_smmu_runtime_suspend/resume and handle it within
the QC driver
Konrad
^ permalink raw reply
* RE: [PATCH v4 3/5] dt-bindings: clock: cix,sky1-audss-clock: add audss clock controller
From: Joakim Zhang @ 2026-06-18 9:27 UTC (permalink / raw)
To: Philipp Zabel, Conor Dooley
Cc: mturquette@baylibre.com, sboyd@kernel.org, bmasney@redhat.com,
robh@kernel.org, krzk+dt@kernel.org, conor+dt@kernel.org,
Gary Yang, cix-kernel-upstream, linux-clk@vger.kernel.org,
devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org
In-Reply-To: <cd63c1ab9831fdfd39aad8d1a40c4702a9cbb158.camel@pengutronix.de>
Hello,
> -----Original Message-----
> From: Philipp Zabel <p.zabel@pengutronix.de>
> Sent: Thursday, June 18, 2026 4:30 PM
> To: Joakim Zhang <joakim.zhang@cixtech.com>; Conor Dooley
> <conor@kernel.org>
> Cc: mturquette@baylibre.com; sboyd@kernel.org; bmasney@redhat.com;
> robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org; Gary Yang
> <Gary.Yang@cixtech.com>; cix-kernel-upstream <cix-kernel-
> upstream@cixtech.com>; linux-clk@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v4 3/5] dt-bindings: clock: cix,sky1-audss-clock: add audss
> clock controller
>
> EXTERNAL EMAIL
>
> CAUTION: Suspicious Email from unusual domain.
>
> On Do, 2026-06-18 at 01:43 +0000, Joakim Zhang wrote:
> > Hello,
> >
> >
> > > -----Original Message-----
> > > From: Conor Dooley <conor@kernel.org>
> > > Sent: Wednesday, June 17, 2026 11:56 PM
> > > To: Joakim Zhang <joakim.zhang@cixtech.com>
> > > Cc: mturquette@baylibre.com; sboyd@kernel.org; bmasney@redhat.com;
> > > robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org;
> > > p.zabel@pengutronix.de; Gary Yang <gary.yang@cixtech.com>;
> > > cix-kernel- upstream <cix-kernel-upstream@cixtech.com>;
> > > linux-clk@vger.kernel.org; devicetree@vger.kernel.org;
> > > linux-kernel@vger.kernel.org; linux-arm- kernel@lists.infradead.org
> > > Subject: Re: [PATCH v4 3/5] dt-bindings: clock:
> > > cix,sky1-audss-clock: add audss clock controller
> > >
> > > On Wed, Jun 17, 2026 at 02:04:35PM +0800, joakim.zhang@cixtech.com
> wrote:
> > > > From: Joakim Zhang <joakim.zhang@cixtech.com>
> > > >
> > > > The AUDSS CRU contains an internal clock tree of muxes, dividers
> > > > and gates for DSP, I2S, HDA, DMAC and related blocks. The clock
> > > > provider is a child node of the cix,sky1-audss-system-control
> > > > syscon and accesses registers through the parent MMIO region.
> > >
> > > Why can this not just be part of the parent syscon node?
> >
> > The clock and reset blocks are handled by different subsystems and
> maintainers (clk vs reset). Putting the clock provider on the parent syscon node
> would mean a single driver has to register both the reset controller and the
> clock provider on one device, which doesn't fit well.
>
> There are many examples of clock and reset drivers sharing the same node, by
> using platform_driver for one (usually clk) and auxiliary_driver for the other
> (usually reset).
OK, I will have a look. If you are also prefer to this, I will refactor the patch.
Thanks,
Joakim
^ permalink raw reply
* Re: [PATCH v2 3/6] arm64: dts: qcom: kodiak: Add GEM_NOC interconnect for adreno SMMU
From: Konrad Dybcio @ 2026-06-18 9:26 UTC (permalink / raw)
To: Bibek Kumar Patro, Dmitry Baryshkov
Cc: Will Deacon, Robin Murphy, Joerg Roedel, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
linux-arm-kernel, iommu, devicetree, linux-kernel, linux-arm-msm
In-Reply-To: <26d51cbb-2d87-4564-b3c6-cc61ab900e19@oss.qualcomm.com>
On 6/8/26 4:37 PM, Bibek Kumar Patro wrote:
>
>
> On 6/8/2026 7:27 PM, Dmitry Baryshkov wrote:
>> On Tue, May 26, 2026 at 08:12:04PM +0530, Bibek Kumar Patro wrote:
>>> On Kodiak platforms, the Adreno SMMU requires a bandwidth vote on
>>> the GEM_NOC path (MASTER_GPU_TCU -> SLAVE_EBI1) before its registers
>>> are accessible. Without this vote, the SMMU may become unreachable,
>>> leading to intermittent probe failures and runtime issues.
>>>
>>> Add the required interconnect to ensure reliable register access.
>>
>> Does it only concern the GPU SMMU? What about the APPS SMMU? Should it
>> be voting on other interconnects too? I guess so, because currently I
>> see that TBUs vote for various interconnects. BTW: should apps_smmu also
>> vote on the power domains?
>>
>
> This race mainly occurs in GPU SMMU, where the GDSC can have an
Mainly or exclusively?
Konrad
> independent vote on the Adreno SMMU. However, the GEM_NOC vote may
> already have been removed by the GPU (or any consumer of adreno_smmu,
> e.g gmu), unless it is explicitly voted by the GPU SMMU (which acts as a
> supplier for the GPU). This mismatch can lead to SHUB timeouts or NoC
> errors.
>
> Mostly this race reported in suspend/resume cycle (when gpu/gmu devices moves to slumber/suspend state before adreno_smmu powers down
> and the later doesn't have explicit interconnect voting).
>
> In the case of APPS SMMU, such a race is not expected for any known
> use case. APPS SMMU is part of a shared infrastructure block, and its
> power is typically kept enabled as long as attached master devices are
> active. Therefore, explicit power-domain voting from APPS SMMU may not
> be required.
>
> Thanks,
> Bibek
>
>
>>>
>>> Signed-off-by: Bibek Kumar Patro <bibek.patro@oss.qualcomm.com>
>>> ---
>>> arch/arm64/boot/dts/qcom/kodiak.dtsi | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/arch/arm64/boot/dts/qcom/kodiak.dtsi b/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> index fa540d8c2615dc02d941eb16bc7253204c2750bd..eefa4b836a81374ff437ab4bbcbc3fecc1590ab6 100644
>>> --- a/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> +++ b/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> @@ -3386,6 +3386,8 @@ adreno_smmu: iommu@3da0000 {
>>> power-domains = <&gpucc GPU_CC_CX_GDSC>;
>>> dma-coherent;
>>> + interconnects = <&gem_noc MASTER_GPU_TCU QCOM_ICC_TAG_ALWAYS
>>> + &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>;
>>> };
>>> gfx_0_tbu: tbu@3dd9000 {
>>>
>>> --
>>> 2.34.1
>>>
>>
>
>
^ permalink raw reply
* [PATCH v2 3/4] arm64: smp: Defer RCU reporting until after local CPU capability checks
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
To support HOTPLUG_PARALLEL on arm64, and to prevent a potential deadlock
on the control CPU, check_local_cpu_capabilities() must be executed
before cpuhp_ap_sync_alive(). This ensures that if an early capability
mismatch occurs and the AP invokes cpu_die_early(), the control CPU
can detect the boot timeout and proceed, rather than hanging
indefinitely.
Furthermore, under parallel bringup, cpuhp_ap_sync_alive() must be called
before rcutree_report_cpu_starting(). This sequence prevents a false
RCU CPU Stall Warning caused by the prolonged spin-waiting/busy-waiting
required during the AP synchronization process.
GICv3: CPU1: using allocated LPI pending table @0x0000000104160000
CPU1: Booted secondary processor 0x0000000001 [0x410fd082]
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 2-O..!: (2 GPs behind) idle=0004/1/0x4000000000000000 softirq=0/0 fqs=2625
rcu: 3-O..!: (2 GPs behind) idle=0004/1/0x4000000000000000 softirq=0/0 fqs=2625
rcu: (detected by 0, t=5252 jiffies, g=-1187, q=1 ncpus=16)
rcu: Offline CPU 2 blocking current GP.
rcu: Offline CPU 3 blocking current GP.
To avoid suspicious RCU usage, commit ce3d31ad3cac ("arm64/smp: Move
rcu_cpu_starting() earlier") move rcutree_report_cpu_starting() earlier
which is before check_local_cpu_capabilities().
But For parallel bringup, the order should be as following:
secondary_start_kernel()
-> check_local_cpu_capabilities()
-> cpu_die_early()
-> cpuhp_ap_sync_alive()
-> rcutree_report_cpu_starting()
And this required order forces standard printk/pr_* statements inside
check_local_cpu_capabilities() to execute while the secondary CPU is still
marked as offline to RCU, triggering a lockdep "suspicious RCU usage" splat
due to console semaphore operations.
So converting early capability logging and failure paths to printk_deferred().
This pushes the logs into the lockless ringbuffer without triggering console
locks or RCU validation on the offline CPU, resolving the lockdep splat while
preserving critical error messages and the strictly required initialization
order.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/cpufeature.c | 20 ++++++++++----------
arch/arm64/kernel/smp.c | 4 ++--
arch/arm64/mm/context.c | 2 +-
3 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 0552202702bf..a5e0bc4d383b 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3546,7 +3546,7 @@ static void update_cpu_capabilities(u16 scope_mask)
* system capabilities are finalised.
*/
if (!match_all && caps->desc && !caps->cpus)
- pr_info("detected: %s\n", caps->desc);
+ printk_deferred(KERN_INFO "detected: %s\n", caps->desc);
__set_bit(caps->capability, system_cpucaps);
@@ -3669,7 +3669,7 @@ static void verify_local_cpu_caps(u16 scope_mask)
}
if (i < ARM64_NCAPS) {
- pr_crit("CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",
+ printk_deferred(KERN_CRIT "CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",
smp_processor_id(), caps->capability,
caps->desc, system_has_cap, cpu_has_cap);
@@ -3697,7 +3697,7 @@ __verify_local_elf_hwcaps(const struct arm64_cpu_capabilities *caps)
for (; caps->matches; caps++)
if (cpus_have_elf_hwcap(caps) && !caps->matches(caps, SCOPE_LOCAL_CPU)) {
- pr_crit("CPU%d: missing HWCAP: %s\n",
+ printk_deferred(KERN_CRIT "CPU%d: missing HWCAP: %s\n",
smp_processor_id(), caps->desc);
cpu_die_early();
}
@@ -3716,7 +3716,7 @@ static void verify_sve_features(void)
unsigned long cpacr = cpacr_save_enable_kernel_sve();
if (vec_verify_vq_map(ARM64_VEC_SVE)) {
- pr_crit("CPU%d: SVE: vector length support mismatch\n",
+ printk_deferred(KERN_CRIT "CPU%d: SVE: vector length support mismatch\n",
smp_processor_id());
cpu_die_early();
}
@@ -3729,7 +3729,7 @@ static void verify_sme_features(void)
unsigned long cpacr = cpacr_save_enable_kernel_sme();
if (vec_verify_vq_map(ARM64_VEC_SME)) {
- pr_crit("CPU%d: SME: vector length support mismatch\n",
+ printk_deferred(KERN_CRIT "CPU%d: SME: vector length support mismatch\n",
smp_processor_id());
cpu_die_early();
}
@@ -3754,7 +3754,7 @@ static void verify_hyp_capabilities(void)
safe_vmid_bits = get_vmid_bits(safe_mmfr1);
vmid_bits = get_vmid_bits(mmfr1);
if (vmid_bits < safe_vmid_bits) {
- pr_crit("CPU%d: VMID width mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: VMID width mismatch\n", smp_processor_id());
cpu_die_early();
}
@@ -3763,7 +3763,7 @@ static void verify_hyp_capabilities(void)
ID_AA64MMFR0_EL1_PARANGE_SHIFT);
ipa_max = id_aa64mmfr0_parange_to_phys_shift(parange);
if (ipa_max < get_kvm_ipa_limit()) {
- pr_crit("CPU%d: IPA range mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: IPA range mismatch\n", smp_processor_id());
cpu_die_early();
}
}
@@ -3776,7 +3776,7 @@ static void verify_mpam_capabilities(void)
if (FIELD_GET(ID_AA64PFR0_EL1_MPAM_MASK, cpu_idr) !=
FIELD_GET(ID_AA64PFR0_EL1_MPAM_MASK, sys_idr)) {
- pr_crit("CPU%d: MPAM version mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: MPAM version mismatch\n", smp_processor_id());
cpu_die_early();
}
@@ -3784,7 +3784,7 @@ static void verify_mpam_capabilities(void)
sys_idr = read_sanitised_ftr_reg(SYS_MPAMIDR_EL1);
if (FIELD_GET(MPAMIDR_EL1_HAS_HCR, cpu_idr) !=
FIELD_GET(MPAMIDR_EL1_HAS_HCR, sys_idr)) {
- pr_crit("CPU%d: Missing MPAM HCR\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: Missing MPAM HCR\n", smp_processor_id());
cpu_die_early();
}
@@ -3793,7 +3793,7 @@ static void verify_mpam_capabilities(void)
sys_partid_max = FIELD_GET(MPAMIDR_EL1_PARTID_MAX, sys_idr);
sys_pmg_max = FIELD_GET(MPAMIDR_EL1_PMG_MAX, sys_idr);
if (cpu_partid_max < sys_partid_max || cpu_pmg_max < sys_pmg_max) {
- pr_crit("CPU%d: MPAM PARTID/PMG max values are mismatched\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: MPAM PARTID/PMG max values are mismatched\n", smp_processor_id());
cpu_die_early();
}
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 6bc90ee4820a..52edabc13d51 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -215,7 +215,6 @@ asmlinkage notrace void secondary_start_kernel(void)
if (system_uses_irq_prio_masking())
init_gic_priority_masking();
- rcutree_report_cpu_starting(cpu);
trace_hardirqs_off();
/*
@@ -224,6 +223,7 @@ asmlinkage notrace void secondary_start_kernel(void)
* fail to come online.
*/
check_local_cpu_capabilities();
+ rcutree_report_cpu_starting(cpu);
ops = get_cpu_ops(cpu);
if (ops->cpu_postboot)
@@ -404,7 +404,7 @@ void __noreturn cpu_die_early(void)
{
int cpu = smp_processor_id();
- pr_crit("CPU%d: will not boot\n", cpu);
+ printk_deferred(KERN_CRIT "CPU%d: will not boot\n", cpu);
/* Mark this CPU absent */
set_cpu_present(cpu, 0);
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 6b8a3245f393..9b5ab56aad5a 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -70,7 +70,7 @@ void verify_cpu_asid_bits(void)
* We cannot decrease the ASID size at runtime, so panic if we support
* fewer ASID bits than the boot CPU.
*/
- pr_crit("CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
+ printk_deferred(KERN_CRIT "CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
smp_processor_id(), asid, asid_bits);
cpu_panic_kernel(smp_processor_id());
}
--
2.34.1
^ permalink raw reply related
* [PATCH v2 4/4] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
Support for parallel secondary CPU bringup is already utilized by x86,
MIPS, and RISC-V. This patch brings this capability to the arm64
architecture.
Rework the global `secondary_data` accessed during early boot into
a per-CPU array `cpu_boot_data` to allow secondary CPUs to boot
in parallel.
And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.
To fully enable HOTPLUG_PARALLEL, this patch implements:
1) An arm64-specific arch_cpuhp_init_parallel_bringup() handler.
2) An arm64-specific arch_cpuhp_kick_ap_alive() handler.
3) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
Tested natively with ATF on QEMU arm64 virt machine with 64 cores
and also tested with KVM arm64 guest with 128 vCPUs.
Bringup Time Comparison (ms, lower is better):
| Platform | Baseline| P=0 | P=1 | Delta(%)|
| -------------------------------- | ------- | ------- | ------ | ------- |
| 128 vCPUs KVM (256-core HIP09) | 1921.5 | 1895.9 | 2776.9 | -44.52% |
| 48 vCPUs KVM (64-core Cortex-A72)| 3644.7 | 3883.6 | 4406.3 | -20.9% |
| 64-core ATF QEMU | 2075.8 | 2080.7 | 1653.4 | 20.34% |
| 192-core server(HIP12) | 14619.2 | 14619.1 | 8589.4 | 41.21% |
| 32-core board | 2776.5 | 2881.0 | 1045.0 | 62.36% |
Tested-by: Michael Kelley <mhklinux@outlook.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/smp.h | 11 ++++++++++
arch/arm64/kernel/asm-offsets.c | 4 ++++
arch/arm64/kernel/head.S | 36 +++++++++++++++++++++++++++++++++
arch/arm64/kernel/smp.c | 33 ++++++++++++++++++++++++++++++
5 files changed, 85 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9091c67e1cc2..8735e9d8ed13 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -113,6 +113,7 @@ config ARM64
select CPUMASK_OFFSTACK if NR_CPUS > 256
select DCACHE_WORD_ACCESS
select HAVE_EXTRA_IPI_TRACEPOINTS
+ select HOTPLUG_PARALLEL if SMP && HOTPLUG_CPU
select DYNAMIC_FTRACE if FUNCTION_TRACER
select DMA_BOUNCE_UNALIGNED_KMALLOC
select DMA_DIRECT_REMAP
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index e2151a01731f..30025030489c 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -92,7 +92,14 @@ struct secondary_data {
long status;
};
+#ifdef CONFIG_HOTPLUG_PARALLEL
+static_assert((sizeof(struct secondary_data) & (sizeof(struct secondary_data) - 1)) == 0,
+ "secondary_data size must be a power of 2 for assembly lsl assembly!");
+
+extern struct secondary_data cpu_boot_data[NR_CPUS];
+#else
extern struct secondary_data secondary_data;
+#endif
extern long __early_cpu_boot_status;
extern void secondary_entry(void);
@@ -124,7 +131,11 @@ static inline void __noreturn cpu_park_loop(void)
static inline void update_cpu_boot_status(unsigned int cpu, int val)
{
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ WRITE_ONCE(cpu_boot_data[cpu].status, val);
+#else
WRITE_ONCE(secondary_data.status, val);
+#endif
/* Ensure the visibility of the status update */
dsb(ishst);
}
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index b6367ff3a49c..0a0aa965dbb4 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -11,6 +11,7 @@
#include <linux/arm_sdei.h>
#include <linux/sched.h>
#include <linux/ftrace.h>
+#include <linux/log2.h>
#include <linux/kexec.h>
#include <linux/mm.h>
#include <linux/kvm_host.h>
@@ -97,6 +98,9 @@ int main(void)
BLANK();
#endif
DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ DEFINE(SECONDARY_DATA_SHIFT, ilog2(sizeof(struct secondary_data)));
+#endif
BLANK();
DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val));
DEFINE(FTR_OVR_MASK_OFFSET, offsetof(struct arm64_ftr_override, mask));
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca..000ab1acf7c5 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -12,6 +12,7 @@
#include <linux/linkage.h>
#include <linux/init.h>
#include <linux/pgtable.h>
+#include <linux/threads.h>
#include <asm/asm_pointer_auth.h>
#include <asm/assembler.h>
@@ -378,6 +379,33 @@ alternative_else_nop_endif
br x8
SYM_FUNC_END(secondary_startup)
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ /*
+ * Convert the physical MPIDR of the current secondary CPU
+ * to its logical CPUID by traversing __cpu_logical_map
+ * in parallel.
+ */
+ .macro mpidr_to_cpuid, mpidr, cpuid, tmp1, tmp2
+ mov_q \tmp1, MPIDR_HWID_BITMASK
+ and \mpidr, \mpidr, \tmp1
+
+ adr_l \tmp1, __cpu_logical_map
+ mov \cpuid, #0
+.Lfind_cpuid\@:
+ ldr \tmp2, [\tmp1, \cpuid, lsl #3]
+ cmp \tmp2, #-1
+ b.eq .Lnext_cpu\@
+ cmp \tmp2, \mpidr
+ b.eq .Lfound_cpuid\@
+.Lnext_cpu\@:
+ add \cpuid, \cpuid, #1
+ cmp \cpuid, #NR_CPUS
+ b.ne .Lfind_cpuid\@
+ b __secondary_too_slow
+.Lfound_cpuid\@:
+ .endm
+#endif
+
.text
SYM_FUNC_START_LOCAL(__secondary_switched)
mov x0, x20
@@ -391,7 +419,15 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
msr vbar_el1, x5
isb
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ mrs x0, mpidr_el1
+ mpidr_to_cpuid mpidr=x0, cpuid=x2, tmp1=x1, tmp2=x3
+
+ adr_l x0, cpu_boot_data
+ add x0, x0, x2, lsl #SECONDARY_DATA_SHIFT
+#else
adr_l x0, secondary_data
+#endif
ldr x2, [x0, #CPU_BOOT_TASK]
cbz x2, __secondary_too_slow
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 52edabc13d51..f7562c38d724 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -61,6 +61,11 @@
* where to place its SVC stack
*/
struct secondary_data secondary_data;
+
+#ifdef CONFIG_HOTPLUG_PARALLEL
+struct secondary_data cpu_boot_data[NR_CPUS] ____cacheline_aligned;
+#endif
+
/* Number of CPUs which aren't online, but looping in kernel text. */
static int cpus_stuck_in_kernel;
@@ -106,8 +111,30 @@ static int boot_secondary(unsigned int cpu, struct task_struct *idle)
return -EOPNOTSUPP;
}
+#ifndef CONFIG_HOTPLUG_PARALLEL
static DECLARE_COMPLETION(cpu_running);
+#endif
+
+#ifdef CONFIG_HOTPLUG_PARALLEL
+extern const struct cpu_operations smp_spin_table_ops;
+
+/* Establish whether parallel bringup can be supported. */
+bool __init arch_cpuhp_init_parallel_bringup(void)
+{
+ int cpu = smp_processor_id();
+ const struct cpu_operations *ops = get_cpu_ops(cpu);
+ return ops && ops != &smp_spin_table_ops;
+}
+
+int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle)
+{
+ cpu_boot_data[cpu].task = tidle;
+ update_cpu_boot_status(cpu, CPU_MMU_OFF);
+
+ return boot_secondary(cpu, tidle);
+}
+#else
int __cpu_up(unsigned int cpu, struct task_struct *idle)
{
int ret;
@@ -172,6 +199,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
return -EIO;
}
+#endif /* CONFIG_HOTPLUG_PARALLEL */
static void init_gic_priority_masking(void)
{
@@ -223,6 +251,9 @@ asmlinkage notrace void secondary_start_kernel(void)
* fail to come online.
*/
check_local_cpu_capabilities();
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ cpuhp_ap_sync_alive();
+#endif
rcutree_report_cpu_starting(cpu);
ops = get_cpu_ops(cpu);
@@ -254,7 +285,9 @@ asmlinkage notrace void secondary_start_kernel(void)
read_cpuid_id());
update_cpu_boot_status(cpu, CPU_BOOT_SUCCESS);
set_cpu_online(cpu, true);
+#ifndef CONFIG_HOTPLUG_PARALLEL
complete(&cpu_running);
+#endif
/*
* Secondary CPUs enter the kernel with all DAIF exceptions masked.
--
2.34.1
^ permalink raw reply related
* [PATCH v2 2/4] arm64: smp: Pass CPU ID to update_cpu_boot_status()
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
To support CONFIG_HOTPLUG_PARALLEL, the CPU boot status tracking must
be refactored from a single global variable (secondary_data.status)
to a per-CPU tracking structure to prevent multi-core race conditions.
Add a 'cpu' parameter to update_cpu_boot_status() and update all its
callsites to pass the corresponding CPU ID. This allows updating the
boot status at a per-CPU granularity during parallel bringup.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/smp.h | 6 +++---
arch/arm64/kernel/cpufeature.c | 2 +-
arch/arm64/kernel/smp.c | 8 ++++----
arch/arm64/mm/context.c | 2 +-
4 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 10ea4f543069..e2151a01731f 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -122,7 +122,7 @@ static inline void __noreturn cpu_park_loop(void)
}
}
-static inline void update_cpu_boot_status(int val)
+static inline void update_cpu_boot_status(unsigned int cpu, int val)
{
WRITE_ONCE(secondary_data.status, val);
/* Ensure the visibility of the status update */
@@ -134,9 +134,9 @@ static inline void update_cpu_boot_status(int val)
* which calls for a kernel panic. Update the boot status and park the calling
* CPU.
*/
-static inline void __noreturn cpu_panic_kernel(void)
+static inline void __noreturn cpu_panic_kernel(unsigned int cpu)
{
- update_cpu_boot_status(CPU_PANIC_KERNEL);
+ update_cpu_boot_status(cpu, CPU_PANIC_KERNEL);
cpu_park_loop();
}
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6d53bb15cf7b..0552202702bf 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3674,7 +3674,7 @@ static void verify_local_cpu_caps(u16 scope_mask)
caps->desc, system_has_cap, cpu_has_cap);
if (cpucap_panic_on_conflict(caps))
- cpu_panic_kernel();
+ cpu_panic_kernel(smp_processor_id());
else
cpu_die_early();
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 24f8448e1fbb..6bc90ee4820a 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -118,7 +118,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
* page tables.
*/
secondary_data.task = idle;
- update_cpu_boot_status(CPU_MMU_OFF);
+ update_cpu_boot_status(cpu, CPU_MMU_OFF);
/* Now bring the CPU into our world */
ret = boot_secondary(cpu, idle);
@@ -252,7 +252,7 @@ asmlinkage notrace void secondary_start_kernel(void)
pr_info("CPU%u: Booted secondary processor 0x%010lx [0x%08x]\n",
cpu, (unsigned long)mpidr,
read_cpuid_id());
- update_cpu_boot_status(CPU_BOOT_SUCCESS);
+ update_cpu_boot_status(cpu, CPU_BOOT_SUCCESS);
set_cpu_online(cpu, true);
complete(&cpu_running);
@@ -411,11 +411,11 @@ void __noreturn cpu_die_early(void)
rcutree_report_cpu_dead();
if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) {
- update_cpu_boot_status(CPU_KILL_ME);
+ update_cpu_boot_status(cpu, CPU_KILL_ME);
__cpu_try_die(cpu);
}
- update_cpu_boot_status(CPU_STUCK_IN_KERNEL);
+ update_cpu_boot_status(cpu, CPU_STUCK_IN_KERNEL);
cpu_park_loop();
}
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 0f4a28b87469..6b8a3245f393 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -72,7 +72,7 @@ void verify_cpu_asid_bits(void)
*/
pr_crit("CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
smp_processor_id(), asid, asid_bits);
- cpu_panic_kernel();
+ cpu_panic_kernel(smp_processor_id());
}
}
--
2.34.1
^ permalink raw reply related
* [PATCH v2 0/4] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
Support for parallel secondary CPU bringup is already utilized by x86,
MIPS, and RISC-V. This patch brings this capability to the arm64
architecture.
Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST to avoid primary SMT threads
to boot first constraint.
And Add a 'cpu' parameter to update_cpu_boot_status() to allow updating the
boot status at a per-CPU granularity during parallel bringup.
Rework the global `secondary_data` accessed during early boot into
a per-CPU array `cpu_boot_data` to allow secondary CPUs to boot
in parallel.
And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.
Changes in v2:
- Remove RFC.
- Add Tested-by.
- Fix AI review issues in [1].
- Add arch_cpuhp_init_parallel_bringup() to check psci boot.
- Reuse `__cpu_logical_map` instead of a new aray.
- Defer rcutree_report_cpu_starting() until after
check_local_cpu_capabilities() to prevent a potential control CPU
deadlock if an early capability check fails.
- Move the assembly in head.S to a macro called `mpidr_to_cpuid`.
- Add `SECONDARY_DATA_SHIFT` for `lsl` to access `cpu_boot_data`.
- Add sizeof(struct secondary_data) power of 2 assert check.
- Expand testing with more data collected from real hardware.
[1] https://sashiko.dev/#/patchset/20260611133809.3854977-1-ruanjinjie%40huawei.com
Jinjie Ruan (4):
cpu/hotplug: Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST
arm64: smp: Pass CPU ID to update_cpu_boot_status()
arm64: smp: Defer RCU reporting until after local CPU capability
checks
arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
arch/Kconfig | 4 +++
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/smp.h | 17 ++++++++++---
arch/arm64/kernel/asm-offsets.c | 4 +++
arch/arm64/kernel/cpufeature.c | 22 ++++++++--------
arch/arm64/kernel/head.S | 36 ++++++++++++++++++++++++++
arch/arm64/kernel/smp.c | 45 ++++++++++++++++++++++++++++-----
arch/arm64/mm/context.c | 4 +--
arch/mips/Kconfig | 1 +
arch/riscv/Kconfig | 1 +
arch/x86/Kconfig | 1 +
kernel/cpu.c | 6 ++++-
12 files changed, 119 insertions(+), 23 deletions(-)
--
2.34.1
^ permalink raw reply
* [PATCH v2 1/4] cpu/hotplug: Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
During parallel CPU bringup, x86 requires primary SMT threads to boot
first to avoid siblings stopping during microcode updates. This constraint
is architecture-specific and unnecessary for other platforms
like arm64.
Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST to decouple this constraint.
Platforms requiring this temporal order (e.g., x86) can select it
in Kconfig. Other architectures (e.g., arm64) can leave it unselected
to entirely bypass the SMT branch via the preprocessor.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/Kconfig | 4 ++++
arch/mips/Kconfig | 1 +
arch/riscv/Kconfig | 1 +
arch/x86/Kconfig | 1 +
kernel/cpu.c | 6 +++++-
5 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index e86880045158..0365d2df2659 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -102,6 +102,10 @@ config HOTPLUG_PARALLEL
bool
select HOTPLUG_SPLIT_STARTUP
+config PARALLEL_SMT_PRIMARY_FIRST
+ bool
+ depends on HOTPLUG_PARALLEL
+
config GENERIC_IRQ_ENTRY
bool
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4364f3dba688..84e11ac0cf71 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -642,6 +642,7 @@ config EYEQ
select MIPS_CPU_SCACHE
select MIPS_GIC
select MIPS_L1_CACHE_SHIFT_7
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DRIVERS_GENERIC
select SMP_UP if SMP
select SWAP_IO_SPACE
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index d235396c4514..0cc49aecc841 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -210,6 +210,7 @@ config RISCV
select OF
select OF_EARLY_FLATTREE
select OF_IRQ
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DOMAINS_GENERIC if PCI
select PCI_ECAM if (ACPI && PCI)
select PCI_MSI if PCI
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f3f7cb01d69d..3ad4115ad051 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -314,6 +314,7 @@ config X86
select NEED_PER_CPU_PAGE_FIRST_CHUNK
select NEED_SG_DMA_LENGTH
select NUMA_MEMBLKS if NUMA
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DOMAINS if PCI
select PCI_LOCKLESS_CONFIG if PCI
select PERF_EVENTS
diff --git a/kernel/cpu.c b/kernel/cpu.c
index bc4f7a9ba64e..7ef8cdf4d239 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1792,6 +1792,7 @@ static int __init parallel_bringup_parse_param(char *arg)
}
early_param("cpuhp.parallel", parallel_bringup_parse_param);
+#ifdef CONFIG_PARALLEL_SMT_PRIMARY_FIRST
#ifdef CONFIG_HOTPLUG_SMT
static inline bool cpuhp_smt_aware(void)
{
@@ -1811,7 +1812,8 @@ static inline const struct cpumask *cpuhp_get_primary_thread_mask(void)
{
return cpu_none_mask;
}
-#endif
+#endif /* CONFIG_HOTPLUG_SMT */
+#endif /* CONFIG_PARALLEL_SMT_PRIMARY_FIRST */
bool __weak arch_cpuhp_init_parallel_bringup(void)
{
@@ -1837,6 +1839,7 @@ static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus)
if (!__cpuhp_parallel_bringup)
return false;
+#ifdef CONFIG_PARALLEL_SMT_PRIMARY_FIRST
if (cpuhp_smt_aware()) {
const struct cpumask *pmask = cpuhp_get_primary_thread_mask();
static struct cpumask tmp_mask __initdata;
@@ -1857,6 +1860,7 @@ static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus)
cpumask_andnot(&tmp_mask, mask, pmask);
mask = &tmp_mask;
}
+#endif /* CONFIG_PARALLEL_SMT_PRIMARY_FIRST */
/* Bring the not-yet started CPUs up */
cpuhp_bringup_mask(mask, ncpus, CPUHP_BP_KICK_AP);
--
2.34.1
^ permalink raw reply related
* Re: [PATCH v1 03/11] KVM: arm64: Use guard()/scoped_guard() in arm64 KVM EL1 code
From: Fuad Tabba @ 2026-06-18 9:24 UTC (permalink / raw)
To: Marc Zyngier
Cc: Oliver Upton, Will Deacon, Catalin Marinas, Quentin Perret,
Vincent Donnefort, Sebastian Ene, Per Larsen, Suzuki K Poulose,
Zenghui Yu, Joey Gouly, Steffen Eiden, Mark Rutland,
Jonathan Cameron, Hyunwoo Kim, linux-arm-kernel, kvmarm,
linux-kernel
In-Reply-To: <86jyrwrymb.wl-maz@kernel.org>
On Thu, 18 Jun 2026 at 10:23, Marc Zyngier <maz@kernel.org> wrote:
>
> On Fri, 12 Jun 2026 07:59:17 +0100,
> tabba@google.com wrote:
> >
> > Convert the manual mutex_lock()/spin_lock() pairs in
> > arch/arm64/kvm/{pkvm,arm,mmu,reset,psci}.c to guard(mutex),
> > guard(spinlock) and scoped_guard(), dropping unlock-only goto labels in
> > favour of direct returns. Centralised cleanup gotos that still serve
> > other resources are preserved.
> >
> > reset.c uses scoped_guard() rather than guard() so the lock covers only
> > the small read/update window inside kvm_reset_vcpu(), leaving the rest
> > of the function outside the critical section.
>
> To be brutally honest, I don't think this sort of widespread changes
> bring us anything. This is just churn.
>
> Sure, if you are reworking a particular bit of code that is goto-heavy
> for the purpose of error handling, this has the potential to cleanup
> the code *while you are changing it*.
>
> But doing it for the sake of doing it? I think we have bigger fish to
> fry right now.
I understand what you mean. Would you like me to drop all of the guard
patches, or only those that go beyond the code changed in this series?
Thanks,
/fuad
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply
* Re: [PATCH RFC v4 10/12] reset: zte: Add a zx297520v3 reset driver
From: Philipp Zabel @ 2026-06-18 9:24 UTC (permalink / raw)
To: Stefan Dösinger, Michael Turquette, Stephen Boyd,
Rob Herring, Krzysztof Kozlowski, Conor Dooley, Brian Masney
Cc: linux-clk, devicetree, linux-kernel, linux-arm-kernel
In-Reply-To: <20260616-zx29clk-v4-10-ca994bd22e9d@gmail.com>
On Di, 2026-06-16 at 23:26 +0300, Stefan Dösinger wrote:
> This drives the auxiliary devices created by the clock driver.
Which auxiliary devices? Which clock driver?
> Signed-off-by: Stefan Dösinger <stefandoesinger@gmail.com>
> ---
> MAINTAINERS | 1 +
> drivers/reset/Kconfig | 11 ++
> drivers/reset/Makefile | 1 +
> drivers/reset/reset-zte-zx297520v3.c | 224 +++++++++++++++++++++++++++++++++++
> 4 files changed, 237 insertions(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f1f0459b2c72..55bf0290343a 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3871,6 +3871,7 @@ F: Documentation/devicetree/zte,zx297520v3-*
> F: arch/arm/boot/dts/zte/
> F: arch/arm/mach-zte/
> F: drivers/clk/zte/
> +F: drivers/reset/reset-zte-zx297520v3.c
> F: include/dt-bindings/clock/zte,zx297520v3-clk.h
>
> ARM/ZYNQ ARCHITECTURE
> diff --git a/drivers/reset/Kconfig b/drivers/reset/Kconfig
> index d009eb0849a3..116dd23f1b8e 100644
> --- a/drivers/reset/Kconfig
> +++ b/drivers/reset/Kconfig
> @@ -404,6 +404,17 @@ config RESET_UNIPHIER_GLUE
> on UniPhier SoCs. Say Y if you want to control reset signals
> provided by the glue layer.
>
> +config RESET_ZTE_ZX297520V3
> + tristate "ZTE zx297520v3 Reset Driver"
> + depends on (ARCH_ZTE || COMPILE_TEST)
> + default CLK_ZTE_ZX297520V3
> + select AUXILIARY_BUS
> + help
> + This enables the reset controller for ZTE zx297520v3 SoCs. The reset
> + controller is part of the clock controller on this SoC. This driver
> + operates on an auxiliary device exposed by the clock driver. Enable
> + this driver if you plan to boot the kernel on a zx297520v3 based SoC.
> +
> config RESET_ZYNQ
> bool "ZYNQ Reset Driver" if COMPILE_TEST
> default ARCH_ZYNQ
> diff --git a/drivers/reset/Makefile b/drivers/reset/Makefile
> index 3e52569bd276..9a8a48d44dc4 100644
> --- a/drivers/reset/Makefile
> +++ b/drivers/reset/Makefile
> @@ -50,5 +50,6 @@ obj-$(CONFIG_RESET_TI_TPS380X) += reset-tps380x.o
> obj-$(CONFIG_RESET_TN48M_CPLD) += reset-tn48m.o
> obj-$(CONFIG_RESET_UNIPHIER) += reset-uniphier.o
> obj-$(CONFIG_RESET_UNIPHIER_GLUE) += reset-uniphier-glue.o
> +obj-$(CONFIG_RESET_ZTE_ZX297520V3) += reset-zte-zx297520v3.o
> obj-$(CONFIG_RESET_ZYNQ) += reset-zynq.o
> obj-$(CONFIG_RESET_ZYNQMP) += reset-zynqmp.o
> diff --git a/drivers/reset/reset-zte-zx297520v3.c b/drivers/reset/reset-zte-zx297520v3.c
> new file mode 100644
> index 000000000000..2022f4df2ebd
> --- /dev/null
> +++ b/drivers/reset/reset-zte-zx297520v3.c
> @@ -0,0 +1,224 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2026 Stefan Dösinger
> + */
> +#include <dt-bindings/clock/zte,zx297520v3-clk.h>
> +#include <linux/reset-controller.h>
> +#include <linux/platform_device.h>
What is this used for?
> +#include <linux/auxiliary_bus.h>
> +#include <linux/clk-provider.h>
What is this used for?
> +#include <linux/mfd/syscon.h>
> +#include <linux/regmap.h>
> +#include <linux/iopoll.h>
> +#include <linux/delay.h>
> +
> +struct zte_reset_reg {
> + u32 mask, wait_mask;
> + u16 reg;
> +};
> +
> +struct zte_reset_info {
> + const struct zte_reset_reg *resets;
> + unsigned int num;
> +};
> +
> +struct zte_reset {
> + struct reset_controller_dev rcdev;
> + struct regmap *map;
> + const struct zte_reset_reg *resets;
> +};
> +
> +static inline struct zte_reset *to_zte_reset(struct reset_controller_dev *rcdev)
> +{
> + return container_of(rcdev, struct zte_reset, rcdev);
> +}
> +
> +static int zx29_rst_assert(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> +
> + return regmap_clear_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> +}
> +
> +static int zx29_rst_deassert(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> + int res;
> + u32 val;
> +
> + res = regmap_set_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> + if (res)
> + return res;
> +
> + /* This is a special case used only by USB reset */
> + if (rst->resets[id].wait_mask) {
> + return regmap_read_poll_timeout(rst->map, rst->resets[id].reg + 4, val,
> + val & rst->resets[id].wait_mask, 1, 100);
> + }
> +
> + return 0;
> +}
> +
> +static int zx29_rst_status(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> + int res;
> +
> + res = regmap_test_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> + if (res < 0)
> + return res;
> +
> + return !res;
> +}
> +
> +static const struct reset_control_ops zx29_rst_ops = {
> + .assert = zx29_rst_assert,
> + .deassert = zx29_rst_deassert,
> + .status = zx29_rst_status,
> +};
> +
> +static const struct zte_reset_reg zx297520v3_top_resets[] = {
> + /* This bit is set by ZTE's cpko.ko blob, it looks like a reset bit for the LTE DSP
> + * coprocessor. Clocks for it are in matrixclk.
> + */
> + [ZX297520V3_ZSP_RESET] = { .reg = 0x13c, .mask = BIT(0) },
> +
> + [ZX297520V3_UART0_RESET] = { .reg = 0x78, .mask = BIT(6) | BIT(7) },
Is this a single reset line controlled by two bits (do you know what
they are)? Or might these actually be two different reset controls that
are just always set together?
> + [ZX297520V3_I2C0_RESET] = { .reg = 0x74, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_RTC_RESET] = { .reg = 0x74, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T08_RESET] = { .reg = 0x78, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T09_RESET] = { .reg = 0x78, .mask = BIT(2) | BIT(3) },
> + [ZX297520V3_PMM_RESET] = { .reg = 0x74, .mask = BIT(0) | BIT(1) },
> +
> + /* I haven't found any clocks for GPIO. It probably wouldn't make much
> + * sense anyway. Only one reset bit per controller.
> + */
> + [ZX297520V3_GPIO_RESET] = { .reg = 0x74, .mask = BIT(3) },
> + [ZX297520V3_GPIO8_RESET] = { .reg = 0x74, .mask = BIT(2) },
> +
> + [ZX297520V3_TIMER_T12_RESET] = { .reg = 0x74, .mask = BIT(6) | BIT(7) },
> + [ZX297520V3_TIMER_T13_RESET] = { .reg = 0x7c, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_TIMER_T14_RESET] = { .reg = 0x7c, .mask = BIT(2) | BIT(3) },
> + [ZX297520V3_TIMER_T15_RESET] = { .reg = 0x74, .mask = BIT(10) | BIT(11) },
> + [ZX297520V3_TIMER_T16_RESET] = { .reg = 0x7c, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T17_RESET] = { .reg = 0x12c, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_WDT_T18_RESET] = { .reg = 0x74, .mask = BIT(12) | BIT(13) },
> + [ZX297520V3_USIM1_RESET] = { .reg = 0x74, .mask = BIT(14) | BIT(15) },
> + [ZX297520V3_AHB_RESET] = { .reg = 0x70, .mask = BIT(0) | BIT(1) },
> +
> + /* USB reset. This is slightly special because it needs to wait for a ready bit after
> + * deasserting.
> + */
> + [ZX297520V3_USB_RESET] = { .reg = 0x80, .mask = BIT(3) | BIT(4) | BIT(5),
> + .wait_mask = BIT(1)},
Same as above, are these actually three separate reset lines?
> + [ZX297520V3_HSIC_RESET] = { .reg = 0x80, .mask = BIT(0) | BIT(1) | BIT(2),
> + .wait_mask = BIT(0)},
> +};
> +
> +static const struct zte_reset_info zx297520v3_top_info = {
> + .resets = zx297520v3_top_resets,
> + .num = ARRAY_SIZE(zx297520v3_top_resets),
> +};
> +
> +static const struct zte_reset_reg zx297520v3_matrix_resets[] = {
> + [ZX297520V3_CPU_RESET] = { .reg = 0x28, .mask = BIT(1) },
> + [ZX297520V3_EDCP_RESET] = { .reg = 0x68, .mask = BIT(0) },
> + [ZX297520V3_SD0_RESET] = { .reg = 0x58, .mask = BIT(1) },
> + [ZX297520V3_SD1_RESET] = { .reg = 0x58, .mask = BIT(0) },
> + [ZX297520V3_NAND_RESET] = { .reg = 0x58, .mask = BIT(4) },
> + [ZX297520V3_PDCFG_RESET] = { .reg = 0x94, .mask = BIT(20) },
> + [ZX297520V3_SSC_RESET] = { .reg = 0x94, .mask = BIT(24) },
> + [ZX297520V3_GMAC_RESET] = { .reg = 0x114, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_VOU_RESET] = { .reg = 0x16c, .mask = BIT(0) },
> +};
> +
> +static const struct zte_reset_info zx297520v3_matrix_info = {
> + .resets = zx297520v3_matrix_resets,
> + .num = ARRAY_SIZE(zx297520v3_matrix_resets),
> +};
> +
> +static const struct zte_reset_reg zx297520v3_lsp_resets[] = {
> + [ZX297520V3_TIMER_L1_RESET] = { .reg = 0x04, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_L2_RESET] = { .reg = 0x08, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_L3_RESET] = { .reg = 0x0c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_PWM_RESET] = { .reg = 0x10, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_I2S0_RESET] = { .reg = 0x14, .mask = BIT(8) | BIT(9) },
> + /* 0x18: Not writeable */
> + [ZX297520V3_I2S1_RESET] = { .reg = 0x1c, .mask = BIT(8) | BIT(9) },
> + /* 0x20: Not writeable */
> + [ZX297520V3_QSPI_RESET] = { .reg = 0x24, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_UART1_RESET] = { .reg = 0x28, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_I2C1_RESET] = { .reg = 0x2c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_SPI0_RESET] = { .reg = 0x30, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LB_RESET] = { .reg = 0x34, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LC_RESET] = { .reg = 0x38, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_UART2_RESET] = { .reg = 0x3c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_LE_RESET] = { .reg = 0x40, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LF_RESET] = { .reg = 0x44, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_SPI1_RESET] = { .reg = 0x48, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_L11_RESET] = { .reg = 0x4c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TDM_RESET] = { .reg = 0x50, .mask = BIT(8) | BIT(9) },
> +};
> +
> +static const struct zte_reset_info zx297520v3_lsp_info = {
> + .resets = zx297520v3_lsp_resets,
> + .num = ARRAY_SIZE(zx297520v3_lsp_resets),
> +};
> +
> +static int reset_zx297520v3_probe(struct auxiliary_device *adev,
> + const struct auxiliary_device_id *id)
> +{
> + const struct zte_reset_info *drv_info;
> + struct device *dev = &adev->dev;
> + struct zte_reset *rst;
> +
> + drv_info = (struct zte_reset_info *)id->driver_data;
> +
> + rst = devm_kzalloc(dev, sizeof(*rst), GFP_KERNEL);
> + if (!rst)
> + return -ENOMEM;
> +
> + rst->resets = drv_info->resets;
> + rst->rcdev.owner = THIS_MODULE;
> + rst->rcdev.nr_resets = drv_info->num;
> + rst->rcdev.ops = &zx29_rst_ops;
> + rst->rcdev.of_node = dev->of_node;
> + rst->rcdev.dev = dev;
> + rst->rcdev.of_reset_n_cells = 1;
No need to set of_reset_n_cells if of_xlate is not set. Here
reset_controller_register will use fwnode_n_cells and set it to 1
anyway.
> +
> + rst->map = device_node_to_regmap(dev->of_node);
> + if (IS_ERR(rst->map))
> + return dev_err_probe(rdev, PTR_ERR(rst->map), "Cannot get parent syscon regmap\n");
> +
> + return devm_reset_controller_register(dev, &rst->rcdev);
> +}
> +
> +static const struct auxiliary_device_id reset_zx297520v3_ids[] = {
> + {
> + .name = "clk_zte.zx297520v3_toprst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_top_info,
> + },
> + {
> + .name = "clk_zte.zx297520v3_matrixrst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_matrix_info,
> + },
> + {
> + .name = "clk_zte.zx297520v3_lsprst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_lsp_info,
> + },
> + { },
> +};
> +
Drop this empty line.
> +MODULE_DEVICE_TABLE(auxiliary, reset_zx297520v3_ids);
> +
> +static struct auxiliary_driver reset_zx297520v3_drv = {
> + .name = "zx297520v3_reset",
> + .id_table = reset_zx297520v3_ids,
> + .probe = reset_zx297520v3_probe,
> +};
> +
Drop this empty line.
> +module_auxiliary_driver(reset_zx297520v3_drv);
> +
> +MODULE_AUTHOR("Stefan Dösinger <stefandoesinger@gmail.com>");
> +MODULE_DESCRIPTION("ZTE zx297520v3 reset driver");
> +MODULE_LICENSE("GPL");
regards
Philipp
^ permalink raw reply
* Re: [PATCH v1 03/11] KVM: arm64: Use guard()/scoped_guard() in arm64 KVM EL1 code
From: Marc Zyngier @ 2026-06-18 9:23 UTC (permalink / raw)
To: tabba
Cc: Oliver Upton, Will Deacon, Catalin Marinas, Quentin Perret,
Vincent Donnefort, Sebastian Ene, Per Larsen, Suzuki K Poulose,
Zenghui Yu, Joey Gouly, Steffen Eiden, Mark Rutland,
Jonathan Cameron, Hyunwoo Kim, linux-arm-kernel, kvmarm,
linux-kernel
In-Reply-To: <20260612065925.755562-4-tabba@google.com>
On Fri, 12 Jun 2026 07:59:17 +0100,
tabba@google.com wrote:
>
> Convert the manual mutex_lock()/spin_lock() pairs in
> arch/arm64/kvm/{pkvm,arm,mmu,reset,psci}.c to guard(mutex),
> guard(spinlock) and scoped_guard(), dropping unlock-only goto labels in
> favour of direct returns. Centralised cleanup gotos that still serve
> other resources are preserved.
>
> reset.c uses scoped_guard() rather than guard() so the lock covers only
> the small read/update window inside kvm_reset_vcpu(), leaving the rest
> of the function outside the critical section.
To be brutally honest, I don't think this sort of widespread changes
bring us anything. This is just churn.
Sure, if you are reworking a particular bit of code that is goto-heavy
for the purpose of error handling, this has the potential to cleanup
the code *while you are changing it*.
But doing it for the sake of doing it? I think we have bigger fish to
fry right now.
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply
* [PATCH v15 9/9] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
memcpy_mc() is the Machine-Check safe memcpy variant that returns the
number of bytes NOT copied on a hardware memory error, or 0 on success.
Add two test cases modeled after the existing memcpy_test() and
memcpy_large_test() implementations.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
lib/tests/memcpy_kunit.c | 121 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 120 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index 812c1fa20fd9..87585fbe78c7 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -554,6 +554,121 @@ static void copy_mc_page_test(struct kunit *test)
}
#endif /* __HAVE_ARCH_COPY_MC_PAGE */
+#ifdef __HAVE_ARCH_MEMCPY_MC
+/*
+ * memcpy_mc() is a Machine-Check safe memcpy variant.
+ * Signature: int memcpy_mc(void *dst, const void *src, size_t len)
+ * Returns: 0 on success, or number of bytes NOT copied on MC error.
+ *
+ * In the normal (no-poison) path it must behave identically to memcpy()
+ * and always return 0.
+ */
+static void memcpy_mc_test(struct kunit *test)
+{
+#define TEST_OP "memcpy_mc"
+ struct some_bytes control = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes zero = { };
+ struct some_bytes middle = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes three = {
+ .data = { 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes dest = { };
+ unsigned long ret;
+ int count;
+ u8 *ptr;
+
+ /* Verify static initializers. */
+ check(control, 0x20);
+ check(zero, 0);
+ compare("static initializers", dest, zero);
+
+ /* Verify assignment. */
+ dest = control;
+ compare("direct assignment", dest, control);
+
+ /* Verify complete overwrite. */
+ ret = memcpy_mc(dest.data, zero.data, sizeof(dest.data));
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("complete overwrite", dest, zero);
+
+ /* Verify middle overwrite: 7 bytes at offset 12. */
+ dest = control;
+ ret = memcpy_mc(dest.data + 12, zero.data, 7);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("middle overwrite", dest, middle);
+
+ /* Verify zero-length copy is a no-op. */
+ dest = control;
+ ret = memcpy_mc(dest.data, zero.data, 0);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("zero length", dest, control);
+
+ /* Verify argument side-effects aren't repeated. */
+ dest = control;
+ ptr = dest.data;
+ count = 1;
+ ret = memcpy_mc(ptr++, zero.data, count++);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ ptr += 8;
+ ret = memcpy_mc(ptr++, zero.data, count++);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("argument side-effects", dest, three);
+#undef TEST_OP
+}
+
+static void memcpy_mc_large_test(struct kunit *test)
+{
+ init_large(test);
+
+ /* Sweep 1..1024 bytes x shifting offset to cover all template paths. */
+ for (int bytes = 1; bytes <= ARRAY_SIZE(large_src); bytes++) {
+ for (int offset = 0; offset < ARRAY_SIZE(large_src); offset++) {
+ int right_zero_pos = offset + bytes;
+ int right_zero_size = ARRAY_SIZE(large_dst) - right_zero_pos;
+ int ret;
+
+ ret = memcpy_mc(large_dst + offset, large_src, bytes);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "memcpy_mc returned %d with size %d at offset %d",
+ ret, bytes, offset);
+
+ /* No write before copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst, large_zero, offset), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* No write after copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(&large_dst[right_zero_pos], large_zero,
+ right_zero_size), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* Byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst + offset, large_src, bytes), 0,
+ "with size %d at offset %d", bytes, offset);
+
+ memset(large_dst + offset, 0, bytes);
+ }
+ cond_resched();
+ }
+}
+#endif /* __HAVE_ARCH_MEMCPY_MC */
+
static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(memset_test),
KUNIT_CASE(memcpy_test),
@@ -564,6 +679,10 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(copy_page_test),
#ifdef __HAVE_ARCH_COPY_MC_PAGE
KUNIT_CASE(copy_mc_page_test),
+#endif
+#ifdef __HAVE_ARCH_MEMCPY_MC
+ KUNIT_CASE(memcpy_mc_test),
+ KUNIT_CASE_SLOW(memcpy_mc_large_test),
#endif
{}
};
@@ -575,5 +694,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset(), copy_page() and memcpy_mc()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related
* [PATCH v15 8/9] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
Add KUnit tests for copy_page() and copy_mc_page(), modeled after
the existing memcpy_test() style: a static page-aligned src and a
two-page dst, filled with random bytes plus non-zero edges, then
verify byte-for-byte equality and that the adjacent page is
untouched.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
lib/tests/memcpy_kunit.c | 67 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 66 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index d36933554e46..812c1fa20fd9 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -493,6 +493,67 @@ static void memmove_overlap_test(struct kunit *test)
}
}
+/* --- Page-sized copy tests --- */
+
+static u8 page_src[PAGE_SIZE] __aligned(PAGE_SIZE);
+static u8 page_dst[PAGE_SIZE * 2] __aligned(PAGE_SIZE);
+static const u8 page_pattern[PAGE_SIZE] __aligned(PAGE_SIZE);
+
+static void init_page(struct kunit *test)
+{
+ /* Get many bit patterns. */
+ get_random_bytes(page_src, PAGE_SIZE);
+
+ /* Make sure we have non-zero edges. */
+ set_random_nonzero(test, &page_src[0]);
+ set_random_nonzero(test, &page_src[PAGE_SIZE - 1]);
+
+ memset(page_dst, 0xA5, ARRAY_SIZE(page_dst));
+ memset(page_pattern, 0xA5, PAGE_SIZE);
+}
+
+static void copy_page_test(struct kunit *test)
+{
+ init_page(test);
+
+ /* Copy. */
+ copy_page(page_dst, page_src);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_pattern, PAGE_SIZE), 0,
+ "copy_page overflow into adjacent page");
+}
+
+#ifdef __HAVE_ARCH_COPY_MC_PAGE
+static void copy_mc_page_test(struct kunit *test)
+{
+ int ret;
+
+ init_page(test);
+
+ /* Copy and check return value. */
+ ret = copy_mc_page(page_dst, page_src);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "copy_mc_page returned %d on clean memory", ret);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_mc_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_pattern, PAGE_SIZE), 0,
+ "copy_mc_page overflow into adjacent page");
+}
+#endif /* __HAVE_ARCH_COPY_MC_PAGE */
+
static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(memset_test),
KUNIT_CASE(memcpy_test),
@@ -500,6 +561,10 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE_SLOW(memmove_test),
KUNIT_CASE_SLOW(memmove_large_test),
KUNIT_CASE_SLOW(memmove_overlap_test),
+ KUNIT_CASE(copy_page_test),
+#ifdef __HAVE_ARCH_COPY_MC_PAGE
+ KUNIT_CASE(copy_mc_page_test),
+#endif
{}
};
@@ -510,5 +575,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), and memset()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related
* [PATCH v15 7/9] arm64: introduce copy_mc_to_kernel() implementation
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
The copy_mc_to_kernel() helper is memory copy implementation that handles
source exceptions. It can be used in memory copy scenarios that tolerate
hardware memory errors(e.g: pmem_read/dax_copy_to_iter).
Currently, only x86 and ppc support this helper, Add this for ARM64 as
well, if ARCH_HAS_COPY_MC is defined, by implementing copy_mc_to_kernel()
and memcpy_mc() functions.
Because there is no caller-saved GPR is available for saving "bytes not
copied" in memcpy(), the memcpy_mc() is referenced to the implementation
of copy_from_user(). In addition, the fixup of MOPS insn is not considered
at present.
[Ruidong: refactor memcpy_mc on top of the new memcpy implementation.]
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/include/asm/string.h | 5 +
arch/arm64/include/asm/uaccess.h | 17 +++
arch/arm64/lib/Makefile | 2 +-
arch/arm64/lib/memcpy.S | 251 +++----------------------------
arch/arm64/lib/memcpy_mc.S | 56 +++++++
arch/arm64/lib/memcpy_template.S | 250 ++++++++++++++++++++++++++++++
mm/kasan/shadow.c | 12 ++
7 files changed, 359 insertions(+), 234 deletions(-)
create mode 100644 arch/arm64/lib/memcpy_mc.S
create mode 100644 arch/arm64/lib/memcpy_template.S
diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h
index 3a3264ff47b9..2e81f6c00cdd 100644
--- a/arch/arm64/include/asm/string.h
+++ b/arch/arm64/include/asm/string.h
@@ -35,6 +35,10 @@ extern void *memchr(const void *, int, __kernel_size_t);
extern void *memcpy(void *, const void *, __kernel_size_t);
extern void *__memcpy(void *, const void *, __kernel_size_t);
+#define __HAVE_ARCH_MEMCPY_MC
+extern unsigned long memcpy_mc(void *, const void *, __kernel_size_t);
+extern unsigned long __memcpy_mc(void *, const void *, __kernel_size_t);
+
#define __HAVE_ARCH_MEMMOVE
extern void *memmove(void *, const void *, __kernel_size_t);
extern void *__memmove(void *, const void *, __kernel_size_t);
@@ -57,6 +61,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt);
*/
#define memcpy(dst, src, len) __memcpy(dst, src, len)
+#define memcpy_mc(dst, src, len) __memcpy_mc(dst, src, len)
#define memmove(dst, src, len) __memmove(dst, src, len)
#define memset(s, c, n) __memset(s, c, n)
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index b0c83a08dda9..93277eca2268 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -499,5 +499,22 @@ static inline size_t probe_subpage_writeable(const char __user *uaddr,
}
#endif /* CONFIG_ARCH_HAS_SUBPAGE_FAULTS */
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/**
+ * copy_mc_to_kernel - memory copy that handles source exceptions
+ *
+ * @to: destination address
+ * @from: source address
+ * @size: number of bytes to copy
+ *
+ * Return 0 for success, or bytes not copied.
+ */
+static inline unsigned long __must_check
+copy_mc_to_kernel(void *to, const void *from, unsigned long size)
+{
+ return memcpy_mc(to, from, size);
+}
+#define copy_mc_to_kernel copy_mc_to_kernel
+#endif
#endif /* __ASM_UACCESS_H */
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 1f4c3f743a20..a5820e6c33d4 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,7 +7,7 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
-lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o memcpy_mc.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
index 9b99106fb95f..ab48c5c798d1 100644
--- a/arch/arm64/lib/memcpy.S
+++ b/arch/arm64/lib/memcpy.S
@@ -15,247 +15,32 @@
*
*/
-#define L(label) .L ## label
+ .macro ldrb1 reg, addr:vararg
+ ldrb \reg, \addr
+ .endm
-#define dstin x0
-#define src x1
-#define count x2
-#define dst x3
-#define srcend x4
-#define dstend x5
-#define A_l x6
-#define A_lw w6
-#define A_h x7
-#define B_l x8
-#define B_lw w8
-#define B_h x9
-#define C_l x10
-#define C_lw w10
-#define C_h x11
-#define D_l x12
-#define D_h x13
-#define E_l x14
-#define E_h x15
-#define F_l x16
-#define F_h x17
-#define G_l count
-#define G_h dst
-#define H_l src
-#define H_h srcend
-#define tmp1 x14
+ .macro ldr1 reg, addr:vararg
+ ldr \reg, \addr
+ .endm
-/* This implementation handles overlaps and supports both memcpy and memmove
- from a single entry point. It uses unaligned accesses and branchless
- sequences to keep the code small, simple and improve performance.
+ .macro ldp1 reg1, reg2, addr:vararg
+ ldp \reg1, \reg2, \addr
+ .endm
- Copies are split into 3 main cases: small copies of up to 32 bytes, medium
- copies of up to 128 bytes, and large copies. The overhead of the overlap
- check is negligible since it is only required for large copies.
-
- Large copies use a software pipelined loop processing 64 bytes per iteration.
- The destination pointer is 16-byte aligned to minimize unaligned accesses.
- The loop tail is handled by always copying 64 bytes from the end.
-*/
-
-SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
- add srcend, src, count
- add dstend, dstin, count
- cmp count, 128
- b.hi L(copy_long)
- cmp count, 32
- b.hi L(copy32_128)
-
- /* Small copies: 0..32 bytes. */
- cmp count, 16
- b.lo L(copy16)
- ldp A_l, A_h, [src]
- ldp D_l, D_h, [srcend, -16]
- stp A_l, A_h, [dstin]
- stp D_l, D_h, [dstend, -16]
- ret
-
- /* Copy 8-15 bytes. */
-L(copy16):
- tbz count, 3, L(copy8)
- ldr A_l, [src]
- ldr A_h, [srcend, -8]
- str A_l, [dstin]
- str A_h, [dstend, -8]
- ret
-
- .p2align 3
- /* Copy 4-7 bytes. */
-L(copy8):
- tbz count, 2, L(copy4)
- ldr A_lw, [src]
- ldr B_lw, [srcend, -4]
- str A_lw, [dstin]
- str B_lw, [dstend, -4]
+ .macro ret1
ret
+ .endm
- /* Copy 0..3 bytes using a branchless sequence. */
-L(copy4):
- cbz count, L(copy0)
- lsr tmp1, count, 1
- ldrb A_lw, [src]
- ldrb C_lw, [srcend, -1]
- ldrb B_lw, [src, tmp1]
- strb A_lw, [dstin]
- strb B_lw, [dstin, tmp1]
- strb C_lw, [dstend, -1]
-L(copy0):
- ret
-
- .p2align 4
- /* Medium copies: 33..128 bytes. */
-L(copy32_128):
- ldp A_l, A_h, [src]
- ldp B_l, B_h, [src, 16]
- ldp C_l, C_h, [srcend, -32]
- ldp D_l, D_h, [srcend, -16]
- cmp count, 64
- b.hi L(copy128)
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy 65..128 bytes. */
-L(copy128):
- ldp E_l, E_h, [src, 32]
- ldp F_l, F_h, [src, 48]
- cmp count, 96
- b.ls L(copy96)
- ldp G_l, G_h, [srcend, -64]
- ldp H_l, H_h, [srcend, -48]
- stp G_l, G_h, [dstend, -64]
- stp H_l, H_h, [dstend, -48]
-L(copy96):
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp E_l, E_h, [dstin, 32]
- stp F_l, F_h, [dstin, 48]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy more than 128 bytes. */
-L(copy_long):
- /* Use backwards copy if there is an overlap. */
- sub tmp1, dstin, src
- cbz tmp1, L(copy0)
- cmp tmp1, count
- b.lo L(copy_long_backwards)
-
- /* Copy 16 bytes and then align dst to 16-byte alignment. */
-
- ldp D_l, D_h, [src]
- and tmp1, dstin, 15
- bic dst, dstin, 15
- sub src, src, tmp1
- add count, count, tmp1 /* Count is now 16 too large. */
- ldp A_l, A_h, [src, 16]
- stp D_l, D_h, [dstin]
- ldp B_l, B_h, [src, 32]
- ldp C_l, C_h, [src, 48]
- ldp D_l, D_h, [src, 64]!
- subs count, count, 128 + 16 /* Test and readjust count. */
- b.ls L(copy64_from_end)
-
-L(loop64):
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [src, 16]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [src, 32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [src, 48]
- stp D_l, D_h, [dst, 64]!
- ldp D_l, D_h, [src, 64]!
- subs count, count, 64
- b.hi L(loop64)
-
- /* Write the last iteration and copy 64 bytes from the end. */
-L(copy64_from_end):
- ldp E_l, E_h, [srcend, -64]
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [srcend, -48]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [srcend, -16]
- stp D_l, D_h, [dst, 64]
- stp E_l, E_h, [dstend, -64]
- stp A_l, A_h, [dstend, -48]
- stp B_l, B_h, [dstend, -32]
- stp C_l, C_h, [dstend, -16]
- ret
-
- .p2align 4
-
- /* Large backwards copy for overlapping copies.
- Copy 16 bytes and then align dst to 16-byte alignment. */
-L(copy_long_backwards):
- ldp D_l, D_h, [srcend, -16]
- and tmp1, dstend, 15
- sub srcend, srcend, tmp1
- sub count, count, tmp1
- ldp A_l, A_h, [srcend, -16]
- stp D_l, D_h, [dstend, -16]
- ldp B_l, B_h, [srcend, -32]
- ldp C_l, C_h, [srcend, -48]
- ldp D_l, D_h, [srcend, -64]!
- sub dstend, dstend, tmp1
- subs count, count, 128
- b.ls L(copy64_from_start)
-
-L(loop64_backwards):
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [srcend, -16]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [srcend, -48]
- stp D_l, D_h, [dstend, -64]!
- ldp D_l, D_h, [srcend, -64]!
- subs count, count, 64
- b.hi L(loop64_backwards)
-
- /* Write the last iteration and copy 64 bytes from the start. */
-L(copy64_from_start):
- ldp G_l, G_h, [src, 48]
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [src, 32]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [src, 16]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [src]
- stp D_l, D_h, [dstend, -64]
- stp G_l, G_h, [dstin, 48]
- stp A_l, A_h, [dstin, 32]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstin]
- ret
-SYM_FUNC_END(__pi_memcpy_generic)
-
-#ifdef CONFIG_AS_HAS_MOPS
+ .macro cpy1 dst, src, count
.arch_extension mops
-SYM_FUNC_START(__pi_memcpy)
-alternative_if_not ARM64_HAS_MOPS
- b __pi_memcpy_generic
-alternative_else_nop_endif
+ cpyp [\dst]!, [\src]!, \count!
+ cpym [\dst]!, [\src]!, \count!
+ cpye [\dst]!, [\src]!, \count!
+ .endm
- mov dst, dstin
- cpyp [dst]!, [src]!, count!
- cpym [dst]!, [src]!, count!
- cpye [dst]!, [src]!, count!
- ret
+SYM_FUNC_START(__pi_memcpy)
+#include "memcpy_template.S"
SYM_FUNC_END(__pi_memcpy)
-#else
-SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
-#endif
SYM_FUNC_ALIAS(__memcpy, __pi_memcpy)
EXPORT_SYMBOL(__memcpy)
diff --git a/arch/arm64/lib/memcpy_mc.S b/arch/arm64/lib/memcpy_mc.S
new file mode 100644
index 000000000000..d9ce8279d91f
--- /dev/null
+++ b/arch/arm64/lib/memcpy_mc.S
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/asm-uaccess.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+ .macro ldrb1 reg, addr:vararg
+ KERNEL_SEA(9998f, ldrb \reg, \addr)
+ .endm
+
+ .macro ldr1 reg, addr:vararg
+ KERNEL_SEA(9998f, ldr \reg, \addr)
+ .endm
+
+ .macro ldp1 reg1, reg2, addr:vararg
+ KERNEL_SEA(9998f, ldp \reg1, \reg2, \addr)
+ .endm
+
+ .macro ret1
+ mov x0, #0
+ ret
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ KERNEL_SEA(9998f, cpyp [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9996f, cpym [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9996f, cpye [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__memcpy_mc)
+#include "memcpy_template.S"
+
+ // Exception fixups
+9996: b.cs 9998f
+ // Registers are in Option A format
+ add dst, dst, count
+9998: sub x0, dstend, dstin // bytes not copied
+ ret
+SYM_FUNC_END(__memcpy_mc)
+
+EXPORT_SYMBOL(__memcpy_mc)
+SYM_FUNC_ALIAS_WEAK(memcpy_mc, __memcpy_mc)
+EXPORT_SYMBOL(memcpy_mc)
diff --git a/arch/arm64/lib/memcpy_template.S b/arch/arm64/lib/memcpy_template.S
new file mode 100644
index 000000000000..a8b496f8f651
--- /dev/null
+++ b/arch/arm64/lib/memcpy_template.S
@@ -0,0 +1,250 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+#define L(label) .L ## label
+
+#define dstin x0
+#define src x1
+#define count x2
+#define dst x3
+#define srcend x4
+#define dstend x5
+#define A_l x6
+#define A_lw w6
+#define A_h x7
+#define B_l x8
+#define B_lw w8
+#define B_h x9
+#define C_l x10
+#define C_lw w10
+#define C_h x11
+#define D_l x12
+#define D_h x13
+#define E_l x14
+#define E_h x15
+#define F_l x16
+#define F_h x17
+#define G_l count
+#define G_h dst
+#define H_l src
+#define H_h srcend
+#define tmp1 x14
+
+/* This implementation handles overlaps and supports both memcpy and memmove
+ from a single entry point. It uses unaligned accesses and branchless
+ sequences to keep the code small, simple and improve performance.
+
+ Copies are split into 3 main cases: small copies of up to 32 bytes, medium
+ copies of up to 128 bytes, and large copies. The overhead of the overlap
+ check is negligible since it is only required for large copies.
+
+ Large copies use a software pipelined loop processing 64 bytes per iteration.
+ The destination pointer is 16-byte aligned to minimize unaligned accesses.
+ The loop tail is handled by always copying 64 bytes from the end.
+*/
+
+ add dstend, dstin, count
+
+#ifdef CONFIG_AS_HAS_MOPS
+alternative_if_not ARM64_HAS_MOPS
+ b L(no_mops)
+alternative_else_nop_endif
+ mov dst, dstin
+ cpy1 dst, src, count
+ ret1
+#endif
+
+L(no_mops):
+ add srcend, src, count
+ cmp count, 128
+ b.hi L(copy_long)
+ cmp count, 32
+ b.hi L(copy32_128)
+
+ /* Small copies: 0..32 bytes. */
+ cmp count, 16
+ b.lo L(copy16)
+ ldp1 A_l, A_h, [src]
+ ldp1 D_l, D_h, [srcend, -16]
+ stp A_l, A_h, [dstin]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ /* Copy 8-15 bytes. */
+L(copy16):
+ tbz count, 3, L(copy8)
+ ldr1 A_l, [src]
+ ldr1 A_h, [srcend, -8]
+ str A_l, [dstin]
+ str A_h, [dstend, -8]
+ ret1
+
+ .p2align 3
+ /* Copy 4-7 bytes. */
+L(copy8):
+ tbz count, 2, L(copy4)
+ ldr1 A_lw, [src]
+ ldr1 B_lw, [srcend, -4]
+ str A_lw, [dstin]
+ str B_lw, [dstend, -4]
+ ret1
+
+ /* Copy 0..3 bytes using a branchless sequence. */
+L(copy4):
+ cbz count, L(copy0)
+ lsr tmp1, count, 1
+ ldrb1 A_lw, [src]
+ ldrb1 C_lw, [srcend, -1]
+ ldrb1 B_lw, [src, tmp1]
+ strb A_lw, [dstin]
+ strb B_lw, [dstin, tmp1]
+ strb C_lw, [dstend, -1]
+L(copy0):
+ ret1
+
+ .p2align 4
+ /* Medium copies: 33..128 bytes. */
+L(copy32_128):
+ ldp1 A_l, A_h, [src]
+ ldp1 B_l, B_h, [src, 16]
+ ldp1 C_l, C_h, [srcend, -32]
+ ldp1 D_l, D_h, [srcend, -16]
+ cmp count, 64
+ b.hi L(copy128)
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy 65..128 bytes. */
+L(copy128):
+ ldp1 E_l, E_h, [src, 32]
+ ldp1 F_l, F_h, [src, 48]
+ cmp count, 96
+ b.ls L(copy96)
+ ldp1 G_l, G_h, [srcend, -64]
+ ldp1 H_l, H_h, [srcend, -48]
+ stp G_l, G_h, [dstend, -64]
+ stp H_l, H_h, [dstend, -48]
+L(copy96):
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp E_l, E_h, [dstin, 32]
+ stp F_l, F_h, [dstin, 48]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy more than 128 bytes. */
+L(copy_long):
+ /* Use backwards copy if there is an overlap. */
+ sub tmp1, dstin, src
+ cbz tmp1, L(copy0)
+ cmp tmp1, count
+ b.lo L(copy_long_backwards)
+
+ /* Copy 16 bytes and then align dst to 16-byte alignment. */
+
+ ldp1 D_l, D_h, [src]
+ and tmp1, dstin, 15
+ bic dst, dstin, 15
+ sub src, src, tmp1
+ add count, count, tmp1 /* Count is now 16 too large. */
+ ldp1 A_l, A_h, [src, 16]
+ stp D_l, D_h, [dstin]
+ ldp1 B_l, B_h, [src, 32]
+ ldp1 C_l, C_h, [src, 48]
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 128 + 16 /* Test and readjust count. */
+ b.ls L(copy64_from_end)
+
+L(loop64):
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [src, 16]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [src, 32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [src, 48]
+ stp D_l, D_h, [dst, 64]!
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 64
+ b.hi L(loop64)
+
+ /* Write the last iteration and copy 64 bytes from the end. */
+L(copy64_from_end):
+ ldp1 E_l, E_h, [srcend, -64]
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [srcend, -48]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [srcend, -16]
+ stp D_l, D_h, [dst, 64]
+ stp E_l, E_h, [dstend, -64]
+ stp A_l, A_h, [dstend, -48]
+ stp B_l, B_h, [dstend, -32]
+ stp C_l, C_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+
+ /* Large backwards copy for overlapping copies.
+ Copy 16 bytes and then align dst to 16-byte alignment. */
+L(copy_long_backwards):
+ ldp1 D_l, D_h, [srcend, -16]
+ and tmp1, dstend, 15
+ sub srcend, srcend, tmp1
+ sub count, count, tmp1
+ ldp1 A_l, A_h, [srcend, -16]
+ stp D_l, D_h, [dstend, -16]
+ ldp1 B_l, B_h, [srcend, -32]
+ ldp1 C_l, C_h, [srcend, -48]
+ ldp1 D_l, D_h, [srcend, -64]!
+ sub dstend, dstend, tmp1
+ subs count, count, 128
+ b.ls L(copy64_from_start)
+
+L(loop64_backwards):
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [srcend, -16]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [srcend, -48]
+ stp D_l, D_h, [dstend, -64]!
+ ldp1 D_l, D_h, [srcend, -64]!
+ subs count, count, 64
+ b.hi L(loop64_backwards)
+
+ /* Write the last iteration and copy 64 bytes from the start. */
+L(copy64_from_start):
+ ldp1 G_l, G_h, [src, 48]
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [src, 32]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [src, 16]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [src]
+ stp D_l, D_h, [dstend, -64]
+ stp G_l, G_h, [dstin, 48]
+ stp A_l, A_h, [dstin, 32]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstin]
+ ret1
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d286e0a04543..da21a13151b9 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -79,6 +79,18 @@ void *memcpy(void *dest, const void *src, size_t len)
}
#endif
+#ifdef __HAVE_ARCH_MEMCPY_MC
+#undef memcpy_mc
+unsigned long memcpy_mc(void *dest, const void *src, size_t len)
+{
+ if (!kasan_check_range(src, len, false, _RET_IP_) ||
+ !kasan_check_range(dest, len, true, _RET_IP_))
+ return len;
+
+ return __memcpy_mc(dest, src, len);
+}
+#endif
+
void *__asan_memset(void *addr, int c, ssize_t len)
{
if (!kasan_check_range(addr, len, true, _RET_IP_))
--
2.39.3
^ permalink raw reply related
* [PATCH v15 6/9] arm64: support copy_mc_[user]_highpage()
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
Currently, many scenarios that can tolerate memory errors when copying page
have been supported in the kernel[1~9], all of which are implemented by
copy_mc_[user]_highpage(). arm64 should also support this mechanism.
Due to mte, arm64 needs to have its own copy_mc_[user]_highpage()
architecture implementation, macros __HAVE_ARCH_COPY_MC_HIGHPAGE and
__HAVE_ARCH_COPY_MC_USER_HIGHPAGE have been added to control it.
Add new helper copy_mc_page() which provide a page copy implementation with
hardware memory error safe. The code logic of copy_mc_page() is the same as
copy_page(), the main difference is that the ldp insn of copy_mc_page()
contains the fixup type EX_TYPE_KACCESS_SEA, therefore, the
main logic is extracted to copy_page_template.S. In addition, the fixup of
MOPS insn is not considered at present.
[Ruidong: add FEAT_MOPS support]
[1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
[2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
[3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
[4] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
[5] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
[6] commit 658be46520ce ("mm: support poison recovery from copy_present_page()")
[7] commit aa549f923f5e ("mm: support poison recovery from do_cow_fault()")
[8] commit f00b295b9b61 ("fs: hugetlbfs: support poisoned recover from hugetlbfs_migrate_folio()")
[9] commit 060913999d7a ("mm: migrate: support poisoned recover from migrate folio")
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/asm-extable.h | 4 ++
arch/arm64/include/asm/mte.h | 9 ++++
arch/arm64/include/asm/page.h | 12 +++++
arch/arm64/lib/Makefile | 2 +
arch/arm64/lib/copy_mc_page.S | 44 +++++++++++++++
arch/arm64/lib/copy_page.S | 67 ++++-------------------
arch/arm64/lib/copy_page_template.S | 70 ++++++++++++++++++++++++
arch/arm64/lib/mte.S | 29 ++++++++++
arch/arm64/mm/copypage.c | 80 ++++++++++++++++++++++++++++
include/linux/highmem.h | 8 +++
11 files changed, 270 insertions(+), 56 deletions(-)
create mode 100644 arch/arm64/lib/copy_mc_page.S
create mode 100644 arch/arm64/lib/copy_page_template.S
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..831b20d45893 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -21,6 +21,7 @@ config ARM64
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CC_PLATFORM
select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
+ select ARCH_HAS_COPY_MC if ACPI_APEI_GHES
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
index 8450ec5a3af6..9305ea77482a 100644
--- a/arch/arm64/include/asm/asm-extable.h
+++ b/arch/arm64/include/asm/asm-extable.h
@@ -10,6 +10,10 @@
#define EX_TYPE_ACCESS_ERR_ZERO 2
#define EX_TYPE_UACCESS_CPY 3
#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4
+/*
+ * Kernel access: used in kernel context for both regular load/store
+ * instructions and MOPS (memory copy/set) instructions.
+ */
#define EX_TYPE_KACCESS_SEA 5
/* Data fields for EX_TYPE_ACCESS_ERR_ZERO */
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 7f7b97e09996..a0b1757f4847 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -98,6 +98,11 @@ static inline bool try_page_mte_tagging(struct page *page)
void mte_zero_clear_page_tags(void *addr);
void mte_sync_tags(pte_t pte, unsigned int nr_pages);
void mte_copy_page_tags(void *kto, const void *kfrom);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int mte_copy_mc_page_tags(void *kto, const void *kfrom);
+#endif
+
void mte_thread_init_user(void);
void mte_thread_switch(struct task_struct *next);
void mte_cpu_setup(void);
@@ -134,6 +139,10 @@ static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages)
static inline void mte_copy_page_tags(void *kto, const void *kfrom)
{
}
+static inline int mte_copy_mc_page_tags(void *kto, const void *kfrom)
+{
+ return 0;
+}
static inline void mte_thread_init_user(void)
{
}
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index e25d0d18f6d7..5c4c9f974b68 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -29,6 +29,18 @@ void copy_user_highpage(struct page *to, struct page *from,
void copy_highpage(struct page *to, struct page *from);
#define __HAVE_ARCH_COPY_HIGHPAGE
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int copy_mc_page(void *to, const void *from);
+#define __HAVE_ARCH_COPY_MC_PAGE
+
+int copy_mc_highpage(struct page *to, struct page *from);
+#define __HAVE_ARCH_COPY_MC_HIGHPAGE
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma);
+#define __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
+#endif
+
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr);
#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 448c917494f3..1f4c3f743a20 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,6 +7,8 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
obj-$(CONFIG_ARM64_MTE) += mte.o
diff --git a/arch/arm64/lib/copy_mc_page.S b/arch/arm64/lib/copy_mc_page.S
new file mode 100644
index 000000000000..f936e0c98611
--- /dev/null
+++ b/arch/arm64/lib/copy_mc_page.S
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#include <linux/linkage.h>
+#include <linux/const.h>
+#include <asm/assembler.h>
+#include <asm/page.h>
+#include <asm/cpufeature.h>
+#include <asm/alternative.h>
+#include <asm/asm-extable.h>
+#include <asm/asm-uaccess.h>
+
+/*
+ * Copy a page from src to dest (both are page aligned) with memory error safe
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ * Returns:
+ * x0 - Return 0 if copy success, or -EFAULT if anything goes wrong
+ * while copying.
+ */
+ .macro ldp1 reg1, reg2, ptr, val
+ KERNEL_SEA(9998f, ldp \reg1, \reg2, [\ptr, \val])
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ KERNEL_SEA(9998f, cpypwn [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9998f, cpymwn [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9998f, cpyewn [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__pi_copy_mc_page)
+#include "copy_page_template.S"
+
+ mov x0, #0
+ ret
+
+9998: mov x0, #-EFAULT
+ ret
+
+SYM_FUNC_END(__pi_copy_mc_page)
+SYM_FUNC_ALIAS(copy_mc_page, __pi_copy_mc_page)
+EXPORT_SYMBOL(copy_mc_page)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index e6374e7e5511..e520777b5150 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -17,65 +17,20 @@
* x0 - dest
* x1 - src
*/
-SYM_FUNC_START(__pi_copy_page)
-#ifdef CONFIG_AS_HAS_MOPS
- .arch_extension mops
-alternative_if_not ARM64_HAS_MOPS
- b .Lno_mops
-alternative_else_nop_endif
-
- mov x2, #PAGE_SIZE
- cpypwn [x0]!, [x1]!, x2!
- cpymwn [x0]!, [x1]!, x2!
- cpyewn [x0]!, [x1]!, x2!
- ret
-.Lno_mops:
-#endif
- ldp x2, x3, [x1]
- ldp x4, x5, [x1, #16]
- ldp x6, x7, [x1, #32]
- ldp x8, x9, [x1, #48]
- ldp x10, x11, [x1, #64]
- ldp x12, x13, [x1, #80]
- ldp x14, x15, [x1, #96]
- ldp x16, x17, [x1, #112]
-
- add x0, x0, #256
- add x1, x1, #128
-1:
- tst x0, #(PAGE_SIZE - 1)
- stnp x2, x3, [x0, #-256]
- ldp x2, x3, [x1]
- stnp x4, x5, [x0, #16 - 256]
- ldp x4, x5, [x1, #16]
- stnp x6, x7, [x0, #32 - 256]
- ldp x6, x7, [x1, #32]
- stnp x8, x9, [x0, #48 - 256]
- ldp x8, x9, [x1, #48]
- stnp x10, x11, [x0, #64 - 256]
- ldp x10, x11, [x1, #64]
- stnp x12, x13, [x0, #80 - 256]
- ldp x12, x13, [x1, #80]
- stnp x14, x15, [x0, #96 - 256]
- ldp x14, x15, [x1, #96]
- stnp x16, x17, [x0, #112 - 256]
- ldp x16, x17, [x1, #112]
+ .macro ldp1 reg1, reg2, ptr, val
+ ldp \reg1, \reg2, [\ptr, \val]
+ .endm
- add x0, x0, #128
- add x1, x1, #128
-
- b.ne 1b
-
- stnp x2, x3, [x0, #-256]
- stnp x4, x5, [x0, #16 - 256]
- stnp x6, x7, [x0, #32 - 256]
- stnp x8, x9, [x0, #48 - 256]
- stnp x10, x11, [x0, #64 - 256]
- stnp x12, x13, [x0, #80 - 256]
- stnp x14, x15, [x0, #96 - 256]
- stnp x16, x17, [x0, #112 - 256]
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ cpypwn [\dst]!, [\src]!, \count!
+ cpymwn [\dst]!, [\src]!, \count!
+ cpyewn [\dst]!, [\src]!, \count!
+ .endm
+SYM_FUNC_START(__pi_copy_page)
+#include "copy_page_template.S"
ret
SYM_FUNC_END(__pi_copy_page)
SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
diff --git a/arch/arm64/lib/copy_page_template.S b/arch/arm64/lib/copy_page_template.S
new file mode 100644
index 000000000000..e5afbeaaad25
--- /dev/null
+++ b/arch/arm64/lib/copy_page_template.S
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ */
+
+/*
+ * Copy a page from src to dest (both are page aligned)
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ */
+dstin .req x0
+src .req x1
+
+#ifdef CONFIG_AS_HAS_MOPS
+alternative_if_not ARM64_HAS_MOPS
+ b .Lno_mops
+alternative_else_nop_endif
+ mov x2, #PAGE_SIZE
+ cpy1 dst, src, x2
+ b .Lexitfunc
+.Lno_mops:
+#endif
+
+ ldp1 x2, x3, x1, #0
+ ldp1 x4, x5, x1, #16
+ ldp1 x6, x7, x1, #32
+ ldp1 x8, x9, x1, #48
+ ldp1 x10, x11, x1, #64
+ ldp1 x12, x13, x1, #80
+ ldp1 x14, x15, x1, #96
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #256
+ add x1, x1, #128
+1:
+ tst x0, #(PAGE_SIZE - 1)
+
+ stnp x2, x3, [x0, #-256]
+ ldp1 x2, x3, x1, #0
+ stnp x4, x5, [x0, #16 - 256]
+ ldp1 x4, x5, x1, #16
+ stnp x6, x7, [x0, #32 - 256]
+ ldp1 x6, x7, x1, #32
+ stnp x8, x9, [x0, #48 - 256]
+ ldp1 x8, x9, x1, #48
+ stnp x10, x11, [x0, #64 - 256]
+ ldp1 x10, x11, x1, #64
+ stnp x12, x13, [x0, #80 - 256]
+ ldp1 x12, x13, x1, #80
+ stnp x14, x15, [x0, #96 - 256]
+ ldp1 x14, x15, x1, #96
+ stnp x16, x17, [x0, #112 - 256]
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #128
+ add x1, x1, #128
+
+ b.ne 1b
+
+ stnp x2, x3, [x0, #-256]
+ stnp x4, x5, [x0, #16 - 256]
+ stnp x6, x7, [x0, #32 - 256]
+ stnp x8, x9, [x0, #48 - 256]
+ stnp x10, x11, [x0, #64 - 256]
+ stnp x12, x13, [x0, #80 - 256]
+ stnp x14, x15, [x0, #96 - 256]
+ stnp x16, x17, [x0, #112 - 256]
+.Lexitfunc:
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 5018ac03b6bf..1afe3ef1502c 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -80,6 +80,35 @@ SYM_FUNC_START(mte_copy_page_tags)
ret
SYM_FUNC_END(mte_copy_page_tags)
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Copy the tags from the source page to the destination one with memory error safe
+ * x0 - address of the destination page
+ * x1 - address of the source page
+ * Returns:
+ * x0 - Return 0 if copy success, or
+ * -EFAULT if anything goes wrong while copying.
+ */
+SYM_FUNC_START(mte_copy_mc_page_tags)
+ mov x2, x0
+ mov x3, x1
+ multitag_transfer_size x5, x6
+1:
+KERNEL_SEA(2f, ldgm x4, [x3])
+ stgm x4, [x2]
+ add x2, x2, x5
+ add x3, x3, x5
+ tst x2, #(PAGE_SIZE - 1)
+ b.ne 1b
+
+ mov x0, #0
+ ret
+
+2: mov x0, #-EFAULT
+ ret
+SYM_FUNC_END(mte_copy_mc_page_tags)
+#endif
+
/*
* Read tags from a user buffer (one tag per byte) and set the corresponding
* tags at the given kernel address. Used by PTRACE_POKEMTETAGS.
diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c
index cd5912ba617b..c22918ed0f3c 100644
--- a/arch/arm64/mm/copypage.c
+++ b/arch/arm64/mm/copypage.c
@@ -72,3 +72,83 @@ void copy_user_highpage(struct page *to, struct page *from,
flush_dcache_page(to);
}
EXPORT_SYMBOL_GPL(copy_user_highpage);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Return -EFAULT if anything goes wrong while copying page or mte.
+ */
+int copy_mc_highpage(struct page *to, struct page *from)
+{
+ void *kto = page_address(to);
+ void *kfrom = page_address(from);
+ struct folio *src = page_folio(from);
+ struct folio *dst = page_folio(to);
+ unsigned int i, nr_pages;
+ int ret;
+
+ ret = copy_mc_page(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+
+ if (kasan_hw_tags_enabled())
+ page_kasan_tag_reset(to);
+
+ if (!system_supports_mte())
+ return 0;
+
+ if (folio_test_hugetlb(src)) {
+ if (!folio_test_hugetlb_mte_tagged(src) ||
+ from != folio_page(src, 0))
+ return 0;
+
+ WARN_ON_ONCE(!folio_try_hugetlb_mte_tagging(dst));
+
+ /*
+ * Populate tags for all subpages.
+ *
+ * Don't assume the first page is head page since
+ * huge page copy may start from any subpage.
+ */
+ nr_pages = folio_nr_pages(src);
+ for (i = 0; i < nr_pages; i++) {
+ kfrom = page_address(folio_page(src, i));
+ kto = page_address(folio_page(dst, i));
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ }
+ folio_set_hugetlb_mte_tagged(dst);
+ } else if (page_mte_tagged(from)) {
+ /* It's a new page, shouldn't have been tagged yet */
+ WARN_ON_ONCE(!try_page_mte_tagging(to));
+
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ set_page_mte_tagged(to);
+ }
+ /*
+ * memory_failure_queue() is not called here because on arm64
+ * the firmware (GHES) has already reported the hardware memory
+ * error and queued the page for memory_failure() handling via
+ * ghes_do_memory_failure().
+ */
+ return 0;
+}
+EXPORT_SYMBOL(copy_mc_highpage);
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma)
+{
+ int ret;
+
+ ret = copy_mc_highpage(to, from);
+ if (ret)
+ return ret;
+
+ flush_dcache_page(to);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(copy_mc_user_highpage);
+#endif
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 18dc4aca4aa1..f168c9d4ad0e 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -424,6 +424,7 @@ static inline void copy_highpage(struct page *to, struct page *from)
#endif
#ifdef copy_mc_to_kernel
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
@@ -449,7 +450,9 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
return ret ? -EFAULT : 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
unsigned long ret;
@@ -468,20 +471,25 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
return ret ? -EFAULT : 0;
}
+#endif
#else
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
{
copy_user_highpage(to, from, vaddr, vma);
return 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
copy_highpage(to, from);
return 0;
}
#endif
+#endif
static inline void memcpy_page(struct page *dst_page, size_t dst_off,
struct page *src_page, size_t src_off,
--
2.39.3
^ permalink raw reply related
* [PATCH v15 5/9] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage()
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong, Jonathan Cameron, Mauro Carvalho Chehab
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
Currently, copy_mc_[user]_highpage() returns zero on success, or in case
of failures, the number of bytes that weren't copied.
While tracking the number of not copied works fine for x86 and PPC, There
are some difficulties in doing the same thing on ARM64 because there is no
available caller-saved register in copy_page()(lib/copy_page.S) to save
"bytes not copied", and the following copy_mc_page() will also encounter
the same problem.
Consider the caller of copy_mc_[user]_highpage() cannot do any processing
on the remaining data(The page has hardware errors), they only check if
copy was succeeded or not, make the interface more generic by using an
error code when copy fails (-EFAULT) or return zero on success.
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Shuai Xue <xueshuai@linux.alibaba.com>
---
include/linux/highmem.h | 8 ++++----
mm/khugepaged.c | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index af03db851a1d..18dc4aca4aa1 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -427,8 +427,8 @@ static inline void copy_highpage(struct page *to, struct page *from)
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
- * page with #MC in source page (@from) handled, and return the number
- * of bytes not copied if there was a #MC, otherwise 0 for success.
+ * page with #MC in source page (@from) handled, and return -EFAULT if there
+ * was a #MC, otherwise 0 for success.
*/
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
@@ -447,7 +447,7 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
static inline int copy_mc_highpage(struct page *to, struct page *from)
@@ -466,7 +466,7 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
#else
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b8452dbdb043..cf1b78eed3c3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -810,7 +810,7 @@ static enum scan_result __collapse_huge_page_copy(pte_t *pte, struct folio *foli
continue;
}
src_page = pte_page(pteval);
- if (copy_mc_user_highpage(page, src_page, src_addr, vma) > 0) {
+ if (copy_mc_user_highpage(page, src_page, src_addr, vma)) {
result = SCAN_COPY_MC;
break;
}
@@ -2143,7 +2143,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
}
for (i = 0; i < nr_pages; i++) {
- if (copy_mc_highpage(dst, folio_page(folio, i)) > 0) {
+ if (copy_mc_highpage(dst, folio_page(folio, i))) {
result = SCAN_COPY_MC;
goto rollback;
}
--
2.39.3
^ permalink raw reply related
* [PATCH v15 4/9] arm64: enable recover from synchronous external abort in kernel context
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
For the arm64 kernel, when it processes hardware memory errors for
synchronize notifications(do_sea()), if the errors is consumed within the
kernel, the current processing is panic. However, it is not optimal.
Take copy_from/to_user for example, If ld* triggers a memory error, even in
kernel mode, only the associated process is affected. Killing the user
process and isolating the corrupt page is a better choice.
Add new fixup type EX_TYPE_KACCESS_SEA to identify insn that can recover
from memory errors triggered by access to kernel memory, and this fixup
type is used in __arch_copy_to_user(), This make the regular copy_to_user()
will handle kernel memory errors.
[Ruidong: modify subject and rename EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR to
EX_TYPE_KACCESS_SEA]
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/include/asm/asm-extable.h | 5 +++++
arch/arm64/include/asm/asm-uaccess.h | 4 ++++
arch/arm64/include/asm/extable.h | 1 +
arch/arm64/lib/copy_to_user.S | 10 +++++-----
arch/arm64/mm/extable.c | 28 ++++++++++++++++++++++++++
arch/arm64/mm/fault.c | 30 ++++++++++++++++++++--------
6 files changed, 65 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
index 06b19023939b..8450ec5a3af6 100644
--- a/arch/arm64/include/asm/asm-extable.h
+++ b/arch/arm64/include/asm/asm-extable.h
@@ -10,6 +10,7 @@
#define EX_TYPE_ACCESS_ERR_ZERO 2
#define EX_TYPE_UACCESS_CPY 3
#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4
+#define EX_TYPE_KACCESS_SEA 5
/* Data fields for EX_TYPE_ACCESS_ERR_ZERO */
#define EX_DATA_REG_ERR_SHIFT 0
@@ -76,6 +77,10 @@
__ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_UACCESS_CPY, \uaccess_is_write)
.endm
+ .macro _asm_extable_kaccess_sea, insn, fixup
+ __ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_KACCESS_SEA, 0)
+ .endm
+
#else /* __ASSEMBLER__ */
#include <linux/stringify.h>
diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h
index 12aa6a283249..27bf8edbf597 100644
--- a/arch/arm64/include/asm/asm-uaccess.h
+++ b/arch/arm64/include/asm/asm-uaccess.h
@@ -57,6 +57,10 @@ alternative_else_nop_endif
.endm
#endif
+#define KERNEL_SEA(l, x...) \
+9999: x; \
+ _asm_extable_kaccess_sea 9999b, l
+
#define USER(l, x...) \
9999: x; \
_asm_extable_uaccess 9999b, l
diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/asm/extable.h
index 9dc39612bdf5..47c851d7df4f 100644
--- a/arch/arm64/include/asm/extable.h
+++ b/arch/arm64/include/asm/extable.h
@@ -48,4 +48,5 @@ bool ex_handler_bpf(const struct exception_table_entry *ex,
#endif /* !CONFIG_BPF_JIT */
bool fixup_exception(struct pt_regs *regs, unsigned long esr);
+bool fixup_exception_me(struct pt_regs *regs);
#endif
diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S
index 819f2e3fc7a9..6103f5b0a2d0 100644
--- a/arch/arm64/lib/copy_to_user.S
+++ b/arch/arm64/lib/copy_to_user.S
@@ -20,7 +20,7 @@
* x0 - bytes not copied
*/
.macro ldrb1 reg, ptr, val
- ldrb \reg, [\ptr], \val
+ KERNEL_SEA(9998f, ldrb \reg, [\ptr], \val)
.endm
.macro strb1 reg, ptr, val
@@ -28,7 +28,7 @@
.endm
.macro ldrh1 reg, ptr, val
- ldrh \reg, [\ptr], \val
+ KERNEL_SEA(9998f, ldrh \reg, [\ptr], \val)
.endm
.macro strh1 reg, ptr, val
@@ -36,7 +36,7 @@
.endm
.macro ldr1 reg, ptr, val
- ldr \reg, [\ptr], \val
+ KERNEL_SEA(9998f, ldr \reg, [\ptr], \val)
.endm
.macro str1 reg, ptr, val
@@ -44,7 +44,7 @@
.endm
.macro ldp1 reg1, reg2, ptr, val
- ldp \reg1, \reg2, [\ptr], \val
+ KERNEL_SEA(9998f, ldp \reg1, \reg2, [\ptr], \val)
.endm
.macro stp1 reg1, reg2, ptr, val
@@ -74,7 +74,7 @@ SYM_FUNC_START(__arch_copy_to_user)
9997: cmp dst, dstin
b.ne 9998f
// Before being absolutely sure we couldn't copy anything, try harder
- ldrb tmp1w, [srcin]
+KERNEL_SEA(9998f, ldrb tmp1w, [srcin])
USER(9998f, sttrb tmp1w, [dst])
add dst, dst, #1
9998: sub x0, end, dst // bytes not copied
diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c
index 76b18780f1f9..20a7a9eeed94 100644
--- a/arch/arm64/mm/extable.c
+++ b/arch/arm64/mm/extable.c
@@ -109,7 +109,35 @@ bool fixup_exception(struct pt_regs *regs, unsigned long esr)
return ex_handler_uaccess_cpy(ex, regs, esr);
case EX_TYPE_LOAD_UNALIGNED_ZEROPAD:
return ex_handler_load_unaligned_zeropad(ex, regs);
+ /*
+ * Kernel address faults (e.g. copy_to_user reading from kernel src).
+ * Do not fixup here: a translation fault on a kernel address is a
+ * kernel bug (e.g. NULL pointer dereference) and must oops.
+ * Only SEA (hardware memory error) should be fixed up, which is
+ * handled by fixup_exception_me() through the do_sea path.
+ */
+ case EX_TYPE_KACCESS_SEA:
+ return false;
}
BUG();
}
+
+bool fixup_exception_me(struct pt_regs *regs)
+{
+ const struct exception_table_entry *ex;
+
+ ex = search_exception_tables(instruction_pointer(regs));
+ if (!ex)
+ return false;
+
+ switch (ex->type) {
+ case EX_TYPE_ACCESS_ERR_ZERO:
+ return ex_handler_access_err_zero(ex, regs);
+ case EX_TYPE_KACCESS_SEA:
+ regs->pc = get_ex_fixup(ex);
+ return true;
+ }
+
+ return false;
+}
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0f3c5c7ca054..b775c0928a53 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -858,21 +858,35 @@ static int do_bad(unsigned long far, unsigned long esr, struct pt_regs *regs)
return 1; /* "fault" */
}
+/*
+ * APEI claimed this as a firmware-first notification.
+ * Some processing deferred to task_work before ret_to_user().
+ */
+static int do_apei_claim_sea(struct pt_regs *regs)
+{
+ int ret;
+
+ ret = apei_claim_sea(regs);
+ if (ret)
+ return ret;
+
+ if (!user_mode(regs)) {
+ if (!fixup_exception_me(regs))
+ return -ENOENT;
+ }
+
+ return ret;
+}
+
static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *regs)
{
const struct fault_info *inf;
unsigned long siaddr;
- inf = esr_to_fault_info(esr);
-
- if (user_mode(regs) && apei_claim_sea(regs) == 0) {
- /*
- * APEI claimed this as a firmware-first notification.
- * Some processing deferred to task_work before ret_to_user().
- */
+ if (do_apei_claim_sea(regs) == 0)
return 0;
- }
+ inf = esr_to_fault_info(esr);
if (esr & ESR_ELx_FnV) {
siaddr = 0;
} else {
--
2.39.3
^ permalink raw reply related
* [PATCH v15 3/9] arm64: extable: merge UACCESS_ERR_ZERO and KACCESS_ERR_ZERO into ACCESS_ERR_ZERO
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
EX_TYPE_UACCESS_ERR_ZERO and EX_TYPE_KACCESS_ERR_ZERO have identical
handling in fixup_exception(): both unconditionally invoke
ex_handler_uaccess_err_zero() to set the error register to -EFAULT,
zero the destination register, and branch to the fixup address.
Merge them into a single EX_TYPE_ACCESS_ERR_ZERO to reduce redundancy
and renumber the subsequent types accordingly.
The _ASM_EXTABLE_UACCESS_ERR_ZERO and _ASM_EXTABLE_KACCESS_ERR_ZERO
helper macros are preserved as-is for caller readability, but both now
emit the unified EX_TYPE_ACCESS_ERR_ZERO type.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/include/asm/asm-extable.h | 15 +++++++--------
arch/arm64/mm/extable.c | 7 +++----
2 files changed, 10 insertions(+), 12 deletions(-)
diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
index d67e2fdd1aee..06b19023939b 100644
--- a/arch/arm64/include/asm/asm-extable.h
+++ b/arch/arm64/include/asm/asm-extable.h
@@ -7,12 +7,11 @@
#define EX_TYPE_NONE 0
#define EX_TYPE_BPF 1
-#define EX_TYPE_UACCESS_ERR_ZERO 2
-#define EX_TYPE_KACCESS_ERR_ZERO 3
-#define EX_TYPE_UACCESS_CPY 4
-#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 5
+#define EX_TYPE_ACCESS_ERR_ZERO 2
+#define EX_TYPE_UACCESS_CPY 3
+#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4
-/* Data fields for EX_TYPE_UACCESS_ERR_ZERO */
+/* Data fields for EX_TYPE_ACCESS_ERR_ZERO */
#define EX_DATA_REG_ERR_SHIFT 0
#define EX_DATA_REG_ERR GENMASK(4, 0)
#define EX_DATA_REG_ZERO_SHIFT 5
@@ -43,7 +42,7 @@
#define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
__ASM_EXTABLE_RAW(insn, fixup, \
- EX_TYPE_UACCESS_ERR_ZERO, \
+ EX_TYPE_ACCESS_ERR_ZERO, \
( \
EX_DATA_REG(ERR, err) | \
EX_DATA_REG(ZERO, zero) \
@@ -96,7 +95,7 @@
#define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
__DEFINE_ASM_GPR_NUMS \
__ASM_EXTABLE_RAW(#insn, #fixup, \
- __stringify(EX_TYPE_UACCESS_ERR_ZERO), \
+ __stringify(EX_TYPE_ACCESS_ERR_ZERO), \
"(" \
EX_DATA_REG(ERR, err) " | " \
EX_DATA_REG(ZERO, zero) \
@@ -105,7 +104,7 @@
#define _ASM_EXTABLE_KACCESS_ERR_ZERO(insn, fixup, err, zero) \
__DEFINE_ASM_GPR_NUMS \
__ASM_EXTABLE_RAW(#insn, #fixup, \
- __stringify(EX_TYPE_KACCESS_ERR_ZERO), \
+ __stringify(EX_TYPE_ACCESS_ERR_ZERO), \
"(" \
EX_DATA_REG(ERR, err) " | " \
EX_DATA_REG(ZERO, zero) \
diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c
index 6e0528831cd3..76b18780f1f9 100644
--- a/arch/arm64/mm/extable.c
+++ b/arch/arm64/mm/extable.c
@@ -41,7 +41,7 @@ get_ex_fixup(const struct exception_table_entry *ex)
return ((unsigned long)&ex->fixup + ex->fixup);
}
-static bool ex_handler_uaccess_err_zero(const struct exception_table_entry *ex,
+static bool ex_handler_access_err_zero(const struct exception_table_entry *ex,
struct pt_regs *regs)
{
int reg_err = FIELD_GET(EX_DATA_REG_ERR, ex->data);
@@ -103,9 +103,8 @@ bool fixup_exception(struct pt_regs *regs, unsigned long esr)
switch (ex->type) {
case EX_TYPE_BPF:
return ex_handler_bpf(ex, regs);
- case EX_TYPE_UACCESS_ERR_ZERO:
- case EX_TYPE_KACCESS_ERR_ZERO:
- return ex_handler_uaccess_err_zero(ex, regs);
+ case EX_TYPE_ACCESS_ERR_ZERO:
+ return ex_handler_access_err_zero(ex, regs);
case EX_TYPE_UACCESS_CPY:
return ex_handler_uaccess_cpy(ex, regs, esr);
case EX_TYPE_LOAD_UNALIGNED_ZEROPAD:
--
2.39.3
^ permalink raw reply related
* [PATCH v15 2/9] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
When a GHES SEA (Synchronous External Abort) fires while the CPU
was executing in kernel mode, it typically means that kernel code
itself consumed a poisoned memory location -- e.g. copy_from_user()
/ copy_to_user() invoked from a ioctl() or write() syscall touched
a poisoned user page or page-cache page on behalf of the task.
The expected behaviour in that case is that the faulting kernel
helper returns via its extable fixup and the syscall returns an
error (e.g. -EFAULT) to user space. It is NOT appropriate to deliver
SIGBUS to the current task: the task did not directly dereference
the poisoned address, the kernel did on its behalf, and the kernel
is able to recover.
Up to now ghes_handle_memory_failure() unconditionally promoted any
synchronous recoverable memory error to MF_ACTION_REQUIRED, which
ends up SIGBUS on current -- regardless of whether the poison was
consumed from user space or from inside the kernel on the task's
behalf. That kills tasks that should instead have seen a plain
syscall error.
To fix this, the execution mode in which the exception was taken
must be captured at the arch-level entry point, where pt_regs (and
hence user_mode(regs)) are still available. The estatus node that
later drains the error in IRQ / process context no longer has
access to the original regs.
Introduce:
enum context { ... };
and plumb the value all the way down to the queued estatus node:
* Add an 'enum context context' field to struct ghes_estatus_node
and record it in ghes_in_nmi_queue_one_entry().
* Extend ghes_notify_sea() and the internal
ghes_in_nmi_spool_from_list() with an enum context parameter.
Then consume the recorded context in ghes_handle_memory_failure()
for the GHES_SEV_RECOVERABLE / sync path:
flags = sync && context == GHES_CTX_USER ? MF_ACTION_REQUIRED : 0;
i.e. MF_ACTION_REQUIRED (and thus SIGBUS via the task_work path) is
only raised for user-mode poison consumption. Synchronous errors
taken in kernel mode fall back to memory_failure_queue() with
flags=0, asynchronously isolating the poisoned page while letting
the faulting kernel helper's extable fixup return -EFAULT
to user space.
Paths that pass NO_USE are unaffected:
sync is false for them, so flags stays 0 as before.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/kernel/acpi.c | 2 +-
drivers/acpi/apei/ghes.c | 36 ++++++++++++++++++++----------------
include/acpi/ghes.h | 15 +++++++++++++--
3 files changed, 34 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 5891f92c2035..fa74f32c6e8c 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -409,7 +409,7 @@ int apei_claim_sea(struct pt_regs *regs)
*/
local_daif_restore(DAIF_ERRCTX);
nmi_enter();
- err = ghes_notify_sea();
+ err = ghes_notify_sea(GHES_CTX(regs));
nmi_exit();
/*
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index 3236a3ce79d6..2c39adfb584a 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -529,7 +529,7 @@ static bool ghes_do_memory_failure(u64 physical_addr, int flags)
}
static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
- int sev, bool sync)
+ int sev, bool sync, enum ghes_exec_ctx context)
{
int flags = -1;
int sec_sev = ghes_severity(gdata->error_severity);
@@ -543,7 +543,7 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
(gdata->flags & CPER_SEC_ERROR_THRESHOLD_EXCEEDED))
flags = MF_SOFT_OFFLINE;
if (sev == GHES_SEV_RECOVERABLE && sec_sev == GHES_SEV_RECOVERABLE)
- flags = sync ? MF_ACTION_REQUIRED : 0;
+ flags = sync && context == GHES_CTX_USER ? MF_ACTION_REQUIRED : 0;
if (flags != -1)
return ghes_do_memory_failure(mem_err->physical_addr, flags);
@@ -552,10 +552,10 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
}
static bool ghes_handle_arm_hw_error(struct acpi_hest_generic_data *gdata,
- int sev, bool sync)
+ int sev, bool sync, enum ghes_exec_ctx context)
{
struct cper_sec_proc_arm *err = acpi_hest_get_payload(gdata);
- int flags = sync ? MF_ACTION_REQUIRED : 0;
+ int flags = sync && context == GHES_CTX_USER ? MF_ACTION_REQUIRED : 0;
int length = gdata->error_data_length;
char error_type[120];
bool queued = false;
@@ -910,7 +910,8 @@ static void ghes_log_hwerr(int sev, guid_t *sec_type)
}
static void ghes_do_proc(struct ghes *ghes,
- const struct acpi_hest_generic_status *estatus)
+ const struct acpi_hest_generic_status *estatus,
+ enum ghes_exec_ctx context)
{
int sev, sec_sev;
struct acpi_hest_generic_data *gdata;
@@ -937,11 +938,11 @@ static void ghes_do_proc(struct ghes *ghes,
atomic_notifier_call_chain(&ghes_report_chain, sev, mem_err);
arch_apei_report_mem_error(sev, mem_err);
- queued = ghes_handle_memory_failure(gdata, sev, sync);
+ queued = ghes_handle_memory_failure(gdata, sev, sync, context);
} else if (guid_equal(sec_type, &CPER_SEC_PCIE)) {
ghes_handle_aer(gdata);
} else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) {
- queued = ghes_handle_arm_hw_error(gdata, sev, sync);
+ queued = ghes_handle_arm_hw_error(gdata, sev, sync, context);
} else if (guid_equal(sec_type, &CPER_SEC_CXL_PROT_ERR)) {
struct cxl_cper_sec_prot_err *prot_err = acpi_hest_get_payload(gdata);
@@ -1190,7 +1191,7 @@ static int ghes_proc(struct ghes *ghes)
if (ghes_print_estatus(NULL, ghes->generic, estatus))
ghes_estatus_cache_add(ghes->generic, estatus);
}
- ghes_do_proc(ghes, estatus);
+ ghes_do_proc(ghes, estatus, GHES_CTX_NA);
out:
ghes_clear_estatus(ghes, estatus, buf_paddr, FIX_APEI_GHES_IRQ);
@@ -1297,7 +1298,7 @@ static void ghes_proc_in_irq(struct irq_work *irq_work)
len = cper_estatus_len(estatus);
node_len = GHES_ESTATUS_NODE_LEN(len);
- ghes_do_proc(estatus_node->ghes, estatus);
+ ghes_do_proc(estatus_node->ghes, estatus, estatus_node->context);
if (!ghes_estatus_cached(estatus)) {
generic = estatus_node->generic;
@@ -1335,7 +1336,8 @@ static void ghes_print_queued_estatus(void)
}
static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
- enum fixed_addresses fixmap_idx)
+ enum fixed_addresses fixmap_idx,
+ enum ghes_exec_ctx context)
{
struct acpi_hest_generic_status *estatus, tmp_header;
struct ghes_estatus_node *estatus_node;
@@ -1364,6 +1366,7 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
if (!estatus_node)
return -ENOMEM;
+ estatus_node->context = context;
estatus_node->ghes = ghes;
estatus_node->generic = ghes->generic;
estatus = GHES_ESTATUS_FROM_NODE(estatus_node);
@@ -1398,14 +1401,15 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
}
static int ghes_in_nmi_spool_from_list(struct list_head *rcu_list,
- enum fixed_addresses fixmap_idx)
+ enum fixed_addresses fixmap_idx,
+ enum ghes_exec_ctx context)
{
int ret = -ENOENT;
struct ghes *ghes;
rcu_read_lock();
list_for_each_entry_rcu(ghes, rcu_list, list) {
- if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx))
+ if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, context))
ret = 0;
}
rcu_read_unlock();
@@ -1488,7 +1492,7 @@ static LIST_HEAD(ghes_sea);
* Return 0 only if one of the SEA error sources successfully reported an error
* record sent from the firmware.
*/
-int ghes_notify_sea(void)
+int ghes_notify_sea(enum ghes_exec_ctx context)
{
static DEFINE_RAW_SPINLOCK(ghes_notify_lock_sea);
int rv;
@@ -1497,7 +1501,7 @@ int ghes_notify_sea(void)
return -ENOENT;
raw_spin_lock(&ghes_notify_lock_sea);
- rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA);
+ rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA, context);
raw_spin_unlock(&ghes_notify_lock_sea);
return rv;
@@ -1552,7 +1556,7 @@ static int ghes_notify_nmi(unsigned int cmd, struct pt_regs *regs)
return ret;
raw_spin_lock(&ghes_notify_lock_nmi);
- if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI))
+ if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI, GHES_CTX_NA))
ret = NMI_HANDLED;
raw_spin_unlock(&ghes_notify_lock_nmi);
@@ -1606,7 +1610,7 @@ static void ghes_nmi_init_cxt(void)
static int __ghes_sdei_callback(struct ghes *ghes,
enum fixed_addresses fixmap_idx)
{
- if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx)) {
+ if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, GHES_CTX_NA)) {
irq_work_queue(&ghes_proc_irq_work);
return 0;
diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h
index 8d7e5caef3f1..8460707ea4b0 100644
--- a/include/acpi/ghes.h
+++ b/include/acpi/ghes.h
@@ -33,10 +33,21 @@ struct ghes {
void __iomem *error_status_vaddr;
};
+enum ghes_exec_ctx {
+ GHES_CTX_NA = -1,
+ GHES_CTX_KERNEL = 0,
+ GHES_CTX_USER = 1
+};
+
+#define GHES_CTX(regs) ((regs) ? (user_mode(regs) ? GHES_CTX_USER \
+ : GHES_CTX_KERNEL) \
+ : GHES_CTX_NA)
+
struct ghes_estatus_node {
struct llist_node llnode;
struct acpi_hest_generic *generic;
struct ghes *ghes;
+ enum ghes_exec_ctx context;
};
struct ghes_estatus_cache {
@@ -135,9 +146,9 @@ static inline void *acpi_hest_get_next(struct acpi_hest_generic_data *gdata)
section = acpi_hest_get_next(section))
#ifdef CONFIG_ACPI_APEI_SEA
-int ghes_notify_sea(void);
+int ghes_notify_sea(enum ghes_exec_ctx context);
#else
-static inline int ghes_notify_sea(void) { return -ENOENT; }
+static inline int ghes_notify_sea(enum ghes_exec_ctx context) { return -ENOENT; }
#endif
struct notifier_block;
--
2.39.3
^ permalink raw reply related
* [PATCH v15 1/9] uaccess: add generic fallback version of copy_mc_to_user()
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong, Mauro Carvalho Chehab, Jonathan Cameron
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
x86/powerpc has it's implementation of copy_mc_to_user(), we add generic
fallback in include/linux/uaccess.h prepare for other architechures to
enable CONFIG_ARCH_HAS_COPY_MC.
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Shuai Xue <xueshuai@linux.alibaba.com>
---
arch/powerpc/include/asm/uaccess.h | 1 +
arch/x86/include/asm/uaccess.h | 1 +
include/linux/uaccess.h | 8 ++++++++
3 files changed, 10 insertions(+)
diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
index e98c628e3899..073de098d45a 100644
--- a/arch/powerpc/include/asm/uaccess.h
+++ b/arch/powerpc/include/asm/uaccess.h
@@ -432,6 +432,7 @@ copy_mc_to_user(void __user *to, const void *from, unsigned long n)
return n;
}
+#define copy_mc_to_user copy_mc_to_user
#endif
extern size_t copy_from_user_flushcache(void *dst, const void __user *src, size_t size);
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 3a0dd3c2b233..308b0854d1d5 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -496,6 +496,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned len);
unsigned long __must_check
copy_mc_to_user(void __user *to, const void *from, unsigned len);
+#define copy_mc_to_user copy_mc_to_user
#endif
/*
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index 56328601218c..13b4a3a15437 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -250,6 +250,14 @@ copy_mc_to_kernel(void *dst, const void *src, size_t cnt)
}
#endif
+#ifndef copy_mc_to_user
+static inline unsigned long __must_check
+copy_mc_to_user(void __user *dst, const void *src, unsigned long cnt)
+{
+ return copy_to_user(dst, src, cnt);
+}
+#endif
+
static __always_inline void pagefault_disabled_inc(void)
{
current->pagefault_disabled++;
--
2.39.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox