* Re: [PATCH v1 03/11] KVM: arm64: Use guard()/scoped_guard() in arm64 KVM EL1 code
From: Marc Zyngier @ 2026-06-18 9:41 UTC (permalink / raw)
To: Fuad Tabba
Cc: Oliver Upton, Will Deacon, Catalin Marinas, Quentin Perret,
Vincent Donnefort, Sebastian Ene, Per Larsen, Suzuki K Poulose,
Zenghui Yu, Joey Gouly, Steffen Eiden, Mark Rutland,
Jonathan Cameron, Hyunwoo Kim, linux-arm-kernel, kvmarm,
linux-kernel
In-Reply-To: <CA+EHjTz-+8M5oSh+7A2vkQ-zPmDkaKptYMoBhLz=rReufKsAHg@mail.gmail.com>
On Thu, 18 Jun 2026 10:24:55 +0100,
Fuad Tabba <tabba@google.com> wrote:
>
> On Thu, 18 Jun 2026 at 10:23, Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Fri, 12 Jun 2026 07:59:17 +0100,
> > tabba@google.com wrote:
> > >
> > > Convert the manual mutex_lock()/spin_lock() pairs in
> > > arch/arm64/kvm/{pkvm,arm,mmu,reset,psci}.c to guard(mutex),
> > > guard(spinlock) and scoped_guard(), dropping unlock-only goto labels in
> > > favour of direct returns. Centralised cleanup gotos that still serve
> > > other resources are preserved.
> > >
> > > reset.c uses scoped_guard() rather than guard() so the lock covers only
> > > the small read/update window inside kvm_reset_vcpu(), leaving the rest
> > > of the function outside the critical section.
> >
> > To be brutally honest, I don't think this sort of widespread changes
> > bring us anything. This is just churn.
> >
> > Sure, if you are reworking a particular bit of code that is goto-heavy
> > for the purpose of error handling, this has the potential to cleanup
> > the code *while you are changing it*.
> >
> > But doing it for the sake of doing it? I think we have bigger fish to
> > fry right now.
>
> I understand what you mean. Would you like me to drop all of the guard
> patches, or only those that go beyond the code changed in this series?
The latter. I'm fine with conversion that happens while the code is
being reworked.
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply
* Re: [PATCH 2/9] drm/rockchip: vop2: Reset AXI and DCLK to improve robustness
From: Philipp Zabel @ 2026-06-18 9:39 UTC (permalink / raw)
To: Cristian Ciocaltea, Sandy Huang, Heiko Stübner, Andy Yan,
David Airlie, Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Rob Herring, Krzysztof Kozlowski, Conor Dooley,
Andrzej Hajda, Neil Armstrong, Robert Foss, Laurent Pinchart,
Jonas Karlman, Jernej Skrabec, Luca Ceresoli
Cc: kernel, Andy Yan, dri-devel, devicetree, linux-arm-kernel,
linux-rockchip, linux-kernel
In-Reply-To: <20260617-dw-hdmi-qp-yuv-v1-2-a665cfd06d7d@collabora.com>
On Mi, 2026-06-17 at 21:51 +0300, Cristian Ciocaltea wrote:
> Assert the AXI reset in the CRTC disable path, and the VP DCLK reset in
> the enable path.
>
> These resets are intended to leave the hardware in a clean state for the
> next use, helping recover from exceptions such as IOMMU page faults, as
> well as to prevent random display output glitches, such as a blank
> image, observed when switching modes that also change the color format,
> e.g. from RGB to YUV420 and vice versa.
>
> For now this seems to affect only the RK3588, hence the resets are
> optional and will be provided in the device tree for this SoC only.
>
> Co-developed-by: Andy Yan <andy.yan@rock-chips.com>
> Signed-off-by: Andy Yan <andy.yan@rock-chips.com>
> Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
> ---
> drivers/gpu/drm/rockchip/rockchip_drm_vop2.c | 35 ++++++++++++++++++++++++++++
> drivers/gpu/drm/rockchip/rockchip_drm_vop2.h | 4 ++++
> 2 files changed, 39 insertions(+)
>
> diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
> index 4cce3e336f5b..2833fb49ad81 100644
> --- a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
> +++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
> @@ -17,6 +17,7 @@
> #include <linux/platform_device.h>
> #include <linux/pm_runtime.h>
> #include <linux/regmap.h>
> +#include <linux/reset.h>
> #include <linux/swab.h>
>
> #include <drm/drm.h>
> @@ -860,6 +861,26 @@ static int vop2_core_clks_prepare_enable(struct vop2 *vop2)
> return ret;
> }
>
> +static void vop2_clk_reset(struct vop2 *vop2, struct reset_control *rstc)
The _clk part of the function name is misleading ...
[...]
> @@ -938,6 +959,8 @@ static void vop2_disable(struct vop2 *vop2)
> {
> rockchip_drm_dma_detach_device(vop2->drm, vop2->dev);
>
> + vop2_clk_reset(vop2, vop2->axi_rst);
... because this function is also called with the AXI reset control.
> +
> pm_runtime_put_sync(vop2->dev);
>
> regcache_drop_region(vop2->map, 0, vop2_regmap_config.max_register);
> @@ -1948,6 +1971,8 @@ static void vop2_crtc_atomic_enable(struct drm_crtc *crtc,
>
> vop2_crtc_atomic_try_set_gamma(vop2, vp, crtc, crtc_state);
>
> + vop2_clk_reset(vop2, vp->dclk_rst);
> +
> drm_crtc_vblank_on(crtc);
>
> vop2_unlock(vop2);
> @@ -2531,6 +2556,11 @@ static int vop2_create_crtcs(struct vop2 *vop2)
> return dev_err_probe(drm->dev, PTR_ERR(vp->dclk),
> "failed to get %s\n", dclk_name);
>
> + vp->dclk_rst = devm_reset_control_get_optional(vop2->dev, dclk_name);
Please use devm_reset_control_get_optional_exclusive() directly.
> + if (IS_ERR(vp->dclk_rst))
> + return dev_err_probe(drm->dev, PTR_ERR(vp->dclk_rst),
> + "failed to get %s reset\n", dclk_name);
> +
> np = of_graph_get_remote_node(dev->of_node, i, -1);
> if (!np) {
> drm_dbg(vop2->drm, "%s: No remote for vp%d\n", __func__, i);
> @@ -2890,6 +2920,11 @@ static int vop2_bind(struct device *dev, struct device *master, void *data)
> return dev_err_probe(drm->dev, PTR_ERR(vop2->pll_hdmiphy1),
> "failed to get pll_hdmiphy1\n");
>
> + vop2->axi_rst = devm_reset_control_get_optional(vop2->dev, "axi");
Same as above, devm_reset_control_get_optional_exclusive().
regards
Philipp
^ permalink raw reply
* [PATCH v10 4/4] media: nxp: Add i.MX95 CSI pixel formatter v4l2 driver
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou, Frank Li
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
The CSI pixel formatter is a module found on i.MX95 used to reformat
packet info, pixel and non-pixel data from CSI-2 host controller to
match Pixel Link(PL) definition.
Add data formatting support.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Use u8 for vc in csi_formatter_get_vc() and drop vc < 0 check
- Add MFD_SYSCON dependency to Kconfig
- Fix stream/VC mapping potential mismatch in start/stop_stream functions
Changes in v8:
- Remove fmt field and look up format from subdev state instead
- Unify function and structure naming to use csi_formatter_ prefix
- Remove misleading alignment comment from set_fmt function
- Optimize get_frame_desc to call once per start_stream
- Replace V4L2_FRAME_DESC_ENTRY_MAX with CSI_FORMATTER_VC_NUM in loops
- Remove redundant debug message in enable_streams
- Use MEDIA_PAD_FL_MUST_CONNECT flag instead of manual link check
- Fix typo: Formater -> Formatter in Kconfig help text
- Improve grammar in data type index mapping comment
Changes in v7:
- Update references from imx9 to imx95 for consistency with dt-bindings
- Enable PM runtime before async registration
Changes in v6:
- Remove unused header includes
- Unify macro naming: VCx/VCX -> VC and parameter x -> vc
- Remove unused format field from csi_formatter struct
- Use compact initialization for formats array
- Make find_csi_format() return NULL instead of default format
- Use unsigned int for array index in find_csi_format()
- Add err_ prefix to error handling labels
- Add v4l2_subdev_cleanup() and reorder cleanup sequence
- Update enable_streams debug output format
- Rename VC_MAX to VC_NUM and fix boundary check
- Update CSI formatter Kconfig description
- Use v4l2_subdev_get_frame_desc_passthrough() helper
- Fix error paths in async registration and probe
- Add mutex to protect enabled_streams
- Switch to devm_pm_runtime_enable()
- Remove redundant num_routes check in set_routing
- Optimize get_index_by_dt() and add warning for unsupported type
- csi_formatter_start/stop_stream: Process all streams in mask
---
MAINTAINERS | 8 +
drivers/media/platform/nxp/Kconfig | 15 +
drivers/media/platform/nxp/Makefile | 1 +
drivers/media/platform/nxp/imx95-csi-formatter.c | 775 +++++++++++++++++++++++
4 files changed, 799 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index efbf808063e5..05009228b162 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19275,6 +19275,14 @@ S: Maintained
F: Documentation/devicetree/bindings/media/nxp,imx8-jpeg.yaml
F: drivers/media/platform/nxp/imx-jpeg
+NXP i.MX 95 CSI PIXEL FORMATTER V4L2 DRIVER
+M: Guoniu Zhou <guoniu.zhou@nxp.com>
+L: imx@lists.linux.dev
+L: linux-media@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml
+F: drivers/media/platform/nxp/imx95-csi-formatter.c
+
NXP i.MX CLOCK DRIVERS
M: Abel Vesa <abelvesa@kernel.org>
R: Peng Fan <peng.fan@nxp.com>
diff --git a/drivers/media/platform/nxp/Kconfig b/drivers/media/platform/nxp/Kconfig
index 40e3436669e2..8f49908b0022 100644
--- a/drivers/media/platform/nxp/Kconfig
+++ b/drivers/media/platform/nxp/Kconfig
@@ -28,6 +28,21 @@ config VIDEO_IMX8MQ_MIPI_CSI2
Video4Linux2 driver for the MIPI CSI-2 receiver found on the i.MX8MQ
SoC.
+config VIDEO_IMX95_CSI_FORMATTER
+ tristate "NXP i.MX95 CSI Pixel Formatter driver"
+ depends on ARCH_MXC || COMPILE_TEST
+ depends on MFD_SYSCON
+ depends on VIDEO_DEV
+ select MEDIA_CONTROLLER
+ select V4L2_FWNODE
+ select VIDEO_V4L2_SUBDEV_API
+ help
+ This driver provides support for the CSI Pixel Formatter found on
+ i.MX95 series SoCs. This module unpacks the pixels received from the
+ CSI-2 interface and reformats them to meet pixel link requirements.
+
+ Say Y here to enable CSI Pixel Formatter module for i.MX95 SoC.
+
config VIDEO_IMX_MIPI_CSIS
tristate "NXP MIPI CSI-2 CSIS receiver found on i.MX7 and i.MX8 models"
depends on ARCH_MXC || COMPILE_TEST
diff --git a/drivers/media/platform/nxp/Makefile b/drivers/media/platform/nxp/Makefile
index 4d90eb713652..6410115d870e 100644
--- a/drivers/media/platform/nxp/Makefile
+++ b/drivers/media/platform/nxp/Makefile
@@ -6,6 +6,7 @@ obj-y += imx8-isi/
obj-$(CONFIG_VIDEO_IMX7_CSI) += imx7-media-csi.o
obj-$(CONFIG_VIDEO_IMX8MQ_MIPI_CSI2) += imx8mq-mipi-csi2.o
+obj-$(CONFIG_VIDEO_IMX95_CSI_FORMATTER) += imx95-csi-formatter.o
obj-$(CONFIG_VIDEO_IMX_MIPI_CSIS) += imx-mipi-csis.o
obj-$(CONFIG_VIDEO_IMX_PXP) += imx-pxp.o
obj-$(CONFIG_VIDEO_MX2_EMMAPRP) += mx2_emmaprp.o
diff --git a/drivers/media/platform/nxp/imx95-csi-formatter.c b/drivers/media/platform/nxp/imx95-csi-formatter.c
new file mode 100644
index 000000000000..cfe448fedd37
--- /dev/null
+++ b/drivers/media/platform/nxp/imx95-csi-formatter.c
@@ -0,0 +1,775 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2025 NXP
+ */
+
+#include <linux/bits.h>
+#include <linux/clk.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/regmap.h>
+
+#include <media/mipi-csi2.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fwnode.h>
+#include <media/v4l2-mc.h>
+#include <media/v4l2-subdev.h>
+
+/* CSI Pixel Formatter registers map */
+
+#define CSI_VC_INTERLACED_LINE_CNT(vc) (0x00 + (vc) * 0x04)
+#define INTERLACED_ODD_LINE_CNT_SET(x) FIELD_PREP(GENMASK(13, 0), (x))
+#define INTERLACED_EVEN_LINE_CNT_SET(x) FIELD_PREP(GENMASK(29, 16), (x))
+
+#define CSI_VC_INTERLACED_CTRL 0x20
+
+#define CSI_VC_INTERLACED_ERR 0x24
+#define CSI_VC_ERR_MASK GENMASK(7, 0)
+#define CSI_VC_ERR(vc) BIT((vc))
+
+#define CSI_VC_YUV420_FIRST_LINE_EVEN 0x28
+#define YUV420_FIRST_LINE_EVEN(vc) BIT((vc))
+
+#define CSI_RAW32_CTRL 0x30
+#define CSI_VC_RAW32_MODE(vc) BIT((vc))
+#define CSI_VC_RAW32_SWAP_MODE(vc) BIT((vc) + 8)
+
+#define CSI_STREAM_FENCING_CTRL 0x34
+#define CSI_VC_STREAM_FENCING(vc) BIT((vc))
+#define CSI_VC_STREAM_FENCING_RST(vc) BIT((vc) + 8)
+
+#define CSI_STREAM_FENCING_STS 0x38
+#define CSI_STREAM_FENCING_STS_MASK GENMASK(7, 0)
+
+#define CSI_VC_NON_PIXEL_DATA_TYPE(vc) (0x40 + (vc) * 0x04)
+
+#define CSI_VC_PIXEL_DATA_CTRL(vc) (0x60 + (vc) * 0x04)
+#define NEW_VC(vc) FIELD_PREP(GENMASK(3, 1), vc)
+#define REROUTE_VC_ENABLE BIT(0)
+
+#define CSI_VC_ROUTE_PIXEL_DATA_TYPE(vc) (0x80 + (vc) * 0x04)
+
+#define CSI_VC_NON_PIXEL_DATA_CTRL(vc) (0xa0 + (vc) * 0x04)
+
+#define CSI_VC_PIXEL_DATA_TYPE(vc) (0xc0 + (vc) * 0x04)
+
+#define CSI_VC_PIXEL_DATA_TYPE_ERR(vc) (0xe0 + (vc) * 0x04)
+
+#define CSI_FORMATTER_PAD_SINK 0
+#define CSI_FORMATTER_PAD_SOURCE 1
+#define CSI_FORMATTER_PAD_NUM 2
+
+#define CSI_FORMATTER_VC_NUM 8 /* Number of virtual channels */
+
+struct csi_formatter_pix_format {
+ u32 code;
+ u32 data_type;
+};
+
+struct csi_formatter {
+ struct device *dev;
+ struct regmap *regs;
+ struct clk *clk;
+
+ struct v4l2_subdev sd;
+ struct v4l2_subdev *csi_sd;
+ struct v4l2_async_notifier notifier;
+ struct media_pad pads[CSI_FORMATTER_PAD_NUM];
+
+ u32 remote_pad;
+ u32 reg_offset;
+
+ /* Protects enabled_streams */
+ struct mutex lock;
+ u64 enabled_streams;
+
+ u8 stream_to_vc[CSI_FORMATTER_VC_NUM];
+};
+
+struct csi_formatter_dt_index {
+ u8 dtype;
+ u8 index;
+};
+
+/*
+ * The index corresponds to the bit index in the register that enables
+ * the data type of pixel data transported by the Formatter.
+ */
+static const struct csi_formatter_dt_index formatter_dt_to_index_map[] = {
+ { .dtype = MIPI_CSI2_DT_YUV420_8B, .index = 0 },
+ { .dtype = MIPI_CSI2_DT_YUV420_8B_LEGACY, .index = 2 },
+ { .dtype = MIPI_CSI2_DT_YUV422_8B, .index = 6 },
+ { .dtype = MIPI_CSI2_DT_RGB444, .index = 8 },
+ { .dtype = MIPI_CSI2_DT_RGB555, .index = 9 },
+ { .dtype = MIPI_CSI2_DT_RGB565, .index = 10 },
+ { .dtype = MIPI_CSI2_DT_RGB666, .index = 11 },
+ { .dtype = MIPI_CSI2_DT_RGB888, .index = 12 },
+ { .dtype = MIPI_CSI2_DT_RAW6, .index = 16 },
+ { .dtype = MIPI_CSI2_DT_RAW7, .index = 17 },
+ { .dtype = MIPI_CSI2_DT_RAW8, .index = 18 },
+ { .dtype = MIPI_CSI2_DT_RAW10, .index = 19 },
+ { .dtype = MIPI_CSI2_DT_RAW12, .index = 20 },
+ { .dtype = MIPI_CSI2_DT_RAW14, .index = 21 },
+ { .dtype = MIPI_CSI2_DT_RAW16, .index = 22 },
+};
+
+static const struct csi_formatter_pix_format formats[] = {
+ /* YUV formats */
+ { MEDIA_BUS_FMT_UYVY8_1X16, MIPI_CSI2_DT_YUV422_8B },
+ /* RGB formats */
+ { MEDIA_BUS_FMT_RGB565_1X16, MIPI_CSI2_DT_RGB565 },
+ { MEDIA_BUS_FMT_RGB888_1X24, MIPI_CSI2_DT_RGB888 },
+ /* RAW (Bayer and greyscale) formats */
+ { MEDIA_BUS_FMT_SBGGR8_1X8, MIPI_CSI2_DT_RAW8 },
+ { MEDIA_BUS_FMT_SGBRG8_1X8, MIPI_CSI2_DT_RAW8 },
+ { MEDIA_BUS_FMT_SGRBG8_1X8, MIPI_CSI2_DT_RAW8 },
+ { MEDIA_BUS_FMT_SRGGB8_1X8, MIPI_CSI2_DT_RAW8 },
+ { MEDIA_BUS_FMT_Y8_1X8, MIPI_CSI2_DT_RAW8 },
+ { MEDIA_BUS_FMT_SBGGR10_1X10, MIPI_CSI2_DT_RAW10 },
+ { MEDIA_BUS_FMT_SGBRG10_1X10, MIPI_CSI2_DT_RAW10 },
+ { MEDIA_BUS_FMT_SGRBG10_1X10, MIPI_CSI2_DT_RAW10 },
+ { MEDIA_BUS_FMT_SRGGB10_1X10, MIPI_CSI2_DT_RAW10 },
+ { MEDIA_BUS_FMT_Y10_1X10, MIPI_CSI2_DT_RAW10 },
+ { MEDIA_BUS_FMT_SBGGR12_1X12, MIPI_CSI2_DT_RAW12 },
+ { MEDIA_BUS_FMT_SGBRG12_1X12, MIPI_CSI2_DT_RAW12 },
+ { MEDIA_BUS_FMT_SGRBG12_1X12, MIPI_CSI2_DT_RAW12 },
+ { MEDIA_BUS_FMT_SRGGB12_1X12, MIPI_CSI2_DT_RAW12 },
+ { MEDIA_BUS_FMT_Y12_1X12, MIPI_CSI2_DT_RAW12 },
+ { MEDIA_BUS_FMT_SBGGR14_1X14, MIPI_CSI2_DT_RAW14 },
+ { MEDIA_BUS_FMT_SGBRG14_1X14, MIPI_CSI2_DT_RAW14 },
+ { MEDIA_BUS_FMT_SGRBG14_1X14, MIPI_CSI2_DT_RAW14 },
+ { MEDIA_BUS_FMT_SRGGB14_1X14, MIPI_CSI2_DT_RAW14 },
+ { MEDIA_BUS_FMT_SBGGR16_1X16, MIPI_CSI2_DT_RAW16 },
+ { MEDIA_BUS_FMT_SGBRG16_1X16, MIPI_CSI2_DT_RAW16 },
+ { MEDIA_BUS_FMT_SGRBG16_1X16, MIPI_CSI2_DT_RAW16 },
+ { MEDIA_BUS_FMT_SRGGB16_1X16, MIPI_CSI2_DT_RAW16 },
+};
+
+static const struct v4l2_mbus_framefmt formatter_default_fmt = {
+ .code = MEDIA_BUS_FMT_UYVY8_1X16,
+ .width = 1920U,
+ .height = 1080U,
+ .field = V4L2_FIELD_NONE,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(V4L2_COLORSPACE_SMPTE170M),
+ .ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(V4L2_COLORSPACE_SMPTE170M),
+ .quantization = V4L2_QUANTIZATION_LIM_RANGE,
+};
+
+static const struct csi_formatter_pix_format *csi_formatter_find_format(u32 code)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(formats); i++)
+ if (code == formats[i].code)
+ return &formats[i];
+
+ return NULL;
+}
+
+/* -----------------------------------------------------------------------------
+ * V4L2 subdev operations
+ */
+
+static inline struct csi_formatter *sd_to_formatter(struct v4l2_subdev *sdev)
+{
+ return container_of(sdev, struct csi_formatter, sd);
+}
+
+static int __csi_formatter_subdev_set_routing(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ struct v4l2_subdev_krouting *routing)
+{
+ int ret;
+
+ ret = v4l2_subdev_routing_validate(sd, routing,
+ V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
+ if (ret)
+ return ret;
+
+ return v4l2_subdev_set_routing_with_fmt(sd, state, routing,
+ &formatter_default_fmt);
+}
+
+static int csi_formatter_subdev_init_state(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *sd_state)
+{
+ struct v4l2_subdev_route routes[] = {
+ {
+ .sink_pad = CSI_FORMATTER_PAD_SINK,
+ .sink_stream = 0,
+ .source_pad = CSI_FORMATTER_PAD_SOURCE,
+ .source_stream = 0,
+ .flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+ },
+ };
+
+ struct v4l2_subdev_krouting routing = {
+ .num_routes = ARRAY_SIZE(routes),
+ .routes = routes,
+ };
+
+ return __csi_formatter_subdev_set_routing(sd, sd_state, &routing);
+}
+
+static int csi_formatter_subdev_enum_mbus_code(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *sd_state,
+ struct v4l2_subdev_mbus_code_enum *code)
+{
+ if (code->pad == CSI_FORMATTER_PAD_SOURCE) {
+ struct v4l2_mbus_framefmt *fmt;
+
+ if (code->index > 0)
+ return -EINVAL;
+
+ fmt = v4l2_subdev_state_get_format(sd_state, code->pad,
+ code->stream);
+ code->code = fmt->code;
+ return 0;
+ }
+
+ if (code->index >= ARRAY_SIZE(formats))
+ return -EINVAL;
+
+ code->code = formats[code->index].code;
+
+ return 0;
+}
+
+static int csi_formatter_subdev_set_fmt(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *sd_state,
+ struct v4l2_subdev_format *sdformat)
+{
+ struct csi_formatter_pix_format const *format;
+ struct v4l2_mbus_framefmt *fmt;
+
+ if (sdformat->pad == CSI_FORMATTER_PAD_SOURCE)
+ return v4l2_subdev_get_fmt(sd, sd_state, sdformat);
+
+ format = csi_formatter_find_format(sdformat->format.code);
+ if (!format)
+ format = &formats[0];
+
+ v4l_bound_align_image(&sdformat->format.width, 1, 0xffff, 2,
+ &sdformat->format.height, 1, 0xffff, 0, 0);
+
+ fmt = v4l2_subdev_state_get_format(sd_state, sdformat->pad,
+ sdformat->stream);
+ *fmt = sdformat->format;
+
+ /* Set default code if user set an invalid value */
+ fmt->code = format->code;
+
+ /* Propagate the format from sink stream to source stream */
+ fmt = v4l2_subdev_state_get_opposite_stream_format(sd_state, sdformat->pad,
+ sdformat->stream);
+ if (!fmt)
+ return -EINVAL;
+
+ *fmt = sdformat->format;
+
+ return 0;
+}
+
+static int csi_formatter_subdev_set_routing(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ enum v4l2_subdev_format_whence which,
+ struct v4l2_subdev_krouting *routing)
+{
+ if (which == V4L2_SUBDEV_FORMAT_ACTIVE &&
+ media_entity_is_streaming(&sd->entity))
+ return -EBUSY;
+
+ return __csi_formatter_subdev_set_routing(sd, state, routing);
+}
+
+static inline void csi_formatter_write(struct csi_formatter *formatter,
+ unsigned int reg, unsigned int value)
+{
+ u32 offset = formatter->reg_offset;
+
+ regmap_write(formatter->regs, reg + offset, value);
+}
+
+static u8 csi_formatter_get_index_by_dt(u8 data_type)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(formatter_dt_to_index_map); ++i) {
+ const struct csi_formatter_dt_index *entry =
+ &formatter_dt_to_index_map[i];
+
+ if (data_type == entry->dtype)
+ return entry->index;
+ }
+
+ pr_warn_once("Unsupported data type 0x%x, using default\n", data_type);
+
+ return formatter_dt_to_index_map[0].index;
+}
+
+static int csi_formatter_get_vc(struct csi_formatter *formatter,
+ struct v4l2_mbus_frame_desc *fd,
+ unsigned int stream)
+{
+ struct v4l2_mbus_frame_desc_entry *entry = NULL;
+ unsigned int i;
+ u8 vc;
+
+ for (i = 0; i < fd->num_entries; ++i) {
+ if (fd->entry[i].stream == stream) {
+ entry = &fd->entry[i];
+ break;
+ }
+ }
+
+ if (!entry) {
+ dev_err(formatter->dev,
+ "No frame desc entry for stream %u\n", stream);
+ return -EPIPE;
+ }
+
+ vc = entry->bus.csi2.vc;
+
+ if (vc >= CSI_FORMATTER_VC_NUM) {
+ dev_err(formatter->dev, "Invalid virtual channel %u\n", vc);
+ return -EINVAL;
+ }
+
+ return vc;
+}
+
+static void csi_formatter_stop_stream(struct csi_formatter *formatter,
+ u64 stream_mask)
+{
+ unsigned int i;
+ u8 vc;
+
+ for (i = 0; i < CSI_FORMATTER_VC_NUM; ++i) {
+ if (!(stream_mask & BIT(i)))
+ continue;
+
+ /* Use the VC that was configured in start_stream */
+ vc = formatter->stream_to_vc[i];
+ if (vc >= CSI_FORMATTER_VC_NUM)
+ continue;
+
+ csi_formatter_write(formatter, CSI_VC_PIXEL_DATA_TYPE(vc), 0);
+
+ /* Clear after use */
+ formatter->stream_to_vc[i] = 0xff;
+ }
+}
+
+static int csi_formatter_start_stream(struct csi_formatter *formatter,
+ struct v4l2_subdev_state *state,
+ u64 stream_mask)
+{
+ const struct csi_formatter_pix_format *pix_fmt;
+ struct v4l2_mbus_framefmt *fmt;
+ struct v4l2_mbus_frame_desc fd = {};
+ u64 configured_streams = 0;
+ unsigned int i;
+ u32 val;
+ int vc;
+ int ret;
+
+ ret = v4l2_subdev_call(formatter->csi_sd, pad, get_frame_desc,
+ formatter->remote_pad, &fd);
+ if (ret < 0 && ret != -ENOIOCTLCMD) {
+ dev_err(formatter->dev, "Failed to get frame desc: %d\n", ret);
+ return ret;
+ }
+
+ for (i = 0; i < CSI_FORMATTER_VC_NUM; ++i) {
+ if (!(stream_mask & BIT(i)))
+ continue;
+
+ fmt = v4l2_subdev_state_get_format(state,
+ CSI_FORMATTER_PAD_SINK, i);
+
+ pix_fmt = csi_formatter_find_format(fmt->code);
+
+ val = BIT(csi_formatter_get_index_by_dt(pix_fmt->data_type));
+
+ if (ret == -ENOIOCTLCMD) {
+ /*
+ * Source doesn't implement get_frame_desc, use
+ * default VC 0
+ */
+ vc = 0;
+ } else {
+ vc = csi_formatter_get_vc(formatter, &fd, i);
+ if (vc < 0) {
+ ret = vc;
+ goto err_cleanup;
+ }
+ }
+
+ /* Store the stream to VC mapping for stop_stream */
+ formatter->stream_to_vc[i] = vc;
+
+ csi_formatter_write(formatter, CSI_VC_PIXEL_DATA_TYPE(vc), val);
+ configured_streams |= BIT(i);
+ }
+
+ return 0;
+
+err_cleanup:
+ csi_formatter_stop_stream(formatter, configured_streams);
+ return ret;
+}
+
+static int csi_formatter_subdev_enable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
+{
+ struct csi_formatter *formatter = sd_to_formatter(sd);
+ struct device *dev = formatter->dev;
+ u64 sink_streams;
+ int ret;
+
+ sink_streams = v4l2_subdev_state_xlate_streams(state,
+ CSI_FORMATTER_PAD_SOURCE,
+ CSI_FORMATTER_PAD_SINK,
+ &streams_mask);
+ if (!sink_streams || !streams_mask)
+ return -EINVAL;
+
+ guard(mutex)(&formatter->lock);
+
+ if (!formatter->enabled_streams) {
+ ret = pm_runtime_resume_and_get(formatter->dev);
+ if (ret < 0) {
+ dev_err(dev, "Failed to resume runtime PM: %d\n", ret);
+ return ret;
+ }
+ }
+
+ ret = csi_formatter_start_stream(formatter, state, streams_mask);
+ if (ret)
+ goto err_runtime_put;
+
+ ret = v4l2_subdev_enable_streams(formatter->csi_sd,
+ formatter->remote_pad,
+ sink_streams);
+ if (ret)
+ goto err_stop_stream;
+
+ formatter->enabled_streams |= streams_mask;
+
+ return 0;
+
+err_stop_stream:
+ csi_formatter_stop_stream(formatter, streams_mask);
+err_runtime_put:
+ if (!formatter->enabled_streams)
+ pm_runtime_put(formatter->dev);
+ return ret;
+}
+
+static int csi_formatter_subdev_disable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
+{
+ struct csi_formatter *formatter = sd_to_formatter(sd);
+ u64 sink_streams;
+ int ret;
+
+ sink_streams = v4l2_subdev_state_xlate_streams(state,
+ CSI_FORMATTER_PAD_SOURCE,
+ CSI_FORMATTER_PAD_SINK,
+ &streams_mask);
+ if (!sink_streams || !streams_mask)
+ return -EINVAL;
+
+ guard(mutex)(&formatter->lock);
+
+ ret = v4l2_subdev_disable_streams(formatter->csi_sd, formatter->remote_pad,
+ sink_streams);
+ if (ret)
+ dev_err(formatter->dev, "Failed to disable streams: %d\n", ret);
+
+ csi_formatter_stop_stream(formatter, streams_mask);
+
+ formatter->enabled_streams &= ~streams_mask;
+
+ if (!formatter->enabled_streams)
+ pm_runtime_put(formatter->dev);
+
+ return ret;
+}
+
+static const struct v4l2_subdev_pad_ops formatter_subdev_pad_ops = {
+ .enum_mbus_code = csi_formatter_subdev_enum_mbus_code,
+ .get_fmt = v4l2_subdev_get_fmt,
+ .set_fmt = csi_formatter_subdev_set_fmt,
+ .get_frame_desc = v4l2_subdev_get_frame_desc_passthrough,
+ .set_routing = csi_formatter_subdev_set_routing,
+ .enable_streams = csi_formatter_subdev_enable_streams,
+ .disable_streams = csi_formatter_subdev_disable_streams,
+};
+
+static const struct v4l2_subdev_ops formatter_subdev_ops = {
+ .pad = &formatter_subdev_pad_ops,
+};
+
+static const struct v4l2_subdev_internal_ops formatter_internal_ops = {
+ .init_state = csi_formatter_subdev_init_state,
+};
+
+/* -----------------------------------------------------------------------------
+ * Media entity operations
+ */
+
+static const struct media_entity_operations formatter_entity_ops = {
+ .link_validate = v4l2_subdev_link_validate,
+ .get_fwnode_pad = v4l2_subdev_get_fwnode_pad_1_to_1,
+};
+
+static int csi_formatter_subdev_init(struct csi_formatter *formatter)
+{
+ struct v4l2_subdev *sd = &formatter->sd;
+ int ret;
+
+ v4l2_subdev_init(sd, &formatter_subdev_ops);
+
+ snprintf(sd->name, sizeof(sd->name), "%s", dev_name(formatter->dev));
+ sd->internal_ops = &formatter_internal_ops;
+
+ sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
+ V4L2_SUBDEV_FL_HAS_EVENTS |
+ V4L2_SUBDEV_FL_STREAMS;
+ sd->entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+ sd->entity.ops = &formatter_entity_ops;
+ sd->dev = formatter->dev;
+
+ formatter->pads[CSI_FORMATTER_PAD_SINK].flags = MEDIA_PAD_FL_SINK
+ | MEDIA_PAD_FL_MUST_CONNECT;
+ formatter->pads[CSI_FORMATTER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+
+ ret = media_entity_pads_init(&sd->entity, CSI_FORMATTER_PAD_NUM,
+ formatter->pads);
+ if (ret) {
+ dev_err(formatter->dev, "Failed to init pads\n");
+ return ret;
+ }
+
+ ret = v4l2_subdev_init_finalize(sd);
+ if (ret)
+ media_entity_cleanup(&sd->entity);
+
+ return ret;
+}
+
+static inline struct csi_formatter *
+notifier_to_csi_formatter(struct v4l2_async_notifier *n)
+{
+ return container_of(n, struct csi_formatter, notifier);
+}
+
+static int csi_formatter_notify_bound(struct v4l2_async_notifier *notifier,
+ struct v4l2_subdev *sd,
+ struct v4l2_async_connection *asc)
+{
+ const unsigned int link_flags = MEDIA_LNK_FL_IMMUTABLE
+ | MEDIA_LNK_FL_ENABLED;
+ struct csi_formatter *formatter = notifier_to_csi_formatter(notifier);
+ struct v4l2_subdev *sdev = &formatter->sd;
+ struct media_pad *sink = &sdev->entity.pads[CSI_FORMATTER_PAD_SINK];
+ struct media_pad *remote_pad;
+ int ret;
+
+ formatter->csi_sd = sd;
+
+ dev_dbg(formatter->dev, "Bound subdev: %s pad\n", sd->name);
+
+ ret = v4l2_create_fwnode_links_to_pad(sd, sink, link_flags);
+ if (ret < 0)
+ return ret;
+
+ remote_pad = media_pad_remote_pad_first(sink);
+ if (!remote_pad) {
+ dev_err(formatter->dev, "Pipe not setup correctly\n");
+ return -EPIPE;
+ }
+ formatter->remote_pad = remote_pad->index;
+
+ return 0;
+}
+
+static const struct v4l2_async_notifier_operations formatter_notify_ops = {
+ .bound = csi_formatter_notify_bound,
+};
+
+static int csi_formatter_async_register(struct csi_formatter *formatter)
+{
+ struct device *dev = formatter->dev;
+ struct v4l2_async_connection *asc;
+ int ret;
+
+ struct fwnode_handle *ep __free(fwnode_handle) =
+ fwnode_graph_get_endpoint_by_id(dev_fwnode(dev), 0, 0,
+ FWNODE_GRAPH_ENDPOINT_NEXT);
+ if (!ep)
+ return -ENOTCONN;
+
+ v4l2_async_subdev_nf_init(&formatter->notifier, &formatter->sd);
+
+ asc = v4l2_async_nf_add_fwnode_remote(&formatter->notifier, ep,
+ struct v4l2_async_connection);
+ if (IS_ERR(asc)) {
+ ret = PTR_ERR(asc);
+ goto err_cleanup_notifier;
+ }
+
+ formatter->notifier.ops = &formatter_notify_ops;
+
+ ret = v4l2_async_nf_register(&formatter->notifier);
+ if (ret)
+ goto err_cleanup_notifier;
+
+ ret = v4l2_async_register_subdev(&formatter->sd);
+ if (ret)
+ goto err_unregister_notifier;
+
+ return 0;
+
+err_unregister_notifier:
+ v4l2_async_nf_unregister(&formatter->notifier);
+err_cleanup_notifier:
+ v4l2_async_nf_cleanup(&formatter->notifier);
+ return ret;
+}
+
+static void csi_formatter_async_unregister(struct csi_formatter *formatter)
+{
+ v4l2_async_unregister_subdev(&formatter->sd);
+ v4l2_async_nf_unregister(&formatter->notifier);
+ v4l2_async_nf_cleanup(&formatter->notifier);
+}
+
+/* -----------------------------------------------------------------------------
+ * Suspend/resume
+ */
+
+static int csi_formatter_runtime_suspend(struct device *dev)
+{
+ struct v4l2_subdev *sd = dev_get_drvdata(dev);
+ struct csi_formatter *formatter = sd_to_formatter(sd);
+
+ clk_disable_unprepare(formatter->clk);
+
+ return 0;
+}
+
+static int csi_formatter_runtime_resume(struct device *dev)
+{
+ struct v4l2_subdev *sd = dev_get_drvdata(dev);
+ struct csi_formatter *formatter = sd_to_formatter(sd);
+
+ return clk_prepare_enable(formatter->clk);
+}
+
+static DEFINE_RUNTIME_DEV_PM_OPS(csi_formatter_pm_ops,
+ csi_formatter_runtime_suspend,
+ csi_formatter_runtime_resume, NULL);
+
+static int csi_formatter_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct csi_formatter *formatter;
+ u32 val;
+ int ret;
+
+ formatter = devm_kzalloc(dev, sizeof(*formatter), GFP_KERNEL);
+ if (!formatter)
+ return -ENOMEM;
+
+ /* Initialize stream to VC mapping to invalid */
+ memset(formatter->stream_to_vc, 0xff, sizeof(formatter->stream_to_vc));
+
+ formatter->dev = dev;
+
+ ret = devm_mutex_init(dev, &formatter->lock);
+ if (ret)
+ return ret;
+
+ formatter->regs = syscon_node_to_regmap(dev->parent->of_node);
+ if (IS_ERR(formatter->regs))
+ return dev_err_probe(dev, PTR_ERR(formatter->regs),
+ "Failed to get csi formatter regmap\n");
+
+ ret = of_property_read_u32(dev->of_node, "reg", &val);
+ if (ret < 0)
+ return dev_err_probe(dev, ret,
+ "Failed to get csi formatter reg property\n");
+
+ formatter->reg_offset = val;
+
+ formatter->clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(formatter->clk))
+ return dev_err_probe(dev, PTR_ERR(formatter->clk),
+ "Failed to get pixel clock\n");
+
+ ret = csi_formatter_subdev_init(formatter);
+ if (ret < 0)
+ return dev_err_probe(dev, ret, "Failed to initialize formatter subdev\n");
+
+ platform_set_drvdata(pdev, &formatter->sd);
+
+ /* Enable runtime PM. */
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ goto err_cleanup_subdev;
+
+ ret = csi_formatter_async_register(formatter);
+ if (ret < 0) {
+ dev_err_probe(dev, ret, "Failed to register async subdevice\n");
+ goto err_cleanup_subdev;
+ }
+
+ return 0;
+
+err_cleanup_subdev:
+ v4l2_subdev_cleanup(&formatter->sd);
+ media_entity_cleanup(&formatter->sd.entity);
+ return ret;
+}
+
+static void csi_formatter_remove(struct platform_device *pdev)
+{
+ struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+ struct csi_formatter *formatter = sd_to_formatter(sd);
+
+ csi_formatter_async_unregister(formatter);
+
+ v4l2_subdev_cleanup(&formatter->sd);
+ media_entity_cleanup(&formatter->sd.entity);
+}
+
+static const struct of_device_id csi_formatter_of_match[] = {
+ { .compatible = "fsl,imx95-csi-formatter" },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, csi_formatter_of_match);
+
+static struct platform_driver csi_formatter_device_driver = {
+ .driver = {
+ .name = "csi-pixel-formatter",
+ .of_match_table = csi_formatter_of_match,
+ .pm = pm_ptr(&csi_formatter_pm_ops),
+ },
+ .probe = csi_formatter_probe,
+ .remove = csi_formatter_remove,
+};
+
+module_platform_driver(csi_formatter_device_driver);
+
+MODULE_AUTHOR("NXP Semiconductor, Inc.");
+MODULE_DESCRIPTION("NXP i.MX95 CSI Pixel Formatter driver");
+MODULE_LICENSE("GPL");
--
2.34.1
^ permalink raw reply related
* [PATCH v10 3/4] dt-bindings: clock: imx95-blk-ctl: Define formatter child node schema
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
The Camera CSR contains control registers for multiple CSI formatter IPs
at different register offsets. Each formatter is an independent hardware
block with its own clock input and media pipeline connection.
Define schema to allow formatter child nodes under nxp,imx95-camera-csr,
with 'reg' property specifying the formatter's register offset within the
CSR address space.
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Use single quotes for regex pattern to be consistent (Krzysztof Kozlowski)
- Add formatter subnode binding and camera-csr syscon example
- Update commit title and message
Changes in v9:
- New patch to address the issue of formatter acting as a child node of syscon
---
.../bindings/clock/nxp,imx95-blk-ctl.yaml | 64 +++++++++++++++++++++-
1 file changed, 63 insertions(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
index 534fa219d9f9..b4d0a7670fac 100644
--- a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
+++ b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
@@ -46,7 +46,27 @@ required:
- power-domains
- clocks
-additionalProperties: false
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: nxp,imx95-camera-csr
+ then:
+ properties:
+ '#address-cells':
+ const: 1
+ '#size-cells':
+ const: 1
+ required:
+ - '#address-cells'
+ - '#size-cells'
+ patternProperties:
+ '^formatter@[0-9a-f]+$':
+ type: object
+ $ref: /schemas/media/fsl,imx95-csi-formatter.yaml#
+
+unevaluatedProperties: false
examples:
- |
@@ -57,4 +77,46 @@ examples:
clocks = <&scmi_clk 114>;
power-domains = <&scmi_devpd 21>;
};
+
+ - |
+ #include <dt-bindings/clock/nxp,imx95-clock.h>
+
+ syscon@4ac10000 {
+ compatible = "nxp,imx95-camera-csr", "syscon";
+ reg = <0x4ac10000 0x10000>;
+ #address-cells = <1>;
+ #size-cells = <1>;
+ #clock-cells = <1>;
+ clocks = <&scmi_clk 62>;
+ power-domains = <&scmi_devpd 3>;
+
+ formatter@20 {
+ compatible = "fsl,imx95-csi-formatter";
+ reg = <0x20 0x100>;
+ clocks = <&cameramix_csr IMX95_CLK_CAMBLK_CSI2_FOR0>;
+ power-domains = <&scmi_devpd 3>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 {
+ reg = <0>;
+
+ endpoint {
+ remote-endpoint = <&mipi_csi_0_out>;
+ };
+
+ };
+
+ port@1 {
+ reg = <1>;
+
+ endpoint {
+ remote-endpoint = <&isi_in_2>;
+ };
+ };
+ };
+ };
+ };
...
--
2.34.1
^ permalink raw reply related
* [PATCH v10 2/4] media: dt-bindings: Add CSI Pixel Formatter DT bindings
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
The i.MX95 CSI pixel formatting module uses packet info, pixel and
non-pixel data from the CSI-2 host controller and reformat them to
match Pixel Link(PL) definition.
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Drop syscon parent node from example
- Drop Reviewed-by tags from Frank and Krzysztof due to binding changes
- Add description for reg property
- Add space after formatter@20 before opening brace in example
- Enhance the port description with more detailed information
- Delete the blank line immediately following the endpoint in example
Changes in v9:
- Use direct node instead of syscon wrapper in example
Changes in v8:
- Use standard port reference instead of video-interfaces.yaml
- Add parent syscon node in example to show device integration
- Add required constraints for port@0 and port@1 in ports node
Changes in v7:
- Change compatible to imx95-csi-formatter as IP is i.MX95 specific per Marco's suggestion
Link: https://lore.kernel.org/linux-media/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com/T/#mcd135b3de179b3cb69daa1fd6e0e8e27c85b3332
---
.../bindings/media/fsl,imx95-csi-formatter.yaml | 88 ++++++++++++++++++++++
1 file changed, 88 insertions(+)
diff --git a/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml b/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml
new file mode 100644
index 000000000000..58c4e1cc056b
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/fsl,imx95-csi-formatter.yaml
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/fsl,imx95-csi-formatter.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: i.MX95 CSI Pixel Formatter
+
+maintainers:
+ - Guoniu Zhou <guoniu.zhou@nxp.com>
+
+description:
+ The CSI pixel formatting module found on i.MX95 uses packet info, pixel
+ and non-pixel data from the CSI-2 host controller and reformat them to
+ match Pixel Link(PL) definition.
+
+properties:
+ compatible:
+ const: fsl,imx95-csi-formatter
+
+ reg:
+ maxItems: 1
+ description: Register offset and size within the parent syscon
+
+ clocks:
+ maxItems: 1
+
+ power-domains:
+ maxItems: 1
+
+ ports:
+ $ref: /schemas/graph.yaml#/properties/ports
+
+ properties:
+ port@0:
+ $ref: /schemas/graph.yaml#/properties/port
+ description:
+ Input port, connects to MIPI CSI-2 receiver output (IDI interface)
+
+ port@1:
+ $ref: /schemas/graph.yaml#/properties/port
+ description:
+ Output port, connects to ISI input via Pixel Link (PL)
+
+ required:
+ - port@0
+ - port@1
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - power-domains
+ - ports
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/nxp,imx95-clock.h>
+
+ formatter@20 {
+ compatible = "fsl,imx95-csi-formatter";
+ reg = <0x20 0x100>;
+ clocks = <&cameramix_csr IMX95_CLK_CAMBLK_CSI2_FOR0>;
+ power-domains = <&scmi_devpd 3>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 {
+ reg = <0>;
+
+ endpoint {
+ remote-endpoint = <&mipi_csi_0_out>;
+ };
+ };
+
+ port@1 {
+ reg = <1>;
+
+ endpoint {
+ remote-endpoint = <&isi_in_2>;
+ };
+ };
+ };
+ };
--
2.34.1
^ permalink raw reply related
* [PATCH v10 1/4] dt-bindings: clock: imx95-blk-ctl: Use single quotes consistently
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou
In-Reply-To: <20260618-csi_formatter-v10-0-f23830312ba5@oss.nxp.com>
From: Guoniu Zhou <guoniu.zhou@nxp.com>
Change "clocks" to 'clocks' in the description to match the quote style
used for property names like '#clock-cells' throughout the file.
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- New patch to fix inconsistent quote usage (Krzysztof Kozlowski)
---
Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
index 27403b4c52d6..534fa219d9f9 100644
--- a/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
+++ b/Documentation/devicetree/bindings/clock/nxp,imx95-blk-ctl.yaml
@@ -36,7 +36,7 @@ properties:
const: 1
description:
The clock consumer should specify the desired clock by having the clock
- ID in its "clocks" phandle cell. See
+ ID in its 'clocks' phandle cell. See
include/dt-bindings/clock/nxp,imx95-clock.h
required:
--
2.34.1
^ permalink raw reply related
* [PATCH v10 0/4] media: nxp: Add CSI Pixel Formatter support
From: guoniu.zhou @ 2026-06-18 9:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Shawn Guo, Sascha Hauer, Pengutronix Kernel Team,
Fabio Estevam, Laurent Pinchart, Frank Li, Abel Vesa, Peng Fan,
Michael Turquette, Stephen Boyd
Cc: imx, linux-media, devicetree, linux-arm-kernel, linux-kernel,
linux-clk, Guoniu Zhou, Frank Li
CSI Pixel Formatter is a module found on i.MX95. It could unpack the
pixels received by the formatter and reformat them to meet the pixel
link format requirement.
This patch series adds a new V4L2 driver for CSI Pixel Formatter.
Background
----------
The Camera CSR binding was upstreamed during the pre-silicon phase when
the hardware integration details were not fully finalized. At that time,
the syscon binding focused on the CSR's primary role as a shared register
provider for clock gating, QoS, and error handling across Camera domain IPs.
However, the Camera CSR hardware is actually a mixed-function register
container with varying control levels over different IP blocks:
- ISP/CSI: Clock gating only
- ISI: QoS configuration
- CSI: Buffer overflow handling
- LPCAC: Error handling only
- ISP: Pixel link selection
- Formatter: Complete IP control (full register set)
Why Child Nodes Now
-------------------
The CSI formatter is unique among these IPs - it's an independent hardware
block whose complete control registers reside within the CSR address
space, not just a few configuration bits. This architectural detail was
not reflected in the original binding because:
1. The pre-silicon binding focused on simple register-level controls
2. The syscon pattern for complete IP control was not initially considered
Discussion with original CSR author(Peng Fan) confirmed this reflects the
intended hardware design that wasn't fully captured during pre-silicon
upstreaming.
v4l2-compliance 1.28.1-5233, 64 bits, 64-bit time_t
v4l2-compliance SHA: fc15e229d9d3 2024-07-23 19:22:15
Compliance test for device /dev/v4l-subdev9:
Driver Info:
Driver version : 7.1.0
Capabilities : 0x00000002
Streams Support
Client Capabilities: 0x0000000000000003
streams interval-uses-which
Required ioctls:
test VIDIOC_SUDBEV_QUERYCAP: OK
test invalid ioctls: OK
Allow for multiple opens:
test second /dev/v4l-subdev9 open: OK
test VIDIOC_SUBDEV_QUERYCAP: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_LOG_STATUS: OK (Not Supported)
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
test VIDIOC_G/S_EDID: OK (Not Supported)
Sub-Device routing ioctls:
test Try VIDIOC_SUBDEV_G_ROUTING/VIDIOC_SUBDEV_S_ROUTING: OK
test Active VIDIOC_SUBDEV_G_ROUTING/VIDIOC_SUBDEV_S_ROUTING: OK
Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported)
test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported)
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported)
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 0 Private Controls: 0
Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK (Not Supported)
test VIDIOC_G/S_PARM: OK (Not Supported)
test VIDIOC_G_FBUF: OK (Not Supported)
test VIDIOC_G_FMT: OK (Not Supported)
test VIDIOC_TRY_FMT: OK (Not Supported)
test VIDIOC_S_FMT: OK (Not Supported)
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK (Not Supported)
test CREATE_BUFS maximum buffers: OK
test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK (Not Supported)
test Requests: OK (Not Supported)
Total for device /dev/v4l-subdev9: 47, Succeeded: 47, Failed: 0, Warnings: 0
Signed-off-by: Guoniu Zhou <guoniu.zhou@nxp.com>
---
Changes in v10:
- Rebase to latest media/next
- [NEW PATCH] Use single quotes consistently (Krzysztof Kozlowski)
- Drop syscon parent node from example
- Drop Reviewed-by tags from Frank and Krzysztof due to binding changes,
requesting re-review
- Add description for reg property
- Add space after formatter@20 before opening brace in example
- Enhance the port description with more detailed information
- Delete the blank line immediately following the endpoint in example
- Use single quotes for regex pattern to be consistent (Krzysztof Kozlowski)
- Add formatter subnode binding and camera-csr syscon example
- Update commit title and message
- Use u8 for vc in csi_formatter_get_vc() and drop vc < 0 check
- Add MFD_SYSCON dependency to Kconfig
- Fix stream/VC mapping potential mismatch in start/stop_stream functions
- Link to v9: https://lore.kernel.org/r/20260526-csi_formatter-v9-0-ca3d8c334c39@oss.nxp.com
Changes in v9:
- [NEW PATCH] Fix formatter as syscon child node issue
- Link to v8: https://lore.kernel.org/r/20260525-csi_formatter-v8-0-6b646231224b@oss.nxp.com
Changes in v8:
- Rebase to latest media/next
- Use standard port reference instead of video-interfaces.yaml
- Add parent syscon node in example to show device integration
- Remove fmt field and look up format from subdev state instead
- Unify function and structure naming to use csi_formatter_ prefix
- Remove misleading alignment comment from set_fmt function
- Optimize get_frame_desc to call once per start_stream
- Replace V4L2_FRAME_DESC_ENTRY_MAX with CSI_FORMATTER_VC_NUM in loops
- Remove redundant debug message in enable_streams
- Use MEDIA_PAD_FL_MUST_CONNECT flag instead of manual link check
- Link to v7: https://lore.kernel.org/r/20260518-csi_formatter-v7-0-562b750557e3@oss.nxp.com
Changes in v7:
- Change compatible to imx95-csi-formatter as IP is i.MX95 specific per Marco's suggestion
Link: https://lore.kernel.org/linux-media/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com/T/#mcd135b3de179b3cb69daa1fd6e0e8e27c85b3332
- Update references from imx9 to imx95 for consistency with dt-bindings
- Enable PM runtime before async registration
- Link to v6: https://lore.kernel.org/r/20260511-csi_formatter-v6-0-01028e312e2b@oss.nxp.com
Changes in v6:
- Rebase to latest media/next
- Update v4l2-compliace test
- Remove unused header includes
- Unify macro naming: VCx/VCX -> VC and parameter x -> vc
- Remove unused format field from csi_formatter struct
- Use compact initialization for formats array
- Make find_csi_format() return NULL instead of default format
- Use unsigned int for array index in find_csi_format()
- Add err_ prefix to error handling labels
- Add v4l2_subdev_cleanup() and reorder cleanup sequence
- Update enable_streams debug output format
- Rename VC_MAX to VC_NUM and fix boundary check
- Update CSI formatter Kconfig description
- Use v4l2_subdev_get_frame_desc_passthrough() helper
- Fix error paths in async registration and probe
- Add mutex to protect enabled_streams
- Switch to devm_pm_runtime_enable()
- Remove redundant num_routes check in set_routing
- Optimize get_index_by_dt() and add warning for unsupported type
- csi_formatter_start/stop_stream: Process all streams in mask
- Link to v5: https://lore.kernel.org/r/20260123-csi_formatter-v5-0-d5b803f867bf@nxp.com
Changes in v5:
- Remove CSI_FORMATTER_DRV_NAME macro since only use once.
- Remove sd->owner = THIS_MODULE;
- Simplify code by using DEFINE_RUNTIME_DEV_PM_OPS macro.
- Link to v4: https://lore.kernel.org/r/20260122-csi_formatter-v4-0-6f6fcad1c33a@nxp.com
Changes in v4:
- Rebase to latest media/next.
- Add comments to describe the index field in formatter_dt_to_index_map array.
- Link to v3: https://lore.kernel.org/r/20251219-csi_formatter-v3-0-8680d6d87091@nxp.com
Changes in v3:
- Rename nxp,imx9-csi-formatter.yaml to fsl,imx9-csi-formatter.yaml.
- Drop clock-names property.
- Drop macro IMX95_PD_CAMERA definition and use a constant directly.
[PATCH 1/2] media: dt-bindings: Add CSI Pixel Formatter DT bindings
- Remove the assignment driver.owner = THIS_MODULE.
- Assign struct fwnode_handle *ep __free(fwnode_handle) when definition.
- Update yaml file name for csi formatter in MAINTAINERS.
[PATCH 2/2] media: nxp: Add i.MX9 CSI pixel formatter v4l2 driver
- Link to v2: https://lore.kernel.org/r/20251217-csi_formatter-v2-0-62168af80210@nxp.com
Changes in v2:
- Delete "|" for description key.
- Add empty line between child node and property.
- Delete labels for endpoint of child nodes.
[PATCH 1/2] media: dt-bindings: Add CSI Pixel Formatter DT bindings
- Update commit message.
- Use the value defined by bellow macros directly since they are used only once.
#define CSI_FORMATTER_DEF_MBUS_CODE MEDIA_BUS_FMT_UYVY8_1X16
#define CSI_FORMATTER_DEF_PIX_WIDTH 1920U
#define CSI_FORMATTER_DEF_PIX_HEIGHT 1080U
#define CSI_FORMATTER_MAX_PIX_WIDTH 0xffff
#define CSI_FORMATTER_MAX_PIX_HEIGHT 0xffff
- Use macro pm_ptr() to fix build warning when CONFIG_PM is disabled.
- Finish route loop by break statement, instead of goto.
- Return dev_err_probe() when meet errors in probe() function instead of dev_err().
- Remove MODULE_ALIAS().
- Refine .enable(.dsable)_stream callback implementation, include bellow changes:
Add stream checking.
Fix potential pm runtime count unbalance issue.
Add stop stream error handling when enabling remote subdev stream.
- Use __free(fwnode_handle) to drop reference to a device node automatically.
[PATCH 2/2] media: nxp: Add i.MX9 CSI pixel formatter v4l2 driver
- Link to v1: https://lore.kernel.org/r/20251203-csi_formatter-v1-0-eb9e1147b49e@nxp.com
---
Guoniu Zhou (4):
dt-bindings: clock: imx95-blk-ctl: Use single quotes consistently
media: dt-bindings: Add CSI Pixel Formatter DT bindings
dt-bindings: clock: imx95-blk-ctl: Define formatter child node schema
media: nxp: Add i.MX95 CSI pixel formatter v4l2 driver
.../bindings/clock/nxp,imx95-blk-ctl.yaml | 66 +-
.../bindings/media/fsl,imx95-csi-formatter.yaml | 88 +++
MAINTAINERS | 8 +
drivers/media/platform/nxp/Kconfig | 15 +
drivers/media/platform/nxp/Makefile | 1 +
drivers/media/platform/nxp/imx95-csi-formatter.c | 775 +++++++++++++++++++++
6 files changed, 951 insertions(+), 2 deletions(-)
---
base-commit: 06cb687a5132fcffe624c0070576ab852ac6b568
change-id: 20251125-csi_formatter-e6d29316dce6
Best regards,
--
Guoniu Zhou <guoniu.zhou@oss.nxp.com>
^ permalink raw reply
* [PATCH net v2] net: ethernet: ti: icssg: guard PA stat lookups
From: Philippe Schenker @ 2026-06-18 9:30 UTC (permalink / raw)
To: netdev
Cc: Philippe Schenker, Simon Horman, danishanwar, rogerq,
linux-arm-kernel, stable, Andrew Lunn, David Carlier,
David S. Miller, Eric Dumazet, Jacob Keller, Jakub Kicinski,
Kevin Hao, Meghana Malladi, Paolo Abeni, Vadim Fedorenko,
linux-kernel
From: Philippe Schenker <philippe.schenker@impulsing.ch>
icssg_ndo_get_stats64() unconditionally calls emac_get_stat_by_name()
with FW PA stat names regardless of whether the PA stats block is
present on the hardware. emac_get_stat_by_name() already guards the
PA stats lookup with `if (emac->prueth->pa_stats)`; when that pointer
is NULL the lookup falls through to netdev_err() and returns -EINVAL.
Because ndo_get_stats64 is polled regularly by the networking stack
this produces thousands of log entries of the form:
icssg-prueth icssg1-eth end0: Invalid stats FW_RX_ERROR
A secondary consequence is that the int(-EINVAL) return value is
implicitly widened to a near-ULLONG_MAX unsigned value when accumulated
into the __u64 fields of rtnl_link_stats64, silently corrupting the
rx_errors, rx_dropped and tx_dropped counters reported by `ip -s link`.
Every other PA-aware code path in the driver is already guarded with
the same `if (emac->prueth->pa_stats)` check. Apply the same guard
here.
Fixes: 0d15a26b247d ("net: ti: icssg-prueth: Add ICSSG FW Stats")
Signed-off-by: Philippe Schenker <philippe.schenker@impulsing.ch>
Reviewed-by: Simon Horman <horms@kernel.org>
Cc: danishanwar@ti.com
Cc: rogerq@kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: stable@vger.kernel.org
---
Changes in v2:
- Removed newline between Fixes tag and Signed-off-by
- Use return in if statement to guard so we get rid
of the 80 char warnings.
- Added Simon's Reviewed-by. Thanks!
drivers/net/ethernet/ti/icssg/icssg_common.c | 49 +++++++++++---------
1 file changed, 28 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ethernet/ti/icssg/icssg_common.c b/drivers/net/ethernet/ti/icssg/icssg_common.c
index a28a608f9bf4..d9af6419e032 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_common.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_common.c
@@ -1628,28 +1628,35 @@ void icssg_ndo_get_stats64(struct net_device *ndev,
stats->rx_over_errors = emac_get_stat_by_name(emac, "rx_over_errors");
stats->multicast = emac_get_stat_by_name(emac, "rx_multicast_frames");
- stats->rx_errors = ndev->stats.rx_errors +
- emac_get_stat_by_name(emac, "FW_RX_ERROR") +
- emac_get_stat_by_name(emac, "FW_RX_EOF_SHORT_FRMERR") +
- emac_get_stat_by_name(emac, "FW_RX_B0_DROP_EARLY_EOF") +
- emac_get_stat_by_name(emac, "FW_RX_EXP_FRAG_Q_DROP") +
- emac_get_stat_by_name(emac, "FW_RX_FIFO_OVERRUN");
- stats->rx_dropped = ndev->stats.rx_dropped +
- emac_get_stat_by_name(emac, "FW_DROPPED_PKT") +
- emac_get_stat_by_name(emac, "FW_INF_PORT_DISABLED") +
- emac_get_stat_by_name(emac, "FW_INF_SAV") +
- emac_get_stat_by_name(emac, "FW_INF_SA_DL") +
- emac_get_stat_by_name(emac, "FW_INF_PORT_BLOCKED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_TAGGED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_PRIOTAGGED") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_NOTAG") +
- emac_get_stat_by_name(emac, "FW_INF_DROP_NOTMEMBER");
+ stats->rx_errors = ndev->stats.rx_errors;
+ stats->rx_dropped = ndev->stats.rx_dropped;
stats->tx_errors = ndev->stats.tx_errors;
- stats->tx_dropped = ndev->stats.tx_dropped +
- emac_get_stat_by_name(emac, "FW_RTU_PKT_DROP") +
- emac_get_stat_by_name(emac, "FW_TX_DROPPED_PACKET") +
- emac_get_stat_by_name(emac, "FW_TX_TS_DROPPED_PACKET") +
- emac_get_stat_by_name(emac, "FW_TX_JUMBO_FRM_CUTOFF");
+ stats->tx_dropped = ndev->stats.tx_dropped;
+
+ if (!emac->prueth->pa_stats)
+ return;
+
+ stats->rx_errors +=
+ emac_get_stat_by_name(emac, "FW_RX_ERROR") +
+ emac_get_stat_by_name(emac, "FW_RX_EOF_SHORT_FRMERR") +
+ emac_get_stat_by_name(emac, "FW_RX_B0_DROP_EARLY_EOF") +
+ emac_get_stat_by_name(emac, "FW_RX_EXP_FRAG_Q_DROP") +
+ emac_get_stat_by_name(emac, "FW_RX_FIFO_OVERRUN");
+ stats->rx_dropped +=
+ emac_get_stat_by_name(emac, "FW_DROPPED_PKT") +
+ emac_get_stat_by_name(emac, "FW_INF_PORT_DISABLED") +
+ emac_get_stat_by_name(emac, "FW_INF_SAV") +
+ emac_get_stat_by_name(emac, "FW_INF_SA_DL") +
+ emac_get_stat_by_name(emac, "FW_INF_PORT_BLOCKED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_TAGGED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_PRIOTAGGED") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_NOTAG") +
+ emac_get_stat_by_name(emac, "FW_INF_DROP_NOTMEMBER");
+ stats->tx_dropped +=
+ emac_get_stat_by_name(emac, "FW_RTU_PKT_DROP") +
+ emac_get_stat_by_name(emac, "FW_TX_DROPPED_PACKET") +
+ emac_get_stat_by_name(emac, "FW_TX_TS_DROPPED_PACKET") +
+ emac_get_stat_by_name(emac, "FW_TX_JUMBO_FRM_CUTOFF");
}
EXPORT_SYMBOL_GPL(icssg_ndo_get_stats64);
--
2.54.0
base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
branch: fix-icssg_common-pa-stats-errors__master-7-1
^ permalink raw reply related
* Re: [PATCH net] net: ethernet: ti: icssg: guard PA stat lookups
From: Philippe Schenker @ 2026-06-18 9:29 UTC (permalink / raw)
To: Simon Horman
Cc: netdev, danishanwar, rogerq, linux-arm-kernel, stable,
Andrew Lunn, David Carlier, David S. Miller, Eric Dumazet,
Jacob Keller, Jakub Kicinski, Kevin Hao, Meghana Malladi,
Paolo Abeni, Vadim Fedorenko, linux-kernel
In-Reply-To: <20260618091004.GG827683@horms.kernel.org>
[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]
Hi Simon
Thanks for the review and I'll send a v2 with that blank line removed.
Saw it right after sending the patch.
Philippe
On Thu, 2026-06-18 at 10:10 +0100, Simon Horman wrote:
> On Tue, Jun 16, 2026 at 04:35:34PM +0200, Philippe Schenker wrote:
> > From: Philippe Schenker <philippe.schenker@impulsing.ch>
> >
> > icssg_ndo_get_stats64() unconditionally calls
> > emac_get_stat_by_name()
> > with FW PA stat names regardless of whether the PA stats block is
> > present on the hardware. emac_get_stat_by_name() already guards
> > the
> > PA stats lookup with `if (emac->prueth->pa_stats)`; when that
> > pointer
> > is NULL the lookup falls through to netdev_err() and returns -
> > EINVAL.
> > Because ndo_get_stats64 is polled regularly by the networking stack
> > this produces thousands of log entries of the form:
> >
> > icssg-prueth icssg1-eth end0: Invalid stats FW_RX_ERROR
> >
> > A secondary consequence is that the int(-EINVAL) return value is
> > implicitly widened to a near-ULLONG_MAX unsigned value when
> > accumulated
> > into the __u64 fields of rtnl_link_stats64, silently corrupting the
> > rx_errors, rx_dropped and tx_dropped counters reported by `ip -s
> > link`.
> >
> > Every other PA-aware code path in the driver is already guarded
> > with
> > the same `if (emac->prueth->pa_stats)` check. Apply the same guard
> > here.
> >
> > Fixes: 0d15a26b247d ("net: ti: icssg-prueth: Add ICSSG FW Stats")
>
> nit: no blank line between tags
>
> >
> > Signed-off-by: Philippe Schenker <philippe.schenker@impulsing.ch>
> >
> > Cc: danishanwar@ti.com
> > Cc: rogerq@kernel.org
> > Cc: linux-arm-kernel@lists.infradead.org
> > Cc: stable@vger.kernel.org
>
> Reviewed-by: Simon Horman <horms@kernel.org>
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply
* Re: [PATCH v2 2/6] iommu/arm-smmu: Add interconnect bandwidth voting support
From: Konrad Dybcio @ 2026-06-18 9:28 UTC (permalink / raw)
To: Bibek Kumar Patro, Dmitry Baryshkov
Cc: Will Deacon, Robin Murphy, Joerg Roedel, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
linux-arm-kernel, iommu, devicetree, linux-kernel, linux-arm-msm
In-Reply-To: <ea345832-711d-4e32-af0f-bc74d8646aef@oss.qualcomm.com>
On 6/17/26 4:26 PM, Bibek Kumar Patro wrote:
>
>
> On 6/16/2026 5:51 AM, Dmitry Baryshkov wrote:
>> On Mon, Jun 15, 2026 at 06:36:51PM +0530, Bibek Kumar Patro wrote:
>>>
>>>
>>> On 6/8/2026 7:25 PM, Dmitry Baryshkov wrote:
>>>> On Tue, May 26, 2026 at 08:12:03PM +0530, Bibek Kumar Patro wrote:
>>>>> On some SoCs the SMMU registers require an active interconnect
>>>>> bandwidth vote to be accessible. While other clients typically
>>>>> satisfy this requirement implicitly, certain corner cases (e.g.
>>>>> during sleep/wakeup transitions) can leave the SMMU without a
>>>>> vote, causing intermittent register access failures.
>>>>>
>>>>> Add support for an optional interconnect path to the arm-smmu
>>>>> driver and vote for bandwidth while the SMMU is active. The path
>>>>> is acquired from DT if present and ignored otherwise.
>>>>>
>>>>> The bandwidth vote is enabled before accessing SMMU registers
>>>>> during probe and runtime resume, and released during runtime
>>>>> suspend and on error paths.
>>>>>
>>>>> Generally, from an architectural perspective, GEM_NOC and DDR are
>>>>> expected to have an active vote whenever the adreno_smmu block is
>>>>> powered on. In most common use cases, this requirement is implicitly
>>>>> satisfied because other GPU-related clients (for example, the GMU
>>>>> device) already hold a GEM_NOC vote when adreno_smmu is enabled.
>>>>>
>>>>> However, there are certain corner cases, such as during sleep/wakeup
>>>>> transitions, where the GEM_NOC vote can be removed before adreno_smmu
>>>>> is powered down. If adreno_smmu is then accessed while the interconnect
>>>>> vote is missing, it can lead to the observed failures. Because of the
>>>>> precise ordering involved, this scenario is difficult to reproduce
>>>>> consistently.
>>>>> (also GDSC is involved in adreno usecases can have an independent vote)
>>>>>
>>>>> Signed-off-by: Bibek Kumar Patro <bibek.patro@oss.qualcomm.com>
>>>>> ---
>>>>> drivers/iommu/arm/arm-smmu/arm-smmu.c | 57 +++++++++++++++++++++++++++++++++--
>>>>> drivers/iommu/arm/arm-smmu/arm-smmu.h | 2 ++
>>>>> 2 files changed, 57 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> index 0bd21d206eb3e75c3b9fb1364cdc92e82c5aa499..07c7e44ec6a5bd1488f00f87d859a20495e46601 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
>>>>> @@ -53,6 +53,11 @@
>>>>> #define MSI_IOVA_BASE 0x8000000
>>>>> #define MSI_IOVA_LENGTH 0x100000
>>>>> +/* Interconnect bandwidth vote values for the SMMU register access path */
>>>>> +#define ARM_SMMU_ICC_AVG_BW 0
>>>>> +#define ARM_SMMU_ICC_PEAK_BW_HIGH 1000
>>>>
>>>> totally random numbers, which might be different for non-Qualcomm platform.
>>>>
>>>
>>> Ideally, any non-zero value would be enough to keep the path active.
>>
>> This is true for Qualcomm devices. However, you are adding this to a
>> generic code.
>>
>>> Here 1 Would be enough to keep the path active, but might be too small to
>>> reliably keep the bus active.
>>> Other is UINT_MAX, which will reliably keep the bus active but might cause a
>>> power penalty.
>>>
>>> #define ARM_SMMU_ICC_PEAK_BW_HIGH UINT_MAX
>>>
>>> seems to be suitable here to reliably keep the bus active by BCM
>>> for both Qualcomm and non-Qualcomm platforms (with some power penalty).
>>>
>>> LMK, if you feel otherwise.
>>
>> Shift it to the qcom instance or provide platform-specific values? (My
>> preference would be towards the first solution).
>>
>
>
> To support platform-specific values, we may need to introduce a LUT-based approach in the driver. (Bandwidth voting values cannot be placed in device-tree property IIRC ?)
>
> Currently, all Qualcomm platforms use 0x1000 for SMMU ICC voting. I
(you used decimal 1000)
> can evaluate if this could be moved to a Qualcomm-specific
> implementation.
Add a vendor hook to arm_smmu_runtime_suspend/resume and handle it within
the QC driver
Konrad
^ permalink raw reply
* RE: [PATCH v4 3/5] dt-bindings: clock: cix,sky1-audss-clock: add audss clock controller
From: Joakim Zhang @ 2026-06-18 9:27 UTC (permalink / raw)
To: Philipp Zabel, Conor Dooley
Cc: mturquette@baylibre.com, sboyd@kernel.org, bmasney@redhat.com,
robh@kernel.org, krzk+dt@kernel.org, conor+dt@kernel.org,
Gary Yang, cix-kernel-upstream, linux-clk@vger.kernel.org,
devicetree@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org
In-Reply-To: <cd63c1ab9831fdfd39aad8d1a40c4702a9cbb158.camel@pengutronix.de>
Hello,
> -----Original Message-----
> From: Philipp Zabel <p.zabel@pengutronix.de>
> Sent: Thursday, June 18, 2026 4:30 PM
> To: Joakim Zhang <joakim.zhang@cixtech.com>; Conor Dooley
> <conor@kernel.org>
> Cc: mturquette@baylibre.com; sboyd@kernel.org; bmasney@redhat.com;
> robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org; Gary Yang
> <Gary.Yang@cixtech.com>; cix-kernel-upstream <cix-kernel-
> upstream@cixtech.com>; linux-clk@vger.kernel.org;
> devicetree@vger.kernel.org; linux-kernel@vger.kernel.org; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: [PATCH v4 3/5] dt-bindings: clock: cix,sky1-audss-clock: add audss
> clock controller
>
> EXTERNAL EMAIL
>
> CAUTION: Suspicious Email from unusual domain.
>
> On Do, 2026-06-18 at 01:43 +0000, Joakim Zhang wrote:
> > Hello,
> >
> >
> > > -----Original Message-----
> > > From: Conor Dooley <conor@kernel.org>
> > > Sent: Wednesday, June 17, 2026 11:56 PM
> > > To: Joakim Zhang <joakim.zhang@cixtech.com>
> > > Cc: mturquette@baylibre.com; sboyd@kernel.org; bmasney@redhat.com;
> > > robh@kernel.org; krzk+dt@kernel.org; conor+dt@kernel.org;
> > > p.zabel@pengutronix.de; Gary Yang <gary.yang@cixtech.com>;
> > > cix-kernel- upstream <cix-kernel-upstream@cixtech.com>;
> > > linux-clk@vger.kernel.org; devicetree@vger.kernel.org;
> > > linux-kernel@vger.kernel.org; linux-arm- kernel@lists.infradead.org
> > > Subject: Re: [PATCH v4 3/5] dt-bindings: clock:
> > > cix,sky1-audss-clock: add audss clock controller
> > >
> > > On Wed, Jun 17, 2026 at 02:04:35PM +0800, joakim.zhang@cixtech.com
> wrote:
> > > > From: Joakim Zhang <joakim.zhang@cixtech.com>
> > > >
> > > > The AUDSS CRU contains an internal clock tree of muxes, dividers
> > > > and gates for DSP, I2S, HDA, DMAC and related blocks. The clock
> > > > provider is a child node of the cix,sky1-audss-system-control
> > > > syscon and accesses registers through the parent MMIO region.
> > >
> > > Why can this not just be part of the parent syscon node?
> >
> > The clock and reset blocks are handled by different subsystems and
> maintainers (clk vs reset). Putting the clock provider on the parent syscon node
> would mean a single driver has to register both the reset controller and the
> clock provider on one device, which doesn't fit well.
>
> There are many examples of clock and reset drivers sharing the same node, by
> using platform_driver for one (usually clk) and auxiliary_driver for the other
> (usually reset).
OK, I will have a look. If you are also prefer to this, I will refactor the patch.
Thanks,
Joakim
^ permalink raw reply
* Re: [PATCH v2 3/6] arm64: dts: qcom: kodiak: Add GEM_NOC interconnect for adreno SMMU
From: Konrad Dybcio @ 2026-06-18 9:26 UTC (permalink / raw)
To: Bibek Kumar Patro, Dmitry Baryshkov
Cc: Will Deacon, Robin Murphy, Joerg Roedel, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
linux-arm-kernel, iommu, devicetree, linux-kernel, linux-arm-msm
In-Reply-To: <26d51cbb-2d87-4564-b3c6-cc61ab900e19@oss.qualcomm.com>
On 6/8/26 4:37 PM, Bibek Kumar Patro wrote:
>
>
> On 6/8/2026 7:27 PM, Dmitry Baryshkov wrote:
>> On Tue, May 26, 2026 at 08:12:04PM +0530, Bibek Kumar Patro wrote:
>>> On Kodiak platforms, the Adreno SMMU requires a bandwidth vote on
>>> the GEM_NOC path (MASTER_GPU_TCU -> SLAVE_EBI1) before its registers
>>> are accessible. Without this vote, the SMMU may become unreachable,
>>> leading to intermittent probe failures and runtime issues.
>>>
>>> Add the required interconnect to ensure reliable register access.
>>
>> Does it only concern the GPU SMMU? What about the APPS SMMU? Should it
>> be voting on other interconnects too? I guess so, because currently I
>> see that TBUs vote for various interconnects. BTW: should apps_smmu also
>> vote on the power domains?
>>
>
> This race mainly occurs in GPU SMMU, where the GDSC can have an
Mainly or exclusively?
Konrad
> independent vote on the Adreno SMMU. However, the GEM_NOC vote may
> already have been removed by the GPU (or any consumer of adreno_smmu,
> e.g gmu), unless it is explicitly voted by the GPU SMMU (which acts as a
> supplier for the GPU). This mismatch can lead to SHUB timeouts or NoC
> errors.
>
> Mostly this race reported in suspend/resume cycle (when gpu/gmu devices moves to slumber/suspend state before adreno_smmu powers down
> and the later doesn't have explicit interconnect voting).
>
> In the case of APPS SMMU, such a race is not expected for any known
> use case. APPS SMMU is part of a shared infrastructure block, and its
> power is typically kept enabled as long as attached master devices are
> active. Therefore, explicit power-domain voting from APPS SMMU may not
> be required.
>
> Thanks,
> Bibek
>
>
>>>
>>> Signed-off-by: Bibek Kumar Patro <bibek.patro@oss.qualcomm.com>
>>> ---
>>> arch/arm64/boot/dts/qcom/kodiak.dtsi | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/arch/arm64/boot/dts/qcom/kodiak.dtsi b/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> index fa540d8c2615dc02d941eb16bc7253204c2750bd..eefa4b836a81374ff437ab4bbcbc3fecc1590ab6 100644
>>> --- a/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> +++ b/arch/arm64/boot/dts/qcom/kodiak.dtsi
>>> @@ -3386,6 +3386,8 @@ adreno_smmu: iommu@3da0000 {
>>> power-domains = <&gpucc GPU_CC_CX_GDSC>;
>>> dma-coherent;
>>> + interconnects = <&gem_noc MASTER_GPU_TCU QCOM_ICC_TAG_ALWAYS
>>> + &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>;
>>> };
>>> gfx_0_tbu: tbu@3dd9000 {
>>>
>>> --
>>> 2.34.1
>>>
>>
>
>
^ permalink raw reply
* [PATCH v2 3/4] arm64: smp: Defer RCU reporting until after local CPU capability checks
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
To support HOTPLUG_PARALLEL on arm64, and to prevent a potential deadlock
on the control CPU, check_local_cpu_capabilities() must be executed
before cpuhp_ap_sync_alive(). This ensures that if an early capability
mismatch occurs and the AP invokes cpu_die_early(), the control CPU
can detect the boot timeout and proceed, rather than hanging
indefinitely.
Furthermore, under parallel bringup, cpuhp_ap_sync_alive() must be called
before rcutree_report_cpu_starting(). This sequence prevents a false
RCU CPU Stall Warning caused by the prolonged spin-waiting/busy-waiting
required during the AP synchronization process.
GICv3: CPU1: using allocated LPI pending table @0x0000000104160000
CPU1: Booted secondary processor 0x0000000001 [0x410fd082]
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 2-O..!: (2 GPs behind) idle=0004/1/0x4000000000000000 softirq=0/0 fqs=2625
rcu: 3-O..!: (2 GPs behind) idle=0004/1/0x4000000000000000 softirq=0/0 fqs=2625
rcu: (detected by 0, t=5252 jiffies, g=-1187, q=1 ncpus=16)
rcu: Offline CPU 2 blocking current GP.
rcu: Offline CPU 3 blocking current GP.
To avoid suspicious RCU usage, commit ce3d31ad3cac ("arm64/smp: Move
rcu_cpu_starting() earlier") move rcutree_report_cpu_starting() earlier
which is before check_local_cpu_capabilities().
But For parallel bringup, the order should be as following:
secondary_start_kernel()
-> check_local_cpu_capabilities()
-> cpu_die_early()
-> cpuhp_ap_sync_alive()
-> rcutree_report_cpu_starting()
And this required order forces standard printk/pr_* statements inside
check_local_cpu_capabilities() to execute while the secondary CPU is still
marked as offline to RCU, triggering a lockdep "suspicious RCU usage" splat
due to console semaphore operations.
So converting early capability logging and failure paths to printk_deferred().
This pushes the logs into the lockless ringbuffer without triggering console
locks or RCU validation on the offline CPU, resolving the lockdep splat while
preserving critical error messages and the strictly required initialization
order.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/cpufeature.c | 20 ++++++++++----------
arch/arm64/kernel/smp.c | 4 ++--
arch/arm64/mm/context.c | 2 +-
3 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 0552202702bf..a5e0bc4d383b 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3546,7 +3546,7 @@ static void update_cpu_capabilities(u16 scope_mask)
* system capabilities are finalised.
*/
if (!match_all && caps->desc && !caps->cpus)
- pr_info("detected: %s\n", caps->desc);
+ printk_deferred(KERN_INFO "detected: %s\n", caps->desc);
__set_bit(caps->capability, system_cpucaps);
@@ -3669,7 +3669,7 @@ static void verify_local_cpu_caps(u16 scope_mask)
}
if (i < ARM64_NCAPS) {
- pr_crit("CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",
+ printk_deferred(KERN_CRIT "CPU%d: Detected conflict for capability %d (%s), System: %d, CPU: %d\n",
smp_processor_id(), caps->capability,
caps->desc, system_has_cap, cpu_has_cap);
@@ -3697,7 +3697,7 @@ __verify_local_elf_hwcaps(const struct arm64_cpu_capabilities *caps)
for (; caps->matches; caps++)
if (cpus_have_elf_hwcap(caps) && !caps->matches(caps, SCOPE_LOCAL_CPU)) {
- pr_crit("CPU%d: missing HWCAP: %s\n",
+ printk_deferred(KERN_CRIT "CPU%d: missing HWCAP: %s\n",
smp_processor_id(), caps->desc);
cpu_die_early();
}
@@ -3716,7 +3716,7 @@ static void verify_sve_features(void)
unsigned long cpacr = cpacr_save_enable_kernel_sve();
if (vec_verify_vq_map(ARM64_VEC_SVE)) {
- pr_crit("CPU%d: SVE: vector length support mismatch\n",
+ printk_deferred(KERN_CRIT "CPU%d: SVE: vector length support mismatch\n",
smp_processor_id());
cpu_die_early();
}
@@ -3729,7 +3729,7 @@ static void verify_sme_features(void)
unsigned long cpacr = cpacr_save_enable_kernel_sme();
if (vec_verify_vq_map(ARM64_VEC_SME)) {
- pr_crit("CPU%d: SME: vector length support mismatch\n",
+ printk_deferred(KERN_CRIT "CPU%d: SME: vector length support mismatch\n",
smp_processor_id());
cpu_die_early();
}
@@ -3754,7 +3754,7 @@ static void verify_hyp_capabilities(void)
safe_vmid_bits = get_vmid_bits(safe_mmfr1);
vmid_bits = get_vmid_bits(mmfr1);
if (vmid_bits < safe_vmid_bits) {
- pr_crit("CPU%d: VMID width mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: VMID width mismatch\n", smp_processor_id());
cpu_die_early();
}
@@ -3763,7 +3763,7 @@ static void verify_hyp_capabilities(void)
ID_AA64MMFR0_EL1_PARANGE_SHIFT);
ipa_max = id_aa64mmfr0_parange_to_phys_shift(parange);
if (ipa_max < get_kvm_ipa_limit()) {
- pr_crit("CPU%d: IPA range mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: IPA range mismatch\n", smp_processor_id());
cpu_die_early();
}
}
@@ -3776,7 +3776,7 @@ static void verify_mpam_capabilities(void)
if (FIELD_GET(ID_AA64PFR0_EL1_MPAM_MASK, cpu_idr) !=
FIELD_GET(ID_AA64PFR0_EL1_MPAM_MASK, sys_idr)) {
- pr_crit("CPU%d: MPAM version mismatch\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: MPAM version mismatch\n", smp_processor_id());
cpu_die_early();
}
@@ -3784,7 +3784,7 @@ static void verify_mpam_capabilities(void)
sys_idr = read_sanitised_ftr_reg(SYS_MPAMIDR_EL1);
if (FIELD_GET(MPAMIDR_EL1_HAS_HCR, cpu_idr) !=
FIELD_GET(MPAMIDR_EL1_HAS_HCR, sys_idr)) {
- pr_crit("CPU%d: Missing MPAM HCR\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: Missing MPAM HCR\n", smp_processor_id());
cpu_die_early();
}
@@ -3793,7 +3793,7 @@ static void verify_mpam_capabilities(void)
sys_partid_max = FIELD_GET(MPAMIDR_EL1_PARTID_MAX, sys_idr);
sys_pmg_max = FIELD_GET(MPAMIDR_EL1_PMG_MAX, sys_idr);
if (cpu_partid_max < sys_partid_max || cpu_pmg_max < sys_pmg_max) {
- pr_crit("CPU%d: MPAM PARTID/PMG max values are mismatched\n", smp_processor_id());
+ printk_deferred(KERN_CRIT "CPU%d: MPAM PARTID/PMG max values are mismatched\n", smp_processor_id());
cpu_die_early();
}
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 6bc90ee4820a..52edabc13d51 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -215,7 +215,6 @@ asmlinkage notrace void secondary_start_kernel(void)
if (system_uses_irq_prio_masking())
init_gic_priority_masking();
- rcutree_report_cpu_starting(cpu);
trace_hardirqs_off();
/*
@@ -224,6 +223,7 @@ asmlinkage notrace void secondary_start_kernel(void)
* fail to come online.
*/
check_local_cpu_capabilities();
+ rcutree_report_cpu_starting(cpu);
ops = get_cpu_ops(cpu);
if (ops->cpu_postboot)
@@ -404,7 +404,7 @@ void __noreturn cpu_die_early(void)
{
int cpu = smp_processor_id();
- pr_crit("CPU%d: will not boot\n", cpu);
+ printk_deferred(KERN_CRIT "CPU%d: will not boot\n", cpu);
/* Mark this CPU absent */
set_cpu_present(cpu, 0);
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 6b8a3245f393..9b5ab56aad5a 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -70,7 +70,7 @@ void verify_cpu_asid_bits(void)
* We cannot decrease the ASID size at runtime, so panic if we support
* fewer ASID bits than the boot CPU.
*/
- pr_crit("CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
+ printk_deferred(KERN_CRIT "CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
smp_processor_id(), asid, asid_bits);
cpu_panic_kernel(smp_processor_id());
}
--
2.34.1
^ permalink raw reply related
* [PATCH v2 4/4] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
Support for parallel secondary CPU bringup is already utilized by x86,
MIPS, and RISC-V. This patch brings this capability to the arm64
architecture.
Rework the global `secondary_data` accessed during early boot into
a per-CPU array `cpu_boot_data` to allow secondary CPUs to boot
in parallel.
And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.
To fully enable HOTPLUG_PARALLEL, this patch implements:
1) An arm64-specific arch_cpuhp_init_parallel_bringup() handler.
2) An arm64-specific arch_cpuhp_kick_ap_alive() handler.
3) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel().
Tested natively with ATF on QEMU arm64 virt machine with 64 cores
and also tested with KVM arm64 guest with 128 vCPUs.
Bringup Time Comparison (ms, lower is better):
| Platform | Baseline| P=0 | P=1 | Delta(%)|
| -------------------------------- | ------- | ------- | ------ | ------- |
| 128 vCPUs KVM (256-core HIP09) | 1921.5 | 1895.9 | 2776.9 | -44.52% |
| 48 vCPUs KVM (64-core Cortex-A72)| 3644.7 | 3883.6 | 4406.3 | -20.9% |
| 64-core ATF QEMU | 2075.8 | 2080.7 | 1653.4 | 20.34% |
| 192-core server(HIP12) | 14619.2 | 14619.1 | 8589.4 | 41.21% |
| 32-core board | 2776.5 | 2881.0 | 1045.0 | 62.36% |
Tested-by: Michael Kelley <mhklinux@outlook.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/smp.h | 11 ++++++++++
arch/arm64/kernel/asm-offsets.c | 4 ++++
arch/arm64/kernel/head.S | 36 +++++++++++++++++++++++++++++++++
arch/arm64/kernel/smp.c | 33 ++++++++++++++++++++++++++++++
5 files changed, 85 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9091c67e1cc2..8735e9d8ed13 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -113,6 +113,7 @@ config ARM64
select CPUMASK_OFFSTACK if NR_CPUS > 256
select DCACHE_WORD_ACCESS
select HAVE_EXTRA_IPI_TRACEPOINTS
+ select HOTPLUG_PARALLEL if SMP && HOTPLUG_CPU
select DYNAMIC_FTRACE if FUNCTION_TRACER
select DMA_BOUNCE_UNALIGNED_KMALLOC
select DMA_DIRECT_REMAP
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index e2151a01731f..30025030489c 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -92,7 +92,14 @@ struct secondary_data {
long status;
};
+#ifdef CONFIG_HOTPLUG_PARALLEL
+static_assert((sizeof(struct secondary_data) & (sizeof(struct secondary_data) - 1)) == 0,
+ "secondary_data size must be a power of 2 for assembly lsl assembly!");
+
+extern struct secondary_data cpu_boot_data[NR_CPUS];
+#else
extern struct secondary_data secondary_data;
+#endif
extern long __early_cpu_boot_status;
extern void secondary_entry(void);
@@ -124,7 +131,11 @@ static inline void __noreturn cpu_park_loop(void)
static inline void update_cpu_boot_status(unsigned int cpu, int val)
{
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ WRITE_ONCE(cpu_boot_data[cpu].status, val);
+#else
WRITE_ONCE(secondary_data.status, val);
+#endif
/* Ensure the visibility of the status update */
dsb(ishst);
}
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index b6367ff3a49c..0a0aa965dbb4 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -11,6 +11,7 @@
#include <linux/arm_sdei.h>
#include <linux/sched.h>
#include <linux/ftrace.h>
+#include <linux/log2.h>
#include <linux/kexec.h>
#include <linux/mm.h>
#include <linux/kvm_host.h>
@@ -97,6 +98,9 @@ int main(void)
BLANK();
#endif
DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ DEFINE(SECONDARY_DATA_SHIFT, ilog2(sizeof(struct secondary_data)));
+#endif
BLANK();
DEFINE(FTR_OVR_VAL_OFFSET, offsetof(struct arm64_ftr_override, val));
DEFINE(FTR_OVR_MASK_OFFSET, offsetof(struct arm64_ftr_override, mask));
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca..000ab1acf7c5 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -12,6 +12,7 @@
#include <linux/linkage.h>
#include <linux/init.h>
#include <linux/pgtable.h>
+#include <linux/threads.h>
#include <asm/asm_pointer_auth.h>
#include <asm/assembler.h>
@@ -378,6 +379,33 @@ alternative_else_nop_endif
br x8
SYM_FUNC_END(secondary_startup)
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ /*
+ * Convert the physical MPIDR of the current secondary CPU
+ * to its logical CPUID by traversing __cpu_logical_map
+ * in parallel.
+ */
+ .macro mpidr_to_cpuid, mpidr, cpuid, tmp1, tmp2
+ mov_q \tmp1, MPIDR_HWID_BITMASK
+ and \mpidr, \mpidr, \tmp1
+
+ adr_l \tmp1, __cpu_logical_map
+ mov \cpuid, #0
+.Lfind_cpuid\@:
+ ldr \tmp2, [\tmp1, \cpuid, lsl #3]
+ cmp \tmp2, #-1
+ b.eq .Lnext_cpu\@
+ cmp \tmp2, \mpidr
+ b.eq .Lfound_cpuid\@
+.Lnext_cpu\@:
+ add \cpuid, \cpuid, #1
+ cmp \cpuid, #NR_CPUS
+ b.ne .Lfind_cpuid\@
+ b __secondary_too_slow
+.Lfound_cpuid\@:
+ .endm
+#endif
+
.text
SYM_FUNC_START_LOCAL(__secondary_switched)
mov x0, x20
@@ -391,7 +419,15 @@ SYM_FUNC_START_LOCAL(__secondary_switched)
msr vbar_el1, x5
isb
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ mrs x0, mpidr_el1
+ mpidr_to_cpuid mpidr=x0, cpuid=x2, tmp1=x1, tmp2=x3
+
+ adr_l x0, cpu_boot_data
+ add x0, x0, x2, lsl #SECONDARY_DATA_SHIFT
+#else
adr_l x0, secondary_data
+#endif
ldr x2, [x0, #CPU_BOOT_TASK]
cbz x2, __secondary_too_slow
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 52edabc13d51..f7562c38d724 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -61,6 +61,11 @@
* where to place its SVC stack
*/
struct secondary_data secondary_data;
+
+#ifdef CONFIG_HOTPLUG_PARALLEL
+struct secondary_data cpu_boot_data[NR_CPUS] ____cacheline_aligned;
+#endif
+
/* Number of CPUs which aren't online, but looping in kernel text. */
static int cpus_stuck_in_kernel;
@@ -106,8 +111,30 @@ static int boot_secondary(unsigned int cpu, struct task_struct *idle)
return -EOPNOTSUPP;
}
+#ifndef CONFIG_HOTPLUG_PARALLEL
static DECLARE_COMPLETION(cpu_running);
+#endif
+
+#ifdef CONFIG_HOTPLUG_PARALLEL
+extern const struct cpu_operations smp_spin_table_ops;
+
+/* Establish whether parallel bringup can be supported. */
+bool __init arch_cpuhp_init_parallel_bringup(void)
+{
+ int cpu = smp_processor_id();
+ const struct cpu_operations *ops = get_cpu_ops(cpu);
+ return ops && ops != &smp_spin_table_ops;
+}
+
+int arch_cpuhp_kick_ap_alive(unsigned int cpu, struct task_struct *tidle)
+{
+ cpu_boot_data[cpu].task = tidle;
+ update_cpu_boot_status(cpu, CPU_MMU_OFF);
+
+ return boot_secondary(cpu, tidle);
+}
+#else
int __cpu_up(unsigned int cpu, struct task_struct *idle)
{
int ret;
@@ -172,6 +199,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
return -EIO;
}
+#endif /* CONFIG_HOTPLUG_PARALLEL */
static void init_gic_priority_masking(void)
{
@@ -223,6 +251,9 @@ asmlinkage notrace void secondary_start_kernel(void)
* fail to come online.
*/
check_local_cpu_capabilities();
+#ifdef CONFIG_HOTPLUG_PARALLEL
+ cpuhp_ap_sync_alive();
+#endif
rcutree_report_cpu_starting(cpu);
ops = get_cpu_ops(cpu);
@@ -254,7 +285,9 @@ asmlinkage notrace void secondary_start_kernel(void)
read_cpuid_id());
update_cpu_boot_status(cpu, CPU_BOOT_SUCCESS);
set_cpu_online(cpu, true);
+#ifndef CONFIG_HOTPLUG_PARALLEL
complete(&cpu_running);
+#endif
/*
* Secondary CPUs enter the kernel with all DAIF exceptions masked.
--
2.34.1
^ permalink raw reply related
* [PATCH v2 2/4] arm64: smp: Pass CPU ID to update_cpu_boot_status()
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
To support CONFIG_HOTPLUG_PARALLEL, the CPU boot status tracking must
be refactored from a single global variable (secondary_data.status)
to a per-CPU tracking structure to prevent multi-core race conditions.
Add a 'cpu' parameter to update_cpu_boot_status() and update all its
callsites to pass the corresponding CPU ID. This allows updating the
boot status at a per-CPU granularity during parallel bringup.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/smp.h | 6 +++---
arch/arm64/kernel/cpufeature.c | 2 +-
arch/arm64/kernel/smp.c | 8 ++++----
arch/arm64/mm/context.c | 2 +-
4 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 10ea4f543069..e2151a01731f 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -122,7 +122,7 @@ static inline void __noreturn cpu_park_loop(void)
}
}
-static inline void update_cpu_boot_status(int val)
+static inline void update_cpu_boot_status(unsigned int cpu, int val)
{
WRITE_ONCE(secondary_data.status, val);
/* Ensure the visibility of the status update */
@@ -134,9 +134,9 @@ static inline void update_cpu_boot_status(int val)
* which calls for a kernel panic. Update the boot status and park the calling
* CPU.
*/
-static inline void __noreturn cpu_panic_kernel(void)
+static inline void __noreturn cpu_panic_kernel(unsigned int cpu)
{
- update_cpu_boot_status(CPU_PANIC_KERNEL);
+ update_cpu_boot_status(cpu, CPU_PANIC_KERNEL);
cpu_park_loop();
}
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 6d53bb15cf7b..0552202702bf 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3674,7 +3674,7 @@ static void verify_local_cpu_caps(u16 scope_mask)
caps->desc, system_has_cap, cpu_has_cap);
if (cpucap_panic_on_conflict(caps))
- cpu_panic_kernel();
+ cpu_panic_kernel(smp_processor_id());
else
cpu_die_early();
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 24f8448e1fbb..6bc90ee4820a 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -118,7 +118,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
* page tables.
*/
secondary_data.task = idle;
- update_cpu_boot_status(CPU_MMU_OFF);
+ update_cpu_boot_status(cpu, CPU_MMU_OFF);
/* Now bring the CPU into our world */
ret = boot_secondary(cpu, idle);
@@ -252,7 +252,7 @@ asmlinkage notrace void secondary_start_kernel(void)
pr_info("CPU%u: Booted secondary processor 0x%010lx [0x%08x]\n",
cpu, (unsigned long)mpidr,
read_cpuid_id());
- update_cpu_boot_status(CPU_BOOT_SUCCESS);
+ update_cpu_boot_status(cpu, CPU_BOOT_SUCCESS);
set_cpu_online(cpu, true);
complete(&cpu_running);
@@ -411,11 +411,11 @@ void __noreturn cpu_die_early(void)
rcutree_report_cpu_dead();
if (IS_ENABLED(CONFIG_HOTPLUG_CPU)) {
- update_cpu_boot_status(CPU_KILL_ME);
+ update_cpu_boot_status(cpu, CPU_KILL_ME);
__cpu_try_die(cpu);
}
- update_cpu_boot_status(CPU_STUCK_IN_KERNEL);
+ update_cpu_boot_status(cpu, CPU_STUCK_IN_KERNEL);
cpu_park_loop();
}
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index 0f4a28b87469..6b8a3245f393 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -72,7 +72,7 @@ void verify_cpu_asid_bits(void)
*/
pr_crit("CPU%d: smaller ASID size(%u) than boot CPU (%u)\n",
smp_processor_id(), asid, asid_bits);
- cpu_panic_kernel();
+ cpu_panic_kernel(smp_processor_id());
}
}
--
2.34.1
^ permalink raw reply related
* [PATCH v2 0/4] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
Support for parallel secondary CPU bringup is already utilized by x86,
MIPS, and RISC-V. This patch brings this capability to the arm64
architecture.
Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST to avoid primary SMT threads
to boot first constraint.
And Add a 'cpu' parameter to update_cpu_boot_status() to allow updating the
boot status at a per-CPU granularity during parallel bringup.
Rework the global `secondary_data` accessed during early boot into
a per-CPU array `cpu_boot_data` to allow secondary CPUs to boot
in parallel.
And reuse `__cpu_logical_map` array in the early boot code in head.S
to resolve each secondary CPU's logical ID concurrently.
Changes in v2:
- Remove RFC.
- Add Tested-by.
- Fix AI review issues in [1].
- Add arch_cpuhp_init_parallel_bringup() to check psci boot.
- Reuse `__cpu_logical_map` instead of a new aray.
- Defer rcutree_report_cpu_starting() until after
check_local_cpu_capabilities() to prevent a potential control CPU
deadlock if an early capability check fails.
- Move the assembly in head.S to a macro called `mpidr_to_cpuid`.
- Add `SECONDARY_DATA_SHIFT` for `lsl` to access `cpu_boot_data`.
- Add sizeof(struct secondary_data) power of 2 assert check.
- Expand testing with more data collected from real hardware.
[1] https://sashiko.dev/#/patchset/20260611133809.3854977-1-ruanjinjie%40huawei.com
Jinjie Ruan (4):
cpu/hotplug: Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST
arm64: smp: Pass CPU ID to update_cpu_boot_status()
arm64: smp: Defer RCU reporting until after local CPU capability
checks
arm64: Add HOTPLUG_PARALLEL support for secondary CPUs
arch/Kconfig | 4 +++
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/smp.h | 17 ++++++++++---
arch/arm64/kernel/asm-offsets.c | 4 +++
arch/arm64/kernel/cpufeature.c | 22 ++++++++--------
arch/arm64/kernel/head.S | 36 ++++++++++++++++++++++++++
arch/arm64/kernel/smp.c | 45 ++++++++++++++++++++++++++++-----
arch/arm64/mm/context.c | 4 +--
arch/mips/Kconfig | 1 +
arch/riscv/Kconfig | 1 +
arch/x86/Kconfig | 1 +
kernel/cpu.c | 6 ++++-
12 files changed, 119 insertions(+), 23 deletions(-)
--
2.34.1
^ permalink raw reply
* [PATCH v2 1/4] cpu/hotplug: Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST
From: Jinjie Ruan @ 2026-06-18 9:24 UTC (permalink / raw)
To: catalin.marinas, will, tsbogend, pjw, palmer, aou, alex, tglx,
mingo, bp, dave.hansen, hpa, peterz, kees, nathan, linusw,
jpoimboe, lukas.bulwahn, ryan.roberts, ojeda, maz, timothy.hayes,
lpieralisi, thuth, menglong8.dong, oupton, yeoreum.yun,
miko.lenczewski, broonie, kevin.brodsky, james.clark, tabba,
mrigendra.chaubey, arnd, anshuman.khandual, x86, linux-kernel,
linux-arm-kernel, linux-mips, linux-riscv, apatel, mhklinux
Cc: ruanjinjie
In-Reply-To: <20260618092444.1316336-1-ruanjinjie@huawei.com>
During parallel CPU bringup, x86 requires primary SMT threads to boot
first to avoid siblings stopping during microcode updates. This constraint
is architecture-specific and unnecessary for other platforms
like arm64.
Introduce CONFIG_PARALLEL_SMT_PRIMARY_FIRST to decouple this constraint.
Platforms requiring this temporal order (e.g., x86) can select it
in Kconfig. Other architectures (e.g., arm64) can leave it unselected
to entirely bypass the SMT branch via the preprocessor.
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/Kconfig | 4 ++++
arch/mips/Kconfig | 1 +
arch/riscv/Kconfig | 1 +
arch/x86/Kconfig | 1 +
kernel/cpu.c | 6 +++++-
5 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index e86880045158..0365d2df2659 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -102,6 +102,10 @@ config HOTPLUG_PARALLEL
bool
select HOTPLUG_SPLIT_STARTUP
+config PARALLEL_SMT_PRIMARY_FIRST
+ bool
+ depends on HOTPLUG_PARALLEL
+
config GENERIC_IRQ_ENTRY
bool
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4364f3dba688..84e11ac0cf71 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -642,6 +642,7 @@ config EYEQ
select MIPS_CPU_SCACHE
select MIPS_GIC
select MIPS_L1_CACHE_SHIFT_7
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DRIVERS_GENERIC
select SMP_UP if SMP
select SWAP_IO_SPACE
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index d235396c4514..0cc49aecc841 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -210,6 +210,7 @@ config RISCV
select OF
select OF_EARLY_FLATTREE
select OF_IRQ
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DOMAINS_GENERIC if PCI
select PCI_ECAM if (ACPI && PCI)
select PCI_MSI if PCI
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f3f7cb01d69d..3ad4115ad051 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -314,6 +314,7 @@ config X86
select NEED_PER_CPU_PAGE_FIRST_CHUNK
select NEED_SG_DMA_LENGTH
select NUMA_MEMBLKS if NUMA
+ select PARALLEL_SMT_PRIMARY_FIRST if HOTPLUG_PARALLEL
select PCI_DOMAINS if PCI
select PCI_LOCKLESS_CONFIG if PCI
select PERF_EVENTS
diff --git a/kernel/cpu.c b/kernel/cpu.c
index bc4f7a9ba64e..7ef8cdf4d239 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1792,6 +1792,7 @@ static int __init parallel_bringup_parse_param(char *arg)
}
early_param("cpuhp.parallel", parallel_bringup_parse_param);
+#ifdef CONFIG_PARALLEL_SMT_PRIMARY_FIRST
#ifdef CONFIG_HOTPLUG_SMT
static inline bool cpuhp_smt_aware(void)
{
@@ -1811,7 +1812,8 @@ static inline const struct cpumask *cpuhp_get_primary_thread_mask(void)
{
return cpu_none_mask;
}
-#endif
+#endif /* CONFIG_HOTPLUG_SMT */
+#endif /* CONFIG_PARALLEL_SMT_PRIMARY_FIRST */
bool __weak arch_cpuhp_init_parallel_bringup(void)
{
@@ -1837,6 +1839,7 @@ static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus)
if (!__cpuhp_parallel_bringup)
return false;
+#ifdef CONFIG_PARALLEL_SMT_PRIMARY_FIRST
if (cpuhp_smt_aware()) {
const struct cpumask *pmask = cpuhp_get_primary_thread_mask();
static struct cpumask tmp_mask __initdata;
@@ -1857,6 +1860,7 @@ static bool __init cpuhp_bringup_cpus_parallel(unsigned int ncpus)
cpumask_andnot(&tmp_mask, mask, pmask);
mask = &tmp_mask;
}
+#endif /* CONFIG_PARALLEL_SMT_PRIMARY_FIRST */
/* Bring the not-yet started CPUs up */
cpuhp_bringup_mask(mask, ncpus, CPUHP_BP_KICK_AP);
--
2.34.1
^ permalink raw reply related
* Re: [PATCH v1 03/11] KVM: arm64: Use guard()/scoped_guard() in arm64 KVM EL1 code
From: Fuad Tabba @ 2026-06-18 9:24 UTC (permalink / raw)
To: Marc Zyngier
Cc: Oliver Upton, Will Deacon, Catalin Marinas, Quentin Perret,
Vincent Donnefort, Sebastian Ene, Per Larsen, Suzuki K Poulose,
Zenghui Yu, Joey Gouly, Steffen Eiden, Mark Rutland,
Jonathan Cameron, Hyunwoo Kim, linux-arm-kernel, kvmarm,
linux-kernel
In-Reply-To: <86jyrwrymb.wl-maz@kernel.org>
On Thu, 18 Jun 2026 at 10:23, Marc Zyngier <maz@kernel.org> wrote:
>
> On Fri, 12 Jun 2026 07:59:17 +0100,
> tabba@google.com wrote:
> >
> > Convert the manual mutex_lock()/spin_lock() pairs in
> > arch/arm64/kvm/{pkvm,arm,mmu,reset,psci}.c to guard(mutex),
> > guard(spinlock) and scoped_guard(), dropping unlock-only goto labels in
> > favour of direct returns. Centralised cleanup gotos that still serve
> > other resources are preserved.
> >
> > reset.c uses scoped_guard() rather than guard() so the lock covers only
> > the small read/update window inside kvm_reset_vcpu(), leaving the rest
> > of the function outside the critical section.
>
> To be brutally honest, I don't think this sort of widespread changes
> bring us anything. This is just churn.
>
> Sure, if you are reworking a particular bit of code that is goto-heavy
> for the purpose of error handling, this has the potential to cleanup
> the code *while you are changing it*.
>
> But doing it for the sake of doing it? I think we have bigger fish to
> fry right now.
I understand what you mean. Would you like me to drop all of the guard
patches, or only those that go beyond the code changed in this series?
Thanks,
/fuad
>
> M.
>
> --
> Without deviation from the norm, progress is not possible.
^ permalink raw reply
* Re: [PATCH RFC v4 10/12] reset: zte: Add a zx297520v3 reset driver
From: Philipp Zabel @ 2026-06-18 9:24 UTC (permalink / raw)
To: Stefan Dösinger, Michael Turquette, Stephen Boyd,
Rob Herring, Krzysztof Kozlowski, Conor Dooley, Brian Masney
Cc: linux-clk, devicetree, linux-kernel, linux-arm-kernel
In-Reply-To: <20260616-zx29clk-v4-10-ca994bd22e9d@gmail.com>
On Di, 2026-06-16 at 23:26 +0300, Stefan Dösinger wrote:
> This drives the auxiliary devices created by the clock driver.
Which auxiliary devices? Which clock driver?
> Signed-off-by: Stefan Dösinger <stefandoesinger@gmail.com>
> ---
> MAINTAINERS | 1 +
> drivers/reset/Kconfig | 11 ++
> drivers/reset/Makefile | 1 +
> drivers/reset/reset-zte-zx297520v3.c | 224 +++++++++++++++++++++++++++++++++++
> 4 files changed, 237 insertions(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index f1f0459b2c72..55bf0290343a 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3871,6 +3871,7 @@ F: Documentation/devicetree/zte,zx297520v3-*
> F: arch/arm/boot/dts/zte/
> F: arch/arm/mach-zte/
> F: drivers/clk/zte/
> +F: drivers/reset/reset-zte-zx297520v3.c
> F: include/dt-bindings/clock/zte,zx297520v3-clk.h
>
> ARM/ZYNQ ARCHITECTURE
> diff --git a/drivers/reset/Kconfig b/drivers/reset/Kconfig
> index d009eb0849a3..116dd23f1b8e 100644
> --- a/drivers/reset/Kconfig
> +++ b/drivers/reset/Kconfig
> @@ -404,6 +404,17 @@ config RESET_UNIPHIER_GLUE
> on UniPhier SoCs. Say Y if you want to control reset signals
> provided by the glue layer.
>
> +config RESET_ZTE_ZX297520V3
> + tristate "ZTE zx297520v3 Reset Driver"
> + depends on (ARCH_ZTE || COMPILE_TEST)
> + default CLK_ZTE_ZX297520V3
> + select AUXILIARY_BUS
> + help
> + This enables the reset controller for ZTE zx297520v3 SoCs. The reset
> + controller is part of the clock controller on this SoC. This driver
> + operates on an auxiliary device exposed by the clock driver. Enable
> + this driver if you plan to boot the kernel on a zx297520v3 based SoC.
> +
> config RESET_ZYNQ
> bool "ZYNQ Reset Driver" if COMPILE_TEST
> default ARCH_ZYNQ
> diff --git a/drivers/reset/Makefile b/drivers/reset/Makefile
> index 3e52569bd276..9a8a48d44dc4 100644
> --- a/drivers/reset/Makefile
> +++ b/drivers/reset/Makefile
> @@ -50,5 +50,6 @@ obj-$(CONFIG_RESET_TI_TPS380X) += reset-tps380x.o
> obj-$(CONFIG_RESET_TN48M_CPLD) += reset-tn48m.o
> obj-$(CONFIG_RESET_UNIPHIER) += reset-uniphier.o
> obj-$(CONFIG_RESET_UNIPHIER_GLUE) += reset-uniphier-glue.o
> +obj-$(CONFIG_RESET_ZTE_ZX297520V3) += reset-zte-zx297520v3.o
> obj-$(CONFIG_RESET_ZYNQ) += reset-zynq.o
> obj-$(CONFIG_RESET_ZYNQMP) += reset-zynqmp.o
> diff --git a/drivers/reset/reset-zte-zx297520v3.c b/drivers/reset/reset-zte-zx297520v3.c
> new file mode 100644
> index 000000000000..2022f4df2ebd
> --- /dev/null
> +++ b/drivers/reset/reset-zte-zx297520v3.c
> @@ -0,0 +1,224 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2026 Stefan Dösinger
> + */
> +#include <dt-bindings/clock/zte,zx297520v3-clk.h>
> +#include <linux/reset-controller.h>
> +#include <linux/platform_device.h>
What is this used for?
> +#include <linux/auxiliary_bus.h>
> +#include <linux/clk-provider.h>
What is this used for?
> +#include <linux/mfd/syscon.h>
> +#include <linux/regmap.h>
> +#include <linux/iopoll.h>
> +#include <linux/delay.h>
> +
> +struct zte_reset_reg {
> + u32 mask, wait_mask;
> + u16 reg;
> +};
> +
> +struct zte_reset_info {
> + const struct zte_reset_reg *resets;
> + unsigned int num;
> +};
> +
> +struct zte_reset {
> + struct reset_controller_dev rcdev;
> + struct regmap *map;
> + const struct zte_reset_reg *resets;
> +};
> +
> +static inline struct zte_reset *to_zte_reset(struct reset_controller_dev *rcdev)
> +{
> + return container_of(rcdev, struct zte_reset, rcdev);
> +}
> +
> +static int zx29_rst_assert(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> +
> + return regmap_clear_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> +}
> +
> +static int zx29_rst_deassert(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> + int res;
> + u32 val;
> +
> + res = regmap_set_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> + if (res)
> + return res;
> +
> + /* This is a special case used only by USB reset */
> + if (rst->resets[id].wait_mask) {
> + return regmap_read_poll_timeout(rst->map, rst->resets[id].reg + 4, val,
> + val & rst->resets[id].wait_mask, 1, 100);
> + }
> +
> + return 0;
> +}
> +
> +static int zx29_rst_status(struct reset_controller_dev *rcdev, unsigned long id)
> +{
> + struct zte_reset *rst = to_zte_reset(rcdev);
> + int res;
> +
> + res = regmap_test_bits(rst->map, rst->resets[id].reg, rst->resets[id].mask);
> + if (res < 0)
> + return res;
> +
> + return !res;
> +}
> +
> +static const struct reset_control_ops zx29_rst_ops = {
> + .assert = zx29_rst_assert,
> + .deassert = zx29_rst_deassert,
> + .status = zx29_rst_status,
> +};
> +
> +static const struct zte_reset_reg zx297520v3_top_resets[] = {
> + /* This bit is set by ZTE's cpko.ko blob, it looks like a reset bit for the LTE DSP
> + * coprocessor. Clocks for it are in matrixclk.
> + */
> + [ZX297520V3_ZSP_RESET] = { .reg = 0x13c, .mask = BIT(0) },
> +
> + [ZX297520V3_UART0_RESET] = { .reg = 0x78, .mask = BIT(6) | BIT(7) },
Is this a single reset line controlled by two bits (do you know what
they are)? Or might these actually be two different reset controls that
are just always set together?
> + [ZX297520V3_I2C0_RESET] = { .reg = 0x74, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_RTC_RESET] = { .reg = 0x74, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T08_RESET] = { .reg = 0x78, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T09_RESET] = { .reg = 0x78, .mask = BIT(2) | BIT(3) },
> + [ZX297520V3_PMM_RESET] = { .reg = 0x74, .mask = BIT(0) | BIT(1) },
> +
> + /* I haven't found any clocks for GPIO. It probably wouldn't make much
> + * sense anyway. Only one reset bit per controller.
> + */
> + [ZX297520V3_GPIO_RESET] = { .reg = 0x74, .mask = BIT(3) },
> + [ZX297520V3_GPIO8_RESET] = { .reg = 0x74, .mask = BIT(2) },
> +
> + [ZX297520V3_TIMER_T12_RESET] = { .reg = 0x74, .mask = BIT(6) | BIT(7) },
> + [ZX297520V3_TIMER_T13_RESET] = { .reg = 0x7c, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_TIMER_T14_RESET] = { .reg = 0x7c, .mask = BIT(2) | BIT(3) },
> + [ZX297520V3_TIMER_T15_RESET] = { .reg = 0x74, .mask = BIT(10) | BIT(11) },
> + [ZX297520V3_TIMER_T16_RESET] = { .reg = 0x7c, .mask = BIT(4) | BIT(5) },
> + [ZX297520V3_TIMER_T17_RESET] = { .reg = 0x12c, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_WDT_T18_RESET] = { .reg = 0x74, .mask = BIT(12) | BIT(13) },
> + [ZX297520V3_USIM1_RESET] = { .reg = 0x74, .mask = BIT(14) | BIT(15) },
> + [ZX297520V3_AHB_RESET] = { .reg = 0x70, .mask = BIT(0) | BIT(1) },
> +
> + /* USB reset. This is slightly special because it needs to wait for a ready bit after
> + * deasserting.
> + */
> + [ZX297520V3_USB_RESET] = { .reg = 0x80, .mask = BIT(3) | BIT(4) | BIT(5),
> + .wait_mask = BIT(1)},
Same as above, are these actually three separate reset lines?
> + [ZX297520V3_HSIC_RESET] = { .reg = 0x80, .mask = BIT(0) | BIT(1) | BIT(2),
> + .wait_mask = BIT(0)},
> +};
> +
> +static const struct zte_reset_info zx297520v3_top_info = {
> + .resets = zx297520v3_top_resets,
> + .num = ARRAY_SIZE(zx297520v3_top_resets),
> +};
> +
> +static const struct zte_reset_reg zx297520v3_matrix_resets[] = {
> + [ZX297520V3_CPU_RESET] = { .reg = 0x28, .mask = BIT(1) },
> + [ZX297520V3_EDCP_RESET] = { .reg = 0x68, .mask = BIT(0) },
> + [ZX297520V3_SD0_RESET] = { .reg = 0x58, .mask = BIT(1) },
> + [ZX297520V3_SD1_RESET] = { .reg = 0x58, .mask = BIT(0) },
> + [ZX297520V3_NAND_RESET] = { .reg = 0x58, .mask = BIT(4) },
> + [ZX297520V3_PDCFG_RESET] = { .reg = 0x94, .mask = BIT(20) },
> + [ZX297520V3_SSC_RESET] = { .reg = 0x94, .mask = BIT(24) },
> + [ZX297520V3_GMAC_RESET] = { .reg = 0x114, .mask = BIT(0) | BIT(1) },
> + [ZX297520V3_VOU_RESET] = { .reg = 0x16c, .mask = BIT(0) },
> +};
> +
> +static const struct zte_reset_info zx297520v3_matrix_info = {
> + .resets = zx297520v3_matrix_resets,
> + .num = ARRAY_SIZE(zx297520v3_matrix_resets),
> +};
> +
> +static const struct zte_reset_reg zx297520v3_lsp_resets[] = {
> + [ZX297520V3_TIMER_L1_RESET] = { .reg = 0x04, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_L2_RESET] = { .reg = 0x08, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_L3_RESET] = { .reg = 0x0c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_PWM_RESET] = { .reg = 0x10, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_I2S0_RESET] = { .reg = 0x14, .mask = BIT(8) | BIT(9) },
> + /* 0x18: Not writeable */
> + [ZX297520V3_I2S1_RESET] = { .reg = 0x1c, .mask = BIT(8) | BIT(9) },
> + /* 0x20: Not writeable */
> + [ZX297520V3_QSPI_RESET] = { .reg = 0x24, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_UART1_RESET] = { .reg = 0x28, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_I2C1_RESET] = { .reg = 0x2c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_SPI0_RESET] = { .reg = 0x30, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LB_RESET] = { .reg = 0x34, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LC_RESET] = { .reg = 0x38, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_UART2_RESET] = { .reg = 0x3c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_WDT_LE_RESET] = { .reg = 0x40, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_LF_RESET] = { .reg = 0x44, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_SPI1_RESET] = { .reg = 0x48, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TIMER_L11_RESET] = { .reg = 0x4c, .mask = BIT(8) | BIT(9) },
> + [ZX297520V3_TDM_RESET] = { .reg = 0x50, .mask = BIT(8) | BIT(9) },
> +};
> +
> +static const struct zte_reset_info zx297520v3_lsp_info = {
> + .resets = zx297520v3_lsp_resets,
> + .num = ARRAY_SIZE(zx297520v3_lsp_resets),
> +};
> +
> +static int reset_zx297520v3_probe(struct auxiliary_device *adev,
> + const struct auxiliary_device_id *id)
> +{
> + const struct zte_reset_info *drv_info;
> + struct device *dev = &adev->dev;
> + struct zte_reset *rst;
> +
> + drv_info = (struct zte_reset_info *)id->driver_data;
> +
> + rst = devm_kzalloc(dev, sizeof(*rst), GFP_KERNEL);
> + if (!rst)
> + return -ENOMEM;
> +
> + rst->resets = drv_info->resets;
> + rst->rcdev.owner = THIS_MODULE;
> + rst->rcdev.nr_resets = drv_info->num;
> + rst->rcdev.ops = &zx29_rst_ops;
> + rst->rcdev.of_node = dev->of_node;
> + rst->rcdev.dev = dev;
> + rst->rcdev.of_reset_n_cells = 1;
No need to set of_reset_n_cells if of_xlate is not set. Here
reset_controller_register will use fwnode_n_cells and set it to 1
anyway.
> +
> + rst->map = device_node_to_regmap(dev->of_node);
> + if (IS_ERR(rst->map))
> + return dev_err_probe(rdev, PTR_ERR(rst->map), "Cannot get parent syscon regmap\n");
> +
> + return devm_reset_controller_register(dev, &rst->rcdev);
> +}
> +
> +static const struct auxiliary_device_id reset_zx297520v3_ids[] = {
> + {
> + .name = "clk_zte.zx297520v3_toprst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_top_info,
> + },
> + {
> + .name = "clk_zte.zx297520v3_matrixrst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_matrix_info,
> + },
> + {
> + .name = "clk_zte.zx297520v3_lsprst",
> + .driver_data = (kernel_ulong_t)&zx297520v3_lsp_info,
> + },
> + { },
> +};
> +
Drop this empty line.
> +MODULE_DEVICE_TABLE(auxiliary, reset_zx297520v3_ids);
> +
> +static struct auxiliary_driver reset_zx297520v3_drv = {
> + .name = "zx297520v3_reset",
> + .id_table = reset_zx297520v3_ids,
> + .probe = reset_zx297520v3_probe,
> +};
> +
Drop this empty line.
> +module_auxiliary_driver(reset_zx297520v3_drv);
> +
> +MODULE_AUTHOR("Stefan Dösinger <stefandoesinger@gmail.com>");
> +MODULE_DESCRIPTION("ZTE zx297520v3 reset driver");
> +MODULE_LICENSE("GPL");
regards
Philipp
^ permalink raw reply
* Re: [PATCH v1 03/11] KVM: arm64: Use guard()/scoped_guard() in arm64 KVM EL1 code
From: Marc Zyngier @ 2026-06-18 9:23 UTC (permalink / raw)
To: tabba
Cc: Oliver Upton, Will Deacon, Catalin Marinas, Quentin Perret,
Vincent Donnefort, Sebastian Ene, Per Larsen, Suzuki K Poulose,
Zenghui Yu, Joey Gouly, Steffen Eiden, Mark Rutland,
Jonathan Cameron, Hyunwoo Kim, linux-arm-kernel, kvmarm,
linux-kernel
In-Reply-To: <20260612065925.755562-4-tabba@google.com>
On Fri, 12 Jun 2026 07:59:17 +0100,
tabba@google.com wrote:
>
> Convert the manual mutex_lock()/spin_lock() pairs in
> arch/arm64/kvm/{pkvm,arm,mmu,reset,psci}.c to guard(mutex),
> guard(spinlock) and scoped_guard(), dropping unlock-only goto labels in
> favour of direct returns. Centralised cleanup gotos that still serve
> other resources are preserved.
>
> reset.c uses scoped_guard() rather than guard() so the lock covers only
> the small read/update window inside kvm_reset_vcpu(), leaving the rest
> of the function outside the critical section.
To be brutally honest, I don't think this sort of widespread changes
bring us anything. This is just churn.
Sure, if you are reworking a particular bit of code that is goto-heavy
for the purpose of error handling, this has the potential to cleanup
the code *while you are changing it*.
But doing it for the sake of doing it? I think we have bigger fish to
fry right now.
M.
--
Without deviation from the norm, progress is not possible.
^ permalink raw reply
* [PATCH v15 9/9] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
memcpy_mc() is the Machine-Check safe memcpy variant that returns the
number of bytes NOT copied on a hardware memory error, or 0 on success.
Add two test cases modeled after the existing memcpy_test() and
memcpy_large_test() implementations.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
lib/tests/memcpy_kunit.c | 121 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 120 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index 812c1fa20fd9..87585fbe78c7 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -554,6 +554,121 @@ static void copy_mc_page_test(struct kunit *test)
}
#endif /* __HAVE_ARCH_COPY_MC_PAGE */
+#ifdef __HAVE_ARCH_MEMCPY_MC
+/*
+ * memcpy_mc() is a Machine-Check safe memcpy variant.
+ * Signature: int memcpy_mc(void *dst, const void *src, size_t len)
+ * Returns: 0 on success, or number of bytes NOT copied on MC error.
+ *
+ * In the normal (no-poison) path it must behave identically to memcpy()
+ * and always return 0.
+ */
+static void memcpy_mc_test(struct kunit *test)
+{
+#define TEST_OP "memcpy_mc"
+ struct some_bytes control = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes zero = { };
+ struct some_bytes middle = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes three = {
+ .data = { 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes dest = { };
+ unsigned long ret;
+ int count;
+ u8 *ptr;
+
+ /* Verify static initializers. */
+ check(control, 0x20);
+ check(zero, 0);
+ compare("static initializers", dest, zero);
+
+ /* Verify assignment. */
+ dest = control;
+ compare("direct assignment", dest, control);
+
+ /* Verify complete overwrite. */
+ ret = memcpy_mc(dest.data, zero.data, sizeof(dest.data));
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("complete overwrite", dest, zero);
+
+ /* Verify middle overwrite: 7 bytes at offset 12. */
+ dest = control;
+ ret = memcpy_mc(dest.data + 12, zero.data, 7);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("middle overwrite", dest, middle);
+
+ /* Verify zero-length copy is a no-op. */
+ dest = control;
+ ret = memcpy_mc(dest.data, zero.data, 0);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("zero length", dest, control);
+
+ /* Verify argument side-effects aren't repeated. */
+ dest = control;
+ ptr = dest.data;
+ count = 1;
+ ret = memcpy_mc(ptr++, zero.data, count++);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ ptr += 8;
+ ret = memcpy_mc(ptr++, zero.data, count++);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("argument side-effects", dest, three);
+#undef TEST_OP
+}
+
+static void memcpy_mc_large_test(struct kunit *test)
+{
+ init_large(test);
+
+ /* Sweep 1..1024 bytes x shifting offset to cover all template paths. */
+ for (int bytes = 1; bytes <= ARRAY_SIZE(large_src); bytes++) {
+ for (int offset = 0; offset < ARRAY_SIZE(large_src); offset++) {
+ int right_zero_pos = offset + bytes;
+ int right_zero_size = ARRAY_SIZE(large_dst) - right_zero_pos;
+ int ret;
+
+ ret = memcpy_mc(large_dst + offset, large_src, bytes);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "memcpy_mc returned %d with size %d at offset %d",
+ ret, bytes, offset);
+
+ /* No write before copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst, large_zero, offset), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* No write after copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(&large_dst[right_zero_pos], large_zero,
+ right_zero_size), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* Byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst + offset, large_src, bytes), 0,
+ "with size %d at offset %d", bytes, offset);
+
+ memset(large_dst + offset, 0, bytes);
+ }
+ cond_resched();
+ }
+}
+#endif /* __HAVE_ARCH_MEMCPY_MC */
+
static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(memset_test),
KUNIT_CASE(memcpy_test),
@@ -564,6 +679,10 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(copy_page_test),
#ifdef __HAVE_ARCH_COPY_MC_PAGE
KUNIT_CASE(copy_mc_page_test),
+#endif
+#ifdef __HAVE_ARCH_MEMCPY_MC
+ KUNIT_CASE(memcpy_mc_test),
+ KUNIT_CASE_SLOW(memcpy_mc_large_test),
#endif
{}
};
@@ -575,5 +694,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset(), copy_page() and memcpy_mc()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related
* [PATCH v15 8/9] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
Add KUnit tests for copy_page() and copy_mc_page(), modeled after
the existing memcpy_test() style: a static page-aligned src and a
two-page dst, filled with random bytes plus non-zero edges, then
verify byte-for-byte equality and that the adjacent page is
untouched.
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
lib/tests/memcpy_kunit.c | 67 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 66 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index d36933554e46..812c1fa20fd9 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -493,6 +493,67 @@ static void memmove_overlap_test(struct kunit *test)
}
}
+/* --- Page-sized copy tests --- */
+
+static u8 page_src[PAGE_SIZE] __aligned(PAGE_SIZE);
+static u8 page_dst[PAGE_SIZE * 2] __aligned(PAGE_SIZE);
+static const u8 page_pattern[PAGE_SIZE] __aligned(PAGE_SIZE);
+
+static void init_page(struct kunit *test)
+{
+ /* Get many bit patterns. */
+ get_random_bytes(page_src, PAGE_SIZE);
+
+ /* Make sure we have non-zero edges. */
+ set_random_nonzero(test, &page_src[0]);
+ set_random_nonzero(test, &page_src[PAGE_SIZE - 1]);
+
+ memset(page_dst, 0xA5, ARRAY_SIZE(page_dst));
+ memset(page_pattern, 0xA5, PAGE_SIZE);
+}
+
+static void copy_page_test(struct kunit *test)
+{
+ init_page(test);
+
+ /* Copy. */
+ copy_page(page_dst, page_src);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_pattern, PAGE_SIZE), 0,
+ "copy_page overflow into adjacent page");
+}
+
+#ifdef __HAVE_ARCH_COPY_MC_PAGE
+static void copy_mc_page_test(struct kunit *test)
+{
+ int ret;
+
+ init_page(test);
+
+ /* Copy and check return value. */
+ ret = copy_mc_page(page_dst, page_src);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "copy_mc_page returned %d on clean memory", ret);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_mc_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_pattern, PAGE_SIZE), 0,
+ "copy_mc_page overflow into adjacent page");
+}
+#endif /* __HAVE_ARCH_COPY_MC_PAGE */
+
static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(memset_test),
KUNIT_CASE(memcpy_test),
@@ -500,6 +561,10 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE_SLOW(memmove_test),
KUNIT_CASE_SLOW(memmove_large_test),
KUNIT_CASE_SLOW(memmove_overlap_test),
+ KUNIT_CASE(copy_page_test),
+#ifdef __HAVE_ARCH_COPY_MC_PAGE
+ KUNIT_CASE(copy_mc_page_test),
+#endif
{}
};
@@ -510,5 +575,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), and memset()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related
* [PATCH v15 7/9] arm64: introduce copy_mc_to_kernel() implementation
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
The copy_mc_to_kernel() helper is memory copy implementation that handles
source exceptions. It can be used in memory copy scenarios that tolerate
hardware memory errors(e.g: pmem_read/dax_copy_to_iter).
Currently, only x86 and ppc support this helper, Add this for ARM64 as
well, if ARCH_HAS_COPY_MC is defined, by implementing copy_mc_to_kernel()
and memcpy_mc() functions.
Because there is no caller-saved GPR is available for saving "bytes not
copied" in memcpy(), the memcpy_mc() is referenced to the implementation
of copy_from_user(). In addition, the fixup of MOPS insn is not considered
at present.
[Ruidong: refactor memcpy_mc on top of the new memcpy implementation.]
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/include/asm/string.h | 5 +
arch/arm64/include/asm/uaccess.h | 17 +++
arch/arm64/lib/Makefile | 2 +-
arch/arm64/lib/memcpy.S | 251 +++----------------------------
arch/arm64/lib/memcpy_mc.S | 56 +++++++
arch/arm64/lib/memcpy_template.S | 250 ++++++++++++++++++++++++++++++
mm/kasan/shadow.c | 12 ++
7 files changed, 359 insertions(+), 234 deletions(-)
create mode 100644 arch/arm64/lib/memcpy_mc.S
create mode 100644 arch/arm64/lib/memcpy_template.S
diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h
index 3a3264ff47b9..2e81f6c00cdd 100644
--- a/arch/arm64/include/asm/string.h
+++ b/arch/arm64/include/asm/string.h
@@ -35,6 +35,10 @@ extern void *memchr(const void *, int, __kernel_size_t);
extern void *memcpy(void *, const void *, __kernel_size_t);
extern void *__memcpy(void *, const void *, __kernel_size_t);
+#define __HAVE_ARCH_MEMCPY_MC
+extern unsigned long memcpy_mc(void *, const void *, __kernel_size_t);
+extern unsigned long __memcpy_mc(void *, const void *, __kernel_size_t);
+
#define __HAVE_ARCH_MEMMOVE
extern void *memmove(void *, const void *, __kernel_size_t);
extern void *__memmove(void *, const void *, __kernel_size_t);
@@ -57,6 +61,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt);
*/
#define memcpy(dst, src, len) __memcpy(dst, src, len)
+#define memcpy_mc(dst, src, len) __memcpy_mc(dst, src, len)
#define memmove(dst, src, len) __memmove(dst, src, len)
#define memset(s, c, n) __memset(s, c, n)
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index b0c83a08dda9..93277eca2268 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -499,5 +499,22 @@ static inline size_t probe_subpage_writeable(const char __user *uaddr,
}
#endif /* CONFIG_ARCH_HAS_SUBPAGE_FAULTS */
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/**
+ * copy_mc_to_kernel - memory copy that handles source exceptions
+ *
+ * @to: destination address
+ * @from: source address
+ * @size: number of bytes to copy
+ *
+ * Return 0 for success, or bytes not copied.
+ */
+static inline unsigned long __must_check
+copy_mc_to_kernel(void *to, const void *from, unsigned long size)
+{
+ return memcpy_mc(to, from, size);
+}
+#define copy_mc_to_kernel copy_mc_to_kernel
+#endif
#endif /* __ASM_UACCESS_H */
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 1f4c3f743a20..a5820e6c33d4 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,7 +7,7 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
-lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o memcpy_mc.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
index 9b99106fb95f..ab48c5c798d1 100644
--- a/arch/arm64/lib/memcpy.S
+++ b/arch/arm64/lib/memcpy.S
@@ -15,247 +15,32 @@
*
*/
-#define L(label) .L ## label
+ .macro ldrb1 reg, addr:vararg
+ ldrb \reg, \addr
+ .endm
-#define dstin x0
-#define src x1
-#define count x2
-#define dst x3
-#define srcend x4
-#define dstend x5
-#define A_l x6
-#define A_lw w6
-#define A_h x7
-#define B_l x8
-#define B_lw w8
-#define B_h x9
-#define C_l x10
-#define C_lw w10
-#define C_h x11
-#define D_l x12
-#define D_h x13
-#define E_l x14
-#define E_h x15
-#define F_l x16
-#define F_h x17
-#define G_l count
-#define G_h dst
-#define H_l src
-#define H_h srcend
-#define tmp1 x14
+ .macro ldr1 reg, addr:vararg
+ ldr \reg, \addr
+ .endm
-/* This implementation handles overlaps and supports both memcpy and memmove
- from a single entry point. It uses unaligned accesses and branchless
- sequences to keep the code small, simple and improve performance.
+ .macro ldp1 reg1, reg2, addr:vararg
+ ldp \reg1, \reg2, \addr
+ .endm
- Copies are split into 3 main cases: small copies of up to 32 bytes, medium
- copies of up to 128 bytes, and large copies. The overhead of the overlap
- check is negligible since it is only required for large copies.
-
- Large copies use a software pipelined loop processing 64 bytes per iteration.
- The destination pointer is 16-byte aligned to minimize unaligned accesses.
- The loop tail is handled by always copying 64 bytes from the end.
-*/
-
-SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
- add srcend, src, count
- add dstend, dstin, count
- cmp count, 128
- b.hi L(copy_long)
- cmp count, 32
- b.hi L(copy32_128)
-
- /* Small copies: 0..32 bytes. */
- cmp count, 16
- b.lo L(copy16)
- ldp A_l, A_h, [src]
- ldp D_l, D_h, [srcend, -16]
- stp A_l, A_h, [dstin]
- stp D_l, D_h, [dstend, -16]
- ret
-
- /* Copy 8-15 bytes. */
-L(copy16):
- tbz count, 3, L(copy8)
- ldr A_l, [src]
- ldr A_h, [srcend, -8]
- str A_l, [dstin]
- str A_h, [dstend, -8]
- ret
-
- .p2align 3
- /* Copy 4-7 bytes. */
-L(copy8):
- tbz count, 2, L(copy4)
- ldr A_lw, [src]
- ldr B_lw, [srcend, -4]
- str A_lw, [dstin]
- str B_lw, [dstend, -4]
+ .macro ret1
ret
+ .endm
- /* Copy 0..3 bytes using a branchless sequence. */
-L(copy4):
- cbz count, L(copy0)
- lsr tmp1, count, 1
- ldrb A_lw, [src]
- ldrb C_lw, [srcend, -1]
- ldrb B_lw, [src, tmp1]
- strb A_lw, [dstin]
- strb B_lw, [dstin, tmp1]
- strb C_lw, [dstend, -1]
-L(copy0):
- ret
-
- .p2align 4
- /* Medium copies: 33..128 bytes. */
-L(copy32_128):
- ldp A_l, A_h, [src]
- ldp B_l, B_h, [src, 16]
- ldp C_l, C_h, [srcend, -32]
- ldp D_l, D_h, [srcend, -16]
- cmp count, 64
- b.hi L(copy128)
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy 65..128 bytes. */
-L(copy128):
- ldp E_l, E_h, [src, 32]
- ldp F_l, F_h, [src, 48]
- cmp count, 96
- b.ls L(copy96)
- ldp G_l, G_h, [srcend, -64]
- ldp H_l, H_h, [srcend, -48]
- stp G_l, G_h, [dstend, -64]
- stp H_l, H_h, [dstend, -48]
-L(copy96):
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp E_l, E_h, [dstin, 32]
- stp F_l, F_h, [dstin, 48]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy more than 128 bytes. */
-L(copy_long):
- /* Use backwards copy if there is an overlap. */
- sub tmp1, dstin, src
- cbz tmp1, L(copy0)
- cmp tmp1, count
- b.lo L(copy_long_backwards)
-
- /* Copy 16 bytes and then align dst to 16-byte alignment. */
-
- ldp D_l, D_h, [src]
- and tmp1, dstin, 15
- bic dst, dstin, 15
- sub src, src, tmp1
- add count, count, tmp1 /* Count is now 16 too large. */
- ldp A_l, A_h, [src, 16]
- stp D_l, D_h, [dstin]
- ldp B_l, B_h, [src, 32]
- ldp C_l, C_h, [src, 48]
- ldp D_l, D_h, [src, 64]!
- subs count, count, 128 + 16 /* Test and readjust count. */
- b.ls L(copy64_from_end)
-
-L(loop64):
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [src, 16]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [src, 32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [src, 48]
- stp D_l, D_h, [dst, 64]!
- ldp D_l, D_h, [src, 64]!
- subs count, count, 64
- b.hi L(loop64)
-
- /* Write the last iteration and copy 64 bytes from the end. */
-L(copy64_from_end):
- ldp E_l, E_h, [srcend, -64]
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [srcend, -48]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [srcend, -16]
- stp D_l, D_h, [dst, 64]
- stp E_l, E_h, [dstend, -64]
- stp A_l, A_h, [dstend, -48]
- stp B_l, B_h, [dstend, -32]
- stp C_l, C_h, [dstend, -16]
- ret
-
- .p2align 4
-
- /* Large backwards copy for overlapping copies.
- Copy 16 bytes and then align dst to 16-byte alignment. */
-L(copy_long_backwards):
- ldp D_l, D_h, [srcend, -16]
- and tmp1, dstend, 15
- sub srcend, srcend, tmp1
- sub count, count, tmp1
- ldp A_l, A_h, [srcend, -16]
- stp D_l, D_h, [dstend, -16]
- ldp B_l, B_h, [srcend, -32]
- ldp C_l, C_h, [srcend, -48]
- ldp D_l, D_h, [srcend, -64]!
- sub dstend, dstend, tmp1
- subs count, count, 128
- b.ls L(copy64_from_start)
-
-L(loop64_backwards):
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [srcend, -16]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [srcend, -48]
- stp D_l, D_h, [dstend, -64]!
- ldp D_l, D_h, [srcend, -64]!
- subs count, count, 64
- b.hi L(loop64_backwards)
-
- /* Write the last iteration and copy 64 bytes from the start. */
-L(copy64_from_start):
- ldp G_l, G_h, [src, 48]
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [src, 32]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [src, 16]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [src]
- stp D_l, D_h, [dstend, -64]
- stp G_l, G_h, [dstin, 48]
- stp A_l, A_h, [dstin, 32]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstin]
- ret
-SYM_FUNC_END(__pi_memcpy_generic)
-
-#ifdef CONFIG_AS_HAS_MOPS
+ .macro cpy1 dst, src, count
.arch_extension mops
-SYM_FUNC_START(__pi_memcpy)
-alternative_if_not ARM64_HAS_MOPS
- b __pi_memcpy_generic
-alternative_else_nop_endif
+ cpyp [\dst]!, [\src]!, \count!
+ cpym [\dst]!, [\src]!, \count!
+ cpye [\dst]!, [\src]!, \count!
+ .endm
- mov dst, dstin
- cpyp [dst]!, [src]!, count!
- cpym [dst]!, [src]!, count!
- cpye [dst]!, [src]!, count!
- ret
+SYM_FUNC_START(__pi_memcpy)
+#include "memcpy_template.S"
SYM_FUNC_END(__pi_memcpy)
-#else
-SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
-#endif
SYM_FUNC_ALIAS(__memcpy, __pi_memcpy)
EXPORT_SYMBOL(__memcpy)
diff --git a/arch/arm64/lib/memcpy_mc.S b/arch/arm64/lib/memcpy_mc.S
new file mode 100644
index 000000000000..d9ce8279d91f
--- /dev/null
+++ b/arch/arm64/lib/memcpy_mc.S
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/asm-uaccess.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+ .macro ldrb1 reg, addr:vararg
+ KERNEL_SEA(9998f, ldrb \reg, \addr)
+ .endm
+
+ .macro ldr1 reg, addr:vararg
+ KERNEL_SEA(9998f, ldr \reg, \addr)
+ .endm
+
+ .macro ldp1 reg1, reg2, addr:vararg
+ KERNEL_SEA(9998f, ldp \reg1, \reg2, \addr)
+ .endm
+
+ .macro ret1
+ mov x0, #0
+ ret
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ KERNEL_SEA(9998f, cpyp [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9996f, cpym [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9996f, cpye [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__memcpy_mc)
+#include "memcpy_template.S"
+
+ // Exception fixups
+9996: b.cs 9998f
+ // Registers are in Option A format
+ add dst, dst, count
+9998: sub x0, dstend, dstin // bytes not copied
+ ret
+SYM_FUNC_END(__memcpy_mc)
+
+EXPORT_SYMBOL(__memcpy_mc)
+SYM_FUNC_ALIAS_WEAK(memcpy_mc, __memcpy_mc)
+EXPORT_SYMBOL(memcpy_mc)
diff --git a/arch/arm64/lib/memcpy_template.S b/arch/arm64/lib/memcpy_template.S
new file mode 100644
index 000000000000..a8b496f8f651
--- /dev/null
+++ b/arch/arm64/lib/memcpy_template.S
@@ -0,0 +1,250 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+#define L(label) .L ## label
+
+#define dstin x0
+#define src x1
+#define count x2
+#define dst x3
+#define srcend x4
+#define dstend x5
+#define A_l x6
+#define A_lw w6
+#define A_h x7
+#define B_l x8
+#define B_lw w8
+#define B_h x9
+#define C_l x10
+#define C_lw w10
+#define C_h x11
+#define D_l x12
+#define D_h x13
+#define E_l x14
+#define E_h x15
+#define F_l x16
+#define F_h x17
+#define G_l count
+#define G_h dst
+#define H_l src
+#define H_h srcend
+#define tmp1 x14
+
+/* This implementation handles overlaps and supports both memcpy and memmove
+ from a single entry point. It uses unaligned accesses and branchless
+ sequences to keep the code small, simple and improve performance.
+
+ Copies are split into 3 main cases: small copies of up to 32 bytes, medium
+ copies of up to 128 bytes, and large copies. The overhead of the overlap
+ check is negligible since it is only required for large copies.
+
+ Large copies use a software pipelined loop processing 64 bytes per iteration.
+ The destination pointer is 16-byte aligned to minimize unaligned accesses.
+ The loop tail is handled by always copying 64 bytes from the end.
+*/
+
+ add dstend, dstin, count
+
+#ifdef CONFIG_AS_HAS_MOPS
+alternative_if_not ARM64_HAS_MOPS
+ b L(no_mops)
+alternative_else_nop_endif
+ mov dst, dstin
+ cpy1 dst, src, count
+ ret1
+#endif
+
+L(no_mops):
+ add srcend, src, count
+ cmp count, 128
+ b.hi L(copy_long)
+ cmp count, 32
+ b.hi L(copy32_128)
+
+ /* Small copies: 0..32 bytes. */
+ cmp count, 16
+ b.lo L(copy16)
+ ldp1 A_l, A_h, [src]
+ ldp1 D_l, D_h, [srcend, -16]
+ stp A_l, A_h, [dstin]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ /* Copy 8-15 bytes. */
+L(copy16):
+ tbz count, 3, L(copy8)
+ ldr1 A_l, [src]
+ ldr1 A_h, [srcend, -8]
+ str A_l, [dstin]
+ str A_h, [dstend, -8]
+ ret1
+
+ .p2align 3
+ /* Copy 4-7 bytes. */
+L(copy8):
+ tbz count, 2, L(copy4)
+ ldr1 A_lw, [src]
+ ldr1 B_lw, [srcend, -4]
+ str A_lw, [dstin]
+ str B_lw, [dstend, -4]
+ ret1
+
+ /* Copy 0..3 bytes using a branchless sequence. */
+L(copy4):
+ cbz count, L(copy0)
+ lsr tmp1, count, 1
+ ldrb1 A_lw, [src]
+ ldrb1 C_lw, [srcend, -1]
+ ldrb1 B_lw, [src, tmp1]
+ strb A_lw, [dstin]
+ strb B_lw, [dstin, tmp1]
+ strb C_lw, [dstend, -1]
+L(copy0):
+ ret1
+
+ .p2align 4
+ /* Medium copies: 33..128 bytes. */
+L(copy32_128):
+ ldp1 A_l, A_h, [src]
+ ldp1 B_l, B_h, [src, 16]
+ ldp1 C_l, C_h, [srcend, -32]
+ ldp1 D_l, D_h, [srcend, -16]
+ cmp count, 64
+ b.hi L(copy128)
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy 65..128 bytes. */
+L(copy128):
+ ldp1 E_l, E_h, [src, 32]
+ ldp1 F_l, F_h, [src, 48]
+ cmp count, 96
+ b.ls L(copy96)
+ ldp1 G_l, G_h, [srcend, -64]
+ ldp1 H_l, H_h, [srcend, -48]
+ stp G_l, G_h, [dstend, -64]
+ stp H_l, H_h, [dstend, -48]
+L(copy96):
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp E_l, E_h, [dstin, 32]
+ stp F_l, F_h, [dstin, 48]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy more than 128 bytes. */
+L(copy_long):
+ /* Use backwards copy if there is an overlap. */
+ sub tmp1, dstin, src
+ cbz tmp1, L(copy0)
+ cmp tmp1, count
+ b.lo L(copy_long_backwards)
+
+ /* Copy 16 bytes and then align dst to 16-byte alignment. */
+
+ ldp1 D_l, D_h, [src]
+ and tmp1, dstin, 15
+ bic dst, dstin, 15
+ sub src, src, tmp1
+ add count, count, tmp1 /* Count is now 16 too large. */
+ ldp1 A_l, A_h, [src, 16]
+ stp D_l, D_h, [dstin]
+ ldp1 B_l, B_h, [src, 32]
+ ldp1 C_l, C_h, [src, 48]
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 128 + 16 /* Test and readjust count. */
+ b.ls L(copy64_from_end)
+
+L(loop64):
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [src, 16]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [src, 32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [src, 48]
+ stp D_l, D_h, [dst, 64]!
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 64
+ b.hi L(loop64)
+
+ /* Write the last iteration and copy 64 bytes from the end. */
+L(copy64_from_end):
+ ldp1 E_l, E_h, [srcend, -64]
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [srcend, -48]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [srcend, -16]
+ stp D_l, D_h, [dst, 64]
+ stp E_l, E_h, [dstend, -64]
+ stp A_l, A_h, [dstend, -48]
+ stp B_l, B_h, [dstend, -32]
+ stp C_l, C_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+
+ /* Large backwards copy for overlapping copies.
+ Copy 16 bytes and then align dst to 16-byte alignment. */
+L(copy_long_backwards):
+ ldp1 D_l, D_h, [srcend, -16]
+ and tmp1, dstend, 15
+ sub srcend, srcend, tmp1
+ sub count, count, tmp1
+ ldp1 A_l, A_h, [srcend, -16]
+ stp D_l, D_h, [dstend, -16]
+ ldp1 B_l, B_h, [srcend, -32]
+ ldp1 C_l, C_h, [srcend, -48]
+ ldp1 D_l, D_h, [srcend, -64]!
+ sub dstend, dstend, tmp1
+ subs count, count, 128
+ b.ls L(copy64_from_start)
+
+L(loop64_backwards):
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [srcend, -16]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [srcend, -48]
+ stp D_l, D_h, [dstend, -64]!
+ ldp1 D_l, D_h, [srcend, -64]!
+ subs count, count, 64
+ b.hi L(loop64_backwards)
+
+ /* Write the last iteration and copy 64 bytes from the start. */
+L(copy64_from_start):
+ ldp1 G_l, G_h, [src, 48]
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [src, 32]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [src, 16]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [src]
+ stp D_l, D_h, [dstend, -64]
+ stp G_l, G_h, [dstin, 48]
+ stp A_l, A_h, [dstin, 32]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstin]
+ ret1
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d286e0a04543..da21a13151b9 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -79,6 +79,18 @@ void *memcpy(void *dest, const void *src, size_t len)
}
#endif
+#ifdef __HAVE_ARCH_MEMCPY_MC
+#undef memcpy_mc
+unsigned long memcpy_mc(void *dest, const void *src, size_t len)
+{
+ if (!kasan_check_range(src, len, false, _RET_IP_) ||
+ !kasan_check_range(dest, len, true, _RET_IP_))
+ return len;
+
+ return __memcpy_mc(dest, src, len);
+}
+#endif
+
void *__asan_memset(void *addr, int c, ssize_t len)
{
if (!kasan_check_range(addr, len, true, _RET_IP_))
--
2.39.3
^ permalink raw reply related
* [PATCH v15 6/9] arm64: support copy_mc_[user]_highpage()
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
Currently, many scenarios that can tolerate memory errors when copying page
have been supported in the kernel[1~9], all of which are implemented by
copy_mc_[user]_highpage(). arm64 should also support this mechanism.
Due to mte, arm64 needs to have its own copy_mc_[user]_highpage()
architecture implementation, macros __HAVE_ARCH_COPY_MC_HIGHPAGE and
__HAVE_ARCH_COPY_MC_USER_HIGHPAGE have been added to control it.
Add new helper copy_mc_page() which provide a page copy implementation with
hardware memory error safe. The code logic of copy_mc_page() is the same as
copy_page(), the main difference is that the ldp insn of copy_mc_page()
contains the fixup type EX_TYPE_KACCESS_SEA, therefore, the
main logic is extracted to copy_page_template.S. In addition, the fixup of
MOPS insn is not considered at present.
[Ruidong: add FEAT_MOPS support]
[1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
[2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
[3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
[4] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
[5] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
[6] commit 658be46520ce ("mm: support poison recovery from copy_present_page()")
[7] commit aa549f923f5e ("mm: support poison recovery from do_cow_fault()")
[8] commit f00b295b9b61 ("fs: hugetlbfs: support poisoned recover from hugetlbfs_migrate_folio()")
[9] commit 060913999d7a ("mm: migrate: support poisoned recover from migrate folio")
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Signed-off-by: Ruidong Tian <tianruidong@linux.alibaba.com>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/asm-extable.h | 4 ++
arch/arm64/include/asm/mte.h | 9 ++++
arch/arm64/include/asm/page.h | 12 +++++
arch/arm64/lib/Makefile | 2 +
arch/arm64/lib/copy_mc_page.S | 44 +++++++++++++++
arch/arm64/lib/copy_page.S | 67 ++++-------------------
arch/arm64/lib/copy_page_template.S | 70 ++++++++++++++++++++++++
arch/arm64/lib/mte.S | 29 ++++++++++
arch/arm64/mm/copypage.c | 80 ++++++++++++++++++++++++++++
include/linux/highmem.h | 8 +++
11 files changed, 270 insertions(+), 56 deletions(-)
create mode 100644 arch/arm64/lib/copy_mc_page.S
create mode 100644 arch/arm64/lib/copy_page_template.S
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..831b20d45893 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -21,6 +21,7 @@ config ARM64
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CC_PLATFORM
select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
+ select ARCH_HAS_COPY_MC if ACPI_APEI_GHES
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
index 8450ec5a3af6..9305ea77482a 100644
--- a/arch/arm64/include/asm/asm-extable.h
+++ b/arch/arm64/include/asm/asm-extable.h
@@ -10,6 +10,10 @@
#define EX_TYPE_ACCESS_ERR_ZERO 2
#define EX_TYPE_UACCESS_CPY 3
#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 4
+/*
+ * Kernel access: used in kernel context for both regular load/store
+ * instructions and MOPS (memory copy/set) instructions.
+ */
#define EX_TYPE_KACCESS_SEA 5
/* Data fields for EX_TYPE_ACCESS_ERR_ZERO */
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 7f7b97e09996..a0b1757f4847 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -98,6 +98,11 @@ static inline bool try_page_mte_tagging(struct page *page)
void mte_zero_clear_page_tags(void *addr);
void mte_sync_tags(pte_t pte, unsigned int nr_pages);
void mte_copy_page_tags(void *kto, const void *kfrom);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int mte_copy_mc_page_tags(void *kto, const void *kfrom);
+#endif
+
void mte_thread_init_user(void);
void mte_thread_switch(struct task_struct *next);
void mte_cpu_setup(void);
@@ -134,6 +139,10 @@ static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages)
static inline void mte_copy_page_tags(void *kto, const void *kfrom)
{
}
+static inline int mte_copy_mc_page_tags(void *kto, const void *kfrom)
+{
+ return 0;
+}
static inline void mte_thread_init_user(void)
{
}
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index e25d0d18f6d7..5c4c9f974b68 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -29,6 +29,18 @@ void copy_user_highpage(struct page *to, struct page *from,
void copy_highpage(struct page *to, struct page *from);
#define __HAVE_ARCH_COPY_HIGHPAGE
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int copy_mc_page(void *to, const void *from);
+#define __HAVE_ARCH_COPY_MC_PAGE
+
+int copy_mc_highpage(struct page *to, struct page *from);
+#define __HAVE_ARCH_COPY_MC_HIGHPAGE
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma);
+#define __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
+#endif
+
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr);
#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 448c917494f3..1f4c3f743a20 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,6 +7,8 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
obj-$(CONFIG_ARM64_MTE) += mte.o
diff --git a/arch/arm64/lib/copy_mc_page.S b/arch/arm64/lib/copy_mc_page.S
new file mode 100644
index 000000000000..f936e0c98611
--- /dev/null
+++ b/arch/arm64/lib/copy_mc_page.S
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#include <linux/linkage.h>
+#include <linux/const.h>
+#include <asm/assembler.h>
+#include <asm/page.h>
+#include <asm/cpufeature.h>
+#include <asm/alternative.h>
+#include <asm/asm-extable.h>
+#include <asm/asm-uaccess.h>
+
+/*
+ * Copy a page from src to dest (both are page aligned) with memory error safe
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ * Returns:
+ * x0 - Return 0 if copy success, or -EFAULT if anything goes wrong
+ * while copying.
+ */
+ .macro ldp1 reg1, reg2, ptr, val
+ KERNEL_SEA(9998f, ldp \reg1, \reg2, [\ptr, \val])
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ KERNEL_SEA(9998f, cpypwn [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9998f, cpymwn [\dst]!, [\src]!, \count!)
+ KERNEL_SEA(9998f, cpyewn [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__pi_copy_mc_page)
+#include "copy_page_template.S"
+
+ mov x0, #0
+ ret
+
+9998: mov x0, #-EFAULT
+ ret
+
+SYM_FUNC_END(__pi_copy_mc_page)
+SYM_FUNC_ALIAS(copy_mc_page, __pi_copy_mc_page)
+EXPORT_SYMBOL(copy_mc_page)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index e6374e7e5511..e520777b5150 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -17,65 +17,20 @@
* x0 - dest
* x1 - src
*/
-SYM_FUNC_START(__pi_copy_page)
-#ifdef CONFIG_AS_HAS_MOPS
- .arch_extension mops
-alternative_if_not ARM64_HAS_MOPS
- b .Lno_mops
-alternative_else_nop_endif
-
- mov x2, #PAGE_SIZE
- cpypwn [x0]!, [x1]!, x2!
- cpymwn [x0]!, [x1]!, x2!
- cpyewn [x0]!, [x1]!, x2!
- ret
-.Lno_mops:
-#endif
- ldp x2, x3, [x1]
- ldp x4, x5, [x1, #16]
- ldp x6, x7, [x1, #32]
- ldp x8, x9, [x1, #48]
- ldp x10, x11, [x1, #64]
- ldp x12, x13, [x1, #80]
- ldp x14, x15, [x1, #96]
- ldp x16, x17, [x1, #112]
-
- add x0, x0, #256
- add x1, x1, #128
-1:
- tst x0, #(PAGE_SIZE - 1)
- stnp x2, x3, [x0, #-256]
- ldp x2, x3, [x1]
- stnp x4, x5, [x0, #16 - 256]
- ldp x4, x5, [x1, #16]
- stnp x6, x7, [x0, #32 - 256]
- ldp x6, x7, [x1, #32]
- stnp x8, x9, [x0, #48 - 256]
- ldp x8, x9, [x1, #48]
- stnp x10, x11, [x0, #64 - 256]
- ldp x10, x11, [x1, #64]
- stnp x12, x13, [x0, #80 - 256]
- ldp x12, x13, [x1, #80]
- stnp x14, x15, [x0, #96 - 256]
- ldp x14, x15, [x1, #96]
- stnp x16, x17, [x0, #112 - 256]
- ldp x16, x17, [x1, #112]
+ .macro ldp1 reg1, reg2, ptr, val
+ ldp \reg1, \reg2, [\ptr, \val]
+ .endm
- add x0, x0, #128
- add x1, x1, #128
-
- b.ne 1b
-
- stnp x2, x3, [x0, #-256]
- stnp x4, x5, [x0, #16 - 256]
- stnp x6, x7, [x0, #32 - 256]
- stnp x8, x9, [x0, #48 - 256]
- stnp x10, x11, [x0, #64 - 256]
- stnp x12, x13, [x0, #80 - 256]
- stnp x14, x15, [x0, #96 - 256]
- stnp x16, x17, [x0, #112 - 256]
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ cpypwn [\dst]!, [\src]!, \count!
+ cpymwn [\dst]!, [\src]!, \count!
+ cpyewn [\dst]!, [\src]!, \count!
+ .endm
+SYM_FUNC_START(__pi_copy_page)
+#include "copy_page_template.S"
ret
SYM_FUNC_END(__pi_copy_page)
SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
diff --git a/arch/arm64/lib/copy_page_template.S b/arch/arm64/lib/copy_page_template.S
new file mode 100644
index 000000000000..e5afbeaaad25
--- /dev/null
+++ b/arch/arm64/lib/copy_page_template.S
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ */
+
+/*
+ * Copy a page from src to dest (both are page aligned)
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ */
+dstin .req x0
+src .req x1
+
+#ifdef CONFIG_AS_HAS_MOPS
+alternative_if_not ARM64_HAS_MOPS
+ b .Lno_mops
+alternative_else_nop_endif
+ mov x2, #PAGE_SIZE
+ cpy1 dst, src, x2
+ b .Lexitfunc
+.Lno_mops:
+#endif
+
+ ldp1 x2, x3, x1, #0
+ ldp1 x4, x5, x1, #16
+ ldp1 x6, x7, x1, #32
+ ldp1 x8, x9, x1, #48
+ ldp1 x10, x11, x1, #64
+ ldp1 x12, x13, x1, #80
+ ldp1 x14, x15, x1, #96
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #256
+ add x1, x1, #128
+1:
+ tst x0, #(PAGE_SIZE - 1)
+
+ stnp x2, x3, [x0, #-256]
+ ldp1 x2, x3, x1, #0
+ stnp x4, x5, [x0, #16 - 256]
+ ldp1 x4, x5, x1, #16
+ stnp x6, x7, [x0, #32 - 256]
+ ldp1 x6, x7, x1, #32
+ stnp x8, x9, [x0, #48 - 256]
+ ldp1 x8, x9, x1, #48
+ stnp x10, x11, [x0, #64 - 256]
+ ldp1 x10, x11, x1, #64
+ stnp x12, x13, [x0, #80 - 256]
+ ldp1 x12, x13, x1, #80
+ stnp x14, x15, [x0, #96 - 256]
+ ldp1 x14, x15, x1, #96
+ stnp x16, x17, [x0, #112 - 256]
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #128
+ add x1, x1, #128
+
+ b.ne 1b
+
+ stnp x2, x3, [x0, #-256]
+ stnp x4, x5, [x0, #16 - 256]
+ stnp x6, x7, [x0, #32 - 256]
+ stnp x8, x9, [x0, #48 - 256]
+ stnp x10, x11, [x0, #64 - 256]
+ stnp x12, x13, [x0, #80 - 256]
+ stnp x14, x15, [x0, #96 - 256]
+ stnp x16, x17, [x0, #112 - 256]
+.Lexitfunc:
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 5018ac03b6bf..1afe3ef1502c 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -80,6 +80,35 @@ SYM_FUNC_START(mte_copy_page_tags)
ret
SYM_FUNC_END(mte_copy_page_tags)
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Copy the tags from the source page to the destination one with memory error safe
+ * x0 - address of the destination page
+ * x1 - address of the source page
+ * Returns:
+ * x0 - Return 0 if copy success, or
+ * -EFAULT if anything goes wrong while copying.
+ */
+SYM_FUNC_START(mte_copy_mc_page_tags)
+ mov x2, x0
+ mov x3, x1
+ multitag_transfer_size x5, x6
+1:
+KERNEL_SEA(2f, ldgm x4, [x3])
+ stgm x4, [x2]
+ add x2, x2, x5
+ add x3, x3, x5
+ tst x2, #(PAGE_SIZE - 1)
+ b.ne 1b
+
+ mov x0, #0
+ ret
+
+2: mov x0, #-EFAULT
+ ret
+SYM_FUNC_END(mte_copy_mc_page_tags)
+#endif
+
/*
* Read tags from a user buffer (one tag per byte) and set the corresponding
* tags at the given kernel address. Used by PTRACE_POKEMTETAGS.
diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c
index cd5912ba617b..c22918ed0f3c 100644
--- a/arch/arm64/mm/copypage.c
+++ b/arch/arm64/mm/copypage.c
@@ -72,3 +72,83 @@ void copy_user_highpage(struct page *to, struct page *from,
flush_dcache_page(to);
}
EXPORT_SYMBOL_GPL(copy_user_highpage);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Return -EFAULT if anything goes wrong while copying page or mte.
+ */
+int copy_mc_highpage(struct page *to, struct page *from)
+{
+ void *kto = page_address(to);
+ void *kfrom = page_address(from);
+ struct folio *src = page_folio(from);
+ struct folio *dst = page_folio(to);
+ unsigned int i, nr_pages;
+ int ret;
+
+ ret = copy_mc_page(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+
+ if (kasan_hw_tags_enabled())
+ page_kasan_tag_reset(to);
+
+ if (!system_supports_mte())
+ return 0;
+
+ if (folio_test_hugetlb(src)) {
+ if (!folio_test_hugetlb_mte_tagged(src) ||
+ from != folio_page(src, 0))
+ return 0;
+
+ WARN_ON_ONCE(!folio_try_hugetlb_mte_tagging(dst));
+
+ /*
+ * Populate tags for all subpages.
+ *
+ * Don't assume the first page is head page since
+ * huge page copy may start from any subpage.
+ */
+ nr_pages = folio_nr_pages(src);
+ for (i = 0; i < nr_pages; i++) {
+ kfrom = page_address(folio_page(src, i));
+ kto = page_address(folio_page(dst, i));
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ }
+ folio_set_hugetlb_mte_tagged(dst);
+ } else if (page_mte_tagged(from)) {
+ /* It's a new page, shouldn't have been tagged yet */
+ WARN_ON_ONCE(!try_page_mte_tagging(to));
+
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ set_page_mte_tagged(to);
+ }
+ /*
+ * memory_failure_queue() is not called here because on arm64
+ * the firmware (GHES) has already reported the hardware memory
+ * error and queued the page for memory_failure() handling via
+ * ghes_do_memory_failure().
+ */
+ return 0;
+}
+EXPORT_SYMBOL(copy_mc_highpage);
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma)
+{
+ int ret;
+
+ ret = copy_mc_highpage(to, from);
+ if (ret)
+ return ret;
+
+ flush_dcache_page(to);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(copy_mc_user_highpage);
+#endif
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 18dc4aca4aa1..f168c9d4ad0e 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -424,6 +424,7 @@ static inline void copy_highpage(struct page *to, struct page *from)
#endif
#ifdef copy_mc_to_kernel
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
@@ -449,7 +450,9 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
return ret ? -EFAULT : 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
unsigned long ret;
@@ -468,20 +471,25 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
return ret ? -EFAULT : 0;
}
+#endif
#else
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
{
copy_user_highpage(to, from, vaddr, vma);
return 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
copy_highpage(to, from);
return 0;
}
#endif
+#endif
static inline void memcpy_page(struct page *dst_page, size_t dst_off,
struct page *src_page, size_t src_off,
--
2.39.3
^ permalink raw reply related
* [PATCH v15 5/9] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage()
From: Ruidong Tian @ 2026-06-18 9:21 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong, Jonathan Cameron, Mauro Carvalho Chehab
In-Reply-To: <20260618092124.3901230-1-tianruidong@linux.alibaba.com>
From: Tong Tiangen <tongtiangen@huawei.com>
Currently, copy_mc_[user]_highpage() returns zero on success, or in case
of failures, the number of bytes that weren't copied.
While tracking the number of not copied works fine for x86 and PPC, There
are some difficulties in doing the same thing on ARM64 because there is no
available caller-saved register in copy_page()(lib/copy_page.S) to save
"bytes not copied", and the following copy_mc_page() will also encounter
the same problem.
Consider the caller of copy_mc_[user]_highpage() cannot do any processing
on the remaining data(The page has hardware errors), they only check if
copy was succeeded or not, make the interface more generic by using an
error code when copy fails (-EFAULT) or return zero on success.
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Shuai Xue <xueshuai@linux.alibaba.com>
---
include/linux/highmem.h | 8 ++++----
mm/khugepaged.c | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index af03db851a1d..18dc4aca4aa1 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -427,8 +427,8 @@ static inline void copy_highpage(struct page *to, struct page *from)
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
- * page with #MC in source page (@from) handled, and return the number
- * of bytes not copied if there was a #MC, otherwise 0 for success.
+ * page with #MC in source page (@from) handled, and return -EFAULT if there
+ * was a #MC, otherwise 0 for success.
*/
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
@@ -447,7 +447,7 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
static inline int copy_mc_highpage(struct page *to, struct page *from)
@@ -466,7 +466,7 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
#else
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b8452dbdb043..cf1b78eed3c3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -810,7 +810,7 @@ static enum scan_result __collapse_huge_page_copy(pte_t *pte, struct folio *foli
continue;
}
src_page = pte_page(pteval);
- if (copy_mc_user_highpage(page, src_page, src_addr, vma) > 0) {
+ if (copy_mc_user_highpage(page, src_page, src_addr, vma)) {
result = SCAN_COPY_MC;
break;
}
@@ -2143,7 +2143,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
}
for (i = 0; i < nr_pages; i++) {
- if (copy_mc_highpage(dst, folio_page(folio, i)) > 0) {
+ if (copy_mc_highpage(dst, folio_page(folio, i))) {
result = SCAN_COPY_MC;
goto rollback;
}
--
2.39.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox