* [RFC PATCH 01/28] dt-bindings: media: add Arm Mali-V76 VPU
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 02/28] media: mali-vpu: add Mali VPU platform driver Charvi Mehta
` (26 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add devicetree binding for the Arm Mali-V76 video processing unit,
a hardware accelerator for video encoding and decoding.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../bindings/media/arm,mali-v76.yaml | 70 +++++++++++++++++++
1 file changed, 70 insertions(+)
create mode 100644 Documentation/devicetree/bindings/media/arm,mali-v76.yaml
diff --git a/Documentation/devicetree/bindings/media/arm,mali-v76.yaml b/Documentation/devicetree/bindings/media/arm,mali-v76.yaml
new file mode 100644
index 0000000000000..2ac4934ddaf4d
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/arm,mali-v76.yaml
@@ -0,0 +1,70 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/arm,mali-v76.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: ARM Mali-V76 Video Processing Unit
+
+maintainers:
+ - Charvi Mehta <charvi.mehta@arm.com>
+
+description:
+ The ARM Mali-V76 VPU is a hardware accelerator for video encoding and
+ decoding present on ARM platforms.
+
+properties:
+ compatible:
+ const: arm,mali-v76
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ items:
+ - description: VPU core clock
+
+ clock-names:
+ items:
+ - const: core
+
+ resets:
+ description:
+ Optional VPU warm reset control. Platforms without this control must
+ ensure that the VPU is released from external reset before the driver
+ accesses it. The driver performs initialization and recovery resets
+ through the VPU's software reset register.
+ items:
+ - description: VPU warm reset
+
+ reset-names:
+ items:
+ - const: warm
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+
+dependencies:
+ resets: [ reset-names ]
+ reset-names: [ resets ]
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+
+ mali_v76: video-codec@6f030000 {
+ compatible = "arm,mali-v76";
+ reg = <0x6f030000 0x300>;
+ interrupts = <GIC_SPI 168 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clock 0>;
+ clock-names = "core";
+ };
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 02/28] media: mali-vpu: add Mali VPU platform driver
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 01/28] dt-bindings: media: add Arm Mali-V76 VPU Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 03/28] media: mali-vpu: register V4L2 decoder device and add querycap ioctl Charvi Mehta
` (25 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
In preparation for adding H.264 video decode support, add the platform
driver for Arm Mali-VPU.
Add probe and remove operations to acquire and manage the MMIO region,
interrupt, clocks and reset controls. Add the variant data for Mali-V76
and validate the hardware product ID during probe.
Restrict the driver to little-endian kernels, as the firmware interface
uses little-endian shared memory structures accessed in native-byte order.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/Kconfig | 1 +
drivers/media/platform/arm/Makefile | 1 +
drivers/media/platform/arm/mali-vpu/Kconfig | 14 ++
drivers/media/platform/arm/mali-vpu/Makefile | 6 +
.../platform/arm/mali-vpu/mali-vpu-device.h | 38 ++++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 215 ++++++++++++++++++
.../arm/mali-vpu/mali-vpu-registers.h | 24 ++
.../arm/mali-vpu/mali-vpu-variant-v76.c | 21 ++
.../platform/arm/mali-vpu/mali-vpu-variant.h | 25 ++
9 files changed, 345 insertions(+)
create mode 100644 drivers/media/platform/arm/mali-vpu/Kconfig
create mode 100644 drivers/media/platform/arm/mali-vpu/Makefile
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
diff --git a/drivers/media/platform/arm/Kconfig b/drivers/media/platform/arm/Kconfig
index 4f0764c329c74..339969d0ef4ce 100644
--- a/drivers/media/platform/arm/Kconfig
+++ b/drivers/media/platform/arm/Kconfig
@@ -3,3 +3,4 @@
comment "ARM media platform drivers"
source "drivers/media/platform/arm/mali-c55/Kconfig"
+source "drivers/media/platform/arm/mali-vpu/Kconfig"
diff --git a/drivers/media/platform/arm/Makefile b/drivers/media/platform/arm/Makefile
index 8cc4918725efa..3c38ad267e59a 100644
--- a/drivers/media/platform/arm/Makefile
+++ b/drivers/media/platform/arm/Makefile
@@ -1,2 +1,3 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-y += mali-c55/
+obj-y += mali-vpu/
diff --git a/drivers/media/platform/arm/mali-vpu/Kconfig b/drivers/media/platform/arm/mali-vpu/Kconfig
new file mode 100644
index 0000000000000..86403d7f424f9
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/Kconfig
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VIDEO_MALI_VPU
+ tristate "ARM Mali VPU driver"
+ depends on VIDEO_DEV
+ depends on !CPU_BIG_ENDIAN
+ help
+ Enable this to support Arm's Mali Video hardware accelerator.
+ This is a V4L2 driver.
+
+ Only little-endian kernels are supported. The firmware interface
+ uses little-endian shared-memory structures.
+
+ To compile this driver as a module, choose M here: the module
+ will be called mali-vpu.
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
new file mode 100644
index 0000000000000..22acf82e99a88
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+mali-vpu-objs := mali-vpu-probe.o \
+ mali-vpu-variant-v76.o
+
+obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
new file mode 100644
index 0000000000000..04160090e64df
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_DEVICE_H__
+#define __MALI_VPU_DEVICE_H__
+
+#include <linux/device.h>
+
+struct clk_bulk_data;
+struct mali_vpu_variant;
+struct reset_control_bulk_data;
+
+/**
+ * struct mali_vpu_dev - Mali VPU device
+ * @dev: Reference to device structure.
+ * @regs_base: MMIO base address for VPU registers.
+ * @clk_list: List of VPU clocks.
+ * @clk_count: Number of clocks in @clk_list.
+ * @rst_list: List of reset controls.
+ * @rst_count: Number of reset controls in @rst_list.
+ * @irq: IRQ number for VPU.
+ * @variant: Structure describing variant-specific features and capabilities.
+ */
+struct mali_vpu_dev {
+ struct device *dev;
+ void __iomem *regs_base;
+ struct clk_bulk_data *clk_list;
+ unsigned int clk_count;
+ struct reset_control_bulk_data *rst_list;
+ unsigned int rst_count;
+ int irq;
+
+ const struct mali_vpu_variant *variant;
+};
+
+#endif /* __MALI_VPU_DEVICE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
new file mode 100644
index 0000000000000..49ecd6ac4554c
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -0,0 +1,215 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/io.h>
+#include <linux/mod_devicetable.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "mali-vpu-device.h"
+#include "mali-vpu-registers.h"
+#include "mali-vpu-variant.h"
+
+/* VPU register access helpers */
+static u32 mali_vpu_read_reg(void __iomem *regs_base, u32 off)
+{
+ return readl(regs_base + off);
+}
+
+static void mali_vpu_read_version(struct mali_vpu_dev *vpu_dev, u32 *product_id, u32 *rev,
+ u32 *patch)
+{
+ u32 pid_prefix, pid_suffix;
+
+ u32 hwver = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_VERSION);
+
+ pid_prefix = FIELD_GET(MALI_VPU_VERSION_PRODUCT_ID_MASK, hwver);
+ pid_suffix = FIELD_GET(MALI_VPU_PID_SUFFIX_MASK, hwver);
+
+ *product_id = (pid_prefix << 4) | pid_suffix;
+ *rev = FIELD_GET(MALI_VPU_VERSION_REVISION_MASK, hwver);
+ *patch = FIELD_GET(MALI_VPU_VERSION_PATCH_MASK, hwver);
+}
+
+static int mali_vpu_get_clocks(struct mali_vpu_dev *vpu_dev)
+{
+ int ret;
+
+ ret = devm_clk_bulk_get_all(vpu_dev->dev, &vpu_dev->clk_list);
+ if (ret < 0)
+ return dev_err_probe(vpu_dev->dev, ret, "failed to get clocks\n");
+ if (!ret)
+ return dev_err_probe(vpu_dev->dev, -EINVAL, "no clocks found\n");
+
+ vpu_dev->clk_count = ret;
+
+ return 0;
+}
+
+static int mali_vpu_get_resets(struct mali_vpu_dev *vpu_dev)
+{
+ unsigned int i;
+ int ret;
+
+ vpu_dev->rst_count = vpu_dev->variant->num_rsts;
+ if (vpu_dev->rst_count == 0)
+ return 0;
+
+ vpu_dev->rst_list = devm_kcalloc(vpu_dev->dev, vpu_dev->rst_count,
+ sizeof(*vpu_dev->rst_list), GFP_KERNEL);
+ if (!vpu_dev->rst_list)
+ return -ENOMEM;
+
+ for (i = 0; i < vpu_dev->rst_count; i++)
+ vpu_dev->rst_list[i].id = vpu_dev->variant->rst_names[i];
+
+ ret = devm_reset_control_bulk_get_optional_exclusive(vpu_dev->dev,
+ vpu_dev->rst_count,
+ vpu_dev->rst_list);
+ if (ret)
+ return dev_err_probe(vpu_dev->dev, ret, "failed to get reset controls\n");
+
+ return 0;
+}
+
+static int mali_vpu_enable_clk(struct mali_vpu_dev *vpu_dev)
+{
+ return clk_bulk_prepare_enable(vpu_dev->clk_count, vpu_dev->clk_list);
+}
+
+static void mali_vpu_disable_clk(struct mali_vpu_dev *vpu_dev)
+{
+ clk_bulk_disable_unprepare(vpu_dev->clk_count, vpu_dev->clk_list);
+}
+
+static void mali_vpu_assert_reset(struct mali_vpu_dev *vpu_dev)
+{
+ if (!vpu_dev->rst_count)
+ return;
+
+ reset_control_bulk_assert(vpu_dev->rst_count, vpu_dev->rst_list);
+}
+
+static int mali_vpu_deassert_reset(struct mali_vpu_dev *vpu_dev)
+{
+ if (!vpu_dev->rst_count)
+ return 0;
+
+ return reset_control_bulk_deassert(vpu_dev->rst_count, vpu_dev->rst_list);
+}
+
+static int mali_vpu_get_resources(struct mali_vpu_dev *vpu_dev)
+{
+ int ret;
+
+ ret = mali_vpu_get_clocks(vpu_dev);
+ if (ret)
+ return ret;
+
+ return mali_vpu_get_resets(vpu_dev);
+}
+
+static int mali_vpu_probe(struct platform_device *pdev)
+{
+ struct mali_vpu_dev *vpu_dev;
+ u32 pid, rev, patch;
+ int ret;
+
+ vpu_dev = devm_kzalloc(&pdev->dev, sizeof(*vpu_dev), GFP_KERNEL);
+ if (!vpu_dev)
+ return -ENOMEM;
+ vpu_dev->dev = &pdev->dev;
+
+ vpu_dev->variant = of_device_get_match_data(&pdev->dev);
+ if (!vpu_dev->variant)
+ return -ENODEV;
+
+ vpu_dev->regs_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(vpu_dev->regs_base))
+ return PTR_ERR(vpu_dev->regs_base);
+
+ ret = platform_get_irq(pdev, 0);
+ if (ret < 0)
+ return ret;
+ vpu_dev->irq = ret;
+
+ ret = mali_vpu_get_resources(vpu_dev);
+ if (ret)
+ return ret;
+
+ ret = mali_vpu_enable_clk(vpu_dev);
+ if (ret) {
+ dev_err(vpu_dev->dev, "failed to enable core clock\n");
+ return ret;
+ }
+
+ ret = mali_vpu_deassert_reset(vpu_dev);
+ if (ret) {
+ dev_err(vpu_dev->dev, "failed to deassert reset\n");
+ goto err_clk;
+ }
+
+ mali_vpu_read_version(vpu_dev, &pid, &rev, &patch);
+ dev_info(&pdev->dev, "Mali VPU device product ID: %#x, revision: %#x, patch: %#x\n",
+ pid, rev, patch);
+
+ if (pid != vpu_dev->variant->expected_hw_pid) {
+ dev_err(vpu_dev->dev, "unexpected hardware product ID: %#x (expected %#x)\n",
+ pid, vpu_dev->variant->expected_hw_pid);
+ ret = -ENODEV;
+ goto err_rst_clk;
+ }
+
+ platform_set_drvdata(pdev, vpu_dev);
+
+ return 0;
+
+err_rst_clk:
+ mali_vpu_assert_reset(vpu_dev);
+err_clk:
+ mali_vpu_disable_clk(vpu_dev);
+
+ return ret;
+}
+
+static void mali_vpu_remove(struct platform_device *pdev)
+{
+ struct mali_vpu_dev *vpu_dev = platform_get_drvdata(pdev);
+
+ mali_vpu_assert_reset(vpu_dev);
+ mali_vpu_disable_clk(vpu_dev);
+
+ platform_set_drvdata(pdev, NULL);
+}
+
+static const struct of_device_id mali_vpu_dt_match[] = {
+ {
+ .compatible = "arm,mali-v76",
+ .data = &mali_vpu_v76_variant,
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(of, mali_vpu_dt_match);
+
+static struct platform_driver mali_vpu_driver = {
+ .probe = mali_vpu_probe,
+ .remove = mali_vpu_remove,
+ .driver = {
+ .name = "mali-vpu",
+ .of_match_table = mali_vpu_dt_match,
+ },
+};
+
+module_platform_driver(mali_vpu_driver);
+MODULE_DESCRIPTION("Arm Mali VPU driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
new file mode 100644
index 0000000000000..471a1f2412a7a
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_REGISTERS_H__
+#define __MALI_VPU_REGISTERS_H__
+
+#include <linux/bits.h>
+
+/*
+ * Mali VPU register offsets and bitfield definitions.
+ *
+ * Offsets are relative to the VPU MMIO base.
+ */
+#define MALI_VPU_REG_VERSION 0x00000
+
+/* Version register fields */
+#define MALI_VPU_VERSION_PRODUCT_ID_MASK GENMASK(31, 16)
+#define MALI_VPU_VERSION_REVISION_MASK GENMASK(15, 8)
+#define MALI_VPU_VERSION_PATCH_MASK GENMASK(7, 0)
+#define MALI_VPU_PID_SUFFIX_MASK GENMASK(15, 12)
+
+#endif /* __MALI_VPU_REGISTERS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
new file mode 100644
index 0000000000000..36509d6a1c5d5
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/array_size.h>
+
+#include "mali-vpu-variant.h"
+
+#define MALI_VPU_HW_PID_V76 0x56627
+
+static const char * const mali_v76_rst_names[] = {
+ "warm",
+};
+
+const struct mali_vpu_variant mali_vpu_v76_variant = {
+ .expected_hw_pid = MALI_VPU_HW_PID_V76,
+
+ .rst_names = mali_v76_rst_names,
+ .num_rsts = ARRAY_SIZE(mali_v76_rst_names),
+};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
new file mode 100644
index 0000000000000..dadfd686275db
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_VARIANT_H__
+#define __MALI_VPU_VARIANT_H__
+
+#include <linux/types.h>
+
+/**
+ * struct mali_vpu_variant - Variant-specific hardware capability descriptor.
+ * @rst_names: Array of reset control names for this variant.
+ * @num_rsts: Number of reset controls for this variant.
+ * @expected_hw_pid: Expected hardware product ID read from the version register.
+ */
+struct mali_vpu_variant {
+ const char * const *rst_names;
+ unsigned int num_rsts;
+ u32 expected_hw_pid;
+};
+
+extern const struct mali_vpu_variant mali_vpu_v76_variant;
+
+#endif /* __MALI_VPU_VARIANT_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 03/28] media: mali-vpu: register V4L2 decoder device and add querycap ioctl
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 01/28] dt-bindings: media: add Arm Mali-V76 VPU Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 02/28] media: mali-vpu: add Mali VPU platform driver Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 04/28] media: mali-vpu: initialize decoder format state Charvi Mehta
` (24 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Register a V4L2 M2M decoder video device for the Mali VPU.
Add open and release file operations that allocate a per-file session.
Implement VIDIOC_QUERYCAP to report the driver and decoder identity.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Kconfig | 1 +
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-device.h | 15 ++++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 59 +++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-session.h | 26 +++++++
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 78 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-vidioc.h | 15 ++++
7 files changed, 195 insertions(+), 2 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
diff --git a/drivers/media/platform/arm/mali-vpu/Kconfig b/drivers/media/platform/arm/mali-vpu/Kconfig
index 86403d7f424f9..d98872f2b7ec0 100644
--- a/drivers/media/platform/arm/mali-vpu/Kconfig
+++ b/drivers/media/platform/arm/mali-vpu/Kconfig
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
config VIDEO_MALI_VPU
tristate "ARM Mali VPU driver"
+ depends on V4L_MEM2MEM_DRIVERS
depends on VIDEO_DEV
depends on !CPU_BIG_ENDIAN
help
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 22acf82e99a88..e659101bd0eac 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
mali-vpu-objs := mali-vpu-probe.o \
- mali-vpu-variant-v76.o
+ mali-vpu-variant-v76.o \
+ mali-vpu-vidioc.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index 04160090e64df..d9b0801cbb520 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -7,11 +7,22 @@
#define __MALI_VPU_DEVICE_H__
#include <linux/device.h>
+#include <media/v4l2-device.h>
+
+#define MALI_VPU_DRV_NAME "mali-vpu"
struct clk_bulk_data;
struct mali_vpu_variant;
struct reset_control_bulk_data;
+/**
+ * enum mali_vpu_session_role - Mali VPU session role
+ * @MALI_VPU_DECODER: Video decoding session
+ */
+enum mali_vpu_session_role {
+ MALI_VPU_DECODER = 0,
+};
+
/**
* struct mali_vpu_dev - Mali VPU device
* @dev: Reference to device structure.
@@ -22,6 +33,8 @@ struct reset_control_bulk_data;
* @rst_count: Number of reset controls in @rst_list.
* @irq: IRQ number for VPU.
* @variant: Structure describing variant-specific features and capabilities.
+ * @v4l2_dev: V4L2 device structure for mali vpu.
+ * @vdev_dec: Video device structure for decoder.
*/
struct mali_vpu_dev {
struct device *dev;
@@ -33,6 +46,8 @@ struct mali_vpu_dev {
int irq;
const struct mali_vpu_variant *variant;
+ struct v4l2_device v4l2_dev;
+ struct video_device *vdev_dec;
};
#endif /* __MALI_VPU_DEVICE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 49ecd6ac4554c..6d21cb731ee1d 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -15,10 +15,14 @@
#include <linux/platform_device.h>
#include <linux/reset.h>
#include <linux/slab.h>
+#include <linux/string.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
#include "mali-vpu-device.h"
#include "mali-vpu-registers.h"
#include "mali-vpu-variant.h"
+#include "mali-vpu-vidioc.h"
/* VPU register access helpers */
static u32 mali_vpu_read_reg(void __iomem *regs_base, u32 off)
@@ -119,6 +123,44 @@ static int mali_vpu_get_resources(struct mali_vpu_dev *vpu_dev)
return mali_vpu_get_resets(vpu_dev);
}
+static int mali_vpu_register_video_device(struct mali_vpu_dev *vpu_dev,
+ enum mali_vpu_session_role role)
+{
+ struct video_device *vdev;
+ int ret;
+
+ vdev = video_device_alloc();
+ if (!vdev)
+ return -ENOMEM;
+
+ vdev->release = video_device_release;
+ vdev->fops = &mali_vpu_fops;
+ vdev->vfl_dir = VFL_DIR_M2M;
+ vdev->v4l2_dev = &vpu_dev->v4l2_dev;
+ vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+
+ if (role == MALI_VPU_DECODER) {
+ strscpy(vdev->name, "mali-vpu-decoder", sizeof(vdev->name));
+ vdev->ioctl_ops = &mali_vpu_ioctl_ops_dec;
+ vpu_dev->vdev_dec = vdev;
+ } else {
+ ret = -EINVAL;
+ goto err_vdev_release;
+ }
+
+ video_set_drvdata(vdev, vpu_dev);
+
+ ret = video_register_device(vdev, VFL_TYPE_VIDEO, -1);
+ if (ret)
+ goto err_vdev_release;
+
+ return 0;
+
+err_vdev_release:
+ video_device_release(vdev);
+ return ret;
+}
+
static int mali_vpu_probe(struct platform_device *pdev)
{
struct mali_vpu_dev *vpu_dev;
@@ -170,10 +212,22 @@ static int mali_vpu_probe(struct platform_device *pdev)
goto err_rst_clk;
}
+ ret = v4l2_device_register(&pdev->dev, &vpu_dev->v4l2_dev);
+ if (ret) {
+ dev_err(vpu_dev->dev, "failed to register V4L2 device\n");
+ goto err_rst_clk;
+ }
+
+ ret = mali_vpu_register_video_device(vpu_dev, MALI_VPU_DECODER);
+ if (ret)
+ goto err_unreg_v4l2;
+
platform_set_drvdata(pdev, vpu_dev);
return 0;
+err_unreg_v4l2:
+ v4l2_device_unregister(&vpu_dev->v4l2_dev);
err_rst_clk:
mali_vpu_assert_reset(vpu_dev);
err_clk:
@@ -186,6 +240,9 @@ static void mali_vpu_remove(struct platform_device *pdev)
{
struct mali_vpu_dev *vpu_dev = platform_get_drvdata(pdev);
+ video_unregister_device(vpu_dev->vdev_dec);
+ v4l2_device_unregister(&vpu_dev->v4l2_dev);
+
mali_vpu_assert_reset(vpu_dev);
mali_vpu_disable_clk(vpu_dev);
@@ -205,7 +262,7 @@ static struct platform_driver mali_vpu_driver = {
.probe = mali_vpu_probe,
.remove = mali_vpu_remove,
.driver = {
- .name = "mali-vpu",
+ .name = MALI_VPU_DRV_NAME,
.of_match_table = mali_vpu_dt_match,
},
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
new file mode 100644
index 0000000000000..7166c53236f1a
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_SESSION_H__
+#define __MALI_VPU_SESSION_H__
+
+#include <media/v4l2-fh.h>
+
+#include "mali-vpu-device.h"
+
+/**
+ * struct mali_vpu_session - holds per session state and resources.
+ * @vpu_dev: Reference to the parent device structure.
+ * @role: Role of this session.
+ * @fh: V4L2 file handle for this session.
+ */
+struct mali_vpu_session {
+ struct mali_vpu_dev *vpu_dev;
+ enum mali_vpu_session_role role;
+
+ struct v4l2_fh fh;
+};
+
+#endif /* __MALI_VPU_SESSION_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
new file mode 100644
index 0000000000000..7a169b7b14bf3
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -0,0 +1,78 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <media/v4l2-dev.h>
+#include <media/v4l2-fh.h>
+#include <media/v4l2-ioctl.h>
+
+#include "mali-vpu-session.h"
+#include "mali-vpu-vidioc.h"
+
+static int mali_vpu_open(struct file *filp)
+{
+ struct video_device *vdev = video_devdata(filp);
+ struct mali_vpu_dev *vpu_dev = video_drvdata(filp);
+ struct mali_vpu_session *sess;
+ int ret;
+
+ sess = kzalloc_obj(*sess);
+ if (!sess)
+ return -ENOMEM;
+
+ if (vdev == vpu_dev->vdev_dec) {
+ sess->role = MALI_VPU_DECODER;
+ } else {
+ ret = -EINVAL;
+ goto err_free_sess;
+ }
+
+ sess->vpu_dev = vpu_dev;
+
+ v4l2_fh_init(&sess->fh, vdev);
+ v4l2_fh_add(&sess->fh, filp);
+
+ return 0;
+
+err_free_sess:
+ kfree(sess);
+ return ret;
+}
+
+static int mali_vpu_release(struct file *filp)
+{
+ struct mali_vpu_session *sess = container_of(file_to_v4l2_fh(filp),
+ struct mali_vpu_session, fh);
+
+ v4l2_fh_del(&sess->fh, filp);
+ v4l2_fh_exit(&sess->fh);
+ kfree(sess);
+ return 0;
+}
+
+const struct v4l2_file_operations mali_vpu_fops = {
+ .owner = THIS_MODULE,
+ .open = mali_vpu_open,
+ .release = mali_vpu_release,
+ .unlocked_ioctl = video_ioctl2,
+};
+
+static int mali_vpu_querycap(struct file *filp, void *fh,
+ struct v4l2_capability *cap)
+{
+ struct video_device *vdev = video_devdata(filp);
+
+ strscpy(cap->driver, MALI_VPU_DRV_NAME, sizeof(cap->driver));
+ strscpy(cap->card, vdev->name, sizeof(cap->card));
+
+ return 0;
+}
+
+const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
+ .vidioc_querycap = mali_vpu_querycap,
+};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
new file mode 100644
index 0000000000000..e1aedea5bbcb6
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_VIDIOC_H__
+#define __MALI_VPU_VIDIOC_H__
+
+struct v4l2_file_operations;
+struct v4l2_ioctl_ops;
+
+extern const struct v4l2_file_operations mali_vpu_fops;
+extern const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec;
+
+#endif /* __MALI_VPU_VIDIOC_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 04/28] media: mali-vpu: initialize decoder format state
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (2 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 03/28] media: mali-vpu: register V4L2 decoder device and add querycap ioctl Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 05/28] media: mali-vpu: add V4L2 mem2mem and VB2 queue support Charvi Mehta
` (23 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Describe the H.264 OUTPUT format and YUV420M, NV12M and NV21M CAPTURE
formats supported by Mali-V76, together with its frame-size limits.
Add source and destination format state to each decoder session and
initialize both ports from variant defaults. Add the MVE format identifiers
required by the format descriptors.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 4 +-
.../platform/arm/mali-vpu/mali-vpu-dec.c | 16 ++++
.../platform/arm/mali-vpu/mali-vpu-dec.h | 13 +++
.../platform/arm/mali-vpu/mali-vpu-formats.h | 27 ++++++
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 51 ++++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 83 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.h | 18 ++++
.../arm/mali-vpu/mali-vpu-variant-v76.c | 63 ++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-variant.h | 30 +++++++
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 7 ++
10 files changed, 311 insertions(+), 1 deletion(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index e659101bd0eac..0a42c53ae5088 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -2,6 +2,8 @@
mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-variant-v76.o \
- mali-vpu-vidioc.o
+ mali-vpu-vidioc.o \
+ mali-vpu-session.o \
+ mali-vpu-dec.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
new file mode 100644
index 0000000000000..cb6eeee2f04df
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include "mali-vpu-dec.h"
+#include "mali-vpu-session.h"
+#include "mali-vpu-variant.h"
+
+void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
+{
+ const struct mali_vpu_format_caps *variant_fmts = &sess->vpu_dev->variant->fmt_caps;
+
+ sess->port_src.supported_fmts = variant_fmts->dec_src;
+ sess->port_dst.supported_fmts = variant_fmts->dec_dst;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
new file mode 100644
index 0000000000000..2514c897587c4
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_DEC_H__
+#define __MALI_VPU_DEC_H__
+
+struct mali_vpu_session;
+
+void mali_vpu_session_init_dec(struct mali_vpu_session *sess);
+
+#endif /* __MALI_VPU_DEC_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
new file mode 100644
index 0000000000000..13031809fdb4d
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_FORMATS_H__
+#define __MALI_VPU_FORMATS_H__
+
+#include <linux/types.h>
+
+enum mali_vpu_codec_fmt {
+ MALI_VPU_FORMAT_CODEC_H264,
+};
+
+struct mali_vpu_format_desc {
+ u32 fourcc;
+ u16 mve_frame_fmt;
+ u16 codec_fmt;
+ u32 flags;
+};
+
+struct mali_vpu_format_list {
+ const struct mali_vpu_format_desc *fmts;
+ u32 num_fmts;
+};
+
+#endif /* __MALI_VPU_FORMATS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
new file mode 100644
index 0000000000000..aed51eec6ca9c
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_PROTOCOL_DEF_V3_H__
+#define __MALI_VPU_PROTOCOL_DEF_V3_H__
+
+/*****************************************************************************
+ *
+ * Communication protocol between the host/driver and the MVE firmware,
+ * the 'host interface'.
+ *
+ * MVE == MALI Video Engine
+ *
+ * Protocol version 3.1
+ *
+ * Note: Message structs may be expanded in the future; the host should
+ * use the 'size' of the message to determine how many bytes to
+ * read from the message queue, rather than a sizeof(struct).
+ *
+ * All multibyte integer fields in the shared-memory interface are little-endian.
+ * The driver supports only little-endian kernels and therefore accesses these
+ * fields in native byte order.
+ *
+ ****************************************************************************/
+
+ #define MVE_CHROMA_FORMAT_420 (0x1)
+
+/* format bitfield: */
+#define MVE_FORMAT_BF_C (0) /* 3 bits, chroma subsampling */
+#define MVE_FORMAT_BF_B (4) /* 4 bits, max bitdepth minus 8 */
+#define MVE_FORMAT_BF_N (8) /* 2 bits, number of planes */
+#define MVE_FORMAT_BF_V (12) /* 2 bits, format variant */
+/* formats: */
+#define MVE_FORMAT_YUV420_I420 ((MVE_CHROMA_FORMAT_420 << MVE_FORMAT_BF_C) | \
+ ((8 - 8) << MVE_FORMAT_BF_B) | \
+ (3 << MVE_FORMAT_BF_N) | \
+ (0 << MVE_FORMAT_BF_V))
+
+#define MVE_FORMAT_YUV420_NV12 ((MVE_CHROMA_FORMAT_420 << MVE_FORMAT_BF_C) | \
+ ((8 - 8) << MVE_FORMAT_BF_B) | \
+ (2 << MVE_FORMAT_BF_N) | \
+ (0 << MVE_FORMAT_BF_V))
+
+#define MVE_FORMAT_YUV420_NV21 ((MVE_CHROMA_FORMAT_420 << MVE_FORMAT_BF_C) | \
+ ((8 - 8) << MVE_FORMAT_BF_B) | \
+ (2 << MVE_FORMAT_BF_N) | \
+ (1 << MVE_FORMAT_BF_V))
+
+#endif /* __MALI_VPU_PROTOCOL_DEF_V3_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
new file mode 100644
index 0000000000000..9bcb3a73a582e
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+#include <linux/minmax.h>
+#include <linux/mm.h>
+#include <linux/string.h>
+#include <media/v4l2-common.h>
+
+#include "mali-vpu-session.h"
+#include "mali-vpu-dec.h"
+#include "mali-vpu-variant.h"
+
+static int mali_vpu_set_port_fmt(struct mali_vpu_port *port,
+ enum v4l2_buf_type type,
+ u32 width, u32 height)
+{
+ struct v4l2_pix_format_mplane *pix_mp = &port->curr_fmt.fmt.pix_mp;
+ const struct mali_vpu_format_desc *fmt_default;
+ int ret;
+
+ if (!port->supported_fmts.fmts || !port->supported_fmts.num_fmts)
+ return -EINVAL;
+
+ fmt_default = &port->supported_fmts.fmts[0];
+
+ memset(&port->curr_fmt, 0, sizeof(port->curr_fmt));
+ port->curr_fmt.type = type;
+
+ if (fmt_default->flags & V4L2_FMT_FLAG_COMPRESSED) {
+ pix_mp->pixelformat = fmt_default->fourcc;
+ pix_mp->num_planes = 1;
+ pix_mp->width = width;
+ pix_mp->height = height;
+ pix_mp->plane_fmt[0].bytesperline = 0;
+
+ /*
+ * Use one byte per pixel as the default compressed sizeimage,
+ * with a minimum of one page, aligned to a page boundary.
+ */
+ pix_mp->plane_fmt[0].sizeimage = PAGE_ALIGN(max_t(u32, PAGE_SIZE, width * height));
+ } else {
+ ret = v4l2_fill_pixfmt_mp(pix_mp, fmt_default->fourcc, width, height);
+ if (ret)
+ return ret;
+ }
+
+ pix_mp->field = V4L2_FIELD_NONE;
+
+ return 0;
+}
+
+static int mali_vpu_set_port_defaults(struct mali_vpu_session *sess)
+{
+ const struct mali_vpu_framesize_caps *fmsz = &sess->vpu_dev->variant->frmsz_caps;
+ int ret;
+
+ ret = mali_vpu_set_port_fmt(&sess->port_src, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE,
+ fmsz->width_default, fmsz->height_default);
+ if (ret)
+ return ret;
+
+ return mali_vpu_set_port_fmt(&sess->port_dst, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE,
+ fmsz->width_default, fmsz->height_default);
+}
+
+int mali_vpu_session_init(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ if (sess->role != MALI_VPU_DECODER)
+ return -EINVAL;
+
+ mali_vpu_session_init_dec(sess);
+
+ ret = mali_vpu_set_port_defaults(sess);
+ if (ret)
+ return ret;
+
+ return 0;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 7166c53236f1a..a2778ca496381 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -6,21 +6,39 @@
#ifndef __MALI_VPU_SESSION_H__
#define __MALI_VPU_SESSION_H__
+#include <linux/videodev2.h>
#include <media/v4l2-fh.h>
#include "mali-vpu-device.h"
+#include "mali-vpu-formats.h"
+
+/**
+ * struct mali_vpu_port - format state of a src/dst queue in a session.
+ * @curr_fmt: V4L2 format currently configured for the port
+ * @supported_fmts: List of format descriptors supported by the queue.
+ */
+struct mali_vpu_port {
+ struct v4l2_format curr_fmt;
+ struct mali_vpu_format_list supported_fmts;
+};
/**
* struct mali_vpu_session - holds per session state and resources.
* @vpu_dev: Reference to the parent device structure.
* @role: Role of this session.
+ * @port_src: Source (OUTPUT) port for this session.
+ * @port_dst: Destination (CAPTURE) port for this session.
* @fh: V4L2 file handle for this session.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
enum mali_vpu_session_role role;
+ struct mali_vpu_port port_src;
+ struct mali_vpu_port port_dst;
struct v4l2_fh fh;
};
+int mali_vpu_session_init(struct mali_vpu_session *sess);
+
#endif /* __MALI_VPU_SESSION_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
index 36509d6a1c5d5..7b04b51eeedc2 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -4,18 +4,81 @@
*/
#include <linux/array_size.h>
+#include <linux/videodev2.h>
+#include "mali-vpu-formats.h"
+#include "mali-vpu-protocol-def-v3.h"
#include "mali-vpu-variant.h"
#define MALI_VPU_HW_PID_V76 0x56627
+#define MALI_V76_MAX_FRAME_WIDTH 8192
+#define MALI_V76_MAX_FRAME_HEIGHT 8192
+#define MALI_V76_MIN_FRAME_WIDTH 2
+#define MALI_V76_MIN_FRAME_HEIGHT 2
+#define MALI_V76_DEFAULT_FRAME_WIDTH 640
+#define MALI_V76_DEFAULT_FRAME_HEIGHT 480
+
static const char * const mali_v76_rst_names[] = {
"warm",
};
+static const struct mali_vpu_format_desc mali_v76_dec_src_fmts[] = {
+ {
+ .fourcc = V4L2_PIX_FMT_H264,
+ .mve_frame_fmt = 0,
+ .codec_fmt = MALI_VPU_FORMAT_CODEC_H264,
+ .flags = V4L2_FMT_FLAG_COMPRESSED | V4L2_FMT_FLAG_DYN_RESOLUTION,
+ },
+};
+
+static const struct mali_vpu_format_desc mali_v76_dec_dst_fmts[] = {
+ {
+ .fourcc = V4L2_PIX_FMT_YUV420M,
+ .mve_frame_fmt = MVE_FORMAT_YUV420_I420,
+ .codec_fmt = 0,
+ .flags = 0,
+ },
+ {
+ .fourcc = V4L2_PIX_FMT_NV12M,
+ .mve_frame_fmt = MVE_FORMAT_YUV420_NV12,
+ .codec_fmt = 0,
+ .flags = 0,
+ },
+ {
+ .fourcc = V4L2_PIX_FMT_NV21M,
+ .mve_frame_fmt = MVE_FORMAT_YUV420_NV21,
+ .codec_fmt = 0,
+ .flags = 0,
+ },
+};
+
const struct mali_vpu_variant mali_vpu_v76_variant = {
.expected_hw_pid = MALI_VPU_HW_PID_V76,
.rst_names = mali_v76_rst_names,
.num_rsts = ARRAY_SIZE(mali_v76_rst_names),
+
+ .fmt_caps = {
+ .dec_src = {
+ .fmts = mali_v76_dec_src_fmts,
+ .num_fmts = ARRAY_SIZE(mali_v76_dec_src_fmts),
+ },
+ .dec_dst = {
+ .fmts = mali_v76_dec_dst_fmts,
+ .num_fmts = ARRAY_SIZE(mali_v76_dec_dst_fmts),
+ },
+ },
+ .frmsz_caps = {
+ .stepwise = {
+ .min_width = MALI_V76_MIN_FRAME_WIDTH,
+ .max_width = MALI_V76_MAX_FRAME_WIDTH,
+ .step_width = 2,
+ .min_height = MALI_V76_MIN_FRAME_HEIGHT,
+ .max_height = MALI_V76_MAX_FRAME_HEIGHT,
+ .step_height = 2,
+ },
+ .width_default = MALI_V76_DEFAULT_FRAME_WIDTH,
+ .height_default = MALI_V76_DEFAULT_FRAME_HEIGHT,
+ },
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
index dadfd686275db..a46c25a94f18b 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -7,17 +7,47 @@
#define __MALI_VPU_VARIANT_H__
#include <linux/types.h>
+#include <linux/videodev2.h>
+
+#include "mali-vpu-formats.h"
+
+/**
+ * struct mali_vpu_framesize_caps - Variant-specific frame size capability.
+ * @stepwise: Stepwise frame size limits and increments supported by the hardware.
+ * @width_default: Default width used when userspace does not request one.
+ * @height_default: Default height used when userspace does not request one.
+ */
+struct mali_vpu_framesize_caps {
+ struct v4l2_frmsize_stepwise stepwise;
+ u32 width_default;
+ u32 height_default;
+};
+
+/**
+ * struct mali_vpu_format_caps - Pixel/coded format capability tables.
+ * @dec_src: List of formats supported on decoder OUTPUT queue (compressed bitstream in).
+ * @dec_dst: List of formats supported on decoder CAPTURE queue (decoded raw frames).
+ */
+struct mali_vpu_format_caps {
+ const struct mali_vpu_format_list dec_src;
+ const struct mali_vpu_format_list dec_dst;
+};
/**
* struct mali_vpu_variant - Variant-specific hardware capability descriptor.
* @rst_names: Array of reset control names for this variant.
* @num_rsts: Number of reset controls for this variant.
* @expected_hw_pid: Expected hardware product ID read from the version register.
+ * @fmt_caps: Format capability tables for encode/decode paths.
+ * @frmsz_caps: Frame dimension limits and defaults.
*/
struct mali_vpu_variant {
const char * const *rst_names;
unsigned int num_rsts;
u32 expected_hw_pid;
+
+ struct mali_vpu_format_caps fmt_caps;
+ struct mali_vpu_framesize_caps frmsz_caps;
};
extern const struct mali_vpu_variant mali_vpu_v76_variant;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index 7a169b7b14bf3..13468c6baba76 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -37,8 +37,15 @@ static int mali_vpu_open(struct file *filp)
v4l2_fh_init(&sess->fh, vdev);
v4l2_fh_add(&sess->fh, filp);
+ ret = mali_vpu_session_init(sess);
+ if (ret)
+ goto err_v4l2_fh_deinit;
+
return 0;
+err_v4l2_fh_deinit:
+ v4l2_fh_del(&sess->fh, filp);
+ v4l2_fh_exit(&sess->fh);
err_free_sess:
kfree(sess);
return ret;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 05/28] media: mali-vpu: add V4L2 mem2mem and VB2 queue support
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (3 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 04/28] media: mali-vpu: initialize decoder format state Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 06/28] media: mali-vpu: implement format query ioctls Charvi Mehta
` (22 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Initialize the V4L2 mem2mem device and create a mem2mem context for each
session.
Add VB2 operations for queue setup, buffer initialization, validation,
preparation and queueing using the DMA scatter-gather allocator. Wire
up the standard mem2mem buffer ioctls together with mmap and poll support.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Kconfig | 2 +
drivers/media/platform/arm/mali-vpu/Makefile | 4 +-
.../platform/arm/mali-vpu/mali-vpu-buffer.c | 51 ++++++++++
.../platform/arm/mali-vpu/mali-vpu-buffer.h | 21 ++++
.../platform/arm/mali-vpu/mali-vpu-device.h | 3 +
.../platform/arm/mali-vpu/mali-vpu-probe.c | 12 ++-
.../platform/arm/mali-vpu/mali-vpu-session.c | 37 ++++++-
.../platform/arm/mali-vpu/mali-vpu-session.h | 8 ++
.../platform/arm/mali-vpu/mali-vpu-vb2.c | 99 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-vb2.h | 13 +++
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 35 ++++++-
.../platform/arm/mali-vpu/mali-vpu-vidioc.h | 2 +
12 files changed, 281 insertions(+), 6 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.h
diff --git a/drivers/media/platform/arm/mali-vpu/Kconfig b/drivers/media/platform/arm/mali-vpu/Kconfig
index d98872f2b7ec0..2728830f11a4a 100644
--- a/drivers/media/platform/arm/mali-vpu/Kconfig
+++ b/drivers/media/platform/arm/mali-vpu/Kconfig
@@ -4,6 +4,8 @@ config VIDEO_MALI_VPU
depends on V4L_MEM2MEM_DRIVERS
depends on VIDEO_DEV
depends on !CPU_BIG_ENDIAN
+ select V4L2_MEM2MEM_DEV
+ select VIDEOBUF2_DMA_SG
help
Enable this to support Arm's Mali Video hardware accelerator.
This is a V4L2 driver.
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 0a42c53ae5088..94ddde02fc528 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -4,6 +4,8 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-variant-v76.o \
mali-vpu-vidioc.o \
mali-vpu-session.o \
- mali-vpu-dec.o
+ mali-vpu-dec.o \
+ mali-vpu-buffer.o \
+ mali-vpu-vb2.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
new file mode 100644
index 0000000000000..7ace0489996a8
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+
+#include "mali-vpu-buffer.h"
+#include "mali-vpu-session.h"
+
+int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
+
+ if (vb2_buf->num_planes > MALI_VPU_BUFFER_MAX_PLANES) {
+ dev_err(sess->vpu_dev->dev,
+ "buffer has too many planes: %u\n",
+ vb2_buf->num_planes);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int mali_vpu_vb2_buffer_prepare(struct vb2_buffer *vb2_buf)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
+ struct mali_vpu_port *port;
+ struct v4l2_pix_format_mplane *pix_mp;
+ u32 i;
+
+ port = V4L2_TYPE_IS_OUTPUT(vb2_buf->vb2_queue->type) ? &sess->port_src :
+ &sess->port_dst;
+ pix_mp = &port->curr_fmt.fmt.pix_mp;
+
+ for (i = 0; i < pix_mp->num_planes; i++) {
+ if (vb2_plane_size(vb2_buf, i) < pix_mp->plane_fmt[i].sizeimage)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+
+ v4l2_buf->field = V4L2_FIELD_NONE;
+
+ return 0;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
new file mode 100644
index 0000000000000..f5d67d48e5921
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_BUFFER_H__
+#define __MALI_VPU_BUFFER_H__
+
+#include <media/v4l2-mem2mem.h>
+
+#define MALI_VPU_BUFFER_MAX_PLANES 3
+
+struct mali_vpu_buffer {
+ struct v4l2_m2m_buffer m2m_buf;
+};
+
+int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf);
+int mali_vpu_vb2_buffer_prepare(struct vb2_buffer *vb2_buf);
+int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf);
+
+#endif /* __MALI_VPU_BUFFER_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index d9b0801cbb520..26b2bf9e1588e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -12,6 +12,7 @@
#define MALI_VPU_DRV_NAME "mali-vpu"
struct clk_bulk_data;
+struct v4l2_m2m_dev;
struct mali_vpu_variant;
struct reset_control_bulk_data;
@@ -35,6 +36,7 @@ enum mali_vpu_session_role {
* @variant: Structure describing variant-specific features and capabilities.
* @v4l2_dev: V4L2 device structure for mali vpu.
* @vdev_dec: Video device structure for decoder.
+ * @m2m_dev: V4L2 mem2mem device structure for mali vpu.
*/
struct mali_vpu_dev {
struct device *dev;
@@ -48,6 +50,7 @@ struct mali_vpu_dev {
const struct mali_vpu_variant *variant;
struct v4l2_device v4l2_dev;
struct video_device *vdev_dec;
+ struct v4l2_m2m_dev *m2m_dev;
};
#endif /* __MALI_VPU_DEVICE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 6d21cb731ee1d..36a25030d7324 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -18,6 +18,7 @@
#include <linux/string.h>
#include <media/v4l2-dev.h>
#include <media/v4l2-device.h>
+#include <media/v4l2-mem2mem.h>
#include "mali-vpu-device.h"
#include "mali-vpu-registers.h"
@@ -218,14 +219,22 @@ static int mali_vpu_probe(struct platform_device *pdev)
goto err_rst_clk;
}
+ vpu_dev->m2m_dev = v4l2_m2m_init(&mali_vpu_m2m_ops);
+ if (IS_ERR(vpu_dev->m2m_dev)) {
+ ret = PTR_ERR(vpu_dev->m2m_dev);
+ goto err_unreg_v4l2;
+ }
+
ret = mali_vpu_register_video_device(vpu_dev, MALI_VPU_DECODER);
if (ret)
- goto err_unreg_v4l2;
+ goto err_release_m2m;
platform_set_drvdata(pdev, vpu_dev);
return 0;
+err_release_m2m:
+ v4l2_m2m_release(vpu_dev->m2m_dev);
err_unreg_v4l2:
v4l2_device_unregister(&vpu_dev->v4l2_dev);
err_rst_clk:
@@ -241,6 +250,7 @@ static void mali_vpu_remove(struct platform_device *pdev)
struct mali_vpu_dev *vpu_dev = platform_get_drvdata(pdev);
video_unregister_device(vpu_dev->vdev_dec);
+ v4l2_m2m_release(vpu_dev->m2m_dev);
v4l2_device_unregister(&vpu_dev->v4l2_dev);
mali_vpu_assert_reset(vpu_dev);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 9bcb3a73a582e..95050296f9904 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -3,15 +3,18 @@
* Copyright (C) 2026 ARM Limited.
*/
+#include <linux/err.h>
#include <linux/errno.h>
#include <linux/minmax.h>
#include <linux/mm.h>
#include <linux/string.h>
#include <media/v4l2-common.h>
+#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
#include "mali-vpu-dec.h"
#include "mali-vpu-variant.h"
+#include "mali-vpu-vb2.h"
static int mali_vpu_set_port_fmt(struct mali_vpu_port *port,
enum v4l2_buf_type type,
@@ -70,14 +73,42 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
{
int ret;
- if (sess->role != MALI_VPU_DECODER)
- return -EINVAL;
+ mutex_init(&sess->vq_lock);
+
+ if (sess->role != MALI_VPU_DECODER) {
+ ret = -EINVAL;
+ goto err_mutex_destroy;
+ }
mali_vpu_session_init_dec(sess);
ret = mali_vpu_set_port_defaults(sess);
if (ret)
- return ret;
+ goto err_mutex_destroy;
+
+ sess->m2m_ctx = v4l2_m2m_ctx_init(sess->vpu_dev->m2m_dev, sess, mali_vpu_queue_init);
+ if (IS_ERR(sess->m2m_ctx)) {
+ ret = PTR_ERR(sess->m2m_ctx);
+ goto err_mutex_destroy;
+ }
return 0;
+
+err_mutex_destroy:
+ mutex_destroy(&sess->vq_lock);
+ return ret;
+}
+
+void mali_vpu_session_cleanup(struct mali_vpu_session *sess)
+{
+ mutex_lock(&sess->vq_lock);
+
+ if (sess->m2m_ctx) {
+ v4l2_m2m_ctx_release(sess->m2m_ctx);
+ sess->m2m_ctx = NULL;
+ sess->fh.m2m_ctx = NULL;
+ }
+
+ mutex_unlock(&sess->vq_lock);
+ mutex_destroy(&sess->vq_lock);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index a2778ca496381..dfd1833f884af 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -6,12 +6,15 @@
#ifndef __MALI_VPU_SESSION_H__
#define __MALI_VPU_SESSION_H__
+#include <linux/mutex.h>
#include <linux/videodev2.h>
#include <media/v4l2-fh.h>
#include "mali-vpu-device.h"
#include "mali-vpu-formats.h"
+struct v4l2_m2m_ctx;
+
/**
* struct mali_vpu_port - format state of a src/dst queue in a session.
* @curr_fmt: V4L2 format currently configured for the port
@@ -28,17 +31,22 @@ struct mali_vpu_port {
* @role: Role of this session.
* @port_src: Source (OUTPUT) port for this session.
* @port_dst: Destination (CAPTURE) port for this session.
+ * @vq_lock: Mutex lock to serialise per-session queue related ioctls.
* @fh: V4L2 file handle for this session.
+ * @m2m_ctx: V4L2 mem2mem context for this session.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
enum mali_vpu_session_role role;
struct mali_vpu_port port_src;
struct mali_vpu_port port_dst;
+ struct mutex vq_lock; /* lock to serialise per-session ioctls */
struct v4l2_fh fh;
+ struct v4l2_m2m_ctx *m2m_ctx;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
+void mali_vpu_session_cleanup(struct mali_vpu_session *sess);
#endif /* __MALI_VPU_SESSION_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
new file mode 100644
index 0000000000000..acdbb06785f6b
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
@@ -0,0 +1,99 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+#include <media/v4l2-mem2mem.h>
+#include <media/videobuf2-dma-sg.h>
+
+#include "mali-vpu-buffer.h"
+#include "mali-vpu-session.h"
+#include "mali-vpu-vb2.h"
+
+#define MALI_VPU_MIN_BUFFERS 1
+
+static int mali_vpu_vb2_queue_setup(struct vb2_queue *q,
+ unsigned int *num_buffers __always_unused,
+ unsigned int *num_planes,
+ unsigned int sizes[],
+ struct device *alloc_devs[] __always_unused)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(q);
+ struct mali_vpu_port *port = V4L2_TYPE_IS_OUTPUT(q->type) ?
+ &sess->port_src :
+ &sess->port_dst;
+ struct v4l2_pix_format_mplane *pix_mp = &port->curr_fmt.fmt.pix_mp;
+ u32 i;
+
+ if (!pix_mp->num_planes || pix_mp->num_planes > MALI_VPU_BUFFER_MAX_PLANES)
+ return -EINVAL;
+
+ for (i = 0; i < pix_mp->num_planes; i++) {
+ if (!pix_mp->plane_fmt[i].sizeimage)
+ return -EINVAL;
+ }
+
+ if (*num_planes) {
+ if (*num_planes != pix_mp->num_planes)
+ return -EINVAL;
+
+ for (i = 0; i < pix_mp->num_planes; i++) {
+ if (sizes[i] < pix_mp->plane_fmt[i].sizeimage)
+ return -EINVAL;
+ }
+ } else {
+ *num_planes = pix_mp->num_planes;
+ for (i = 0; i < pix_mp->num_planes; i++)
+ sizes[i] = pix_mp->plane_fmt[i].sizeimage;
+ }
+
+ return 0;
+}
+
+static void mali_vpu_vb2_buffer_queue(struct vb2_buffer *vb2_buf)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+
+ v4l2_m2m_buf_queue(sess->m2m_ctx, v4l2_buf);
+}
+
+static const struct vb2_ops mali_vpu_vb2_ops = {
+ .queue_setup = mali_vpu_vb2_queue_setup,
+ .buf_out_validate = mali_vpu_vb2_buffer_out_validate,
+ .buf_init = mali_vpu_vb2_buffer_init,
+ .buf_prepare = mali_vpu_vb2_buffer_prepare,
+ .buf_queue = mali_vpu_vb2_buffer_queue,
+};
+
+static void mali_vpu_queue_init_type(struct mali_vpu_session *sess,
+ struct vb2_queue *vq,
+ enum v4l2_buf_type type)
+{
+ vq->type = type;
+ vq->io_modes = VB2_MMAP | VB2_DMABUF;
+ vq->dev = sess->vpu_dev->dev;
+ vq->lock = &sess->vq_lock;
+ vq->ops = &mali_vpu_vb2_ops;
+ vq->mem_ops = &vb2_dma_sg_memops;
+ vq->drv_priv = sess;
+ vq->buf_struct_size = sizeof(struct mali_vpu_buffer);
+ vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+ vq->min_queued_buffers = MALI_VPU_MIN_BUFFERS;
+}
+
+int mali_vpu_queue_init(void *priv, struct vb2_queue *src_vq,
+ struct vb2_queue *dst_vq)
+{
+ struct mali_vpu_session *sess = priv;
+ int ret;
+
+ mali_vpu_queue_init_type(sess, src_vq, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+ ret = vb2_queue_init(src_vq);
+ if (ret)
+ return ret;
+
+ mali_vpu_queue_init_type(sess, dst_vq, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ return vb2_queue_init(dst_vq);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.h
new file mode 100644
index 0000000000000..4aa80d2136c2f
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_VB2_H__
+#define __MALI_VPU_VB2_H__
+
+struct vb2_queue;
+
+int mali_vpu_queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq);
+
+#endif /* __MALI_VPU_VB2_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index 13468c6baba76..aaf604beff93a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -10,6 +10,7 @@
#include <media/v4l2-dev.h>
#include <media/v4l2-fh.h>
#include <media/v4l2-ioctl.h>
+#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
#include "mali-vpu-vidioc.h"
@@ -41,6 +42,8 @@ static int mali_vpu_open(struct file *filp)
if (ret)
goto err_v4l2_fh_deinit;
+ sess->fh.m2m_ctx = sess->m2m_ctx;
+
return 0;
err_v4l2_fh_deinit:
@@ -56,6 +59,7 @@ static int mali_vpu_release(struct file *filp)
struct mali_vpu_session *sess = container_of(file_to_v4l2_fh(filp),
struct mali_vpu_session, fh);
+ mali_vpu_session_cleanup(sess);
v4l2_fh_del(&sess->fh, filp);
v4l2_fh_exit(&sess->fh);
kfree(sess);
@@ -67,6 +71,27 @@ const struct v4l2_file_operations mali_vpu_fops = {
.open = mali_vpu_open,
.release = mali_vpu_release,
.unlocked_ioctl = video_ioctl2,
+ .poll = v4l2_m2m_fop_poll,
+ .mmap = v4l2_m2m_fop_mmap,
+};
+
+static int mali_vpu_m2m_job_ready(void *priv)
+{
+ /*
+ * The Mali scheduler/LSID path owns hardware dispatch.
+ * v4l2-mem2mem is used only for vb2 queue bookkeeping.
+ */
+ return 0;
+}
+
+static void mali_vpu_m2m_device_run(void *priv)
+{
+ WARN_ON_ONCE(1);
+}
+
+const struct v4l2_m2m_ops mali_vpu_m2m_ops = {
+ .job_ready = mali_vpu_m2m_job_ready,
+ .device_run = mali_vpu_m2m_device_run,
};
static int mali_vpu_querycap(struct file *filp, void *fh,
@@ -81,5 +106,13 @@ static int mali_vpu_querycap(struct file *filp, void *fh,
}
const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
- .vidioc_querycap = mali_vpu_querycap,
+ .vidioc_querycap = mali_vpu_querycap,
+ .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
+ .vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
+ .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
+ .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
+ .vidioc_expbuf = v4l2_m2m_ioctl_expbuf,
+ .vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
+ .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
+ .vidioc_remove_bufs = v4l2_m2m_ioctl_remove_bufs,
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
index e1aedea5bbcb6..e967cae8be137 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.h
@@ -8,8 +8,10 @@
struct v4l2_file_operations;
struct v4l2_ioctl_ops;
+struct v4l2_m2m_ops;
extern const struct v4l2_file_operations mali_vpu_fops;
+extern const struct v4l2_m2m_ops mali_vpu_m2m_ops;
extern const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec;
#endif /* __MALI_VPU_VIDIOC_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 06/28] media: mali-vpu: implement format query ioctls
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (4 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 05/28] media: mali-vpu: add V4L2 mem2mem and VB2 queue support Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 07/28] media: mali-vpu: implement TRY_FMT and S_FMT ioctls Charvi Mehta
` (21 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Expose H.264 as the decoder OUTPUT format and YUV420M, NV12M, NV21M as the
CAPTURE formats through VIDIOC_ENUM_FMT.
Implement VIDIOC_ENUM_FRAMESIZES to report the variant-specific stepwise
frame size limits. Add VIDIOC_G_FMT support to return the current source
and destination formats for each session.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-session.h | 5 +
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 152 ++++++++++++++++--
2 files changed, 146 insertions(+), 11 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index dfd1833f884af..f8a602cb59006 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -49,4 +49,9 @@ struct mali_vpu_session {
int mali_vpu_session_init(struct mali_vpu_session *sess);
void mali_vpu_session_cleanup(struct mali_vpu_session *sess);
+static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
+{
+ return container_of(file_to_v4l2_fh(filp), struct mali_vpu_session, fh);
+}
+
#endif /* __MALI_VPU_SESSION_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index aaf604beff93a..b993255b8c817 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -12,7 +12,9 @@
#include <media/v4l2-ioctl.h>
#include <media/v4l2-mem2mem.h>
+#include "mali-vpu-formats.h"
#include "mali-vpu-session.h"
+#include "mali-vpu-variant.h"
#include "mali-vpu-vidioc.h"
static int mali_vpu_open(struct file *filp)
@@ -56,8 +58,7 @@ static int mali_vpu_open(struct file *filp)
static int mali_vpu_release(struct file *filp)
{
- struct mali_vpu_session *sess = container_of(file_to_v4l2_fh(filp),
- struct mali_vpu_session, fh);
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
mali_vpu_session_cleanup(sess);
v4l2_fh_del(&sess->fh, filp);
@@ -105,14 +106,143 @@ static int mali_vpu_querycap(struct file *filp, void *fh,
return 0;
}
+static int mali_vpu_enum_fmt(struct file *filp, void *fh,
+ struct v4l2_fmtdesc *f)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ const struct mali_vpu_format_list *fmt_list;
+
+ switch (f->type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+ fmt_list = &sess->port_src.supported_fmts;
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+ fmt_list = &sess->port_dst.supported_fmts;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (f->index >= fmt_list->num_fmts)
+ return -EINVAL;
+
+ f->pixelformat = fmt_list->fmts[f->index].fourcc;
+ f->flags = fmt_list->fmts[f->index].flags;
+ return 0;
+}
+
+static const struct v4l2_format *mali_vpu_get_port_fmt(struct mali_vpu_session *sess,
+ u32 type)
+{
+ const struct mali_vpu_port *port;
+
+ switch (type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+ port = &sess->port_src;
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+ port = &sess->port_dst;
+ break;
+ default:
+ return NULL;
+ }
+
+ return &port->curr_fmt;
+}
+
+static int mali_vpu_g_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ const struct v4l2_format *port_fmt;
+ int ret = 0;
+
+ mutex_lock(&sess->vq_lock);
+
+ port_fmt = mali_vpu_get_port_fmt(sess, fmt->type);
+ if (port_fmt)
+ *fmt = *port_fmt;
+ else
+ ret = -EINVAL;
+
+ mutex_unlock(&sess->vq_lock);
+ return ret;
+}
+
+static const struct mali_vpu_format_desc *mali_vpu_find_format(struct mali_vpu_session *sess,
+ u32 pixfmt, u32 type)
+{
+ const struct mali_vpu_format_desc *fmt_list;
+ u32 num_fmts;
+ u32 i;
+
+ switch (type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+ fmt_list = sess->port_src.supported_fmts.fmts;
+ num_fmts = sess->port_src.supported_fmts.num_fmts;
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+ fmt_list = sess->port_dst.supported_fmts.fmts;
+ num_fmts = sess->port_dst.supported_fmts.num_fmts;
+ break;
+ default:
+ return NULL;
+ }
+
+ for (i = 0; i < num_fmts; i++) {
+ if (fmt_list[i].fourcc == pixfmt)
+ return &fmt_list[i];
+ }
+
+ return NULL;
+}
+
+static int mali_vpu_validate_format(struct mali_vpu_session *sess, u32 pixfmt)
+{
+ const struct mali_vpu_format_desc *fmt;
+
+ fmt = mali_vpu_find_format(sess, pixfmt, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ if (!fmt) {
+ fmt = mali_vpu_find_format(sess, pixfmt, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+ if (!fmt)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_enum_framesizes(struct file *filp, void *fh,
+ struct v4l2_frmsizeenum *fsize)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ const struct mali_vpu_framesize_caps *fmsz = &sess->vpu_dev->variant->frmsz_caps;
+ int ret;
+
+ if (fsize->index)
+ return -EINVAL;
+
+ ret = mali_vpu_validate_format(sess, fsize->pixel_format);
+ if (ret)
+ return ret;
+
+ fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+ fsize->stepwise = fmsz->stepwise;
+
+ return 0;
+}
+
const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
- .vidioc_querycap = mali_vpu_querycap,
- .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
- .vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
- .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
- .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
- .vidioc_expbuf = v4l2_m2m_ioctl_expbuf,
- .vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
- .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
- .vidioc_remove_bufs = v4l2_m2m_ioctl_remove_bufs,
+ .vidioc_querycap = mali_vpu_querycap,
+ .vidioc_enum_fmt_vid_out = mali_vpu_enum_fmt,
+ .vidioc_enum_fmt_vid_cap = mali_vpu_enum_fmt,
+ .vidioc_g_fmt_vid_out_mplane = mali_vpu_g_fmt,
+ .vidioc_g_fmt_vid_cap_mplane = mali_vpu_g_fmt,
+ .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
+ .vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
+ .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
+ .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf,
+ .vidioc_expbuf = v4l2_m2m_ioctl_expbuf,
+ .vidioc_qbuf = v4l2_m2m_ioctl_qbuf,
+ .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
+ .vidioc_remove_bufs = v4l2_m2m_ioctl_remove_bufs,
+ .vidioc_enum_framesizes = mali_vpu_enum_framesizes,
};
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 07/28] media: mali-vpu: implement TRY_FMT and S_FMT ioctls
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (5 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 06/28] media: mali-vpu: implement format query ioctls Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:55 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 08/28] media: mali-vpu: register V76 decoder controls and introduce UAPI controls Charvi Mehta
` (20 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Implement VIDIOC_TRY_FMT and VIDIOC_S_FMT for the decoder OUTPUT and
CAPTURE queues.
Move format lookup and buffer layout calculation into shared helpers so
session initialization and format negotiation use the same sizing rules.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-formats.c | 81 +++++++++++
.../platform/arm/mali-vpu/mali-vpu-formats.h | 10 ++
.../platform/arm/mali-vpu/mali-vpu-session.c | 24 +---
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 132 +++++++++++++++---
5 files changed, 209 insertions(+), 41 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 94ddde02fc528..1dc62621d9dc0 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -6,6 +6,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-session.o \
mali-vpu-dec.o \
mali-vpu-buffer.o \
- mali-vpu-vb2.o
+ mali-vpu-vb2.o \
+ mali-vpu-formats.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
new file mode 100644
index 0000000000000..d268435432371
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
@@ -0,0 +1,81 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+#include <linux/minmax.h>
+#include <linux/mm.h>
+#include <linux/overflow.h>
+#include <linux/string.h>
+#include <media/v4l2-common.h>
+
+#include "mali-vpu-formats.h"
+#include "mali-vpu-session.h"
+
+const struct mali_vpu_format_desc *
+mali_vpu_find_format(struct mali_vpu_session *sess, u32 pixfmt,
+ enum v4l2_buf_type type)
+{
+ const struct mali_vpu_format_list *fmt_list;
+ u32 i;
+
+ switch (type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+ fmt_list = &sess->port_src.supported_fmts;
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+ fmt_list = &sess->port_dst.supported_fmts;
+ break;
+ default:
+ return NULL;
+ }
+
+ for (i = 0; i < fmt_list->num_fmts; i++) {
+ if (fmt_list->fmts[i].fourcc == pixfmt)
+ return &fmt_list->fmts[i];
+ }
+
+ return NULL;
+}
+
+int mali_vpu_fill_pixmp_fmt(const struct mali_vpu_format_desc *fmt,
+ u32 width, u32 height,
+ struct v4l2_pix_format_mplane *pix_mp)
+{
+ u32 sizeimage;
+ int ret;
+
+ memset(pix_mp->reserved, 0, sizeof(pix_mp->reserved));
+ memset(pix_mp->plane_fmt, 0, sizeof(pix_mp->plane_fmt));
+ pix_mp->flags = 0;
+
+ if (fmt->flags & V4L2_FMT_FLAG_COMPRESSED) {
+ pix_mp->width = width;
+ pix_mp->height = height;
+ pix_mp->pixelformat = fmt->fourcc;
+ pix_mp->num_planes = 1;
+ pix_mp->plane_fmt[0].bytesperline = 0;
+
+ /*
+ * Use one byte per pixel as the default compressed sizeimage,
+ * with a minimum of one page, aligned to a page boundary.
+ */
+ if (check_mul_overflow(width, height, &sizeimage))
+ return -EOVERFLOW;
+
+ sizeimage = max_t(u32, PAGE_SIZE, sizeimage);
+
+ if (sizeimage > U32_MAX - (PAGE_SIZE - 1))
+ return -EOVERFLOW;
+
+ pix_mp->plane_fmt[0].sizeimage = PAGE_ALIGN(sizeimage);
+
+ } else {
+ ret = v4l2_fill_pixfmt_mp(pix_mp, fmt->fourcc, width, height);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
index 13031809fdb4d..5703142b93b46 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
@@ -7,6 +7,9 @@
#define __MALI_VPU_FORMATS_H__
#include <linux/types.h>
+#include <linux/videodev2.h>
+
+struct mali_vpu_session;
enum mali_vpu_codec_fmt {
MALI_VPU_FORMAT_CODEC_H264,
@@ -24,4 +27,11 @@ struct mali_vpu_format_list {
u32 num_fmts;
};
+const struct mali_vpu_format_desc *mali_vpu_find_format(struct mali_vpu_session *sess,
+ u32 pixfmt,
+ enum v4l2_buf_type type);
+int mali_vpu_fill_pixmp_fmt(const struct mali_vpu_format_desc *fmt,
+ u32 width, u32 height,
+ struct v4l2_pix_format_mplane *pix_mp);
+
#endif /* __MALI_VPU_FORMATS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 95050296f9904..6867f51287f48 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -5,14 +5,12 @@
#include <linux/err.h>
#include <linux/errno.h>
-#include <linux/minmax.h>
-#include <linux/mm.h>
#include <linux/string.h>
-#include <media/v4l2-common.h>
#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
#include "mali-vpu-dec.h"
+#include "mali-vpu-formats.h"
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
@@ -32,23 +30,9 @@ static int mali_vpu_set_port_fmt(struct mali_vpu_port *port,
memset(&port->curr_fmt, 0, sizeof(port->curr_fmt));
port->curr_fmt.type = type;
- if (fmt_default->flags & V4L2_FMT_FLAG_COMPRESSED) {
- pix_mp->pixelformat = fmt_default->fourcc;
- pix_mp->num_planes = 1;
- pix_mp->width = width;
- pix_mp->height = height;
- pix_mp->plane_fmt[0].bytesperline = 0;
-
- /*
- * Use one byte per pixel as the default compressed sizeimage,
- * with a minimum of one page, aligned to a page boundary.
- */
- pix_mp->plane_fmt[0].sizeimage = PAGE_ALIGN(max_t(u32, PAGE_SIZE, width * height));
- } else {
- ret = v4l2_fill_pixfmt_mp(pix_mp, fmt_default->fourcc, width, height);
- if (ret)
- return ret;
- }
+ ret = mali_vpu_fill_pixmp_fmt(fmt_default, width, height, pix_mp);
+ if (ret)
+ return ret;
pix_mp->field = V4L2_FIELD_NONE;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index b993255b8c817..dd142527a1a0c 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -7,6 +7,7 @@
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/string.h>
+#include <media/v4l2-common.h>
#include <media/v4l2-dev.h>
#include <media/v4l2-fh.h>
#include <media/v4l2-ioctl.h>
@@ -168,32 +169,119 @@ static int mali_vpu_g_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
return ret;
}
-static const struct mali_vpu_format_desc *mali_vpu_find_format(struct mali_vpu_session *sess,
- u32 pixfmt, u32 type)
+static int mali_vpu_try_format_common(struct mali_vpu_session *sess, struct v4l2_format *fmt)
{
- const struct mali_vpu_format_desc *fmt_list;
- u32 num_fmts;
- u32 i;
+ const struct mali_vpu_framesize_caps *fmsz = &sess->vpu_dev->variant->frmsz_caps;
+ struct v4l2_pix_format_mplane *pix_mp = &fmt->fmt.pix_mp;
+ const struct mali_vpu_format_desc *vpu_fmt;
+ const struct v4l2_format *port_fmt, *port_fmt_dec;
+ struct vb2_queue *src_q;
+ u32 w, h;
+ int ret = 0;
- switch (type) {
- case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
- fmt_list = sess->port_src.supported_fmts.fmts;
- num_fmts = sess->port_src.supported_fmts.num_fmts;
- break;
- case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
- fmt_list = sess->port_dst.supported_fmts.fmts;
- num_fmts = sess->port_dst.supported_fmts.num_fmts;
- break;
- default:
- return NULL;
+ vpu_fmt = mali_vpu_find_format(sess, pix_mp->pixelformat, fmt->type);
+ if (!vpu_fmt) {
+ port_fmt = mali_vpu_get_port_fmt(sess, fmt->type);
+ if (!port_fmt)
+ return -EINVAL;
+
+ *pix_mp = port_fmt->fmt.pix_mp;
+ return 0;
+ }
+
+ /* Requested format found, validate other fields */
+ w = pix_mp->width ? pix_mp->width : fmsz->width_default;
+ h = pix_mp->height ? pix_mp->height : fmsz->height_default;
+
+ v4l2_apply_frmsize_constraints(&w, &h, &fmsz->stepwise);
+
+ /* Decode CAPTURE queue: lock geometry to input if streaming */
+ if (sess->role == MALI_VPU_DECODER && fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+ src_q = v4l2_m2m_get_src_vq(sess->m2m_ctx);
+ if (vb2_is_streaming(src_q)) {
+ port_fmt_dec = &sess->port_src.curr_fmt;
+ w = port_fmt_dec->fmt.pix_mp.width;
+ h = port_fmt_dec->fmt.pix_mp.height;
+ }
+ }
+
+ ret = mali_vpu_fill_pixmp_fmt(vpu_fmt, w, h, pix_mp);
+ if (ret)
+ return ret;
+
+ if (pix_mp->field == V4L2_FIELD_ANY)
+ pix_mp->field = V4L2_FIELD_NONE;
+
+ return 0;
+}
+
+static int mali_vpu_try_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ int ret;
+
+ mutex_lock(&sess->vq_lock);
+
+ ret = mali_vpu_try_format_common(sess, fmt);
+
+ mutex_unlock(&sess->vq_lock);
+ return ret;
+}
+
+static int mali_vpu_s_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ struct vb2_queue *vq;
+ struct v4l2_pix_format_mplane *pixmp_src = &sess->port_src.curr_fmt.fmt.pix_mp;
+ struct v4l2_pix_format_mplane *pixmp_dst = &sess->port_dst.curr_fmt.fmt.pix_mp;
+ int ret = 0;
+
+ mutex_lock(&sess->vq_lock);
+
+ vq = v4l2_m2m_get_vq(sess->m2m_ctx, fmt->type);
+ if (vb2_is_busy(vq)) {
+ ret = -EBUSY;
+ goto mutex_unlock_return;
}
- for (i = 0; i < num_fmts; i++) {
- if (fmt_list[i].fourcc == pixfmt)
- return &fmt_list[i];
+ ret = mali_vpu_try_format_common(sess, fmt);
+ if (ret)
+ goto mutex_unlock_return;
+
+ /* OUTPUT provides initial CAPTURE geometry */
+ if (V4L2_TYPE_IS_OUTPUT(fmt->type)) {
+ const struct mali_vpu_format_desc *cap_fmt;
+ struct v4l2_pix_format_mplane *cap;
+ struct v4l2_pix_format_mplane *fmt_mp = &fmt->fmt.pix_mp;
+
+ cap = &sess->port_dst.curr_fmt.fmt.pix_mp;
+ cap_fmt = mali_vpu_find_format(sess, cap->pixelformat,
+ V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ if (!cap_fmt) {
+ ret = -EINVAL;
+ goto mutex_unlock_return;
+ }
+
+ ret = mali_vpu_fill_pixmp_fmt(cap_fmt, fmt_mp->width, fmt_mp->height, cap);
+ if (ret)
+ goto mutex_unlock_return;
}
- return NULL;
+ if (V4L2_TYPE_IS_OUTPUT(fmt->type))
+ sess->port_src.curr_fmt = *fmt;
+ else
+ sess->port_dst.curr_fmt = *fmt;
+
+ /* Keep peer queue colorimetry aligned with input */
+ if (V4L2_TYPE_IS_OUTPUT(fmt->type)) {
+ pixmp_dst->colorspace = pixmp_src->colorspace;
+ pixmp_dst->ycbcr_enc = pixmp_src->ycbcr_enc;
+ pixmp_dst->quantization = pixmp_src->quantization;
+ pixmp_dst->xfer_func = pixmp_src->xfer_func;
+ }
+mutex_unlock_return:
+ mutex_unlock(&sess->vq_lock);
+ return ret;
}
static int mali_vpu_validate_format(struct mali_vpu_session *sess, u32 pixfmt)
@@ -236,6 +324,10 @@ const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
.vidioc_enum_fmt_vid_cap = mali_vpu_enum_fmt,
.vidioc_g_fmt_vid_out_mplane = mali_vpu_g_fmt,
.vidioc_g_fmt_vid_cap_mplane = mali_vpu_g_fmt,
+ .vidioc_s_fmt_vid_out_mplane = mali_vpu_s_fmt,
+ .vidioc_s_fmt_vid_cap_mplane = mali_vpu_s_fmt,
+ .vidioc_try_fmt_vid_out_mplane = mali_vpu_try_fmt,
+ .vidioc_try_fmt_vid_cap_mplane = mali_vpu_try_fmt,
.vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs,
.vidioc_querybuf = v4l2_m2m_ioctl_querybuf,
.vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs,
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 07/28] media: mali-vpu: implement TRY_FMT and S_FMT ioctls
2026-11-19 15:37 ` [RFC PATCH 07/28] media: mali-vpu: implement TRY_FMT and S_FMT ioctls Charvi Mehta
@ 2026-11-19 15:55 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:55 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 4 potential
issues to consider:
- [Medium] media: mali-vpu: missing CAPTURE busy check in
mali_vpu_s_fmt() on OUTPUT
- [Medium] media: mali-vpu: CAPTURE colorimetry lost in mali_vpu_s_fmt()
and try_fmt
- [Medium] media: mali-vpu: incomplete field check in
mali_vpu_try_format_common()
- [Medium] media: mali-vpu: early return on bad pixfmt in
mali_vpu_try_format_common()
--
commit 5a51818e198650b05f2b6b84f5fbe1b7fdc55bf4
Author: Charvi Mehta <charvi.mehta@arm.com>
media: mali-vpu: implement TRY_FMT and S_FMT ioctls
This commit implements the VIDIOC_TRY_FMT and VIDIOC_S_FMT ioctls for the
decoder OUTPUT and CAPTURE queues and moves format lookup and buffer sizing
into shared helpers.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
> index b993255b8c817..dd142527a1a0c 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
[ ... ]
> @@ -168,32 +169,119 @@ static int mali_vpu_g_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
> return ret;
> }
>
> -static const struct mali_vpu_format_desc *mali_vpu_find_format(struct mali_vpu_session *sess,
> - u32 pixfmt, u32 type)
> +static int mali_vpu_try_format_common(struct mali_vpu_session *sess, struct v4l2_format *fmt)
> {
> - const struct mali_vpu_format_desc *fmt_list;
> - u32 num_fmts;
> - u32 i;
> + const struct mali_vpu_framesize_caps *fmsz = &sess->vpu_dev->variant->frmsz_caps;
> + struct v4l2_pix_format_mplane *pix_mp = &fmt->fmt.pix_mp;
> + const struct mali_vpu_format_desc *vpu_fmt;
> + const struct v4l2_format *port_fmt, *port_fmt_dec;
> + struct vb2_queue *src_q;
> + u32 w, h;
> + int ret = 0;
>
> - switch (type) {
> - case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> - fmt_list = sess->port_src.supported_fmts.fmts;
> - num_fmts = sess->port_src.supported_fmts.num_fmts;
> - break;
> - case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> - fmt_list = sess->port_dst.supported_fmts.fmts;
> - num_fmts = sess->port_dst.supported_fmts.num_fmts;
> - break;
> - default:
> - return NULL;
> + vpu_fmt = mali_vpu_find_format(sess, pix_mp->pixelformat, fmt->type);
> + if (!vpu_fmt) {
> + port_fmt = mali_vpu_get_port_fmt(sess, fmt->type);
> + if (!port_fmt)
> + return -EINVAL;
> +
> + *pix_mp = port_fmt->fmt.pix_mp;
> + return 0;
> + }
[Severity: Medium]
When pix_mp->pixelformat is unrecognized, should mali_vpu_try_format_common()
fall back to a supported format descriptor and continue validating the rest
of pix_mp instead of overwriting *pix_mp with port_fmt->fmt.pix_mp and
returning early?
Returning here discards the caller's requested width, height, and OUTPUT
colorimetry and skips v4l2_apply_frmsize_constraints() as well as the
streaming CAPTURE geometry lock.
> +
> + /* Requested format found, validate other fields */
> + w = pix_mp->width ? pix_mp->width : fmsz->width_default;
> + h = pix_mp->height ? pix_mp->height : fmsz->height_default;
> +
> + v4l2_apply_frmsize_constraints(&w, &h, &fmsz->stepwise);
> +
> + /* Decode CAPTURE queue: lock geometry to input if streaming */
> + if (sess->role == MALI_VPU_DECODER && fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
> + src_q = v4l2_m2m_get_src_vq(sess->m2m_ctx);
> + if (vb2_is_streaming(src_q)) {
> + port_fmt_dec = &sess->port_src.curr_fmt;
> + w = port_fmt_dec->fmt.pix_mp.width;
> + h = port_fmt_dec->fmt.pix_mp.height;
> + }
> + }
> +
> + ret = mali_vpu_fill_pixmp_fmt(vpu_fmt, w, h, pix_mp);
> + if (ret)
> + return ret;
> +
> + if (pix_mp->field == V4L2_FIELD_ANY)
> + pix_mp->field = V4L2_FIELD_NONE;
[Severity: Medium]
Should pix_mp->field be set unconditionally to V4L2_FIELD_NONE? Because the
driver only supports progressive frames, checking only for V4L2_FIELD_ANY
leaves any other field value (such as V4L2_FIELD_INTERLACED or an invalid
value from userspace) intact in fmt and port->curr_fmt.
> +
> + return 0;
> +}
> +
> +static int mali_vpu_try_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
> +{
> + struct mali_vpu_session *sess = mali_vpu_get_session(filp);
> + int ret;
> +
> + mutex_lock(&sess->vq_lock);
> +
> + ret = mali_vpu_try_format_common(sess, fmt);
> +
> + mutex_unlock(&sess->vq_lock);
> + return ret;
> +}
> +
> +static int mali_vpu_s_fmt(struct file *filp, void *fh, struct v4l2_format *fmt)
> +{
> + struct mali_vpu_session *sess = mali_vpu_get_session(filp);
> + struct vb2_queue *vq;
> + struct v4l2_pix_format_mplane *pixmp_src = &sess->port_src.curr_fmt.fmt.pix_mp;
> + struct v4l2_pix_format_mplane *pixmp_dst = &sess->port_dst.curr_fmt.fmt.pix_mp;
> + int ret = 0;
> +
> + mutex_lock(&sess->vq_lock);
> +
> + vq = v4l2_m2m_get_vq(sess->m2m_ctx, fmt->type);
> + if (vb2_is_busy(vq)) {
> + ret = -EBUSY;
> + goto mutex_unlock_return;
> + }
> +
> + ret = mali_vpu_try_format_common(sess, fmt);
> + if (ret)
> + goto mutex_unlock_return;
> +
> + /* OUTPUT provides initial CAPTURE geometry */
> + if (V4L2_TYPE_IS_OUTPUT(fmt->type)) {
> + const struct mali_vpu_format_desc *cap_fmt;
> + struct v4l2_pix_format_mplane *cap;
> + struct v4l2_pix_format_mplane *fmt_mp = &fmt->fmt.pix_mp;
> +
> + cap = &sess->port_dst.curr_fmt.fmt.pix_mp;
> + cap_fmt = mali_vpu_find_format(sess, cap->pixelformat,
> + V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
> + if (!cap_fmt) {
> + ret = -EINVAL;
> + goto mutex_unlock_return;
> + }
> +
> + ret = mali_vpu_fill_pixmp_fmt(cap_fmt, fmt_mp->width, fmt_mp->height, cap);
> + if (ret)
> + goto mutex_unlock_return;
> + }
[Severity: Medium]
Can VIDIOC_S_FMT on the OUTPUT queue overwrite sess->port_dst.curr_fmt while
CAPTURE buffers are already allocated?
Only the queue for fmt->type is checked with vb2_is_busy(vq). If userspace
allocates CAPTURE buffers and then calls VIDIOC_S_FMT on OUTPUT with a new
resolution, sess->port_dst.curr_fmt.fmt.pix_mp is updated in place without
checking whether the destination queue is busy, which violates the V4L2
stateful decoder specification and causes later VIDIOC_PREPARE_BUF or
VIDIOC_QBUF calls on existing CAPTURE buffers to fail size validation in
mali_vpu_vb2_buffer_prepare().
> +
> + if (V4L2_TYPE_IS_OUTPUT(fmt->type))
> + sess->port_src.curr_fmt = *fmt;
> + else
> + sess->port_dst.curr_fmt = *fmt;
> +
> + /* Keep peer queue colorimetry aligned with input */
> + if (V4L2_TYPE_IS_OUTPUT(fmt->type)) {
> + pixmp_dst->colorspace = pixmp_src->colorspace;
> + pixmp_dst->ycbcr_enc = pixmp_src->ycbcr_enc;
> + pixmp_dst->quantization = pixmp_src->quantization;
> + pixmp_dst->xfer_func = pixmp_src->xfer_func;
> + }
[Severity: Medium]
Does calling VIDIOC_TRY_FMT or VIDIOC_S_FMT on the CAPTURE queue lose the
OUTPUT colorimetry?
mali_vpu_try_format_common() does not copy colorspace, ycbcr_enc,
quantization, and xfer_func from sess->port_src.curr_fmt.fmt.pix_mp when
fmt->type is V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE. As a result,
VIDIOC_TRY_FMT on CAPTURE returns the caller's unaligned colorimetry, and
VIDIOC_S_FMT on CAPTURE overwrites sess->port_dst.curr_fmt with *fmt,
clobbering the colorimetry propagated during VIDIOC_S_FMT on OUTPUT.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=7
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 08/28] media: mali-vpu: register V76 decoder controls and introduce UAPI controls
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (6 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 07/28] media: mali-vpu: implement TRY_FMT and S_FMT ioctls Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 09/28] media: mali-vpu: add MMU software page table manager Charvi Mehta
` (19 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Register a V4L2 control handler for each decoder session and attach it to
the session file handle. Describe the controls supported by V76 in the
variant data and maintain their values in per-session state.
Add Mali VPU userspace controls for NALU layout, stream escaping,
intermediate buffer size, stream-header handling, frame reordering and
firmware debug output. Also expose the standard read-only
V4L2_CID_MIN_BUFFERS_FOR_CAPTURE control.
Stage control values set before the firmware is ready and reject changes to
non-dynamic controls while the OUTPUT queue is streaming. Allow userspace
to subscribe to control events.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-controls.c | 339 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-controls.h | 71 ++++
.../platform/arm/mali-vpu/mali-vpu-dec.c | 44 +++
.../platform/arm/mali-vpu/mali-vpu-dec.h | 6 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 15 +
.../platform/arm/mali-vpu/mali-vpu-session.h | 14 +
.../arm/mali-vpu/mali-vpu-variant-v76.c | 99 +++++
.../platform/arm/mali-vpu/mali-vpu-variant.h | 13 +
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 19 +
.../uapi/linux/media/arm/mali-vpu-config.h | 47 +++
include/uapi/linux/v4l2-controls.h | 6 +
12 files changed, 675 insertions(+), 1 deletion(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-controls.h
create mode 100644 include/uapi/linux/media/arm/mali-vpu-config.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 1dc62621d9dc0..95bad821eb466 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -7,6 +7,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-dec.o \
mali-vpu-buffer.o \
mali-vpu-vb2.o \
- mali-vpu-formats.o
+ mali-vpu-formats.o \
+ mali-vpu-controls.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
new file mode 100644
index 0000000000000..d4921eddf0fcf
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
@@ -0,0 +1,339 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/errno.h>
+#include <linux/limits.h>
+#include <linux/slab.h>
+#include <media/v4l2-mem2mem.h>
+
+#include <linux/media/arm/mali-vpu-config.h>
+
+#include "mali-vpu-controls.h"
+#include "mali-vpu-session.h"
+
+static int mali_vpu_ctrl_slot_from_cid(const struct mali_vpu_session *sess,
+ u32 cid, u32 *slot)
+{
+ const struct mali_vpu_ctrl_state *ctrl_state = sess->ctrls_state;
+ u32 i;
+
+ for (i = 0; i < sess->num_ctrls_caps; i++) {
+ if (ctrl_state[i].cid == cid) {
+ *slot = i;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
+
+static int mali_vpu_get_volatile_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct mali_vpu_session *sess = container_of(ctrl->handler,
+ struct mali_vpu_session,
+ ctrl_handler);
+
+ switch (ctrl->id) {
+ case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE:
+ if (sess->role != MALI_VPU_DECODER)
+ return -EINVAL;
+ ctrl->val = sess->min_buffers_required;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_apply_or_stage_ctrl(struct mali_vpu_session *sess,
+ const struct mali_vpu_ctrl_desc *desc,
+ struct mali_vpu_ctrl_state *state,
+ s64 value)
+{
+ bool staged;
+ int ret;
+
+ staged = !sess->is_fw_ready;
+
+ if (!staged) {
+ ret = desc->set(sess, state->cid, value);
+ if (ret)
+ return ret;
+ }
+
+ state->value = value;
+ state->client_set = true;
+ state->is_staged = staged;
+
+ return 0;
+}
+
+static int mali_vpu_set_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct mali_vpu_session *sess = container_of(ctrl->handler,
+ struct mali_vpu_session,
+ ctrl_handler);
+ const struct mali_vpu_ctrl_desc *desc;
+ struct mali_vpu_ctrl_state *state;
+ struct vb2_queue *src_vq;
+ u32 slot;
+ int ret;
+
+ ret = mali_vpu_ctrl_slot_from_cid(sess, ctrl->id, &slot);
+ if (ret)
+ return ret;
+
+ desc = &sess->ctrls_caps[slot];
+ state = &sess->ctrls_state[slot];
+
+ src_vq = v4l2_m2m_get_src_vq(sess->m2m_ctx);
+ if (vb2_is_streaming(src_vq) &&
+ !(desc->flags & MALI_VPU_CTRL_FLAG_DYNAMIC))
+ return -EBUSY;
+
+ return mali_vpu_apply_or_stage_ctrl(sess, desc, state, ctrl->val);
+}
+
+static const struct v4l2_ctrl_ops mali_vpu_ctrl_ops = {
+ .g_volatile_ctrl = mali_vpu_get_volatile_ctrl,
+ .s_ctrl = mali_vpu_set_ctrl,
+};
+
+static int mali_vpu_register_control(struct mali_vpu_session *sess,
+ const struct mali_vpu_ctrl_desc *d)
+{
+ struct v4l2_ctrl *ctrl;
+
+ if (d->is_custom) {
+ struct v4l2_ctrl_config cfg = {
+ .ops = &mali_vpu_ctrl_ops,
+ .id = d->cid,
+ .name = d->name,
+ .type = d->type,
+ .min = d->min_val,
+ .max = d->max_val,
+ .step = (d->type == V4L2_CTRL_TYPE_MENU) ? 0 : d->step_or_mask,
+ .def = d->default_val,
+ .menu_skip_mask = (d->type == V4L2_CTRL_TYPE_MENU) ? ~(d->step_or_mask) : 0,
+ .qmenu = d->qmenu,
+ };
+
+ ctrl = v4l2_ctrl_new_custom(&sess->ctrl_handler, &cfg, NULL);
+
+ } else if (d->type == V4L2_CTRL_TYPE_MENU) {
+ ctrl = v4l2_ctrl_new_std_menu(&sess->ctrl_handler,
+ &mali_vpu_ctrl_ops,
+ d->cid,
+ d->max_val,
+ ~(d->step_or_mask),
+ d->default_val);
+ } else {
+ ctrl = v4l2_ctrl_new_std(&sess->ctrl_handler,
+ &mali_vpu_ctrl_ops,
+ d->cid,
+ d->min_val,
+ d->max_val,
+ d->step_or_mask,
+ d->default_val);
+ }
+
+ if (!ctrl || sess->ctrl_handler.error)
+ return sess->ctrl_handler.error ? sess->ctrl_handler.error : -EINVAL;
+
+ return 0;
+}
+
+static int mali_vpu_validate_ctrl_caps(const struct mali_vpu_session *sess)
+{
+ u32 i, j;
+
+ if (!sess->ctrls_caps || !sess->num_ctrls_caps) {
+ dev_err(sess->vpu_dev->dev, "invalid ctrl caps structure\n");
+ return -EINVAL;
+ }
+
+ for (i = 0; i < sess->num_ctrls_caps; i++) {
+ const struct mali_vpu_ctrl_desc *d = &sess->ctrls_caps[i];
+
+ if (!d->cid || !d->set)
+ return -EINVAL;
+
+ if (d->min_val > d->max_val)
+ return -EINVAL;
+
+ if (d->default_val < d->min_val || d->default_val > d->max_val)
+ return -EINVAL;
+
+ if (d->is_custom && !d->name)
+ return -EINVAL;
+
+ if (d->is_custom && d->type == V4L2_CTRL_TYPE_MENU && !d->qmenu)
+ return -EINVAL;
+
+ /* Reject duplicate entries */
+ for (j = i + 1; j < sess->num_ctrls_caps; j++) {
+ if (d->cid == sess->ctrls_caps[j].cid)
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+int mali_vpu_ctrls_init(struct mali_vpu_session *sess)
+{
+ const struct mali_vpu_ctrl_desc *ctrls_cap = sess->ctrls_caps;
+ struct mali_vpu_ctrl_state *ctrl_state;
+ struct v4l2_ctrl *volatile_ctrl;
+ u32 i;
+ int ret;
+
+ ret = mali_vpu_validate_ctrl_caps(sess);
+ if (ret)
+ return ret;
+
+ sess->ctrls_state = kcalloc(sess->num_ctrls_caps, sizeof(*sess->ctrls_state), GFP_KERNEL);
+ if (!sess->ctrls_state)
+ return -ENOMEM;
+
+ /*
+ * Adding 1 to num_ctrls to include
+ * V4L2_CID_MIN_BUFFERS_FOR_CAPTURE for decoder
+ */
+ ret = v4l2_ctrl_handler_init(&sess->ctrl_handler, sess->num_ctrls_caps + 1);
+ if (ret)
+ goto err_free_state;
+
+ sess->ctrl_handler.lock = &sess->vq_lock;
+
+ ctrl_state = sess->ctrls_state;
+
+ for (i = 0; i < sess->num_ctrls_caps; i++) {
+ const struct mali_vpu_ctrl_desc *d = &ctrls_cap[i];
+
+ ctrl_state[i].cid = d->cid;
+ ctrl_state[i].value = d->default_val;
+ ctrl_state[i].client_set = false;
+ ctrl_state[i].is_staged = false;
+
+ ret = mali_vpu_register_control(sess, d);
+ if (ret)
+ goto err_free_handler;
+ }
+
+ if (sess->role == MALI_VPU_DECODER) {
+ volatile_ctrl = v4l2_ctrl_new_std(&sess->ctrl_handler,
+ &mali_vpu_ctrl_ops,
+ V4L2_CID_MIN_BUFFERS_FOR_CAPTURE,
+ 1, 32, 1, 1);
+ } else {
+ ret = -EINVAL;
+ goto err_free_handler;
+ }
+
+ if (!volatile_ctrl || sess->ctrl_handler.error) {
+ ret = sess->ctrl_handler.error ? sess->ctrl_handler.error : -EINVAL;
+ goto err_free_handler;
+ }
+
+ volatile_ctrl->flags |= V4L2_CTRL_FLAG_VOLATILE | V4L2_CTRL_FLAG_READ_ONLY;
+
+ sess->fh.ctrl_handler = &sess->ctrl_handler;
+
+ return 0;
+
+err_free_handler:
+ v4l2_ctrl_handler_free(&sess->ctrl_handler);
+
+err_free_state:
+ kfree(sess->ctrls_state);
+ sess->ctrls_state = NULL;
+ return ret;
+}
+
+void mali_vpu_ctrls_cleanup(struct mali_vpu_session *sess)
+{
+ v4l2_ctrl_handler_free(&sess->ctrl_handler);
+ kfree(sess->ctrls_state);
+ sess->ctrls_state = NULL;
+ sess->fh.ctrl_handler = NULL;
+}
+
+int mali_vpu_apply_staged_controls(struct mali_vpu_session *sess)
+{
+ u32 i;
+ int ret;
+
+ if (!sess->is_fw_ready)
+ return -EINVAL;
+
+ for (i = 0; i < sess->num_ctrls_caps; i++) {
+ const struct mali_vpu_ctrl_desc *d = &sess->ctrls_caps[i];
+ struct mali_vpu_ctrl_state *s = &sess->ctrls_state[i];
+
+ if (!s->client_set || !s->is_staged)
+ continue;
+
+ ret = mali_vpu_apply_or_stage_ctrl(sess, d, s, s->value);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+int mali_vpu_ctrl_set_nalu_format(struct mali_vpu_session *sess, u32 cid,
+ s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_NALU_FORMAT)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
+}
+
+int mali_vpu_ctrl_set_intbuf_size(struct mali_vpu_session *sess, u32 cid,
+ s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_INTBUF_SIZE || value < 0 || value > INT_MAX)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
+}
+
+int mali_vpu_ctrl_set_stream_escaping(struct mali_vpu_session *sess, u32 cid,
+ s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_STREAM_ESCAPING)
+ return -EINVAL;
+
+ if (value == -1)
+ return 0; /* tristate no-op */
+
+ if (value != 0 && value != 1)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
+}
+
+int mali_vpu_ctrl_set_host_print_enable(struct mali_vpu_session *sess, u32 cid,
+ s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_HOST_PRINT_ENABLE)
+ return -EINVAL;
+
+ if (value != 0 && value != 1)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.h
new file mode 100644
index 0000000000000..d7f5898963478
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.h
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_CONTROLS_H__
+#define __MALI_VPU_CONTROLS_H__
+
+#include <linux/bits.h>
+#include <linux/types.h>
+#include <media/v4l2-ctrls.h>
+
+struct mali_vpu_session;
+
+enum mali_vpu_ctrl_flags {
+ MALI_VPU_CTRL_FLAG_DYNAMIC = BIT(0),
+};
+
+/**
+ * struct mali_vpu_ctrl_desc - static description of one driver control.
+ * @cid: V4L2 control ID.
+ * @type: V4L2 control type.
+ * @name: Control name used for custom registration. NULL for standard controls.
+ * @qmenu: Menu labels for V4L2_CTRL_TYPE_MENU custom controls.
+ * @min_val: Minimum allowed value for this control.
+ * @max_val: Maximum allowed value for this control.
+ * @default_val: Default value for this control at session init.
+ * @step_or_mask: Step for integer controls, or enabled-item mask for menu controls.
+ * @is_custom: Flag to indicate if a control is a custom Mali VPU control.
+ * @flags: Driver-specific behavior flags from enum mali_vpu_ctrl_flags.
+ * @set: Setter callback invoked on client control changes, to apply value to firmware.
+ */
+struct mali_vpu_ctrl_desc {
+ u32 cid;
+ enum v4l2_ctrl_type type;
+ const char *name;
+ const char * const *qmenu;
+ s64 min_val;
+ s64 max_val;
+ s64 default_val;
+ u64 step_or_mask;
+ bool is_custom;
+ enum mali_vpu_ctrl_flags flags;
+ int (*set)(struct mali_vpu_session *sess, u32 cid, s64 value);
+};
+
+/**
+ * struct mali_vpu_ctrl_state - per-session runtime state of one driver control.
+ * @cid: V4L2 control ID.
+ * @value: Last accepted value for the control (from userspace or default at init).
+ * @client_set: True if userspace explicitly set this control at least once.
+ * @is_staged: True if value is accepted but not yet sent to firmware.
+ */
+struct mali_vpu_ctrl_state {
+ u32 cid;
+ s64 value;
+ bool client_set;
+ bool is_staged;
+};
+
+int mali_vpu_ctrls_init(struct mali_vpu_session *sess);
+void mali_vpu_ctrls_cleanup(struct mali_vpu_session *sess);
+int mali_vpu_apply_staged_controls(struct mali_vpu_session *sess);
+
+/* Common controls setters */
+int mali_vpu_ctrl_set_nalu_format(struct mali_vpu_session *sess, u32 cid, s64 value);
+int mali_vpu_ctrl_set_intbuf_size(struct mali_vpu_session *sess, u32 cid, s64 value);
+int mali_vpu_ctrl_set_stream_escaping(struct mali_vpu_session *sess, u32 cid, s64 value);
+int mali_vpu_ctrl_set_host_print_enable(struct mali_vpu_session *sess, u32 cid, s64 value);
+
+#endif /* __MALI_VPU_CONTROLS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
index cb6eeee2f04df..7e78c475a4ff3 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
@@ -3,6 +3,9 @@
* Copyright (C) 2026 ARM Limited.
*/
+#include <linux/errno.h>
+#include <linux/media/arm/mali-vpu-config.h>
+
#include "mali-vpu-dec.h"
#include "mali-vpu-session.h"
#include "mali-vpu-variant.h"
@@ -13,4 +16,45 @@ void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
sess->port_src.supported_fmts = variant_fmts->dec_src;
sess->port_dst.supported_fmts = variant_fmts->dec_dst;
+
+ const struct mali_vpu_control_caps *variant_ctrls = &sess->vpu_dev->variant->ctrl_caps;
+
+ sess->ctrls_caps = variant_ctrls->dec_ctrls;
+ sess->num_ctrls_caps = variant_ctrls->num_dec_ctrls;
+
+ sess->min_buffers_required = 3; /* For CAPTURE/dst queue */
+}
+
+int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess,
+ u32 cid, s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_IGNORE_STREAM_HEADERS)
+ return -EINVAL;
+
+ if (value == -1)
+ return 0; /* tristate no-op */
+
+ if (value != 0 && value != 1)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
+}
+
+int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid,
+ s64 value)
+{
+ if (cid != V4L2_CID_MALI_VPU_FRAME_REORDERING)
+ return -EINVAL;
+
+ if (value == -1)
+ return 0; /* tristate no-op */
+
+ if (value != 0 && value != 1)
+ return -EINVAL;
+
+ (void)sess;
+ (void)value;
+ return 0;
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
index 2514c897587c4..e323b38475c81 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
@@ -6,8 +6,14 @@
#ifndef __MALI_VPU_DEC_H__
#define __MALI_VPU_DEC_H__
+#include <linux/types.h>
+
struct mali_vpu_session;
void mali_vpu_session_init_dec(struct mali_vpu_session *sess);
+/* Decoder controls setters */
+int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess, u32 cid, s64 value);
+int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid, s64 value);
+
#endif /* __MALI_VPU_DEC_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 6867f51287f48..d1e8680de5556 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -3,12 +3,14 @@
* Copyright (C) 2026 ARM Limited.
*/
+#include <linux/device.h>
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/string.h>
#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
+#include "mali-vpu-controls.h"
#include "mali-vpu-dec.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-variant.h"
@@ -76,8 +78,19 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
goto err_mutex_destroy;
}
+ sess->is_fw_ready = false;
+
+ ret = mali_vpu_ctrls_init(sess);
+ if (ret) {
+ dev_err(sess->vpu_dev->dev, "failed to initialise controls: %d\n", ret);
+ goto err_m2m_ctx_release;
+ }
+
return 0;
+err_m2m_ctx_release:
+ v4l2_m2m_ctx_release(sess->m2m_ctx);
+
err_mutex_destroy:
mutex_destroy(&sess->vq_lock);
return ret;
@@ -94,5 +107,7 @@ void mali_vpu_session_cleanup(struct mali_vpu_session *sess)
}
mutex_unlock(&sess->vq_lock);
+
+ mali_vpu_ctrls_cleanup(sess);
mutex_destroy(&sess->vq_lock);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index f8a602cb59006..766912aec0b8e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -8,6 +8,7 @@
#include <linux/mutex.h>
#include <linux/videodev2.h>
+#include <media/v4l2-ctrls.h>
#include <media/v4l2-fh.h>
#include "mali-vpu-device.h"
@@ -32,6 +33,12 @@ struct mali_vpu_port {
* @port_src: Source (OUTPUT) port for this session.
* @port_dst: Destination (CAPTURE) port for this session.
* @vq_lock: Mutex lock to serialise per-session queue related ioctls.
+ * @is_fw_ready: True if the firmware is loaded and ready to accept commands.
+ * @min_buffers_required: Driver-reported minimum buffers for role-specific queue.
+ * @ctrls_caps: Variant-provided immutable control capability table.
+ * @num_ctrls_caps: Number of entries in @ctrls_caps.
+ * @ctrls_state: Per-session control cache and staging state.
+ * @ctrl_handler: V4L2 control handler bound to this session.
* @fh: V4L2 file handle for this session.
* @m2m_ctx: V4L2 mem2mem context for this session.
*/
@@ -41,6 +48,13 @@ struct mali_vpu_session {
struct mali_vpu_port port_src;
struct mali_vpu_port port_dst;
struct mutex vq_lock; /* lock to serialise per-session ioctls */
+ bool is_fw_ready;
+ u32 min_buffers_required;
+
+ const struct mali_vpu_ctrl_desc *ctrls_caps;
+ u32 num_ctrls_caps;
+ struct mali_vpu_ctrl_state *ctrls_state;
+ struct v4l2_ctrl_handler ctrl_handler;
struct v4l2_fh fh;
struct v4l2_m2m_ctx *m2m_ctx;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
index 7b04b51eeedc2..b807f8d888222 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -4,8 +4,14 @@
*/
#include <linux/array_size.h>
+#include <linux/bits.h>
+#include <linux/limits.h>
#include <linux/videodev2.h>
+#include <linux/media/arm/mali-vpu-config.h>
+
+#include "mali-vpu-controls.h"
+#include "mali-vpu-dec.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-protocol-def-v3.h"
#include "mali-vpu-variant.h"
@@ -53,6 +59,95 @@ static const struct mali_vpu_format_desc mali_v76_dec_dst_fmts[] = {
},
};
+static const char *const mali_v76_nalu_format_qmenu[] = {
+ "Start Codes",
+ "One NALU Per Buffer",
+ "One Byte Length Field",
+ "Two Byte Length Field",
+ "Four Byte Length Field",
+ NULL,
+};
+
+static const struct mali_vpu_ctrl_desc mali_v76_dec_ctrls[] = {
+ {
+ .cid = V4L2_CID_MALI_VPU_NALU_FORMAT,
+ .type = V4L2_CTRL_TYPE_MENU,
+ .name = "NALU format",
+ .qmenu = mali_v76_nalu_format_qmenu,
+ .min_val = MALI_VPU_NALU_FORMAT_START_CODES,
+ .max_val = MALI_VPU_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD,
+ .default_val = MALI_VPU_NALU_FORMAT_START_CODES,
+ .step_or_mask = BIT(MALI_VPU_NALU_FORMAT_START_CODES) |
+ BIT(MALI_VPU_NALU_FORMAT_ONE_NALU_PER_BUFFER) |
+ BIT(MALI_VPU_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD) |
+ BIT(MALI_VPU_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD) |
+ BIT(MALI_VPU_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD),
+ .is_custom = true,
+ .flags = 0,
+ .set = mali_vpu_ctrl_set_nalu_format,
+ },
+ {
+ .cid = V4L2_CID_MALI_VPU_INTBUF_SIZE,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .name = "Internal buffer size",
+ .min_val = 0,
+ .max_val = INT_MAX,
+ .default_val = 0,
+ .step_or_mask = 1,
+ .is_custom = true,
+ .flags = 0,
+ .set = mali_vpu_ctrl_set_intbuf_size,
+ },
+ {
+ .cid = V4L2_CID_MALI_VPU_STREAM_ESCAPING,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .name = "Stream Escaping",
+ .min_val = -1,
+ .max_val = 1,
+ .default_val = -1,
+ .step_or_mask = 1,
+ .is_custom = true,
+ .flags = 0,
+ .set = mali_vpu_ctrl_set_stream_escaping,
+ },
+ {
+ .cid = V4L2_CID_MALI_VPU_HOST_PRINT_ENABLE,
+ .type = V4L2_CTRL_TYPE_BOOLEAN,
+ .name = "Firmware debug prints",
+ .min_val = false,
+ .max_val = true,
+ .default_val = false,
+ .step_or_mask = 1,
+ .is_custom = true,
+ .flags = MALI_VPU_CTRL_FLAG_DYNAMIC,
+ .set = mali_vpu_ctrl_set_host_print_enable,
+ },
+ {
+ .cid = V4L2_CID_MALI_VPU_IGNORE_STREAM_HEADERS,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .name = "Ignore Stream Headers",
+ .min_val = -1,
+ .max_val = 1,
+ .default_val = -1,
+ .step_or_mask = 1,
+ .is_custom = true,
+ .flags = 0,
+ .set = mali_vpu_ctrl_set_ignore_stream_headers,
+ },
+ {
+ .cid = V4L2_CID_MALI_VPU_FRAME_REORDERING,
+ .type = V4L2_CTRL_TYPE_INTEGER,
+ .name = "Frame Reordering",
+ .min_val = -1,
+ .max_val = 1,
+ .default_val = -1,
+ .step_or_mask = 1,
+ .is_custom = true,
+ .flags = 0,
+ .set = mali_vpu_ctrl_set_frame_reordering,
+ },
+};
+
const struct mali_vpu_variant mali_vpu_v76_variant = {
.expected_hw_pid = MALI_VPU_HW_PID_V76,
@@ -81,4 +176,8 @@ const struct mali_vpu_variant mali_vpu_v76_variant = {
.width_default = MALI_V76_DEFAULT_FRAME_WIDTH,
.height_default = MALI_V76_DEFAULT_FRAME_HEIGHT,
},
+ .ctrl_caps = {
+ .dec_ctrls = mali_v76_dec_ctrls,
+ .num_dec_ctrls = ARRAY_SIZE(mali_v76_dec_ctrls),
+ },
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
index a46c25a94f18b..ad20896c90b0f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -9,6 +9,7 @@
#include <linux/types.h>
#include <linux/videodev2.h>
+#include "mali-vpu-controls.h"
#include "mali-vpu-formats.h"
/**
@@ -33,6 +34,16 @@ struct mali_vpu_format_caps {
const struct mali_vpu_format_list dec_dst;
};
+/**
+ * struct mali_vpu_control_caps - Control capability tables per role.
+ * @dec_ctrls: Array of controls supported by the decoder.
+ * @num_dec_ctrls: Number of entries in @dec_ctrls.
+ */
+struct mali_vpu_control_caps {
+ const struct mali_vpu_ctrl_desc *dec_ctrls;
+ u32 num_dec_ctrls;
+};
+
/**
* struct mali_vpu_variant - Variant-specific hardware capability descriptor.
* @rst_names: Array of reset control names for this variant.
@@ -40,6 +51,7 @@ struct mali_vpu_format_caps {
* @expected_hw_pid: Expected hardware product ID read from the version register.
* @fmt_caps: Format capability tables for encode/decode paths.
* @frmsz_caps: Frame dimension limits and defaults.
+ * @ctrl_caps: Control capability tables for encode/decode paths.
*/
struct mali_vpu_variant {
const char * const *rst_names;
@@ -48,6 +60,7 @@ struct mali_vpu_variant {
struct mali_vpu_format_caps fmt_caps;
struct mali_vpu_framesize_caps frmsz_caps;
+ struct mali_vpu_control_caps ctrl_caps;
};
extern const struct mali_vpu_variant mali_vpu_v76_variant;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index dd142527a1a0c..88e706be6705e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -8,7 +8,9 @@
#include <linux/slab.h>
#include <linux/string.h>
#include <media/v4l2-common.h>
+#include <media/v4l2-ctrls.h>
#include <media/v4l2-dev.h>
+#include <media/v4l2-event.h>
#include <media/v4l2-fh.h>
#include <media/v4l2-ioctl.h>
#include <media/v4l2-mem2mem.h>
@@ -318,6 +320,21 @@ static int mali_vpu_enum_framesizes(struct file *filp, void *fh,
return 0;
}
+static int mali_vpu_subscribe_event(struct v4l2_fh *fh, const struct v4l2_event_subscription *sub)
+{
+ switch (sub->type) {
+ case V4L2_EVENT_CTRL:
+ return v4l2_ctrl_subscribe_event(fh, sub);
+ case V4L2_EVENT_EOS:
+ return v4l2_event_subscribe(fh, sub, 2, NULL);
+ case V4L2_EVENT_SOURCE_CHANGE:
+ return v4l2_event_subscribe(fh, sub, 2, NULL);
+
+ default:
+ return -EINVAL;
+ }
+}
+
const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
.vidioc_querycap = mali_vpu_querycap,
.vidioc_enum_fmt_vid_out = mali_vpu_enum_fmt,
@@ -337,4 +354,6 @@ const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
.vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf,
.vidioc_remove_bufs = v4l2_m2m_ioctl_remove_bufs,
.vidioc_enum_framesizes = mali_vpu_enum_framesizes,
+ .vidioc_subscribe_event = mali_vpu_subscribe_event,
+ .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
};
diff --git a/include/uapi/linux/media/arm/mali-vpu-config.h b/include/uapi/linux/media/arm/mali-vpu-config.h
new file mode 100644
index 0000000000000..a82fd69f16b75
--- /dev/null
+++ b/include/uapi/linux/media/arm/mali-vpu-config.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Arm Mali-VPU Driver - Userspace API
+ *
+ * (C) COPYRIGHT 2026 ARM Limited.
+ */
+
+#ifndef __UAPI_MALI_VPU_CONFIG_H__
+#define __UAPI_MALI_VPU_CONFIG_H__
+
+#include <linux/v4l2-controls.h>
+
+/* Mali VPU specific V4L2 driver controls */
+
+/* Encoded stream NALU encapsulation mode. */
+#define V4L2_CID_MALI_VPU_NALU_FORMAT (V4L2_CID_USER_MALI_VPU_BASE + 0x00)
+
+/**
+ * enum mali_vpu_nalu_format - Encoded stream NALU OUTPUT format.
+ *
+ * @MALI_VPU_NALU_FORMAT_START_CODES: Each NAL unit is prefixed by an Annex B start code.
+ * @MALI_VPU_NALU_FORMAT_ONE_NALU_PER_BUFFER: Exactly one NALU in each output buffer.
+ * @MALI_VPU_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD: Length-prefixed NALU with 1-byte size field.
+ * @MALI_VPU_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD: Length-prefixed NALU with 2-byte size field.
+ * @MALI_VPU_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD: Length-prefixed NALU with 4-byte size field.
+ *
+ * Options for V4L2_CID_MALI_VPU_NALU_FORMAT control, which determines how the firmware
+ * encapsulates NALUs in the output stream buffers H.264.
+ */
+enum mali_vpu_nalu_format {
+ MALI_VPU_NALU_FORMAT_START_CODES = 0,
+ MALI_VPU_NALU_FORMAT_ONE_NALU_PER_BUFFER = 1,
+ MALI_VPU_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD = 2,
+ MALI_VPU_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD = 3,
+ MALI_VPU_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD = 4,
+};
+
+#define V4L2_CID_MALI_VPU_STREAM_ESCAPING (V4L2_CID_USER_MALI_VPU_BASE + 0x01)
+#define V4L2_CID_MALI_VPU_IGNORE_STREAM_HEADERS (V4L2_CID_USER_MALI_VPU_BASE + 0x02)
+#define V4L2_CID_MALI_VPU_FRAME_REORDERING (V4L2_CID_USER_MALI_VPU_BASE + 0x03)
+/* Suggested firmware intermediate buffer size limit. */
+#define V4L2_CID_MALI_VPU_INTBUF_SIZE (V4L2_CID_USER_MALI_VPU_BASE + 0x04)
+
+/* Enable firmware protocol to send debug prints to host. */
+#define V4L2_CID_MALI_VPU_HOST_PRINT_ENABLE (V4L2_CID_USER_MALI_VPU_BASE + 0x05)
+
+#endif /* __UAPI_MALI_VPU_CONFIG_H__ */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index d17e41d51d2ec..b9f7bc3aac1de 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -255,6 +255,12 @@ enum v4l2_colorfx {
*/
#define V4L2_CID_USER_IT6625_BASE (V4L2_CID_USER_BASE + 0x1250)
+/*
+ * The base for Arm Mali-VPU driver controls.
+ * We reserve 64 controls for this driver.
+ */
+#define V4L2_CID_USER_MALI_VPU_BASE (V4L2_CID_USER_BASE + 0x1260)
+
/* MPEG-class control IDs */
/* The MPEG controls are applicable to all codec controls
* and the 'MPEG' part of the define is historical */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 09/28] media: mali-vpu: add MMU software page table manager
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (7 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 08/28] media: mali-vpu: register V76 decoder controls and introduce UAPI controls Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:59 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 10/28] media: mali-vpu: add firmware cache infrastructure Charvi Mehta
` (18 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Rath Sankaralingam <rath.sankaralingam@arm.com>
Add Mali VPU internal MMU page table manager for managing
firmware-visible virtual address space. The MMU code builds and
maintains the VPU two-level page tables, allocating backing pages and maps
scatter-gather and dma-buf backed memory. It also exposes helpers for
mapping, unmapping, resizing, synchronizing and writing into mapped
regions.
The page table entries describe the addresses consumed by the VPU
peripheral MMU, allowing firmware communication queues and video buffers
to be mapped into the VPU virtual address layout with the required bus
attributes and access permissions.
This also configures the V76 DMA mask for its 40-bit addressing
capability and sets the maximum DMA segment size used by DMA-backed
buffers.
Signed-off-by: Rath Sankaralingam <rath.sankaralingam@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-device.h | 2 +
.../platform/arm/mali-vpu/mali-vpu-mmu.c | 693 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-mmu.h | 152 ++++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 8 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 10 +
.../platform/arm/mali-vpu/mali-vpu-session.h | 3 +
.../arm/mali-vpu/mali-vpu-variant-v76.c | 1 +
.../platform/arm/mali-vpu/mali-vpu-variant.h | 2 +
9 files changed, 873 insertions(+), 1 deletion(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 95bad821eb466..0716a2b0c8712 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -8,6 +8,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-buffer.o \
mali-vpu-vb2.o \
mali-vpu-formats.o \
- mali-vpu-controls.o
+ mali-vpu-controls.o \
+ mali-vpu-mmu.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index 26b2bf9e1588e..20361df05bdcd 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -32,6 +32,7 @@ enum mali_vpu_session_role {
* @clk_count: Number of clocks in @clk_list.
* @rst_list: List of reset controls.
* @rst_count: Number of reset controls in @rst_list.
+ * @dma_mask: DMA mask for the device, indicating supported address width.
* @irq: IRQ number for VPU.
* @variant: Structure describing variant-specific features and capabilities.
* @v4l2_dev: V4L2 device structure for mali vpu.
@@ -45,6 +46,7 @@ struct mali_vpu_dev {
unsigned int clk_count;
struct reset_control_bulk_data *rst_list;
unsigned int rst_count;
+ u64 dma_mask;
int irq;
const struct mali_vpu_variant *variant;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
new file mode 100644
index 0000000000000..00cf5251eff5e
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
@@ -0,0 +1,693 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/build_bug.h>
+#include <linux/dma-mapping.h>
+#include <linux/gfp.h>
+#include <linux/slab.h>
+
+#include "mali-vpu-mmu.h"
+
+/* Number of bits for the VPU physical (DMA) address space. */
+#define MALI_VPU_PA_BITS 40
+#define MALI_VPU_PA_MASK GENMASK_ULL(MALI_VPU_PA_BITS - 1, 0)
+
+/* Number of bits for the virtual address space. */
+#define MALI_VPU_VA_BITS 32
+#define MALI_VPU_VA_MASK GENMASK(MALI_VPU_VA_BITS - 1, 0)
+
+/* Number of bits from the VA used to index a PTE in a page. */
+#define MALI_VPU_INDEX_SHIFT 10
+#define MALI_VPU_INDEX_SIZE BIT(MALI_VPU_INDEX_SHIFT)
+#define MALI_VPU_INDEX_MASK GENMASK(MALI_VPU_INDEX_SHIFT - 1, 0)
+
+/* Access permission defines. */
+#define MALI_VPU_PTE_AP_SHIFT 0
+#define MALI_VPU_PTE_AP_BITS 2
+#define MALI_VPU_PTE_AP_MASK ((1 << MALI_VPU_PTE_AP_BITS) - 1)
+
+/* PTE shift defines. */
+#define MALI_VPU_PTE_PHYSADDR_SHIFT 2
+#define MALI_VPU_PTE_ATTR_SHIFT 30
+
+/* Number of levels for Page Table Walk. */
+#define MALI_VPU_PTW_LEVELS 2
+
+/*
+ * A Linux physical page can be equal in size or larger than the MVE page size.
+ * This define calculates how many MVE pages fit in one Linux page.
+ */
+#define MALI_VPU_PAGES_PER_LINUX_PAGE (PAGE_SIZE / MALI_VPU_PAGE_SIZE)
+
+/*
+ * If the Linux PAGE_SIZE were ever smaller than MALI_VPU_PAGE_SIZE, the define
+ * above would evaluate to 0 and break the page accounting. This static_assert enforces
+ * this invariant at build time.
+ */
+static_assert(MALI_VPU_PAGES_PER_LINUX_PAGE > 0,
+ "PAGE_SIZE must be >= MALI_VPU_PAGE_SIZE");
+
+/*
+ * Return the PTE index for a given level (L1=0, L2=1).
+ * 22 12 0
+ * +-------------------+-------------------+-----------------------+
+ * | Level 1 | Level 2 | Page offset |
+ * +-------------------+-------------------+-----------------------+
+ */
+static unsigned int mali_vpu_get_index(const mali_vpu_mmu_va va, const unsigned int level)
+{
+ return (va >> (MALI_VPU_PAGE_SHIFT + (MALI_VPU_PTW_LEVELS - level - 1) *
+ MALI_VPU_INDEX_SHIFT)) & MALI_VPU_INDEX_MASK;
+}
+
+static mali_vpu_mmu_pte mali_vpu_mmu_set_pte(enum mali_vpu_mmu_attr attr,
+ dma_addr_t dma,
+ enum mali_vpu_mmu_access access)
+{
+ return (attr << MALI_VPU_PTE_ATTR_SHIFT) |
+ ((dma >> MALI_VPU_PAGE_SHIFT) << MALI_VPU_PTE_PHYSADDR_SHIFT) |
+ (access << MALI_VPU_PTE_AP_SHIFT);
+}
+
+static mali_vpu_mmu_pte *mali_vpu_page_table_walk(struct mali_vpu_mmu *mmu, mali_vpu_mmu_va va,
+ bool alloc)
+{
+ struct mali_vpu_mmu_pt_page *l2;
+ mali_vpu_mmu_pte *pte = mmu->root.cpu;
+ unsigned int index;
+
+ /* Level 1. */
+ index = mali_vpu_get_index(va, 0);
+ l2 = &mmu->l2_tables[index];
+
+ /* Map in L2 page if it is missing. */
+ if (!l2->linux_page) {
+ if (!alloc) {
+ dev_err(mmu->dev, "missing L2 page in PTW. va=0x%x", va);
+ return ERR_PTR(-EFAULT);
+ }
+
+ l2->dma = mali_vpu_mmu_alloc_page(mmu->dev, &l2->linux_page, DMA_TO_DEVICE);
+ if (!l2->linux_page) {
+ dev_err(mmu->dev, "failed to allocate L2 page. va=0x%x", va);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ l2->cpu = page_address(l2->linux_page);
+
+ pte[index] = mali_vpu_mmu_set_pte(MALI_VPU_ATTR_PRIVATE, l2->dma,
+ MALI_VPU_ACCESS_READ_ONLY);
+ dma_sync_single_for_device(mmu->dev, mmu->root.dma + index * sizeof(*pte),
+ sizeof(*pte), DMA_TO_DEVICE);
+ }
+
+ /* Level 2. */
+ index = mali_vpu_get_index(va, 1);
+
+ /* The caller must sync the entry for the device after modifying it. */
+ return &l2->cpu[index];
+}
+
+static int mali_vpu_map_page(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ dma_addr_t dma,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access)
+{
+ struct mali_vpu_mmu_pt_page *l2;
+ mali_vpu_mmu_pte *pte;
+
+ /* Check that both VA and DMA address are page aligned. */
+ if ((va | dma) & MALI_VPU_PAGE_MASK) {
+ dev_err(mmu->dev, "VA and DMA address must be page aligned. va=0x%x, dma=%pad",
+ va, &dma);
+ return -EFAULT;
+ }
+
+ /* Check that VA is within valid address range. */
+ if (va & ~MALI_VPU_VA_MASK) {
+ dev_err(mmu->dev, "VA out of valid range. va=0x%x", va);
+ return -EFAULT;
+ }
+
+ /* Check that the DMA address is within valid address range. */
+ if (dma & ~MALI_VPU_PA_MASK) {
+ dev_err(mmu->dev, "DMA address out of valid range. dma=%pad", &dma);
+ return -EFAULT;
+ }
+
+ pte = mali_vpu_page_table_walk(mmu, va, true);
+ if (IS_ERR(pte))
+ return PTR_ERR(pte);
+
+ /* Return error if the entry is already mapped or reserved. */
+ if (*pte)
+ return -EAGAIN;
+
+ /* Map in DMA address and flush data. */
+ *pte = mali_vpu_mmu_set_pte(attr, dma, access);
+ l2 = &mmu->l2_tables[mali_vpu_get_index(va, 0)];
+ dma_sync_single_for_device(mmu->dev,
+ l2->dma + mali_vpu_get_index(va, 1) * sizeof(*pte),
+ sizeof(*pte), DMA_TO_DEVICE);
+
+ return 0;
+}
+
+static void mali_vpu_unmap_page(struct mali_vpu_mmu *mmu, mali_vpu_mmu_va va)
+{
+ struct mali_vpu_mmu_pt_page *l2;
+ mali_vpu_mmu_pte *pte;
+
+ pte = mali_vpu_page_table_walk(mmu, va, false);
+ if (IS_ERR(pte))
+ return;
+
+ /* Unmap virtual address and flush data. */
+ *pte = 0;
+ l2 = &mmu->l2_tables[mali_vpu_get_index(va, 0)];
+ dma_sync_single_for_device(mmu->dev,
+ l2->dma + mali_vpu_get_index(va, 1) * sizeof(*pte),
+ sizeof(*pte), DMA_TO_DEVICE);
+}
+
+static int mali_vpu_remap_page(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ dma_addr_t dma,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access)
+{
+ mali_vpu_unmap_page(mmu, va);
+ return mali_vpu_map_page(mmu, va, dma, attr, access);
+}
+
+static int mali_vpu_remap_pages(struct mali_vpu_mmu_pages *pages, size_t oldcount,
+ size_t newcount)
+{
+ int ret;
+
+ if (!pages->mmu)
+ return 0;
+
+ /* Remap pages to no access if new count is smaller than old count. */
+ while (newcount < oldcount) {
+ oldcount--;
+
+ ret = mali_vpu_remap_page(pages->mmu,
+ pages->va + oldcount * MALI_VPU_PAGE_SIZE,
+ MALI_VPU_PAGE_SIZE, MALI_VPU_ATTR_PRIVATE,
+ MALI_VPU_ACCESS_NO);
+ if (ret)
+ return ret;
+ }
+
+ /* Map up pages if new count is larger than old count. */
+ while (newcount > oldcount) {
+ ret = mali_vpu_remap_page(pages->mmu,
+ pages->va + oldcount * MALI_VPU_PAGE_SIZE,
+ pages->pages[oldcount].dma, pages->attr,
+ pages->access);
+ if (ret)
+ return ret;
+
+ oldcount++;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_mapped_count(const mali_vpu_mmu_pte *l2)
+{
+ int count = 0;
+ unsigned int j;
+
+ for (j = 0; j < MALI_VPU_INDEX_SIZE; j++) {
+ /* Do not count pages with AP as MALI_VPU_ACCESS_NO */
+ if (l2[j] & MALI_VPU_PTE_AP_MASK)
+ count++;
+ }
+
+ return count;
+}
+
+static size_t mali_vpu_get_sg_table_npages(struct sg_table *sgt)
+{
+ struct sg_dma_page_iter piter;
+ size_t count = 0;
+
+ for_each_sg_dma_page(sgt->sgl, &piter, sgt->nents, 0) {
+ count++;
+ }
+
+ return count;
+}
+
+static int mali_vpu_append_sg_table(struct mali_vpu_mmu_pages *pages, struct sg_table *sgt)
+{
+ size_t count;
+ struct sg_dma_page_iter piter;
+
+ /*
+ * SG import must describe complete Linux pages. VB2 DMA-SG MMAP buffers
+ * are allocated as whole pages with zero offset. DMA-BUF attachment
+ * mappings guarantee PAGE_SIZE-aligned DMA addresses and lengths.
+ * USERPTR is disabled on both queues because it can supply partial pages.
+ *
+ * Each Linux page is split into MALI_VPU_PAGES_PER_LINUX_PAGE VPU pages.
+ * Do not round DMA addresses down: the supplied mapping must already
+ * cover every page inserted into the VPU page tables.
+ */
+
+ count = mali_vpu_get_sg_table_npages(sgt) * MALI_VPU_PAGES_PER_LINUX_PAGE;
+
+ if ((pages->count + count) > pages->capacity) {
+ dev_err(pages->dev,
+ "pages capacity too small to append sgt. count=%zu, capacity=%zu, append=%zu",
+ pages->count, pages->capacity, count);
+ return -ENOMEM;
+ }
+
+ for_each_sg_dma_page(sgt->sgl, &piter, sgt->nents, 0) {
+ unsigned int j;
+ dma_addr_t base;
+
+ base = sg_page_iter_dma_address(&piter);
+
+ for (j = 0; j < MALI_VPU_PAGES_PER_LINUX_PAGE; ++j)
+ pages->pages[pages->count++].dma =
+ base + j * MALI_VPU_PAGE_SIZE;
+ }
+
+ return 0;
+}
+
+int mali_vpu_mmu_init(struct mali_vpu_mmu *mmu, struct device *dev)
+{
+ struct page *page;
+ dma_addr_t dma;
+
+ mmu->dev = dev;
+
+ /* Allocate Page Table Base (the L1 table). */
+ dma = mali_vpu_mmu_alloc_page(dev, &page, DMA_TO_DEVICE);
+ if (!page)
+ return -ENOMEM;
+
+ mmu->l2_tables = kvcalloc(MALI_VPU_INDEX_SIZE, sizeof(*mmu->l2_tables), GFP_KERNEL);
+ if (!mmu->l2_tables) {
+ mali_vpu_mmu_free_page(dev, dma, page, DMA_TO_DEVICE);
+ return -ENOMEM;
+ }
+
+ mmu->root.linux_page = page;
+ mmu->root.cpu = page_address(page);
+ mmu->root.dma = dma;
+
+ return 0;
+}
+
+void mali_vpu_mmu_cleanup(struct mali_vpu_mmu *mmu)
+{
+ int count = 0;
+ unsigned int i;
+
+ for (i = 0; i < MALI_VPU_INDEX_SIZE; i++) {
+ struct mali_vpu_mmu_pt_page *l2 = &mmu->l2_tables[i];
+
+ if (!l2->linux_page)
+ continue;
+
+ count += mali_vpu_mapped_count(l2->cpu);
+ mali_vpu_mmu_free_page(mmu->dev, l2->dma, l2->linux_page, DMA_TO_DEVICE);
+ }
+
+ kvfree(mmu->l2_tables);
+ mali_vpu_mmu_free_page(mmu->dev, mmu->root.dma, mmu->root.linux_page,
+ DMA_TO_DEVICE);
+
+ WARN_ON_ONCE(count > 0);
+}
+
+/* Optionally return the original page for callers that track CPU access. */
+dma_addr_t mali_vpu_mmu_alloc_page(struct device *dev, struct page **out_page,
+ enum dma_data_direction dir)
+{
+ struct page *page;
+ dma_addr_t dma_handle;
+
+ if (out_page)
+ *out_page = NULL;
+
+ page = alloc_page(GFP_KERNEL | __GFP_ZERO | __GFP_NORETRY);
+ if (!page)
+ return 0;
+
+ dma_handle = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+ if (dma_mapping_error(dev, dma_handle) != 0) {
+ dev_err(dev, "cannot map page to DMA address space. page=%p", page);
+ goto free_page;
+ }
+
+ dma_sync_single_for_device(dev, dma_handle, PAGE_SIZE, dir);
+
+ if (out_page)
+ *out_page = page;
+
+ return dma_handle;
+
+free_page:
+ __free_page(page);
+ return 0;
+}
+
+void mali_vpu_mmu_free_page(struct device *dev, dma_addr_t dma, struct page *page,
+ enum dma_data_direction dir)
+{
+ if (!page)
+ return;
+
+ dma_unmap_page(dev, dma, PAGE_SIZE, dir);
+ __free_page(page);
+}
+
+struct mali_vpu_mmu_pages *mali_vpu_mmu_alloc_pages(struct device *dev,
+ size_t count,
+ size_t capacity)
+{
+ struct mali_vpu_mmu_pages *pages;
+ int ret;
+
+ count = roundup(count, MALI_VPU_PAGES_PER_LINUX_PAGE);
+ capacity = roundup(capacity, MALI_VPU_PAGES_PER_LINUX_PAGE);
+ capacity = max(count, capacity);
+
+ pages = devm_kzalloc(dev,
+ sizeof(*pages) + sizeof(pages->pages[0]) * capacity,
+ GFP_KERNEL);
+ if (!pages)
+ return ERR_PTR(-ENOMEM);
+
+ pages->dev = dev;
+ pages->capacity = capacity;
+
+ for (pages->count = 0; pages->count < count; ) {
+ struct page *page;
+ dma_addr_t dma;
+ unsigned int i;
+
+ /* Allocate one linux page containing one or more MVE pages. */
+ dma = mali_vpu_mmu_alloc_page(dev, &page, DMA_BIDIRECTIONAL);
+ if (!page) {
+ ret = -ENOMEM;
+ goto release_pages;
+ }
+
+ /*
+ * If the Linux page is larger than the MVE page, then
+ * we iterate and add DMA addresses with an offset from
+ * the Linux page.
+ */
+ for (i = 0; i < MALI_VPU_PAGES_PER_LINUX_PAGE; i++) {
+ pages->pages[pages->count].dma = dma + i * MALI_VPU_PAGE_SIZE;
+ pages->pages[pages->count].linux_page = page;
+ pages->count++;
+ }
+ }
+
+ return pages;
+
+release_pages:
+ mali_vpu_mmu_free_pages(pages);
+
+ return ERR_PTR(ret);
+}
+
+struct mali_vpu_mmu_pages *mali_vpu_mmu_alloc_pages_sg(struct device *dev,
+ struct sg_table *sgt,
+ size_t capacity)
+{
+ struct mali_vpu_mmu_pages *pages;
+ size_t count;
+ int ret;
+
+ count = mali_vpu_get_sg_table_npages(sgt) * MALI_VPU_PAGES_PER_LINUX_PAGE;
+ capacity = roundup(capacity, MALI_VPU_PAGES_PER_LINUX_PAGE);
+ capacity = max(count, capacity);
+
+ pages = devm_kzalloc(dev,
+ sizeof(*pages) + sizeof(pages->pages[0]) * capacity,
+ GFP_KERNEL);
+ if (!pages)
+ return ERR_PTR(-ENOMEM);
+
+ pages->dev = dev;
+ pages->capacity = capacity;
+ pages->is_external = true;
+ pages->offset = sgt->sgl ? sgt->sgl->offset : 0;
+
+ ret = mali_vpu_append_sg_table(pages, sgt);
+ if (ret) {
+ devm_kfree(dev, pages);
+ return ERR_PTR(ret);
+ }
+
+ return pages;
+}
+
+/* Release backing pages only; page-table updates are handled by the caller. */
+static void mali_vpu_mmu_shrink_pages(struct mali_vpu_mmu_pages *pages, size_t npages)
+{
+ while (pages->count > npages) {
+ pages->count--;
+
+ if ((pages->count % MALI_VPU_PAGES_PER_LINUX_PAGE) == 0)
+ mali_vpu_mmu_free_page(pages->dev,
+ pages->pages[pages->count].dma,
+ pages->pages[pages->count].linux_page,
+ DMA_BIDIRECTIONAL);
+
+ pages->pages[pages->count].dma = 0;
+ pages->pages[pages->count].linux_page = NULL;
+ }
+}
+
+int mali_vpu_mmu_resize_pages(struct mali_vpu_mmu_pages *pages, size_t npages)
+{
+ int ret;
+ size_t oldcount = pages->count;
+
+ if (pages->is_external) {
+ dev_err(pages->dev, "externally managed pages cannot be resized");
+ return -EINVAL;
+ }
+
+ npages = roundup(npages, MALI_VPU_PAGES_PER_LINUX_PAGE);
+
+ if (npages > pages->capacity) {
+ dev_err(pages->dev,
+ "new MMU pages size is larger than capacity. npages=%zu, capacity=%zu",
+ npages, pages->capacity);
+ return -ENOMEM;
+ }
+
+ /* Free pages if npage is smaller than allocated pages. */
+ if (pages->count > npages) {
+ /* Replace the PTEs before releasing their backing pages. */
+ ret = mali_vpu_remap_pages(pages, oldcount, npages);
+ if (ret)
+ return ret;
+
+ mali_vpu_mmu_shrink_pages(pages, npages);
+ return 0;
+ }
+
+ /* Allocate pages if npage is larger than allocated pages. */
+ while (pages->count < npages) {
+ struct page *page;
+ dma_addr_t dma;
+ unsigned int i;
+
+ dma = mali_vpu_mmu_alloc_page(pages->dev, &page, DMA_BIDIRECTIONAL);
+ if (!page)
+ goto rollback_alloc;
+
+ for (i = 0; i < MALI_VPU_PAGES_PER_LINUX_PAGE; i++) {
+ pages->pages[pages->count].dma = dma + i * MALI_VPU_PAGE_SIZE;
+ pages->pages[pages->count].linux_page = page;
+ pages->count++;
+ }
+ }
+
+ return mali_vpu_remap_pages(pages, oldcount, pages->count);
+
+rollback_alloc:
+ mali_vpu_mmu_shrink_pages(pages, oldcount);
+ return -ENOMEM;
+}
+
+void mali_vpu_mmu_free_pages(struct mali_vpu_mmu_pages *pages)
+{
+ unsigned int i;
+
+ mali_vpu_mmu_unmap_pages(pages);
+
+ if (!pages->is_external)
+ for (i = 0; i < pages->count; i += MALI_VPU_PAGES_PER_LINUX_PAGE)
+ mali_vpu_mmu_free_page(pages->dev, pages->pages[i].dma,
+ pages->pages[i].linux_page,
+ DMA_BIDIRECTIONAL);
+
+ devm_kfree(pages->dev, pages);
+}
+
+size_t mali_vpu_mmu_size_pages(struct mali_vpu_mmu_pages *pages)
+{
+ return pages->count * MALI_VPU_PAGE_SIZE;
+}
+
+int mali_vpu_mmu_sync_pages(struct mali_vpu_mmu_pages *pages, enum dma_data_direction dir)
+{
+ size_t i;
+
+ if (dir == DMA_FROM_DEVICE) {
+ for (i = 0; i < pages->count; i += MALI_VPU_PAGES_PER_LINUX_PAGE)
+ dma_sync_single_for_cpu(pages->dev, pages->pages[i].dma,
+ PAGE_SIZE, DMA_FROM_DEVICE);
+ } else if (dir == DMA_TO_DEVICE) {
+ for (i = 0; i < pages->count; i += MALI_VPU_PAGES_PER_LINUX_PAGE)
+ dma_sync_single_for_device(pages->dev, pages->pages[i].dma,
+ PAGE_SIZE, DMA_TO_DEVICE);
+ } else {
+ dev_err(pages->dev, "unsupported MMU flush direction. dir=%u", dir);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int mali_vpu_mmu_map_pages(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ struct mali_vpu_mmu_pages *pages,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access)
+{
+ size_t i;
+ int ret;
+
+ /*
+ * A pages object may be mapped at multiple virtual addresses. For example,
+ * each core maps the same physical firmware text pages into its own virtual
+ * address region.
+ */
+ for (i = 0; i < pages->count; i++) {
+ ret = mali_vpu_map_page(mmu, va + i * MALI_VPU_PAGE_SIZE,
+ pages->pages[i].dma, attr, access);
+ if (ret)
+ goto unmap_pages;
+ }
+
+ /*
+ * Reserve the rest of the address range. Adding a dummy page with
+ * DMA address 'MALI_VPU_PAGE_SIZE' should not lead to memory corruption,
+ * because the page is marked as 'no access'.
+ */
+ for (; i < pages->capacity; i++) {
+ ret = mali_vpu_map_page(mmu, va + i * MALI_VPU_PAGE_SIZE, MALI_VPU_PAGE_SIZE,
+ MALI_VPU_ATTR_PRIVATE, MALI_VPU_ACCESS_NO);
+ if (ret)
+ goto unmap_pages;
+ }
+
+ pages->mmu = mmu;
+ pages->va = va;
+ pages->attr = attr;
+ pages->access = access;
+
+ return 0;
+
+unmap_pages:
+ while (i-- > 0)
+ mali_vpu_unmap_page(mmu, va + i * MALI_VPU_PAGE_SIZE);
+
+ return ret;
+}
+
+void mali_vpu_mmu_unmap_pages(struct mali_vpu_mmu_pages *pages)
+{
+ size_t i;
+
+ if (!pages->mmu)
+ return;
+
+ for (i = 0; i < pages->capacity; i++)
+ mali_vpu_unmap_page(pages->mmu, pages->va + i * MALI_VPU_PAGE_SIZE);
+
+ pages->mmu = NULL;
+ pages->va = 0;
+}
+
+int mali_vpu_mmu_map_dma(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ dma_addr_t dma,
+ size_t size,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access)
+{
+ int ret;
+ size_t offset;
+
+ for (offset = 0; offset < size; offset += MALI_VPU_PAGE_SIZE) {
+ ret = mali_vpu_map_page(mmu, va + offset, dma + offset, attr, access);
+ if (ret)
+ goto unmap_pages;
+ }
+
+ return 0;
+
+unmap_pages:
+ /* Unroll mapped pages. */
+ while (offset > 0) {
+ offset -= MALI_VPU_PAGE_SIZE;
+ mali_vpu_unmap_page(mmu, va + offset);
+ }
+
+ return ret;
+}
+
+void mali_vpu_mmu_unmap_va(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ size_t size)
+{
+ size_t offset;
+
+ for (offset = 0; offset < size; offset += MALI_VPU_PAGE_SIZE)
+ mali_vpu_unmap_page(mmu, va + offset);
+}
+
+/* Initialize read-only firmware text before the VPU accesses these pages. */
+int mali_vpu_mmu_write(struct mali_vpu_mmu_pages *pages,
+ const void *data,
+ size_t size)
+{
+ size_t i;
+
+ if (pages->is_external || size > mali_vpu_mmu_size_pages(pages))
+ return -EINVAL;
+
+ for (i = 0; size; i += MALI_VPU_PAGES_PER_LINUX_PAGE) {
+ dma_addr_t dma = pages->pages[i].dma;
+ void *dst = page_address(pages->pages[i].linux_page);
+ size_t n = min_t(size_t, size, PAGE_SIZE);
+
+ memcpy(dst, data, n);
+ dma_sync_single_for_device(pages->dev, dma, PAGE_SIZE,
+ DMA_TO_DEVICE);
+
+ data += n;
+ size -= n;
+ }
+
+ return 0;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
new file mode 100644
index 0000000000000..6ad645a2dc6bb
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
@@ -0,0 +1,152 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_MMU_H__
+#define __MALI_VPU_MMU_H__
+
+#include <linux/dma-mapping.h>
+#include <linux/hashtable.h>
+#include <linux/types.h>
+
+/* Page size is 4 KB - pow(2,12) bytes */
+#define MALI_VPU_PAGE_SHIFT 12
+#define MALI_VPU_PAGE_SIZE BIT(MALI_VPU_PAGE_SHIFT)
+#define MALI_VPU_PAGE_MASK (MALI_VPU_PAGE_SIZE - 1)
+
+/**
+ * typedef mali_vpu_mmu_va - 32 bit virtual address.
+ *
+ * This is the address the firmware/MVE will use.
+ */
+typedef u32 mali_vpu_mmu_va;
+
+/**
+ * typedef mali_vpu_mmu_pte - Page table entry.
+ *
+ * A PTE pointer should always point at a Linux kernel virtual address.
+ *
+ * AT - Attribute.
+ * PA - VPU physical address bits [39:12], supplied as a DMA address.
+ * AP - Access permission.
+ *
+ * 30 2 0
+ * +---+-------------------------------------------------------+---+
+ * | AT| PA | AP|
+ * +---+-------------------------------------------------------+---+
+ */
+typedef u32 mali_vpu_mmu_pte;
+
+enum mali_vpu_mmu_attr {
+ MALI_VPU_ATTR_PRIVATE = 0,
+ MALI_VPU_ATTR_REFFRAME = 1,
+ MALI_VPU_ATTR_SHARED_RO = 2,
+ MALI_VPU_ATTR_SHARED_RW = 3
+};
+
+enum mali_vpu_mmu_access {
+ MALI_VPU_ACCESS_NO = 0,
+ MALI_VPU_ACCESS_READ_ONLY = 1,
+ MALI_VPU_ACCESS_EXECUTABLE = 2,
+ MALI_VPU_ACCESS_READ_WRITE = 3
+};
+
+/**
+ * struct mali_vpu_mmu_pages - Structure used to allocate an array of pages.
+ * @dev: Pointer to device.
+ * @node: Hash table node. Used to keep track of allocated pages objects.
+ * @mmu: Pointer to MMU instance.
+ * @va: MVE virtual address. Set to 0 if objects is unmapped.
+ * @offset: Offset from mapped VA to where the data begins.
+ * @attr: Page table attributes.
+ * @access: Page table access.
+ * @capacity: Maximum number of MVE pages this object can hold.
+ * @count: Current number of allocated pages.
+ * @is_external: If the physical pages have been externally allocated.
+ * @pages: DMA address and original backing Linux page for each MVE page.
+ * The backing page is NULL for external memory and may be shared by
+ * multiple MVE entries when Linux pages are larger than MVE pages.
+ */
+struct mali_vpu_mmu_pages {
+ struct device *dev;
+ struct hlist_node node;
+ struct mali_vpu_mmu *mmu;
+ mali_vpu_mmu_va va;
+ size_t offset;
+ enum mali_vpu_mmu_attr attr;
+ enum mali_vpu_mmu_access access;
+ size_t capacity;
+ size_t count;
+ bool is_external;
+ struct {
+ dma_addr_t dma;
+ struct page *linux_page;
+ } pages[];
+};
+
+/**
+ * struct mali_vpu_mmu_pt_page - Backing memory for one VPU page table.
+ * @linux_page: Original Linux page backing the table.
+ * @cpu: Kernel virtual address used to access the table entries.
+ * @dma: DMA address used by the VPU page-table walker.
+ */
+struct mali_vpu_mmu_pt_page {
+ struct page *linux_page;
+ mali_vpu_mmu_pte *cpu;
+ dma_addr_t dma;
+};
+
+/**
+ * struct mali_vpu_mmu - VPU page tables for one session.
+ * @dev: Device used for DMA mapping and synchronization.
+ * @root: Root (L1) page table.
+ * @l2_tables: Array of L2 table descriptors indexed by the L1 index.
+ * L2 backing pages are allocated on demand.
+ */
+struct mali_vpu_mmu {
+ struct device *dev;
+ struct mali_vpu_mmu_pt_page root;
+ struct mali_vpu_mmu_pt_page *l2_tables;
+};
+
+int mali_vpu_mmu_init(struct mali_vpu_mmu *mmu, struct device *dev);
+void mali_vpu_mmu_cleanup(struct mali_vpu_mmu *mmu);
+
+dma_addr_t mali_vpu_mmu_alloc_page(struct device *dev, struct page **out_page,
+ enum dma_data_direction dir);
+void mali_vpu_mmu_free_page(struct device *dev, dma_addr_t dma, struct page *page,
+ enum dma_data_direction dir);
+
+struct mali_vpu_mmu_pages *mali_vpu_mmu_alloc_pages(struct device *dev,
+ size_t count,
+ size_t capacity);
+
+struct mali_vpu_mmu_pages *mali_vpu_mmu_alloc_pages_sg(struct device *dev,
+ struct sg_table *sgt,
+ size_t capacity);
+
+size_t mali_vpu_mmu_size_pages(struct mali_vpu_mmu_pages *pages);
+int mali_vpu_mmu_resize_pages(struct mali_vpu_mmu_pages *pages, size_t npages);
+void mali_vpu_mmu_free_pages(struct mali_vpu_mmu_pages *pages);
+int mali_vpu_mmu_sync_pages(struct mali_vpu_mmu_pages *pages, enum dma_data_direction dir);
+
+int mali_vpu_mmu_map_pages(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ struct mali_vpu_mmu_pages *pages,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access);
+
+void mali_vpu_mmu_unmap_pages(struct mali_vpu_mmu_pages *pages);
+
+int mali_vpu_mmu_map_dma(struct mali_vpu_mmu *mmu,
+ mali_vpu_mmu_va va,
+ dma_addr_t dma,
+ size_t size,
+ enum mali_vpu_mmu_attr attr,
+ enum mali_vpu_mmu_access access);
+
+void mali_vpu_mmu_unmap_va(struct mali_vpu_mmu *mmu, mali_vpu_mmu_va va, size_t size);
+int mali_vpu_mmu_write(struct mali_vpu_mmu_pages *pages, const void *data, size_t size);
+
+#endif /* __MALI_VPU_MMU_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 36a25030d7324..2d63dc058c91a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -213,6 +213,13 @@ static int mali_vpu_probe(struct platform_device *pdev)
goto err_rst_clk;
}
+ vpu_dev->dma_mask = vpu_dev->variant->dma_mask;
+ ret = dma_set_mask_and_coherent(vpu_dev->dev, vpu_dev->dma_mask);
+ if (ret)
+ goto err_rst_clk;
+
+ dma_set_max_seg_size(vpu_dev->dev, SZ_2G);
+
ret = v4l2_device_register(&pdev->dev, &vpu_dev->v4l2_dev);
if (ret) {
dev_err(vpu_dev->dev, "failed to register V4L2 device\n");
@@ -280,3 +287,4 @@ static struct platform_driver mali_vpu_driver = {
module_platform_driver(mali_vpu_driver);
MODULE_DESCRIPTION("Arm Mali VPU driver");
MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS("DMA_BUF");
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index d1e8680de5556..10481ba16d68a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -86,8 +86,17 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
goto err_m2m_ctx_release;
}
+ ret = mali_vpu_mmu_init(&sess->mmu, sess->vpu_dev->dev);
+ if (ret) {
+ dev_err(sess->vpu_dev->dev, "failed to initialise MMU: %d\n", ret);
+ goto err_ctrls_cleanup;
+ }
+
return 0;
+err_ctrls_cleanup:
+ mali_vpu_ctrls_cleanup(sess);
+
err_m2m_ctx_release:
v4l2_m2m_ctx_release(sess->m2m_ctx);
@@ -108,6 +117,7 @@ void mali_vpu_session_cleanup(struct mali_vpu_session *sess)
mutex_unlock(&sess->vq_lock);
+ mali_vpu_mmu_cleanup(&sess->mmu);
mali_vpu_ctrls_cleanup(sess);
mutex_destroy(&sess->vq_lock);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 766912aec0b8e..4dd6c8bc22dd9 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -13,6 +13,7 @@
#include "mali-vpu-device.h"
#include "mali-vpu-formats.h"
+#include "mali-vpu-mmu.h"
struct v4l2_m2m_ctx;
@@ -32,6 +33,7 @@ struct mali_vpu_port {
* @role: Role of this session.
* @port_src: Source (OUTPUT) port for this session.
* @port_dst: Destination (CAPTURE) port for this session.
+ * @mmu: MMU handle for this session
* @vq_lock: Mutex lock to serialise per-session queue related ioctls.
* @is_fw_ready: True if the firmware is loaded and ready to accept commands.
* @min_buffers_required: Driver-reported minimum buffers for role-specific queue.
@@ -47,6 +49,7 @@ struct mali_vpu_session {
enum mali_vpu_session_role role;
struct mali_vpu_port port_src;
struct mali_vpu_port port_dst;
+ struct mali_vpu_mmu mmu;
struct mutex vq_lock; /* lock to serialise per-session ioctls */
bool is_fw_ready;
u32 min_buffers_required;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
index b807f8d888222..19ad2667f2597 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -149,6 +149,7 @@ static const struct mali_vpu_ctrl_desc mali_v76_dec_ctrls[] = {
};
const struct mali_vpu_variant mali_vpu_v76_variant = {
+ .dma_mask = DMA_BIT_MASK(40),
.expected_hw_pid = MALI_VPU_HW_PID_V76,
.rst_names = mali_v76_rst_names,
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
index ad20896c90b0f..d316e010fff48 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -49,6 +49,7 @@ struct mali_vpu_control_caps {
* @rst_names: Array of reset control names for this variant.
* @num_rsts: Number of reset controls for this variant.
* @expected_hw_pid: Expected hardware product ID read from the version register.
+ * @dma_mask: DMA addressing capability mask.
* @fmt_caps: Format capability tables for encode/decode paths.
* @frmsz_caps: Frame dimension limits and defaults.
* @ctrl_caps: Control capability tables for encode/decode paths.
@@ -57,6 +58,7 @@ struct mali_vpu_variant {
const char * const *rst_names;
unsigned int num_rsts;
u32 expected_hw_pid;
+ u64 dma_mask;
struct mali_vpu_format_caps fmt_caps;
struct mali_vpu_framesize_caps frmsz_caps;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 09/28] media: mali-vpu: add MMU software page table manager
2026-11-19 15:37 ` [RFC PATCH 09/28] media: mali-vpu: add MMU software page table manager Charvi Mehta
@ 2026-11-19 15:59 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:59 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] media: mali-vpu: tautological VA range check and 32-bit VA
truncation
- [Medium] media: mali-vpu: devm_kzalloc() lifetime and size mismatch
for MMU pages
--
commit 635e5f0a32c7b907a3a802ff85c63c68b877a3e5
Author: Rath Sankaralingam <rath.sankaralingam@arm.com>
media: mali-vpu: add MMU software page table manager
This commit adds the software MMU two-level page table manager for the
Mali VPU driver and configures the 40-bit DMA mask and maximum DMA
segment size.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
> new file mode 100644
> index 0000000000000..00cf5251eff5e
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
> @@ -0,0 +1,693 @@
[ ... ]
> +/* Number of bits for the virtual address space. */
> +#define MALI_VPU_VA_BITS 32
> +#define MALI_VPU_VA_MASK GENMASK(MALI_VPU_VA_BITS - 1, 0)
[ ... ]
> +static int mali_vpu_map_page(struct mali_vpu_mmu *mmu,
> + mali_vpu_mmu_va va,
> + dma_addr_t dma,
> + enum mali_vpu_mmu_attr attr,
> + enum mali_vpu_mmu_access access)
> +{
> + struct mali_vpu_mmu_pt_page *l2;
> + mali_vpu_mmu_pte *pte;
> +
> + /* Check that both VA and DMA address are page aligned. */
> + if ((va | dma) & MALI_VPU_PAGE_MASK) {
> + dev_err(mmu->dev, "VA and DMA address must be page aligned. va=0x%x, dma=%pad",
> + va, &dma);
> + return -EFAULT;
> + }
> +
> + /* Check that VA is within valid address range. */
> + if (va & ~MALI_VPU_VA_MASK) {
> + dev_err(mmu->dev, "VA out of valid range. va=0x%x", va);
> + return -EFAULT;
> + }
[Severity: Medium]
Can (va & ~MALI_VPU_VA_MASK) ever evaluate to true here? mali_vpu_mmu_va
is a u32 and MALI_VPU_VA_MASK is GENMASK(31, 0), so the check is always
zero, and 64-bit address arithmetic in callers such as
mali_vpu_mmu_map_pages() (va + i * MALI_VPU_PAGE_SIZE) and
mali_vpu_mmu_map_dma() (va + offset) is truncated to u32 before reaching
mali_vpu_map_page(), wrapping around to address 0 if a mapping crosses
4 GiB.
[ ... ]
> +struct mali_vpu_mmu_pages *mali_vpu_mmu_alloc_pages(struct device *dev,
> + size_t count,
> + size_t capacity)
> +{
> + struct mali_vpu_mmu_pages *pages;
> + int ret;
> +
> + count = roundup(count, MALI_VPU_PAGES_PER_LINUX_PAGE);
> + capacity = roundup(capacity, MALI_VPU_PAGES_PER_LINUX_PAGE);
> + capacity = max(count, capacity);
> +
> + pages = devm_kzalloc(dev,
> + sizeof(*pages) + sizeof(pages->pages[0]) * capacity,
> + GFP_KERNEL);
> + if (!pages)
> + return ERR_PTR(-ENOMEM);
[ ... ]
> +void mali_vpu_mmu_free_pages(struct mali_vpu_mmu_pages *pages)
> +{
> + unsigned int i;
> +
> + mali_vpu_mmu_unmap_pages(pages);
> +
> + if (!pages->is_external)
> + for (i = 0; i < pages->count; i += MALI_VPU_PAGES_PER_LINUX_PAGE)
> + mali_vpu_mmu_free_page(pages->dev, pages->pages[i].dma,
> + pages->pages[i].linux_page,
> + DMA_BIDIRECTIONAL);
> +
> + devm_kfree(pages->dev, pages);
> +}
[Severity: Medium]
Should mali_vpu_mmu_alloc_pages() and mali_vpu_mmu_alloc_pages_sg() use
kvzalloc(struct_size(pages, pages, capacity), GFP_KERNEL) and kvfree()
instead of devm_kzalloc() and devm_kfree()?
Because the backing pages and DMA mappings are not devres-managed,
unbinding the platform device while a V4L2 session fd is open frees the
struct mali_vpu_mmu_pages container in devres_release_all() (leaking the
backing pages), and when the fd is later closed, mali_vpu_mmu_free_pages()
reads the freed pages struct and hits WARN_ON(rc) in devm_kfree(). In
addition, devm_kzalloc() requires physically contiguous slab memory and
will fail above KMALLOC_MAX_SIZE for large buffer capacities.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=9
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 10/28] media: mali-vpu: add firmware cache infrastructure
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (8 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 09/28] media: mali-vpu: add MMU software page table manager Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:50 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 11/28] media: mali-vpu: load firmware for decoder sessions Charvi Mehta
` (17 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add a per-device cache for VPU firmware binaries and wire its lifetime
into probe and remove paths. Key cache entries by codec, session role
and compatible firmware major version, and load new entries asynchronously
through the firmware loader.
Validate the firmware and host protocol versions from the firmware
header and derive the number of text, private BSS and shared BSS pages.
Notify registered waiters when loading completes while retaining each
binary through cache, loader and caller references. Failed entries are
evicted immediately.
For now, keep the cache policy deliberately simple: a background thread
periodically prunes entries that are only held by the cache itself.
This is intentional for initial upstreaming, so lifetime and
concurrency behavior stays easy to reason about before adding a
smarter eviction policy.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-device.h | 3 +
.../arm/mali-vpu/mali-vpu-firmware-cache.c | 461 ++++++++++++++++++
.../arm/mali-vpu/mali-vpu-firmware-cache.h | 172 +++++++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 16 +-
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 3 +
6 files changed, 656 insertions(+), 2 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 0716a2b0c8712..74e1455c01d8b 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -9,6 +9,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-vb2.o \
mali-vpu-formats.o \
mali-vpu-controls.o \
- mali-vpu-mmu.o
+ mali-vpu-mmu.o \
+ mali-vpu-firmware-cache.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index 20361df05bdcd..a35c1b901aa3f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -14,6 +14,7 @@
struct clk_bulk_data;
struct v4l2_m2m_dev;
struct mali_vpu_variant;
+struct mali_vpu_fw_cache;
struct reset_control_bulk_data;
/**
@@ -38,6 +39,7 @@ enum mali_vpu_session_role {
* @v4l2_dev: V4L2 device structure for mali vpu.
* @vdev_dec: Video device structure for decoder.
* @m2m_dev: V4L2 mem2mem device structure for mali vpu.
+ * @fw_cache: Firmware cache for sharing or reusing firmware blobs.
*/
struct mali_vpu_dev {
struct device *dev;
@@ -53,6 +55,7 @@ struct mali_vpu_dev {
struct v4l2_device v4l2_dev;
struct video_device *vdev_dec;
struct v4l2_m2m_dev *m2m_dev;
+ struct mali_vpu_fw_cache *fw_cache;
};
#endif /* __MALI_VPU_DEVICE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
new file mode 100644
index 0000000000000..746d30f179305
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
@@ -0,0 +1,461 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/delay.h>
+#include <linux/kthread.h>
+
+#include "mali-vpu-device.h"
+#include "mali-vpu-firmware-cache.h"
+#include "mali-vpu-mmu.h"
+#include "mali-vpu-protocol-def-v3.h"
+
+#define MALI_VPU_FW_CACHE_CLEANUP_INTERVAL_MS 30000
+
+#define MALI_VPU_GET_MAJOR_VERSION(ver) (((ver) >> 16) & 0xFF)
+
+static inline bool mali_vpu_test_bit_le32(u32 bit, const __le32 *addr)
+{
+ return !!(le32_to_cpu(addr[bit >> 5]) & BIT(bit & 0x1F));
+}
+
+static void mali_vpu_fw_bin_release(struct kref *ref)
+{
+ struct mali_vpu_fw_bin *bin = container_of(ref, struct mali_vpu_fw_bin, refcount);
+
+ /* cache_link must already be detached before final put */
+ WARN_ON_ONCE(!list_empty(&bin->cache_link));
+
+ if (bin->fw)
+ release_firmware(bin->fw);
+
+ kfree(bin);
+}
+
+static int mali_vpu_fw_build_name(char *name, size_t name_sz,
+ const struct mali_vpu_fw_key *key)
+{
+ const char *role_str = NULL;
+ const char *codec = NULL;
+ int n;
+
+ if (!name || !name_sz || !key)
+ return -EINVAL;
+
+ if (key->session_role != MALI_VPU_DECODER)
+ return -EINVAL;
+
+ role_str = "dec";
+
+ if (key->codec_fourcc == V4L2_PIX_FMT_H264)
+ codec = "h264";
+
+ if (!codec)
+ return -ENOENT;
+
+ n = snprintf(name, name_sz, "arm/mali-vpu/v%u/%s%s.fwb",
+ key->fw_ver_major, codec, role_str);
+ if (n < 0)
+ return n;
+ if ((size_t)n >= name_sz)
+ return -ENAMETOOLONG;
+
+ return 0;
+}
+
+static int mali_vpu_fw_parse_header(const struct firmware *fw,
+ struct mali_vpu_fw_bin *vpu_fw_bin)
+{
+ const struct mali_vpu_fw_header *header;
+ u32 header_fw_ver_major;
+ u32 fw_version, text_length, bss_start, bss_bitmap_size;
+ u32 master_rw_start, master_rw_size;
+ mali_vpu_mmu_va va;
+ u32 i;
+
+ /* Validate the fw binary */
+ if (fw->size < sizeof(*header)) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "firmware bin too small: %zu bytes, need at least %zu\n",
+ fw->size, sizeof(*header));
+ return -EINVAL;
+ }
+
+ header = (const struct mali_vpu_fw_header *)fw->data;
+
+ fw_version = le32_to_cpu(header->fw_version);
+ text_length = le32_to_cpu(header->text_length);
+ bss_start = le32_to_cpu(header->bss_start_address);
+ bss_bitmap_size = le32_to_cpu(header->bss_bitmap_size);
+ master_rw_start = le32_to_cpu(header->master_rw_start_address);
+ master_rw_size = le32_to_cpu(header->master_rw_size);
+
+ if (text_length < sizeof(*header) ||
+ text_length > fw->size) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "invalid firmware text length: %u\n",
+ text_length);
+ return -EBADMSG;
+ }
+
+ if (bss_bitmap_size > ARRAY_SIZE(header->bss_bitmap) * 32) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "invalid firmware BSS bitmap size: %u (max %zu)\n",
+ bss_bitmap_size, ARRAY_SIZE(header->bss_bitmap) * 32UL);
+ return -EBADMSG;
+ }
+
+ /* Firmware version mismatch */
+ header_fw_ver_major = MALI_VPU_GET_MAJOR_VERSION(fw_version);
+ if (header_fw_ver_major != vpu_fw_bin->key.fw_ver_major) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "firmware major version mismatch: expected %u, got %u. Binary ver: %#x\n",
+ vpu_fw_bin->key.fw_ver_major, header_fw_ver_major, fw_version);
+ return -EINVAL;
+ }
+
+ /* Match host<->firmware communication protocol version */
+ if (header->protocol_major != MALI_VPU_HOST_PROTOCOL_MAJOR ||
+ header->protocol_minor > MALI_VPU_HOST_PROTOCOL_MINOR) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "firmware protocol version mismatch: expected %u.%u, got %u.%u\n",
+ MALI_VPU_HOST_PROTOCOL_MAJOR, MALI_VPU_HOST_PROTOCOL_MINOR,
+ header->protocol_major, header->protocol_minor);
+ return -EINVAL;
+ }
+
+ vpu_fw_bin->header = header;
+
+ /* Calculate number of pages needed for the text segment. */
+ vpu_fw_bin->text_page_count = DIV_ROUND_UP(text_length, MALI_VPU_PAGE_SIZE);
+
+ /* Calculate number of pages needed for the BSS segments. */
+ vpu_fw_bin->bss_page_count = 0;
+ vpu_fw_bin->shared_bss_page_count = 0;
+
+ va = bss_start;
+ for (i = 0; i < bss_bitmap_size; i++) {
+ if (va >= master_rw_start &&
+ va < (master_rw_start + master_rw_size))
+ vpu_fw_bin->shared_bss_page_count++;
+ else if (mali_vpu_test_bit_le32(i, header->bss_bitmap))
+ vpu_fw_bin->bss_page_count++;
+
+ va += MALI_VPU_PAGE_SIZE;
+ }
+
+ return 0;
+}
+
+static void mali_vpu_fw_notify_event_list(struct list_head *event_list,
+ struct mali_vpu_fw_bin *bin,
+ bool same_thread)
+{
+ struct mali_vpu_fw_event *event, *tmp;
+
+ list_for_each_entry_safe(event, tmp, event_list, link) {
+ list_del_init(&event->link);
+ event->fw_bin_ready(bin, event->arg, same_thread);
+ }
+}
+
+static void mali_vpu_fw_request_done(const struct firmware *fw, void *context)
+{
+ struct mali_vpu_fw_bin *vpu_fw_bin = context;
+ const struct mali_vpu_fw_header *hdr;
+ LIST_HEAD(local_event_list);
+ int ret = 0;
+
+ mutex_lock(&vpu_fw_bin->cache->lock);
+ mutex_lock(&vpu_fw_bin->lock);
+
+ if (!fw) {
+ ret = -ENOENT;
+ vpu_fw_bin->fw = NULL;
+ } else {
+ ret = mali_vpu_fw_parse_header(fw, vpu_fw_bin);
+ if (ret) {
+ release_firmware(fw);
+ vpu_fw_bin->fw = NULL;
+ } else {
+ vpu_fw_bin->fw = fw;
+ }
+ }
+
+ /* We have to notify waiters in failure or success case */
+ vpu_fw_bin->load_status = ret;
+ vpu_fw_bin->state = ret ? MALI_VPU_FW_BIN_FAILED : MALI_VPU_FW_BIN_READY;
+
+ /*
+ * Take a local copy of events/waiters
+ * so that notify callback can be called outside vpu_fw_bin->lock
+ */
+ list_splice_init(&vpu_fw_bin->event_list, &local_event_list);
+
+ if (ret) {
+ /* Firmware load failed, remove cached entry */
+ list_del_init(&vpu_fw_bin->cache_link);
+ kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release); /* Drop cache-ref */
+ }
+
+ mutex_unlock(&vpu_fw_bin->lock);
+ mutex_unlock(&vpu_fw_bin->cache->lock);
+
+ /*
+ * Keep the loader-owned reference until all waiters have been notified
+ * and completion logging has finished. Callbacks may release their
+ * caller-owned reference to vpu_fw_bin; on a load failure, the cache
+ * reference has also already been dropped.
+ */
+ mali_vpu_fw_notify_event_list(&local_event_list, vpu_fw_bin, false);
+
+ if (ret) {
+ dev_warn(vpu_fw_bin->cache->dev, "firmware load failed: %s (status=%d)\n",
+ vpu_fw_bin->filename, ret);
+ } else {
+ hdr = vpu_fw_bin->header;
+ dev_info(vpu_fw_bin->cache->dev, "firmware loaded: %s\n", vpu_fw_bin->filename);
+
+ dev_dbg(vpu_fw_bin->cache->dev,
+ "firmware info: Version %.*s, MVE Protocol Version %u.%u,\n",
+ (int)sizeof(hdr->fw_version_str), (const char *)hdr->fw_version_str,
+ hdr->protocol_major, hdr->protocol_minor);
+ dev_dbg(vpu_fw_bin->cache->dev,
+ "firmware info: \"%.*s\", Rasc jmp = %#x, text_length=%u, bss=%#x\n",
+ (int)sizeof(hdr->info_string), (const char *)hdr->info_string,
+ le32_to_cpu(hdr->rasc_jmp),
+ le32_to_cpu(hdr->text_length),
+ le32_to_cpu(hdr->bss_start_address));
+ dev_dbg(vpu_fw_bin->cache->dev,
+ "firmware info: text pages=%u, shared bss=%u, bss pages=%u\n",
+ vpu_fw_bin->text_page_count, vpu_fw_bin->shared_bss_page_count,
+ vpu_fw_bin->bss_page_count);
+ }
+
+ /* Notification and logging are complete; drop loader-owned reference */
+ kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release);
+}
+
+static struct mali_vpu_fw_bin *
+mali_vpu_fw_bin_create(struct mali_vpu_fw_cache *cache,
+ const struct mali_vpu_fw_key *key,
+ struct mali_vpu_fw_event *event,
+ bool *call_now)
+{
+ struct mali_vpu_fw_bin *vpu_fw_bin;
+ int ret;
+
+ vpu_fw_bin = kzalloc_obj(*vpu_fw_bin);
+ if (!vpu_fw_bin)
+ return ERR_PTR(-ENOMEM);
+
+ /* One cache-owned reference while binary is in cache->fw_bins */
+ kref_init(&vpu_fw_bin->refcount);
+ /* One caller-owned reference returned by this get(). */
+ kref_get(&vpu_fw_bin->refcount);
+
+ vpu_fw_bin->cache = cache;
+ vpu_fw_bin->key = *key;
+ vpu_fw_bin->state = MALI_VPU_FW_BIN_LOADING;
+ vpu_fw_bin->load_status = -EINPROGRESS;
+ mutex_init(&vpu_fw_bin->lock);
+ INIT_LIST_HEAD(&vpu_fw_bin->cache_link);
+ INIT_LIST_HEAD(&vpu_fw_bin->event_list);
+
+ ret = mali_vpu_fw_build_name(vpu_fw_bin->filename,
+ sizeof(vpu_fw_bin->filename), key);
+ if (ret) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "could not build fw name: fourcc=%#x, role=%d, version=%#x\n",
+ key->codec_fourcc, key->session_role, key->fw_ver_major);
+ goto err_put_caller_cache;
+ }
+
+ list_add(&vpu_fw_bin->cache_link, &cache->fw_bins);
+
+ /* One loader-owned reference while firmware is being loaded */
+ kref_get(&vpu_fw_bin->refcount);
+
+ ret = request_firmware_nowait(THIS_MODULE, true, vpu_fw_bin->filename,
+ cache->dev, GFP_KERNEL, vpu_fw_bin,
+ mali_vpu_fw_request_done);
+ if (ret)
+ goto err_put_loader_caller_cache;
+
+ /*
+ * Serialise adding event to the event_list and firmware load completion.
+ * For fast callback where firmware load is completed before this thread can
+ * queue the event to the event_list for notification.
+ */
+ mutex_lock(&vpu_fw_bin->lock);
+ if (vpu_fw_bin->state == MALI_VPU_FW_BIN_LOADING)
+ list_add(&event->link, &vpu_fw_bin->event_list);
+ else
+ *call_now = true;
+ mutex_unlock(&vpu_fw_bin->lock);
+
+ return vpu_fw_bin;
+
+err_put_loader_caller_cache:
+ kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release); /* Loader ref */
+ list_del_init(&vpu_fw_bin->cache_link);
+
+err_put_caller_cache:
+ kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release); /* Caller ref */
+ kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release); /* Cache ref */
+ /* Binary freed in release callback */
+ return ERR_PTR(ret);
+}
+
+static bool mali_vpu_fw_key_match(const struct mali_vpu_fw_key *a,
+ const struct mali_vpu_fw_key *b)
+{
+ return a->codec_fourcc == b->codec_fourcc &&
+ a->session_role == b->session_role &&
+ a->fw_ver_major == b->fw_ver_major;
+}
+
+static struct mali_vpu_fw_bin *mali_vpu_fw_find_locked(struct mali_vpu_fw_cache *cache,
+ const struct mali_vpu_fw_key *key)
+{
+ struct mali_vpu_fw_bin *bin;
+
+ list_for_each_entry(bin, &cache->fw_bins, cache_link) {
+ if (!mali_vpu_fw_key_match(&bin->key, key))
+ continue;
+ if (!kref_get_unless_zero(&bin->refcount))
+ continue;
+
+ return bin;
+ }
+
+ return NULL;
+}
+
+int mali_vpu_fw_cache_get(struct mali_vpu_fw_cache *cache,
+ const struct mali_vpu_fw_key *key,
+ struct mali_vpu_fw_event *event)
+{
+ struct mali_vpu_fw_bin *bin;
+ bool call_now = false;
+ int ret = 0;
+
+ if (!cache || !key || !event || !event->fw_bin_ready)
+ return -EINVAL;
+
+ /*
+ * The callback may run synchronously before this function returns if
+ * firmware loading has already completed. In that case, same_thread is
+ * true. Otherwise, the firmware loader invokes it asynchronously with
+ * same_thread set to false.
+ *
+ * The event must remain valid until the callback runs. Every call that
+ * returns 0 owns one reference to the binary supplied to the callback;
+ * the caller must release it with mali_vpu_fw_cache_put()
+ */
+ INIT_LIST_HEAD(&event->link);
+
+ mutex_lock(&cache->lock);
+ bin = mali_vpu_fw_find_locked(cache, key);
+ if (!bin) {
+ bin = mali_vpu_fw_bin_create(cache, key, event, &call_now);
+ if (IS_ERR(bin)) {
+ ret = PTR_ERR(bin);
+ goto unlock_cache;
+ }
+ } else {
+ mutex_lock(&bin->lock);
+ if (bin->state == MALI_VPU_FW_BIN_LOADING)
+ list_add(&event->link, &bin->event_list);
+ else
+ call_now = true;
+ mutex_unlock(&bin->lock);
+ }
+
+unlock_cache:
+ mutex_unlock(&cache->lock);
+
+ if (ret)
+ return ret;
+
+ if (call_now)
+ event->fw_bin_ready(bin, event->arg, true);
+
+ return 0;
+}
+
+void mali_vpu_fw_cache_put(struct mali_vpu_fw_bin *bin)
+{
+ kref_put(&bin->refcount, mali_vpu_fw_bin_release);
+}
+
+static void mali_vpu_fw_cache_prune(struct mali_vpu_fw_cache *cache)
+{
+ LIST_HEAD(prune_list);
+ struct mali_vpu_fw_bin *bin, *temp;
+
+ mutex_lock(&cache->lock);
+ list_for_each_entry_safe(bin, temp, &cache->fw_bins, cache_link) {
+ if (kref_read(&bin->refcount) != 1) /* If only cache holds ref */
+ continue;
+
+ list_del_init(&bin->cache_link);
+ list_add(&bin->cache_link, &prune_list);
+ }
+ mutex_unlock(&cache->lock);
+
+ list_for_each_entry_safe(bin, temp, &prune_list, cache_link) {
+ list_del_init(&bin->cache_link);
+ kref_put(&bin->refcount, mali_vpu_fw_bin_release);
+ }
+}
+
+static int mali_vpu_fw_cache_thread_fn(void *data)
+{
+ struct mali_vpu_fw_cache *cache = data;
+
+ while (!kthread_should_stop()) {
+ mali_vpu_fw_cache_prune(cache);
+ msleep_interruptible(MALI_VPU_FW_CACHE_CLEANUP_INTERVAL_MS);
+ }
+
+ return 0;
+}
+
+int mali_vpu_fw_cache_init(struct mali_vpu_fw_cache *cache, struct device *dev)
+{
+ int ret;
+
+ cache->dev = dev;
+ mutex_init(&cache->lock);
+ INIT_LIST_HEAD(&cache->fw_bins);
+
+ cache->cache_thread = kthread_run(mali_vpu_fw_cache_thread_fn, cache, "mali_vpu_fw_cache");
+ if (IS_ERR(cache->cache_thread)) {
+ ret = PTR_ERR(cache->cache_thread);
+ cache->cache_thread = NULL;
+ return ret;
+ }
+
+ return 0;
+}
+
+void mali_vpu_fw_cache_cleanup(struct mali_vpu_fw_cache *cache)
+{
+ struct mali_vpu_fw_bin *bin;
+ struct mali_vpu_fw_bin *temp;
+ LIST_HEAD(local);
+
+ if (cache->cache_thread)
+ kthread_stop(cache->cache_thread);
+
+ mutex_lock(&cache->lock);
+ list_splice_init(&cache->fw_bins, &local);
+ mutex_unlock(&cache->lock);
+
+ list_for_each_entry_safe(bin, temp, &local, cache_link) {
+ list_del_init(&bin->cache_link);
+ kref_put(&bin->refcount, mali_vpu_fw_bin_release);
+ }
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
new file mode 100644
index 0000000000000..f8b024222d2d2
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
@@ -0,0 +1,172 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_FIRMWARE_CACHE_H__
+#define __MALI_VPU_FIRMWARE_CACHE_H__
+
+#include <asm/byteorder.h>
+#include <linux/firmware.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/types.h>
+
+struct device;
+struct task_struct;
+
+/**
+ * struct mali_vpu_fw_cache - Per-device firmware cache context.
+ * @dev: Device used for firmware requests and logging.
+ * @lock: Serializes access to @fw_bins and cache-global state.
+ * @fw_bins: List of cached firmware binaries (linked by struct mali_vpu_fw_bin::cache_link).
+ * @cache_thread: Background thread that periodically prunes bins as per cache policy.
+ *
+ * One cache context exists per VPU device.
+ */
+
+struct mali_vpu_fw_cache {
+ struct device *dev;
+ struct mutex lock; /* Protects the fw_bins list */
+ struct list_head fw_bins;
+ struct task_struct *cache_thread;
+};
+
+/**
+ * struct mali_vpu_fw_header - Firmware binary header.
+ * @rasc_jmp: Start address.
+ * @protocol_minor: Host interface protocol minor version.
+ * @protocol_major: Host interface protocol major version.
+ * @reserved: Reserved for future use. Always 0.
+ * @mve_version: MVE version string.
+ * @info_string: Human readable codec information.
+ * @part_number: Part number.
+ * @git_hash: Short git hash of the firmware source.
+ * @fw_version: Semantic firmware version encoded as 0x00MMmmpp where MM = major,
+ * mm = minor, pp = patch.
+ * @fw_version_str: Semantic firmware version as a string.
+ * @text_length: Length in bytes of the read-only part of the firmware.
+ * @bss_start_address: Start address for BSS segment. This is always page-aligned.
+ * @bss_bitmap_size: The number of bits used in 'bss_bitmap'.
+ * @bss_bitmap: Bitmap which pages that shall be allocated and MMU mapped.
+ * If bit N is set, then a page shall be allocated and MMU mapped to VA address
+ * FW_BASE + bss_start_address + N * MVE_PAGE_SIZE.
+ * @master_rw_start_address: Defines a region of shared pages.
+ * @master_rw_size: Defines a region of shared pages.
+ *
+ * This structure is read directly from the firmware binary and describes the layout
+ * of the firmware in memory. All multibyte integer fields are little-endian.
+ */
+struct mali_vpu_fw_header {
+ __le32 rasc_jmp;
+ u8 protocol_minor;
+ u8 protocol_major;
+ u8 reserved[2];
+
+ u8 mve_version[8];
+ u8 info_string[48];
+
+ u8 part_number[8];
+ u8 git_hash[8];
+
+ __le32 fw_version;
+ u8 fw_version_str[16];
+
+ __le32 text_length;
+ __le32 bss_start_address;
+ __le32 bss_bitmap_size;
+ __le32 bss_bitmap[16];
+
+ __le32 master_rw_start_address;
+ __le32 master_rw_size;
+};
+
+/**
+ * struct mali_vpu_fw_key - Structure describing the key fields to identify a firmware binary.
+ * @codec_fourcc: Fourcc code of the codec.
+ * @session_role: Role of the session (encoder/decoder).
+ * @fw_ver_major: Expected major firmware version.
+ *
+ * The key uniquely identifies the firmware variant expected by a session.
+ */
+struct mali_vpu_fw_key {
+ u32 codec_fourcc;
+ u32 session_role;
+ u32 fw_ver_major;
+};
+
+enum mali_vpu_fw_bin_state {
+ MALI_VPU_FW_BIN_LOADING = 0,
+ MALI_VPU_FW_BIN_READY,
+ MALI_VPU_FW_BIN_FAILED,
+};
+
+/**
+ * struct mali_vpu_fw_bin - Structure describing a loaded firmware binary.
+ * @refcount: Lifetime reference count (cache/session/loader references).
+ * @cache: Owning firmware cache context.
+ * @cache_link: Node in mali_vpu_fw_cache::fw_bins list.
+ * @event_list: List of waiters (mali_vpu_fw_event) waiting for this firmware to load.
+ * @lock: Protects load, status, header pointers and @event_list.
+ * @key: Identity key for cache match and filename generation.
+ * @filename: Firmware filename derived from @key.
+ * @state: Current load state of the firmware binary (loading/ready/failed).
+ * @load_status: Final load result (0 on success, negative errno on failure).
+ * @fw: Firmware object returned by firmware loader.
+ * @header: Parsed firmware header within @fw->data.
+ * @text_page_count: Derived page count for firmware text segment.
+ * @bss_page_count: Derived page count for firmware BSS segment.
+ * @shared_bss_page_count: Derived page count for shared BSS area.
+ *
+ * A bin remains in the cache while referenced and may be shared by many sessions
+ * requesting the same @key.
+ */
+struct mali_vpu_fw_bin {
+ struct kref refcount;
+ struct mali_vpu_fw_cache *cache;
+
+ struct list_head cache_link;
+ struct list_head event_list;
+ struct mutex lock; /* protects state/load_status/header/waiters */
+
+ struct mali_vpu_fw_key key;
+ char filename[128];
+
+ enum mali_vpu_fw_bin_state state;
+ int load_status;
+
+ const struct firmware *fw;
+ const struct mali_vpu_fw_header *header;
+
+ /* Derived fields from mali_vpu_fw_header */
+ u32 text_page_count;
+ u32 bss_page_count;
+ u32 shared_bss_page_count;
+};
+
+/**
+ * struct mali_vpu_fw_event - Completion callback registration for firmware loads.
+ * @link: Internal list node used while in queued in a bin's waiter list.
+ * @fw_bin_ready: Callback routine invoked when firmware load finishes. Its same_thread
+ * argument is true when called synchronously from mali_vpu_fw_cache_get(),
+ * and false when called from the asynchronous firmware-loader callback.
+ * @arg: Opaque client context passed to @fw_bin_ready.
+ *
+ * Callback is invoked for both success and failure. The client checks fw_bin->load_status
+ * to determine result.
+ */
+struct mali_vpu_fw_event {
+ struct list_head link;
+ void (*fw_bin_ready)(struct mali_vpu_fw_bin *fw_bin, void *arg, bool same_thread);
+ void *arg;
+};
+
+int mali_vpu_fw_cache_init(struct mali_vpu_fw_cache *cache, struct device *dev);
+void mali_vpu_fw_cache_cleanup(struct mali_vpu_fw_cache *cache);
+int mali_vpu_fw_cache_get(struct mali_vpu_fw_cache *cache,
+ const struct mali_vpu_fw_key *key,
+ struct mali_vpu_fw_event *event);
+void mali_vpu_fw_cache_put(struct mali_vpu_fw_bin *bin);
+
+#endif /* __MALI_VPU_FIRMWARE_CACHE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 2d63dc058c91a..b87725d7d4cab 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -21,6 +21,7 @@
#include <media/v4l2-mem2mem.h>
#include "mali-vpu-device.h"
+#include "mali-vpu-firmware-cache.h"
#include "mali-vpu-registers.h"
#include "mali-vpu-variant.h"
#include "mali-vpu-vidioc.h"
@@ -232,14 +233,26 @@ static int mali_vpu_probe(struct platform_device *pdev)
goto err_unreg_v4l2;
}
- ret = mali_vpu_register_video_device(vpu_dev, MALI_VPU_DECODER);
+ vpu_dev->fw_cache = devm_kzalloc(&pdev->dev, sizeof(*vpu_dev->fw_cache), GFP_KERNEL);
+ if (!vpu_dev->fw_cache) {
+ ret = -ENOMEM;
+ goto err_release_m2m;
+ }
+
+ ret = mali_vpu_fw_cache_init(vpu_dev->fw_cache, vpu_dev->dev);
if (ret)
goto err_release_m2m;
+ ret = mali_vpu_register_video_device(vpu_dev, MALI_VPU_DECODER);
+ if (ret)
+ goto err_cleanup_fw_cache;
+
platform_set_drvdata(pdev, vpu_dev);
return 0;
+err_cleanup_fw_cache:
+ mali_vpu_fw_cache_cleanup(vpu_dev->fw_cache);
err_release_m2m:
v4l2_m2m_release(vpu_dev->m2m_dev);
err_unreg_v4l2:
@@ -257,6 +270,7 @@ static void mali_vpu_remove(struct platform_device *pdev)
struct mali_vpu_dev *vpu_dev = platform_get_drvdata(pdev);
video_unregister_device(vpu_dev->vdev_dec);
+ mali_vpu_fw_cache_cleanup(vpu_dev->fw_cache);
v4l2_m2m_release(vpu_dev->m2m_dev);
v4l2_device_unregister(&vpu_dev->v4l2_dev);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index aed51eec6ca9c..f3a61731191bc 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -25,6 +25,9 @@
*
****************************************************************************/
+#define MALI_VPU_HOST_PROTOCOL_MAJOR 3
+#define MALI_VPU_HOST_PROTOCOL_MINOR 1
+
#define MVE_CHROMA_FORMAT_420 (0x1)
/* format bitfield: */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 10/28] media: mali-vpu: add firmware cache infrastructure
2026-11-19 15:37 ` [RFC PATCH 10/28] media: mali-vpu: add firmware cache infrastructure Charvi Mehta
@ 2026-11-19 15:50 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:50 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] media: mali-vpu: race and UAF in mali_vpu_fw_cache_cleanup()
- [Low] media: mali-vpu: %zu format mismatch in
mali_vpu_fw_parse_header()
--
commit c4904ccc1e8387726744e3d73159243657fc5c24
Author: Charvi Mehta <charvi.mehta@arm.com>
media: mali-vpu: add firmware cache infrastructure
This commit adds a per-device firmware cache for Mali VPU firmware
binaries, loading entries asynchronously and pruning unused binaries in a
background thread.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
> new file mode 100644
> index 0000000000000..746d30f179305
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
> @@ -0,0 +1,461 @@
[ ... ]
> +static int mali_vpu_fw_parse_header(const struct firmware *fw,
> + struct mali_vpu_fw_bin *vpu_fw_bin)
> +{
[ ... ]
> + if (bss_bitmap_size > ARRAY_SIZE(header->bss_bitmap) * 32) {
> + dev_err(vpu_fw_bin->cache->dev,
> + "invalid firmware BSS bitmap size: %u (max %zu)\n",
> + bss_bitmap_size, ARRAY_SIZE(header->bss_bitmap) * 32UL);
> + return -EBADMSG;
> + }
[Severity: Low]
Does multiplying ARRAY_SIZE(header->bss_bitmap) by 32UL promote the
argument to unsigned long, causing a -Wformat warning with %zu on 32-bit
builds where size_t is unsigned int? Multiplying by 32 instead of 32UL
preserves size_t.
[ ... ]
> +static void mali_vpu_fw_request_done(const struct firmware *fw, void *context)
> +{
> + struct mali_vpu_fw_bin *vpu_fw_bin = context;
> + const struct mali_vpu_fw_header *hdr;
> + LIST_HEAD(local_event_list);
> + int ret = 0;
> +
> + mutex_lock(&vpu_fw_bin->cache->lock);
> + mutex_lock(&vpu_fw_bin->lock);
> +
> + if (!fw) {
> + ret = -ENOENT;
> + vpu_fw_bin->fw = NULL;
> + } else {
> + ret = mali_vpu_fw_parse_header(fw, vpu_fw_bin);
> + if (ret) {
> + release_firmware(fw);
> + vpu_fw_bin->fw = NULL;
> + } else {
> + vpu_fw_bin->fw = fw;
> + }
> + }
> +
> + /* We have to notify waiters in failure or success case */
> + vpu_fw_bin->load_status = ret;
> + vpu_fw_bin->state = ret ? MALI_VPU_FW_BIN_FAILED : MALI_VPU_FW_BIN_READY;
> +
> + /*
> + * Take a local copy of events/waiters
> + * so that notify callback can be called outside vpu_fw_bin->lock
> + */
> + list_splice_init(&vpu_fw_bin->event_list, &local_event_list);
> +
> + if (ret) {
> + /* Firmware load failed, remove cached entry */
> + list_del_init(&vpu_fw_bin->cache_link);
> + kref_put(&vpu_fw_bin->refcount, mali_vpu_fw_bin_release); /* Drop cache-ref */
> + }
> +
> + mutex_unlock(&vpu_fw_bin->lock);
> + mutex_unlock(&vpu_fw_bin->cache->lock);
[ ... ]
> +void mali_vpu_fw_cache_cleanup(struct mali_vpu_fw_cache *cache)
> +{
> + struct mali_vpu_fw_bin *bin;
> + struct mali_vpu_fw_bin *temp;
> + LIST_HEAD(local);
> +
> + if (cache->cache_thread)
> + kthread_stop(cache->cache_thread);
> +
> + mutex_lock(&cache->lock);
> + list_splice_init(&cache->fw_bins, &local);
> + mutex_unlock(&cache->lock);
> +
> + list_for_each_entry_safe(bin, temp, &local, cache_link) {
> + list_del_init(&bin->cache_link);
> + kref_put(&bin->refcount, mali_vpu_fw_bin_release);
> + }
> +}
[Severity: High]
Can an in-flight request_firmware_nowait() callback race with
mali_vpu_fw_cache_cleanup() during device removal, causing list
corruption, a double kref_put() of the cache reference, and use-after-free
of both vpu_fw_bin and vpu_dev->fw_cache?
CPU0 CPU1 (firmware loader work)
mali_vpu_remove()
mali_vpu_fw_cache_cleanup()
mutex_lock(&cache->lock)
list_splice_init(&cache->fw_bins,
&local)
mutex_unlock(&cache->lock)
mali_vpu_fw_request_done()
mutex_lock(&bin->cache->lock)
/* load fails: ret != 0 */
list_del_init(&bin->cache_link)
<-- races with local traversal
kref_put(&bin->refcount, ...)
<-- drops cache-ref
list_for_each_entry_safe(...,
&local, ...)
list_del_init(&bin->cache_link)
kref_put(&bin->refcount, ...)
<-- drops cache-ref a second time
/* devres frees vpu_dev->fw_cache */
dev_warn(bin->cache->dev, ...)
<-- use-after-free of cache
kref_put(&bin->refcount, ...)
<-- use-after-free of bin
mali_vpu_fw_cache_cleanup() releases cache->lock while each bin->cache_link
is still linked into the stack-allocated local list, so a failing
mali_vpu_fw_request_done() can mutate bin->cache_link while local is being
walked unlocked. Even if the list operations do not overlap, both
mali_vpu_fw_cache_cleanup() and the failure path in
mali_vpu_fw_request_done() unconditionally drop the single cache-owned
reference on bin->refcount, freeing vpu_fw_bin before
mali_vpu_fw_request_done() finishes logging and dropping its loader
reference. Finally, because mali_vpu_fw_cache_cleanup() does not wait for
pending request_firmware_nowait() callbacks before mali_vpu_remove() lets
devres free vpu_dev->fw_cache, any later callback dereferences freed memory
through vpu_fw_bin->cache.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=10
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 11/28] media: mali-vpu: load firmware for decoder sessions
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (9 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 10/28] media: mali-vpu: add firmware cache infrastructure Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 12/28] media: mali-vpu: prepare firmware address space Charvi Mehta
` (16 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Request the firmware binary during preparation of the decoder coded queue.
Build the cache key from the coded format, session role and
variant firmware major version. Propagate request submission errors
through VB2 prepare_streaming callback.
Track asynchronous firmware loading per session. Retain the cache
reference after a successful load and release failed or late results.
During the session teardown, wait for firmware loading with a timeout.
On timeout, cancel the pending cache waiter. If completion has already
claimed the waiter, wait for its callback before releasing session
resources.
This establishes firmware-binary loading and ownership only. Firmware
address-space construction and execution are added in subsequent
changes.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../arm/mali-vpu/mali-vpu-firmware-cache.c | 45 +++++
.../arm/mali-vpu/mali-vpu-firmware-cache.h | 2 +
.../platform/arm/mali-vpu/mali-vpu-formats.c | 7 +
.../platform/arm/mali-vpu/mali-vpu-formats.h | 1 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 175 +++++++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-session.h | 40 +++-
.../arm/mali-vpu/mali-vpu-variant-v76.c | 2 +
.../platform/arm/mali-vpu/mali-vpu-variant.h | 2 +
.../platform/arm/mali-vpu/mali-vpu-vb2.c | 28 +++
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 5 +-
10 files changed, 303 insertions(+), 4 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
index 746d30f179305..2cc69df71bd93 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
@@ -390,6 +390,51 @@ void mali_vpu_fw_cache_put(struct mali_vpu_fw_bin *bin)
kref_put(&bin->refcount, mali_vpu_fw_bin_release);
}
+bool mali_vpu_fw_cache_cancel_waiter(struct mali_vpu_fw_cache *cache,
+ struct mali_vpu_fw_event *event)
+{
+ struct mali_vpu_fw_bin *cancelled_bin = NULL;
+ struct mali_vpu_fw_event *waiter;
+ struct mali_vpu_fw_bin *bin;
+
+ if (!cache || !event)
+ return false;
+
+ mutex_lock(&cache->lock);
+
+ list_for_each_entry(bin, &cache->fw_bins, cache_link) {
+ mutex_lock(&bin->lock);
+
+ list_for_each_entry(waiter, &bin->event_list, link) {
+ if (waiter != event)
+ continue;
+
+ list_del_init(&event->link);
+ cancelled_bin = bin;
+ break;
+ }
+
+ mutex_unlock(&bin->lock);
+
+ if (cancelled_bin)
+ break;
+ }
+
+ mutex_unlock(&cache->lock);
+
+ if (!cancelled_bin)
+ return false;
+
+ /*
+ * Every successful cache_get() owns one caller reference. Since the
+ * callback will no longer run for this waiter, release that reference
+ * here.
+ */
+ mali_vpu_fw_cache_put(cancelled_bin);
+
+ return true;
+}
+
static void mali_vpu_fw_cache_prune(struct mali_vpu_fw_cache *cache)
{
LIST_HEAD(prune_list);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
index f8b024222d2d2..ea7d35ef8ca60 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
@@ -168,5 +168,7 @@ int mali_vpu_fw_cache_get(struct mali_vpu_fw_cache *cache,
const struct mali_vpu_fw_key *key,
struct mali_vpu_fw_event *event);
void mali_vpu_fw_cache_put(struct mali_vpu_fw_bin *bin);
+bool mali_vpu_fw_cache_cancel_waiter(struct mali_vpu_fw_cache *cache,
+ struct mali_vpu_fw_event *event);
#endif /* __MALI_VPU_FIRMWARE_CACHE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
index d268435432371..57aa1b3baf40f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
@@ -13,6 +13,13 @@
#include "mali-vpu-formats.h"
#include "mali-vpu-session.h"
+bool mali_vpu_is_coded_q(const struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ if (sess->role == MALI_VPU_DECODER)
+ return V4L2_TYPE_IS_OUTPUT(type);
+ return V4L2_TYPE_IS_CAPTURE(type);
+}
+
const struct mali_vpu_format_desc *
mali_vpu_find_format(struct mali_vpu_session *sess, u32 pixfmt,
enum v4l2_buf_type type)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
index 5703142b93b46..37198325ee481 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
@@ -33,5 +33,6 @@ const struct mali_vpu_format_desc *mali_vpu_find_format(struct mali_vpu_session
int mali_vpu_fill_pixmp_fmt(const struct mali_vpu_format_desc *fmt,
u32 width, u32 height,
struct v4l2_pix_format_mplane *pix_mp);
+bool mali_vpu_is_coded_q(const struct mali_vpu_session *sess, enum v4l2_buf_type type);
#endif /* __MALI_VPU_FORMATS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 10481ba16d68a..b9a3960bca4c7 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -6,16 +6,20 @@
#include <linux/device.h>
#include <linux/err.h>
#include <linux/errno.h>
+#include <linux/slab.h>
#include <linux/string.h>
#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
#include "mali-vpu-controls.h"
#include "mali-vpu-dec.h"
+#include "mali-vpu-firmware-cache.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
+static const unsigned int mali_vpu_fw_load_timeout_ms = 10000;
+
static int mali_vpu_set_port_fmt(struct mali_vpu_port *port,
enum v4l2_buf_type type,
u32 width, u32 height)
@@ -55,6 +59,13 @@ static int mali_vpu_set_port_defaults(struct mali_vpu_session *sess)
fmsz->width_default, fmsz->height_default);
}
+static bool
+mali_vpu_both_ports_unrequested(struct mali_vpu_session *sess)
+{
+ return !sess->port_src.stream_requested &&
+ !sess->port_dst.stream_requested;
+}
+
int mali_vpu_session_init(struct mali_vpu_session *sess)
{
int ret;
@@ -79,6 +90,7 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
}
sess->is_fw_ready = false;
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
ret = mali_vpu_ctrls_init(sess);
if (ret) {
@@ -92,8 +104,22 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
goto err_ctrls_cleanup;
}
+ sess->fw_event = kzalloc(sizeof(*sess->fw_event), GFP_KERNEL);
+ if (!sess->fw_event) {
+ ret = -ENOMEM;
+ goto err_mmu_cleanup;
+ }
+
+ sess->fw_bin = NULL;
+ sess->port_src.stream_requested = false;
+ sess->port_dst.stream_requested = false;
+ init_completion(&sess->fw_load_done);
+
return 0;
+err_mmu_cleanup:
+ mali_vpu_mmu_cleanup(&sess->mmu);
+
err_ctrls_cleanup:
mali_vpu_ctrls_cleanup(sess);
@@ -105,10 +131,62 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
return ret;
}
-void mali_vpu_session_cleanup(struct mali_vpu_session *sess)
+static void mali_vpu_session_release_fw_bin(struct mali_vpu_session *sess)
{
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY) {
+ mali_vpu_fw_cache_put(sess->fw_bin);
+ sess->fw_bin = NULL;
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ return;
+ }
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_FAILED)
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+}
+
+int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
+{
+ struct mali_vpu_dev *vpu_dev = sess->vpu_dev;
+ int fw_load_ret = 0;
+ bool waiter_cancelled = false;
+
mutex_lock(&sess->vq_lock);
+ sess->port_src.stream_requested = false;
+ sess->port_dst.stream_requested = false;
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_LOADING) {
+ mutex_unlock(&sess->vq_lock);
+
+ if (!wait_for_completion_timeout(&sess->fw_load_done,
+ msecs_to_jiffies(mali_vpu_fw_load_timeout_ms))) {
+ dev_warn(vpu_dev->dev,
+ "firmware load timed out\n");
+ fw_load_ret = -ETIMEDOUT;
+
+ waiter_cancelled = mali_vpu_fw_cache_cancel_waiter(vpu_dev->fw_cache,
+ sess->fw_event);
+
+ /*
+ * The waiter was not found in the cache list, so
+ * firmware completion may already have claimed it for
+ * callback execution. Wait for the callback to finish
+ * before freeing the session and its event.
+ */
+ if (!waiter_cancelled)
+ wait_for_completion(&sess->fw_load_done);
+ }
+
+ mutex_lock(&sess->vq_lock);
+
+ if (waiter_cancelled)
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ }
+
+ mali_vpu_session_release_fw_bin(sess);
+
if (sess->m2m_ctx) {
v4l2_m2m_ctx_release(sess->m2m_ctx);
sess->m2m_ctx = NULL;
@@ -117,7 +195,102 @@ void mali_vpu_session_cleanup(struct mali_vpu_session *sess)
mutex_unlock(&sess->vq_lock);
+ kfree(sess->fw_event);
mali_vpu_mmu_cleanup(&sess->mmu);
mali_vpu_ctrls_cleanup(sess);
mutex_destroy(&sess->vq_lock);
+
+ return fw_load_ret;
+}
+
+static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
+ void *arg,
+ bool same_thread)
+{
+ struct mali_vpu_session *sess = arg;
+
+ if (!same_thread)
+ mutex_lock(&sess->vq_lock);
+
+ /*
+ * A synchronous cache hit precedes marking the port requested.
+ * An asynchronous callback acquires vq_lock after preparation.
+ */
+ if (!same_thread && mali_vpu_both_ports_unrequested(sess)) {
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ goto err_cache_put;
+ }
+
+ if (vpu_fw_bin->load_status) {
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ goto err_cache_put;
+ }
+
+ sess->fw_bin = vpu_fw_bin;
+
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_READY;
+
+ goto out_complete;
+
+err_cache_put:
+ mali_vpu_fw_cache_put(vpu_fw_bin);
+
+ sess->fw_bin = NULL;
+
+out_complete:
+ complete_all(&sess->fw_load_done);
+
+ if (!same_thread)
+ mutex_unlock(&sess->vq_lock);
+}
+
+static u32 mali_vpu_coded_q_fourcc(const struct mali_vpu_session *sess)
+{
+ if (sess->role == MALI_VPU_DECODER)
+ return sess->port_src.curr_fmt.fmt.pix_mp.pixelformat;
+ return sess->port_dst.curr_fmt.fmt.pix_mp.pixelformat;
+}
+
+static int mali_vpu_session_build_fw_key(const struct mali_vpu_session *sess,
+ u32 codec_fourcc,
+ struct mali_vpu_fw_key *key)
+{
+ if (!key || !sess)
+ return -EINVAL;
+
+ key->codec_fourcc = codec_fourcc;
+ key->session_role = sess->role;
+ key->fw_ver_major = sess->vpu_dev->variant->fw_ver_major;
+
+ return 0;
+}
+
+int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess)
+{
+ struct mali_vpu_fw_key fw_key;
+ int ret;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY ||
+ sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_LOADING)
+ return 0;
+
+ ret = mali_vpu_session_build_fw_key(sess, mali_vpu_coded_q_fourcc(sess), &fw_key);
+ if (ret)
+ return ret;
+
+ sess->fw_event->fw_bin_ready = mali_vpu_session_fw_bin_ready;
+ sess->fw_event->arg = sess;
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_LOADING;
+ reinit_completion(&sess->fw_load_done);
+
+ ret = mali_vpu_fw_cache_get(sess->vpu_dev->fw_cache, &fw_key, sess->fw_event);
+ if (ret) {
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ complete_all(&sess->fw_load_done);
+ return ret;
+ }
+
+ return 0;
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 4dd6c8bc22dd9..0b810d7d6a036 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -6,6 +6,7 @@
#ifndef __MALI_VPU_SESSION_H__
#define __MALI_VPU_SESSION_H__
+#include <linux/completion.h>
#include <linux/mutex.h>
#include <linux/videodev2.h>
#include <media/v4l2-ctrls.h>
@@ -16,15 +17,42 @@
#include "mali-vpu-mmu.h"
struct v4l2_m2m_ctx;
+struct mali_vpu_fw_bin;
+struct mali_vpu_fw_event;
/**
* struct mali_vpu_port - format state of a src/dst queue in a session.
* @curr_fmt: V4L2 format currently configured for the port
* @supported_fmts: List of format descriptors supported by the queue.
+ * @stream_requested: Port/Queue is being prepared for streaming or has
+ * successfully completed preparation. Protected by vq_lock.
*/
struct mali_vpu_port {
struct v4l2_format curr_fmt;
struct mali_vpu_format_list supported_fmts;
+ bool stream_requested;
+};
+
+/*
+ * Firmware binary load lifecycle
+ *
+ * Preparing the coded queue starts an asynchronous firmware request.
+ * The binary request does not require queued buffers and does not wait for
+ * loading to complete. READY and LOADING requests are reused.
+ *
+ * A cache hit may invoke the callback synchronously. An asynchronous callback
+ * discards its result if neither port requests streaming. A successfully
+ * loaded binary is retained until session cleanup.
+ *
+ * Session cleanup waits for an in-flight callback before releasing the
+ * session-owned binary reference and callback event. On timeout, it cancels
+ * the pending cache waiter or waits for an already claimed callback.
+ */
+enum mali_vpu_session_fw_bin_state {
+ MALI_VPU_SESSION_FW_BIN_UNINITIALIZED = 0,
+ MALI_VPU_SESSION_FW_BIN_LOADING,
+ MALI_VPU_SESSION_FW_BIN_READY,
+ MALI_VPU_SESSION_FW_BIN_FAILED,
};
/**
@@ -43,6 +71,10 @@ struct mali_vpu_port {
* @ctrl_handler: V4L2 control handler bound to this session.
* @fh: V4L2 file handle for this session.
* @m2m_ctx: V4L2 mem2mem context for this session.
+ * @fw_bin: Pointer to firmware binary loaded for this session.
+ * @fw_event: Firmware binary load event callback structure for this session.
+ * @fw_bin_state: Session firmware binary status.
+ * @fw_load_done: Completion used to synchronize teardown with an in-flight firmware load.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
@@ -61,10 +93,16 @@ struct mali_vpu_session {
struct v4l2_fh fh;
struct v4l2_m2m_ctx *m2m_ctx;
+
+ struct mali_vpu_fw_bin *fw_bin;
+ struct mali_vpu_fw_event *fw_event;
+ enum mali_vpu_session_fw_bin_state fw_bin_state;
+ struct completion fw_load_done;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
-void mali_vpu_session_cleanup(struct mali_vpu_session *sess);
+int mali_vpu_session_cleanup(struct mali_vpu_session *sess);
+int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
index 19ad2667f2597..0a7481d97c717 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -17,6 +17,7 @@
#include "mali-vpu-variant.h"
#define MALI_VPU_HW_PID_V76 0x56627
+#define MALI_VPU_FW_VER_MAJOR_V76 1
#define MALI_V76_MAX_FRAME_WIDTH 8192
#define MALI_V76_MAX_FRAME_HEIGHT 8192
@@ -151,6 +152,7 @@ static const struct mali_vpu_ctrl_desc mali_v76_dec_ctrls[] = {
const struct mali_vpu_variant mali_vpu_v76_variant = {
.dma_mask = DMA_BIT_MASK(40),
.expected_hw_pid = MALI_VPU_HW_PID_V76,
+ .fw_ver_major = MALI_VPU_FW_VER_MAJOR_V76,
.rst_names = mali_v76_rst_names,
.num_rsts = ARRAY_SIZE(mali_v76_rst_names),
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
index d316e010fff48..3c37cd70bd81f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -50,6 +50,7 @@ struct mali_vpu_control_caps {
* @num_rsts: Number of reset controls for this variant.
* @expected_hw_pid: Expected hardware product ID read from the version register.
* @dma_mask: DMA addressing capability mask.
+ * @fw_ver_major: Major firmware version compatible with this variant.
* @fmt_caps: Format capability tables for encode/decode paths.
* @frmsz_caps: Frame dimension limits and defaults.
* @ctrl_caps: Control capability tables for encode/decode paths.
@@ -59,6 +60,7 @@ struct mali_vpu_variant {
unsigned int num_rsts;
u32 expected_hw_pid;
u64 dma_mask;
+ u32 fw_ver_major;
struct mali_vpu_format_caps fmt_caps;
struct mali_vpu_framesize_caps frmsz_caps;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
index acdbb06785f6b..4262758a5be26 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
@@ -59,12 +59,40 @@ static void mali_vpu_vb2_buffer_queue(struct vb2_buffer *vb2_buf)
v4l2_m2m_buf_queue(sess->m2m_ctx, v4l2_buf);
}
+static int mali_vpu_vb2_prepare_streaming(struct vb2_queue *q)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(q);
+ struct mali_vpu_port *port = V4L2_TYPE_IS_OUTPUT(q->type) ?
+ &sess->port_src : &sess->port_dst;
+ int ret;
+
+ if (mali_vpu_is_coded_q(sess, q->type)) {
+ ret = mali_vpu_session_request_fw_load(sess);
+ if (ret)
+ return ret;
+ }
+
+ port->stream_requested = true;
+ return 0;
+}
+
+static void mali_vpu_vb2_unprepare_streaming(struct vb2_queue *q)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(q);
+ struct mali_vpu_port *port = V4L2_TYPE_IS_OUTPUT(q->type) ?
+ &sess->port_src : &sess->port_dst;
+
+ port->stream_requested = false;
+}
+
static const struct vb2_ops mali_vpu_vb2_ops = {
.queue_setup = mali_vpu_vb2_queue_setup,
.buf_out_validate = mali_vpu_vb2_buffer_out_validate,
.buf_init = mali_vpu_vb2_buffer_init,
.buf_prepare = mali_vpu_vb2_buffer_prepare,
.buf_queue = mali_vpu_vb2_buffer_queue,
+ .prepare_streaming = mali_vpu_vb2_prepare_streaming,
+ .unprepare_streaming = mali_vpu_vb2_unprepare_streaming,
};
static void mali_vpu_queue_init_type(struct mali_vpu_session *sess,
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index 88e706be6705e..d9196cc86cdf4 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -62,12 +62,13 @@ static int mali_vpu_open(struct file *filp)
static int mali_vpu_release(struct file *filp)
{
struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ int ret = 0;
- mali_vpu_session_cleanup(sess);
+ ret = mali_vpu_session_cleanup(sess);
v4l2_fh_del(&sess->fh, filp);
v4l2_fh_exit(&sess->fh);
kfree(sess);
- return 0;
+ return ret;
}
const struct v4l2_file_operations mali_vpu_fops = {
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 12/28] media: mali-vpu: prepare firmware address space
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (10 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 11/28] media: mali-vpu: load firmware for decoder sessions Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:57 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 13/28] media: mali-vpu: add firmware message transport Charvi Mehta
` (15 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add a per-session firmware object containing the memory required to
communicate with and execute the firmware.
Define the fixed firmware virtual-address layout and allocate the host
communication pages. Allocate firmware text pages shared by each core
instance, private BSS pages for each core and shared BSS pages mapped
across all instances. Copy the loaded firmware text into the executable
mapping.
Construct the firmware address space when the firmware binary
finishes loading, using the number of cores reported by the hardware.
Release all the firmware mappings and backing pages during session
teardown.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../arm/mali-vpu/mali-vpu-firmware-cache.c | 55 ++-
.../arm/mali-vpu/mali-vpu-firmware-cache.h | 6 +
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 383 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 108 +++++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 6 -
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 94 +++++
.../arm/mali-vpu/mali-vpu-registers.h | 7 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 20 +
.../platform/arm/mali-vpu/mali-vpu-session.h | 8 +-
10 files changed, 675 insertions(+), 15 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 74e1455c01d8b..168b4bc43d54d 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -10,6 +10,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-formats.o \
mali-vpu-controls.o \
mali-vpu-mmu.o \
- mali-vpu-firmware-cache.o
+ mali-vpu-firmware-cache.o \
+ mali-vpu-firmware.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
index 2cc69df71bd93..40eba8caeb0d7 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.c
@@ -7,6 +7,7 @@
#include <linux/kthread.h>
#include "mali-vpu-device.h"
+#include "mali-vpu-firmware.h"
#include "mali-vpu-firmware-cache.h"
#include "mali-vpu-mmu.h"
#include "mali-vpu-protocol-def-v3.h"
@@ -15,11 +16,6 @@
#define MALI_VPU_GET_MAJOR_VERSION(ver) (((ver) >> 16) & 0xFF)
-static inline bool mali_vpu_test_bit_le32(u32 bit, const __le32 *addr)
-{
- return !!(le32_to_cpu(addr[bit >> 5]) & BIT(bit & 0x1F));
-}
-
static void mali_vpu_fw_bin_release(struct kref *ref)
{
struct mali_vpu_fw_bin *bin = container_of(ref, struct mali_vpu_fw_bin, refcount);
@@ -71,9 +67,13 @@ static int mali_vpu_fw_parse_header(const struct firmware *fw,
u32 header_fw_ver_major;
u32 fw_version, text_length, bss_start, bss_bitmap_size;
u32 master_rw_start, master_rw_size;
+ u64 text_end, bss_end, master_rw_end;
mali_vpu_mmu_va va;
u32 i;
+ u32 region_size = MVE_MEM_REGION_FW_INSTANCE0_ADDR_END -
+ MVE_MEM_REGION_FW_INSTANCE0_ADDR_BEGIN;
+
/* Validate the fw binary */
if (fw->size < sizeof(*header)) {
dev_err(vpu_fw_bin->cache->dev,
@@ -91,8 +91,15 @@ static int mali_vpu_fw_parse_header(const struct firmware *fw,
master_rw_start = le32_to_cpu(header->master_rw_start_address);
master_rw_size = le32_to_cpu(header->master_rw_size);
+ /*
+ * text_length includes header and is measured from file offset zero.
+ * The image is mapped at MALI_VPU_FW_TEXT_BASE_ADDR.
+ */
+ text_end = (u64)MALI_VPU_FW_TEXT_BASE_ADDR + text_length;
+
if (text_length < sizeof(*header) ||
- text_length > fw->size) {
+ text_length > fw->size ||
+ text_end > region_size) {
dev_err(vpu_fw_bin->cache->dev,
"invalid firmware text length: %u\n",
text_length);
@@ -106,6 +113,42 @@ static int mali_vpu_fw_parse_header(const struct firmware *fw,
return -EBADMSG;
}
+ /*
+ * BSS addresses are relative to a firmware instance's base, not offsets
+ * into the file. Each bitmap bit describes one VPU page,
+ *
+ * The BSS span must fit within the instance and must not overlap the
+ * text mapping. Since bss_start is page-aligned, comparing it against
+ * text_end also prevents overlap with the final text page.
+ */
+ bss_end = (u64)bss_start +
+ (u64)bss_bitmap_size * MALI_VPU_PAGE_SIZE;
+
+ if ((bss_start & MALI_VPU_PAGE_MASK) ||
+ bss_start < text_end ||
+ bss_end > region_size) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "invalid firmware BSS layout: start=%#x, pages=%u\n",
+ bss_start, bss_bitmap_size);
+ return -EBADMSG;
+ }
+
+ /*
+ * The shared region must lie within the BSS span. Its start must be
+ * page-aligned, but its size need not be: the final page may be only
+ * partially used.
+ */
+ master_rw_end = (u64)master_rw_start + master_rw_size;
+
+ if ((master_rw_start & MALI_VPU_PAGE_MASK) ||
+ master_rw_start < bss_start ||
+ master_rw_end > bss_end) {
+ dev_err(vpu_fw_bin->cache->dev,
+ "invalid firmware shared region: start=%#x, size=%#x\n",
+ master_rw_start, master_rw_size);
+ return -EBADMSG;
+ }
+
/* Firmware version mismatch */
header_fw_ver_major = MALI_VPU_GET_MAJOR_VERSION(fw_version);
if (header_fw_ver_major != vpu_fw_bin->key.fw_ver_major) {
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
index ea7d35ef8ca60..181bf164b49a7 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware-cache.h
@@ -7,6 +7,7 @@
#define __MALI_VPU_FIRMWARE_CACHE_H__
#include <asm/byteorder.h>
+#include <linux/bits.h>
#include <linux/firmware.h>
#include <linux/kref.h>
#include <linux/list.h>
@@ -171,4 +172,9 @@ void mali_vpu_fw_cache_put(struct mali_vpu_fw_bin *bin);
bool mali_vpu_fw_cache_cancel_waiter(struct mali_vpu_fw_cache *cache,
struct mali_vpu_fw_event *event);
+static inline bool mali_vpu_test_bit_le32(u32 bit, const __le32 *addr)
+{
+ return !!(le32_to_cpu(addr[bit >> 5]) & BIT(bit & 0x1F));
+}
+
#endif /* __MALI_VPU_FIRMWARE_CACHE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
new file mode 100644
index 0000000000000..3c552b5fb2d9b
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -0,0 +1,383 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/string.h>
+
+#include "mali-vpu-firmware.h"
+#include "mali-vpu-firmware-cache.h"
+#include "mali-vpu-mmu.h"
+#include "mali-vpu-protocol-def-v3.h"
+
+#define MALI_VPU_FW_MAX_CORES 8
+
+static int mali_vpu_fw_get_region(enum mali_vpu_fw_region region,
+ mali_vpu_mmu_va *begin, mali_vpu_mmu_va *end)
+{
+ switch (region) {
+ case MALI_VPU_FW_REGION_MSG_HOST:
+ *begin = MVE_COMM_MSG_INQ_ADDR;
+ *end = MVE_COMM_MSG_INQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_MSG_MVE:
+ *begin = MVE_COMM_MSG_OUTQ_ADDR;
+ *end = MVE_COMM_MSG_OUTQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_BUF_IN_HOST:
+ *begin = MVE_COMM_BUF_INQ_ADDR;
+ *end = MVE_COMM_BUF_INQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_BUF_IN_MVE:
+ *begin = MVE_COMM_BUF_INRQ_ADDR;
+ *end = MVE_COMM_BUF_INRQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_BUF_OUT_HOST:
+ *begin = MVE_COMM_BUF_OUTQ_ADDR;
+ *end = MVE_COMM_BUF_OUTQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_BUF_OUT_MVE:
+ *begin = MVE_COMM_BUF_OUTRQ_ADDR;
+ *end = MVE_COMM_BUF_OUTRQ_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_RPC:
+ *begin = MVE_COMM_RPC_ADDR;
+ *end = MVE_COMM_RPC_ADDR + MALI_VPU_PAGE_SIZE;
+ break;
+ case MALI_VPU_FW_REGION_CORE_0:
+ *begin = MVE_MEM_REGION_FW_INSTANCE0_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE0_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_1:
+ *begin = MVE_MEM_REGION_FW_INSTANCE1_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE1_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_2:
+ *begin = MVE_MEM_REGION_FW_INSTANCE2_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE2_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_3:
+ *begin = MVE_MEM_REGION_FW_INSTANCE3_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE3_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_4:
+ *begin = MVE_MEM_REGION_FW_INSTANCE4_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE4_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_5:
+ *begin = MVE_MEM_REGION_FW_INSTANCE5_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE5_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_6:
+ *begin = MVE_MEM_REGION_FW_INSTANCE6_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE6_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_CORE_7:
+ *begin = MVE_MEM_REGION_FW_INSTANCE7_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FW_INSTANCE7_ADDR_END;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_fw_map_region(struct mali_vpu_fw *fw, struct mali_vpu_fw_proto_page *proto_page,
+ enum mali_vpu_fw_region region)
+{
+ mali_vpu_mmu_va begin, end;
+ dma_addr_t dma;
+ int ret;
+
+ /* Get virtual address where the message queue is to be mapped. */
+ ret = mali_vpu_fw_get_region(region, &begin, &end);
+ if (ret)
+ return ret;
+
+ /* Allocate backing memory and retain the original page. */
+ dma = mali_vpu_mmu_alloc_page(fw->dev, &proto_page->linux_page, DMA_BIDIRECTIONAL);
+ if (!proto_page->linux_page)
+ return -ENOMEM;
+
+ /* Memory map region. */
+ ret = mali_vpu_mmu_map_dma(fw->mmu, begin, dma, MALI_VPU_PAGE_SIZE,
+ MALI_VPU_ATTR_SHARED_RW, MALI_VPU_ACCESS_READ_WRITE);
+ if (ret) {
+ mali_vpu_mmu_free_page(fw->dev, dma, proto_page->linux_page, DMA_BIDIRECTIONAL);
+ proto_page->linux_page = NULL;
+ return ret;
+ }
+
+ proto_page->cpu = page_address(proto_page->linux_page);
+ proto_page->dma = dma;
+ proto_page->va = begin;
+
+ return 0;
+}
+
+static void mali_vpu_fw_unmap_region(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_proto_page *proto_page)
+{
+ if (!fw || !proto_page || !proto_page->linux_page)
+ return;
+
+ mali_vpu_mmu_unmap_va(fw->mmu, proto_page->va, MALI_VPU_PAGE_SIZE);
+ mali_vpu_mmu_free_page(fw->dev, proto_page->dma, proto_page->linux_page,
+ DMA_BIDIRECTIONAL);
+ memset(proto_page, 0, sizeof(*proto_page));
+}
+
+static void mali_vpu_fw_unmap_protocol(struct mali_vpu_fw *fw)
+{
+ if (!fw)
+ return;
+
+ mali_vpu_fw_unmap_region(fw, &fw->rpc);
+ mali_vpu_fw_unmap_region(fw, &fw->buf_out_mve);
+ mali_vpu_fw_unmap_region(fw, &fw->buf_out_host);
+ mali_vpu_fw_unmap_region(fw, &fw->buf_in_mve);
+ mali_vpu_fw_unmap_region(fw, &fw->buf_in_host);
+ mali_vpu_fw_unmap_region(fw, &fw->msg_mve);
+ mali_vpu_fw_unmap_region(fw, &fw->msg_host);
+}
+
+static int mali_vpu_fw_map_protocol(struct mali_vpu_fw *fw)
+{
+ int ret = 0;
+
+ if (!fw)
+ return -EINVAL;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->msg_host, MALI_VPU_FW_REGION_MSG_HOST);
+ if (ret)
+ return ret;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->msg_mve, MALI_VPU_FW_REGION_MSG_MVE);
+ if (ret)
+ goto err_unmap_protocol;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->buf_in_host, MALI_VPU_FW_REGION_BUF_IN_HOST);
+ if (ret)
+ goto err_unmap_protocol;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->buf_in_mve, MALI_VPU_FW_REGION_BUF_IN_MVE);
+ if (ret)
+ goto err_unmap_protocol;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->buf_out_host, MALI_VPU_FW_REGION_BUF_OUT_HOST);
+ if (ret)
+ goto err_unmap_protocol;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->buf_out_mve, MALI_VPU_FW_REGION_BUF_OUT_MVE);
+ if (ret)
+ goto err_unmap_protocol;
+
+ ret = mali_vpu_fw_map_region(fw, &fw->rpc, MALI_VPU_FW_REGION_RPC);
+ if (ret)
+ goto err_unmap_protocol;
+
+ return 0;
+
+err_unmap_protocol:
+ mali_vpu_fw_unmap_protocol(fw);
+
+ return ret;
+}
+
+static int mali_vpu_fw_map_one_core(struct mali_vpu_fw *fw,
+ const struct mali_vpu_fw_bin *fw_bin,
+ u32 core)
+{
+ /* Map pages for the text and BSS segments for one core. */
+ enum mali_vpu_fw_region region = MALI_VPU_FW_REGION_CORE_0 + core;
+ mali_vpu_mmu_va begin, end, va;
+ const struct mali_vpu_fw_header *hdr = fw_bin->header;
+ u32 bss_count = core * fw_bin->bss_page_count;
+ u32 shared_bss_count = 0;
+ u32 master_rw_start, master_rw_size, bss_bitmap_size, i;
+ bool is_shared;
+ int ret;
+
+ /*
+ * Get the base address where the pages for this core should be
+ * mapped.
+ */
+ ret = mali_vpu_fw_get_region(region, &begin, &end);
+ if (ret)
+ return ret;
+
+ /* Map text segment. */
+ ret = mali_vpu_mmu_map_pages(fw->mmu, begin + MALI_VPU_FW_TEXT_BASE_ADDR,
+ fw->text_pages, MALI_VPU_ATTR_PRIVATE,
+ MALI_VPU_ACCESS_EXECUTABLE);
+ if (ret)
+ return ret;
+
+ bss_bitmap_size = le32_to_cpu(hdr->bss_bitmap_size);
+ master_rw_start = le32_to_cpu(hdr->master_rw_start_address);
+ master_rw_size = le32_to_cpu(hdr->master_rw_size);
+
+ /* Map bss shared and private pages. */
+ va = le32_to_cpu(hdr->bss_start_address);
+ for (i = 0; i < bss_bitmap_size; i++) {
+ dma_addr_t dma;
+
+ is_shared = (va >= master_rw_start &&
+ va < (master_rw_start + master_rw_size));
+
+ if (is_shared) {
+ dma = fw->shared_bss_pages->pages[shared_bss_count++].dma;
+ } else if (mali_vpu_test_bit_le32(i, hdr->bss_bitmap)) {
+ dma = fw->bss_pages->pages[bss_count++].dma;
+ } else {
+ va += MALI_VPU_PAGE_SIZE;
+ continue;
+ }
+
+ ret = mali_vpu_mmu_map_dma(fw->mmu, begin + va, dma, MALI_VPU_PAGE_SIZE,
+ MALI_VPU_ATTR_PRIVATE, MALI_VPU_ACCESS_READ_WRITE);
+ if (ret)
+ return ret;
+
+ va += MALI_VPU_PAGE_SIZE;
+ }
+
+ return 0;
+}
+
+static void mali_vpu_fw_unmap_core_regions(struct mali_vpu_fw *fw)
+{
+ enum mali_vpu_fw_region region;
+ mali_vpu_mmu_va begin, end;
+ u32 core;
+ int ret;
+
+ if (!fw)
+ return;
+
+ for (core = 0; core < fw->ncores; core++) {
+ region = MALI_VPU_FW_REGION_CORE_0 + core;
+ ret = mali_vpu_fw_get_region(region, &begin, &end);
+ if (!ret)
+ mali_vpu_mmu_unmap_va(fw->mmu, begin, end - begin);
+ }
+
+ if (!IS_ERR_OR_NULL(fw->text_pages)) {
+ mali_vpu_mmu_free_pages(fw->text_pages);
+ fw->text_pages = NULL;
+ }
+
+ if (!IS_ERR_OR_NULL(fw->bss_pages)) {
+ mali_vpu_mmu_free_pages(fw->bss_pages);
+ fw->bss_pages = NULL;
+ }
+
+ if (!IS_ERR_OR_NULL(fw->shared_bss_pages)) {
+ mali_vpu_mmu_free_pages(fw->shared_bss_pages);
+ fw->shared_bss_pages = NULL;
+ }
+}
+
+static int mali_vpu_fw_map_core_regions(struct mali_vpu_fw *fw,
+ const struct mali_vpu_fw_bin *fw_bin)
+{
+ const struct mali_vpu_fw_header *hdr;
+ int ret = 0;
+ u32 core;
+
+ if (!fw || !fw_bin || !fw_bin->header || !fw_bin->fw)
+ return -EINVAL;
+
+ hdr = fw_bin->header;
+
+ fw->text_pages = mali_vpu_mmu_alloc_pages(fw->dev, fw_bin->text_page_count, 0);
+ if (IS_ERR(fw->text_pages)) {
+ ret = PTR_ERR(fw->text_pages);
+ goto err_unmap;
+ }
+
+ if (fw_bin->bss_page_count) {
+ fw->bss_pages = mali_vpu_mmu_alloc_pages(fw->dev,
+ fw_bin->bss_page_count * fw->ncores, 0);
+ if (IS_ERR(fw->bss_pages)) {
+ ret = PTR_ERR(fw->bss_pages);
+ goto err_unmap;
+ }
+ }
+
+ if (fw_bin->shared_bss_page_count) {
+ fw->shared_bss_pages = mali_vpu_mmu_alloc_pages(fw->dev,
+ fw_bin->shared_bss_page_count, 0);
+ if (IS_ERR(fw->shared_bss_pages)) {
+ ret = PTR_ERR(fw->shared_bss_pages);
+ goto err_unmap;
+ }
+ }
+
+ for (core = 0; core < fw->ncores; core++) {
+ ret = mali_vpu_fw_map_one_core(fw, fw_bin, core);
+ if (ret)
+ goto err_unmap;
+ }
+
+ ret = mali_vpu_mmu_write(fw->text_pages, fw_bin->fw->data,
+ le32_to_cpu(hdr->text_length));
+ if (ret)
+ goto err_unmap;
+
+ return 0;
+
+err_unmap:
+ mali_vpu_fw_unmap_core_regions(fw);
+ return ret;
+}
+
+int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
+ struct device *dev, struct mali_vpu_mmu *mmu, u32 ncores)
+{
+ int ret;
+
+ if (!fw || !fw_bin || !dev || !mmu)
+ return -EINVAL;
+
+ if (ncores == 0 || ncores > MALI_VPU_FW_MAX_CORES)
+ return -EINVAL;
+
+ memset(fw, 0, sizeof(*fw));
+ fw->dev = dev;
+ fw->mmu = mmu;
+ fw->ncores = ncores;
+
+ /* Initialise firmware core and protocol regions */
+ ret = mali_vpu_fw_map_core_regions(fw, fw_bin);
+ if (ret) {
+ dev_err(fw->dev, "failed to map firmware core regions: %d\n", ret);
+ return ret;
+ }
+
+ ret = mali_vpu_fw_map_protocol(fw);
+ if (ret) {
+ dev_err(fw->dev, "failed to map firmware protocol regions: %d\n", ret);
+ goto err_unmap_core;
+ }
+
+ return 0;
+
+err_unmap_core:
+ mali_vpu_fw_unmap_core_regions(fw);
+ return ret;
+}
+
+void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw)
+{
+ if (!fw)
+ return;
+
+ mali_vpu_fw_unmap_protocol(fw);
+ mali_vpu_fw_unmap_core_regions(fw);
+ memset(fw, 0, sizeof(*fw));
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
new file mode 100644
index 0000000000000..a40928db774f0
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_FIRMWARE_H__
+#define __MALI_VPU_FIRMWARE_H__
+
+#include <linux/types.h>
+
+#define MALI_VPU_FW_TEXT_BASE_ADDR 0x1000u
+
+struct device;
+struct mali_vpu_mmu;
+struct mali_vpu_fw_bin;
+struct mali_vpu_mmu_pages;
+
+/**
+ * enum mali_vpu_fw_region - Regions in the firmware virtual address space.
+ * @MALI_VPU_FW_REGION_MSG_HOST: Host-to-firmware message queue.
+ * @MALI_VPU_FW_REGION_MSG_MVE: Firmware-to-host message queue.
+ * @MALI_VPU_FW_REGION_BUF_IN_HOST: Host input-buffer queue.
+ * @MALI_VPU_FW_REGION_BUF_IN_MVE: Returned input-buffer queue.
+ * @MALI_VPU_FW_REGION_BUF_OUT_HOST: Host output-buffer queue.
+ * @MALI_VPU_FW_REGION_BUF_OUT_MVE: Returned output-buffer queue.
+ * @MALI_VPU_FW_REGION_RPC: Firmware RPC communication area.
+ * @MALI_VPU_FW_REGION_CORE_0: Firmware instance region for core 0.
+ * @MALI_VPU_FW_REGION_CORE_1: Firmware instance region for core 1.
+ * @MALI_VPU_FW_REGION_CORE_2: Firmware instance region for core 2.
+ * @MALI_VPU_FW_REGION_CORE_3: Firmware instance region for core 3.
+ * @MALI_VPU_FW_REGION_CORE_4: Firmware instance region for core 4.
+ * @MALI_VPU_FW_REGION_CORE_5: Firmware instance region for core 5.
+ * @MALI_VPU_FW_REGION_CORE_6: Firmware instance region for core 6.
+ * @MALI_VPU_FW_REGION_CORE_7: Firmware instance region for core 7.
+ */
+enum mali_vpu_fw_region {
+ MALI_VPU_FW_REGION_MSG_HOST,
+ MALI_VPU_FW_REGION_MSG_MVE,
+ MALI_VPU_FW_REGION_BUF_IN_HOST,
+ MALI_VPU_FW_REGION_BUF_IN_MVE,
+ MALI_VPU_FW_REGION_BUF_OUT_HOST,
+ MALI_VPU_FW_REGION_BUF_OUT_MVE,
+ MALI_VPU_FW_REGION_RPC,
+
+ /* Firmware instance regions (one per core) */
+ MALI_VPU_FW_REGION_CORE_0,
+ MALI_VPU_FW_REGION_CORE_1,
+ MALI_VPU_FW_REGION_CORE_2,
+ MALI_VPU_FW_REGION_CORE_3,
+ MALI_VPU_FW_REGION_CORE_4,
+ MALI_VPU_FW_REGION_CORE_5,
+ MALI_VPU_FW_REGION_CORE_6,
+ MALI_VPU_FW_REGION_CORE_7,
+};
+
+/**
+ * struct mali_vpu_fw_proto_page - Backing memory and mappings for a protocol page.
+ * @linux_page: Original allocated page backing the queue/RPC area.
+ * @cpu: Kernel CPU mapping used by host-side queue/RPC access.
+ * @dma: DMA address of the queue/RPC page, mapped into the VPU address space.
+ * @va: Firmware-visible virtual address used by firmware-side queue/RPC access.
+ */
+struct mali_vpu_fw_proto_page {
+ struct page *linux_page;
+ void *cpu;
+ dma_addr_t dma;
+ u32 va;
+};
+
+/**
+ * struct mali_vpu_fw - Firmware object
+ * @dev: Pointer to device.
+ * @mmu: Pointer to MMU object.
+ * @msg_host: Host message queue.
+ * @msg_mve: MVE message queue.
+ * @buf_in_host: Input buffer queue. Host enqueue filled buffers.
+ * @buf_in_mve: Input buffer queue. MVE return empty buffers.
+ * @buf_out_host: Output buffer queue. Host enqueue empty buffers.
+ * @buf_out_mve: Output buffer queue. MVE return filled buffers.
+ * @rpc: RPC communication area.
+ * @text_pages: Firmware text pages shared by all mapped core instances.
+ * @bss_pages: Private BSS pages allocated separately for each core instance.
+ * @shared_bss_pages: Shared BSS pages mapped into every core instance.
+ * @ncores: Number of firmware core-instance regions constructed.
+ */
+struct mali_vpu_fw {
+ struct device *dev;
+ struct mali_vpu_mmu *mmu;
+
+ struct mali_vpu_fw_proto_page msg_host;
+ struct mali_vpu_fw_proto_page msg_mve;
+ struct mali_vpu_fw_proto_page buf_in_host;
+ struct mali_vpu_fw_proto_page buf_in_mve;
+ struct mali_vpu_fw_proto_page buf_out_host;
+ struct mali_vpu_fw_proto_page buf_out_mve;
+ struct mali_vpu_fw_proto_page rpc;
+
+ struct mali_vpu_mmu_pages *text_pages;
+ struct mali_vpu_mmu_pages *bss_pages;
+ struct mali_vpu_mmu_pages *shared_bss_pages;
+ u32 ncores;
+};
+
+int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
+ struct device *dev, struct mali_vpu_mmu *mmu, u32 ncores);
+void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw);
+
+#endif /* __MALI_VPU_FIRMWARE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index b87725d7d4cab..123ff4d8fc937 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -26,12 +26,6 @@
#include "mali-vpu-variant.h"
#include "mali-vpu-vidioc.h"
-/* VPU register access helpers */
-static u32 mali_vpu_read_reg(void __iomem *regs_base, u32 off)
-{
- return readl(regs_base + off);
-}
-
static void mali_vpu_read_version(struct mali_vpu_dev *vpu_dev, u32 *product_id, u32 *rev,
u32 *patch)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index f3a61731191bc..b6d46f91287c4 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -28,6 +28,100 @@
#define MALI_VPU_HOST_PROTOCOL_MAJOR 3
#define MALI_VPU_HOST_PROTOCOL_MINOR 1
+/*****************************************************************************
+ *
+ * Virtual memory regions
+ *
+ * ..._ADDR_BEGIN gives the starting virtual address of the region,
+ * and ..._ADDR_END the (non-inclusive) ending address, such that
+ * the size of the region is obtained with the subtraction
+ * (..._ADDR_END - ..._ADDR_BEGIN).
+ *
+ ****************************************************************************/
+
+/* Memory region for first firmware instance */
+#define MVE_MEM_REGION_FW_INSTANCE0_ADDR_BEGIN (0x00000000u)
+#define MVE_MEM_REGION_FW_INSTANCE0_ADDR_END (0x000FFFFFu + 1)
+
+/* Memory regions for other firmware instances */
+#define MVE_MEM_REGION_FW_INSTANCE1_ADDR_BEGIN (0x01000000u)
+#define MVE_MEM_REGION_FW_INSTANCE1_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE1_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE2_ADDR_BEGIN (0x02000000u)
+#define MVE_MEM_REGION_FW_INSTANCE2_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE2_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE3_ADDR_BEGIN (0x03000000u)
+#define MVE_MEM_REGION_FW_INSTANCE3_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE3_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE4_ADDR_BEGIN (0x04000000u)
+#define MVE_MEM_REGION_FW_INSTANCE4_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE4_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE5_ADDR_BEGIN (0x05000000u)
+#define MVE_MEM_REGION_FW_INSTANCE5_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE5_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE6_ADDR_BEGIN (0x06000000u)
+#define MVE_MEM_REGION_FW_INSTANCE6_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE6_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+#define MVE_MEM_REGION_FW_INSTANCE7_ADDR_BEGIN (0x07000000u)
+#define MVE_MEM_REGION_FW_INSTANCE7_ADDR_END \
+ (MVE_MEM_REGION_FW_INSTANCE7_ADDR_BEGIN + MVE_MEM_REGION_FW_INSTANCE0_ADDR_END)
+
+/*
+ * Areas for communication between host and MVE are placed in the interval
+ * 0x10079000 - 0x1007FFFF, see special defines further down.
+ */
+
+/*****************************************************************************
+ *
+ * Communication queues between HOST/DRIVER and MVE
+ *
+ ****************************************************************************/
+
+/*
+ * Address for queue for messages in to MVE,
+ * one struct mve_comm_area_host located here
+ */
+#define MVE_COMM_MSG_INQ_ADDR (0x10079000u)
+
+/*
+ * Address for queue for messages out from MVE,
+ * one struct mve_comm_area_mve located here
+ */
+#define MVE_COMM_MSG_OUTQ_ADDR (0x1007A000u)
+
+/*
+ * Address for queue for input buffers in to MVE,
+ * one struct mve_comm_area_host located here
+ */
+#define MVE_COMM_BUF_INQ_ADDR (0x1007B000u)
+
+/*
+ * Address for queue for input buffers returned from MVE,
+ * one struct mve_comm_area_mve located here
+ */
+#define MVE_COMM_BUF_INRQ_ADDR (0x1007C000u)
+
+/*
+ * Address for queue for output buffers in to MVE,
+ * one struct mve_comm_area_host located here
+ */
+#define MVE_COMM_BUF_OUTQ_ADDR (0x1007D000u)
+
+/*
+ * Address for queue for output buffers returned from MVE,
+ * one struct mve_comm_area_mve located here
+ */
+#define MVE_COMM_BUF_OUTRQ_ADDR (0x1007E000u)
+
+/* One struct mve_rpc_communication_area located here */
+#define MVE_COMM_RPC_ADDR (0x1007F000u)
+
#define MVE_CHROMA_FORMAT_420 (0x1)
/* format bitfield: */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
index 471a1f2412a7a..b7883b1ea47f3 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
@@ -7,6 +7,7 @@
#define __MALI_VPU_REGISTERS_H__
#include <linux/bits.h>
+#include <linux/io.h>
/*
* Mali VPU register offsets and bitfield definitions.
@@ -14,6 +15,7 @@
* Offsets are relative to the VPU MMIO base.
*/
#define MALI_VPU_REG_VERSION 0x00000
+#define MALI_VPU_REG_NCORES 0x00008
/* Version register fields */
#define MALI_VPU_VERSION_PRODUCT_ID_MASK GENMASK(31, 16)
@@ -21,4 +23,9 @@
#define MALI_VPU_VERSION_PATCH_MASK GENMASK(7, 0)
#define MALI_VPU_PID_SUFFIX_MASK GENMASK(15, 12)
+static inline u32 mali_vpu_read_reg(void __iomem *regs_base, u32 off)
+{
+ return readl(regs_base + off);
+}
+
#endif /* __MALI_VPU_REGISTERS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index b9a3960bca4c7..c9a2c683b8dab 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -13,8 +13,10 @@
#include "mali-vpu-session.h"
#include "mali-vpu-controls.h"
#include "mali-vpu-dec.h"
+#include "mali-vpu-firmware.h"
#include "mali-vpu-firmware-cache.h"
#include "mali-vpu-formats.h"
+#include "mali-vpu-registers.h"
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
@@ -136,6 +138,7 @@ static void mali_vpu_session_release_fw_bin(struct mali_vpu_session *sess)
lockdep_assert_held(&sess->vq_lock);
if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY) {
+ mali_vpu_fw_cleanup(&sess->fw);
mali_vpu_fw_cache_put(sess->fw_bin);
sess->fw_bin = NULL;
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
@@ -208,6 +211,8 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
bool same_thread)
{
struct mali_vpu_session *sess = arg;
+ u32 ncores;
+ int ret;
if (!same_thread)
mutex_lock(&sess->vq_lock);
@@ -228,6 +233,21 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
sess->fw_bin = vpu_fw_bin;
+ /*
+ * Get total number of cores on VPU element.
+ * This will be scheduler decision in subsequent patches.
+ */
+ ncores = mali_vpu_read_reg(sess->vpu_dev->regs_base, MALI_VPU_REG_NCORES);
+
+ ret = mali_vpu_fw_construct(&sess->fw, sess->fw_bin, sess->vpu_dev->dev,
+ &sess->mmu, ncores);
+ if (ret) {
+ dev_err(sess->vpu_dev->dev,
+ "failed to initialise firmware: %d\n", ret);
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ goto err_cache_put;
+ }
+
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_READY;
goto out_complete;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 0b810d7d6a036..ecd048e3e5c01 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -13,6 +13,7 @@
#include <media/v4l2-fh.h>
#include "mali-vpu-device.h"
+#include "mali-vpu-firmware.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-mmu.h"
@@ -41,8 +42,9 @@ struct mali_vpu_port {
* loading to complete. READY and LOADING requests are reused.
*
* A cache hit may invoke the callback synchronously. An asynchronous callback
- * discards its result if neither port requests streaming. A successfully
- * loaded binary is retained until session cleanup.
+ * discards its result if neither port requests streaming. Successful callbacks
+ * construct the firmware address space before making the binary READY. The
+ * session retains its binary reference until cleanup.
*
* Session cleanup waits for an in-flight callback before releasing the
* session-owned binary reference and callback event. On timeout, it cancels
@@ -71,6 +73,7 @@ enum mali_vpu_session_fw_bin_state {
* @ctrl_handler: V4L2 control handler bound to this session.
* @fh: V4L2 file handle for this session.
* @m2m_ctx: V4L2 mem2mem context for this session.
+ * @fw: Per-session firmware address space object.
* @fw_bin: Pointer to firmware binary loaded for this session.
* @fw_event: Firmware binary load event callback structure for this session.
* @fw_bin_state: Session firmware binary status.
@@ -94,6 +97,7 @@ struct mali_vpu_session {
struct v4l2_fh fh;
struct v4l2_m2m_ctx *m2m_ctx;
+ struct mali_vpu_fw fw;
struct mali_vpu_fw_bin *fw_bin;
struct mali_vpu_fw_event *fw_event;
enum mali_vpu_session_fw_bin_state fw_bin_state;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 12/28] media: mali-vpu: prepare firmware address space
2026-11-19 15:37 ` [RFC PATCH 12/28] media: mali-vpu: prepare firmware address space Charvi Mehta
@ 2026-11-19 15:57 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:57 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] media: mali-vpu: rounded-up text_pages mapping in
mali_vpu_fw_map_one_core()
- [Medium] media: mali-vpu: unmapped core walk in
mali_vpu_fw_unmap_core_regions()
--
commit df911b80c99884b502e4cb4ba2acac413ae6c21d
Author: Charvi Mehta <charvi.mehta@arm.com>
media: mali-vpu: prepare firmware address space
This commit adds per-session firmware address space management, allocating
and mapping firmware text, BSS, shared BSS, and host communication protocol
pages across the hardware cores.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
> new file mode 100644
> index 0000000000000..3c552b5fb2d9b
> --- /dev/null
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
[ ... ]
> +static int mali_vpu_fw_map_one_core(struct mali_vpu_fw *fw,
> + const struct mali_vpu_fw_bin *fw_bin,
> + u32 core)
> +{
> + /* Map pages for the text and BSS segments for one core. */
> + enum mali_vpu_fw_region region = MALI_VPU_FW_REGION_CORE_0 + core;
> + mali_vpu_mmu_va begin, end, va;
> + const struct mali_vpu_fw_header *hdr = fw_bin->header;
> + u32 bss_count = core * fw_bin->bss_page_count;
> + u32 shared_bss_count = 0;
> + u32 master_rw_start, master_rw_size, bss_bitmap_size, i;
> + bool is_shared;
> + int ret;
> +
> + /*
> + * Get the base address where the pages for this core should be
> + * mapped.
> + */
> + ret = mali_vpu_fw_get_region(region, &begin, &end);
> + if (ret)
> + return ret;
> +
> + /* Map text segment. */
> + ret = mali_vpu_mmu_map_pages(fw->mmu, begin + MALI_VPU_FW_TEXT_BASE_ADDR,
> + fw->text_pages, MALI_VPU_ATTR_PRIVATE,
> + MALI_VPU_ACCESS_EXECUTABLE);
> + if (ret)
> + return ret;
[Severity: Medium]
Can mali_vpu_mmu_map_pages() map trailing VPU subpages of fw->text_pages
into the BSS range or past the end of the core region on 16K/64K PAGE_SIZE
kernels?
mali_vpu_mmu_alloc_pages() rounds fw->text_pages->count up to a multiple of
MALI_VPU_PAGES_PER_LINUX_PAGE, whereas mali_vpu_fw_parse_header() validates
bss_start >= text_end and text_end <= region_size at 4 KB VPU page
granularity:
drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c:mali_vpu_mmu_alloc_pages() {
...
count = roundup(count, MALI_VPU_PAGES_PER_LINUX_PAGE);
capacity = roundup(capacity, MALI_VPU_PAGES_PER_LINUX_PAGE);
...
}
When fw_bin->text_page_count is not aligned to
MALI_VPU_PAGES_PER_LINUX_PAGE and the 4 KB-aligned bss_start falls within
the trailing subpages of the last Linux page, mali_vpu_mmu_map_dma() below
fails with -EAGAIN because the PTE is already occupied. If
fw_bin->text_page_count lies in the final MALI_VPU_PAGES_PER_LINUX_PAGE - 1
pages of the 256-page core region, the rounded-up mapping at
begin + MALI_VPU_FW_TEXT_BASE_ADDR also spills into begin + 0x100000
(outside [begin, end)), leaving dangling PTEs on teardown for cores
0 .. ncores - 2 because mali_vpu_fw_unmap_core_regions() only unmaps
[begin, end) and mali_vpu_mmu_free_pages() only unmaps the single
fw->text_pages->va recorded by the last core.
> +
> + bss_bitmap_size = le32_to_cpu(hdr->bss_bitmap_size);
> + master_rw_start = le32_to_cpu(hdr->master_rw_start_address);
> + master_rw_size = le32_to_cpu(hdr->master_rw_size);
> +
> + /* Map bss shared and private pages. */
> + va = le32_to_cpu(hdr->bss_start_address);
> + for (i = 0; i < bss_bitmap_size; i++) {
> + dma_addr_t dma;
> +
> + is_shared = (va >= master_rw_start &&
> + va < (master_rw_start + master_rw_size));
> +
> + if (is_shared) {
> + dma = fw->shared_bss_pages->pages[shared_bss_count++].dma;
> + } else if (mali_vpu_test_bit_le32(i, hdr->bss_bitmap)) {
> + dma = fw->bss_pages->pages[bss_count++].dma;
> + } else {
> + va += MALI_VPU_PAGE_SIZE;
> + continue;
> + }
> +
> + ret = mali_vpu_mmu_map_dma(fw->mmu, begin + va, dma, MALI_VPU_PAGE_SIZE,
> + MALI_VPU_ATTR_PRIVATE, MALI_VPU_ACCESS_READ_WRITE);
> + if (ret)
> + return ret;
> +
> + va += MALI_VPU_PAGE_SIZE;
> + }
> +
> + return 0;
> +}
> +
> +static void mali_vpu_fw_unmap_core_regions(struct mali_vpu_fw *fw)
> +{
> + enum mali_vpu_fw_region region;
> + mali_vpu_mmu_va begin, end;
> + u32 core;
> + int ret;
> +
> + if (!fw)
> + return;
> +
> + for (core = 0; core < fw->ncores; core++) {
> + region = MALI_VPU_FW_REGION_CORE_0 + core;
> + ret = mali_vpu_fw_get_region(region, &begin, &end);
> + if (!ret)
> + mali_vpu_mmu_unmap_va(fw->mmu, begin, end - begin);
> + }
> +
> + if (!IS_ERR_OR_NULL(fw->text_pages)) {
> + mali_vpu_mmu_free_pages(fw->text_pages);
> + fw->text_pages = NULL;
> + }
> +
> + if (!IS_ERR_OR_NULL(fw->bss_pages)) {
> + mali_vpu_mmu_free_pages(fw->bss_pages);
> + fw->bss_pages = NULL;
> + }
> +
> + if (!IS_ERR_OR_NULL(fw->shared_bss_pages)) {
> + mali_vpu_mmu_free_pages(fw->shared_bss_pages);
> + fw->shared_bss_pages = NULL;
> + }
> +}
[Severity: Medium]
Does calling mali_vpu_fw_unmap_core_regions() from the err_unmap path in
mali_vpu_fw_map_core_regions() walk unallocated L2 page tables for cores
that were never mapped?
mali_vpu_fw_construct() sets fw->ncores before calling
mali_vpu_fw_map_core_regions(), and each core's L2 page table is only
allocated on demand when mali_vpu_fw_map_one_core() maps its pages. If page
allocation or per-core mapping fails before all fw->ncores cores are mapped,
mali_vpu_mmu_unmap_va() walks 256 pages per unmapped core and
mali_vpu_page_table_walk() logs a dev_err() on every 4 KB page:
drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c:mali_vpu_page_table_walk() {
...
if (!l2->linux_page) {
if (!alloc) {
dev_err(mmu->dev, "missing L2 page in PTW. va=0x%x", va);
return ERR_PTR(-EFAULT);
}
...
}
Also, when mali_vpu_mmu_alloc_pages() fails, the !IS_ERR_OR_NULL() checks
above leave the ERR_PTR value stored in fw->text_pages, fw->bss_pages, or
fw->shared_bss_pages instead of clearing it to NULL.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=12
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 13/28] media: mali-vpu: add firmware message transport
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (11 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 12/28] media: mali-vpu: prepare firmware address space Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 14/28] media: mali-vpu: route decoder controls to firmware Charvi Mehta
` (14 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add transport support for exchanging messages with the firmware over
the shared circular message queues.
Provide helpers for submitting job, ping and state-change requests and
for parsing firmware state-change, error, job, switch and pong responses.
Validate queue positions and message boundaries, and synchronize
the shared pages around CPU and device access.
This provides the common message transport needed by the subsequent
firmware control and scheduler support.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 370 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 83 ++++
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 145 +++++++
3 files changed, 598 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 3c552b5fb2d9b..4aead9a062260 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -4,6 +4,7 @@
*/
#include <linux/device.h>
+#include <linux/dma-mapping.h>
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/string.h>
@@ -381,3 +382,372 @@ void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw)
mali_vpu_fw_unmap_core_regions(fw);
memset(fw, 0, sizeof(*fw));
}
+
+/*
+ * Read arbitrary byte count from circular u32 queue payload area.
+ * Return new read position in the queue.
+ */
+static u16 mali_vpu_fw_read32n(u32 *src, u16 offset, u32 *dst, size_t size)
+{
+ u32 v;
+
+ for (; size >= sizeof(u32); size -= sizeof(u32)) {
+ *dst++ = READ_ONCE(src[offset]);
+ offset = (offset + 1) % MVE_COMM_QUEUE_SIZE_IN_WORDS;
+ }
+
+ if (size) {
+ v = READ_ONCE(src[offset]);
+ memcpy(dst, &v, size);
+ offset = (offset + 1) % MVE_COMM_QUEUE_SIZE_IN_WORDS;
+ }
+
+ return offset;
+}
+
+static int mali_vpu_fw_read_msg(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_proto_page *host_page,
+ struct mali_vpu_fw_proto_page *mve_page,
+ u16 *code, void *payload, size_t *size)
+{
+ struct mve_comm_area_host *host = host_page->cpu;
+ struct mve_comm_area_mve *mve = mve_page->cpu;
+ struct mve_msg_header hdr;
+ u16 rpos, out_wpos;
+ ssize_t capacity;
+
+ if (!host || !mve)
+ return -EINVAL;
+
+ dma_sync_single_for_cpu(fw->dev, mve_page->dma, MALI_VPU_PAGE_SIZE,
+ DMA_FROM_DEVICE);
+
+ rpos = READ_ONCE(host->out_rpos);
+ out_wpos = READ_ONCE(mve->out_wpos);
+
+ if (rpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS ||
+ out_wpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS)
+ return -EPROTO;
+
+ /* Calculate how much data that is available in the buffer. */
+ capacity = out_wpos - rpos;
+ if (capacity < 0)
+ capacity += MVE_COMM_QUEUE_SIZE_IN_WORDS;
+
+ if (capacity <= 0)
+ return 0;
+
+ /* Firmware publishes out_wpos after writing the message. */
+ dma_rmb();
+
+ rpos = mali_vpu_fw_read32n(mve->out_data, rpos, (u32 *)&hdr, sizeof(hdr));
+
+ /* Make sure there is enough space for both header and message. */
+ capacity -= DIV_ROUND_UP(sizeof(hdr) + hdr.size, sizeof(u32));
+ if (capacity < 0) {
+ dev_err(fw->dev,
+ "fw msg larger than capacity code=%u size=%u, wpos=%u, rpos=%u\n",
+ hdr.code, hdr.size, out_wpos, rpos);
+ return -EPROTO;
+ }
+
+ if (hdr.size > *size)
+ return -EMSGSIZE;
+
+ rpos = mali_vpu_fw_read32n(mve->out_data, rpos,
+ (u32 *)payload, hdr.size);
+
+ /*
+ * Finish reading the message before publishing updated consumed queue
+ * positions to the firmware.
+ */
+ dma_mb();
+ WRITE_ONCE(host->out_rpos, rpos);
+
+ dma_sync_single_for_device(fw->dev,
+ host_page->dma + offsetof(struct mve_comm_area_host, out_rpos),
+ sizeof(host->out_rpos), DMA_TO_DEVICE);
+
+ *code = hdr.code;
+ *size = hdr.size;
+
+ return 1;
+}
+
+/*
+ * Write arbitrary byte count to circular u32 queue payload area.
+ * Return new write position in the queue.
+ */
+static u16 mali_vpu_fw_write32n(u32 *dst, u16 offset, const u32 *src, size_t size)
+{
+ u32 v;
+
+ for (; size >= sizeof(u32); size -= sizeof(u32)) {
+ WRITE_ONCE(dst[offset], *src++);
+ offset = (offset + 1) % MVE_COMM_QUEUE_SIZE_IN_WORDS;
+ }
+
+ if (size) {
+ v = 0;
+ memcpy(&v, src, size);
+ WRITE_ONCE(dst[offset], v);
+ offset = (offset + 1) % MVE_COMM_QUEUE_SIZE_IN_WORDS;
+ }
+
+ return offset;
+}
+
+static int mali_vpu_fw_write_msg(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_proto_page *host_page,
+ struct mali_vpu_fw_proto_page *mve_page,
+ u16 code, const void *payload, size_t size)
+{
+ struct mve_comm_area_host *host = host_page->cpu;
+ struct mve_comm_area_mve *mve = mve_page->cpu;
+ struct mve_msg_header hdr = { .code = code, .size = size};
+ u16 wpos, in_rpos;
+ int capacity;
+
+ if (!host || !mve)
+ return -EINVAL;
+
+ dma_sync_single_for_cpu(fw->dev,
+ mve_page->dma + offsetof(struct mve_comm_area_mve, in_rpos),
+ sizeof(mve->in_rpos), DMA_FROM_DEVICE);
+
+ wpos = READ_ONCE(host->in_wpos);
+ in_rpos = READ_ONCE(mve->in_rpos);
+
+ if (wpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS || in_rpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS)
+ return -EPROTO;
+
+ /* Calculate how much space that is available in the buffer. */
+ capacity = in_rpos - wpos;
+ if (capacity <= 0)
+ capacity += MVE_COMM_QUEUE_SIZE_IN_WORDS;
+
+ /* Make sure there is enough space for both header and message. */
+ capacity -= DIV_ROUND_UP(sizeof(hdr) + size, sizeof(u32));
+ if (capacity <= 0)
+ return -ENOMEM;
+
+ wpos = mali_vpu_fw_write32n(host->in_data, wpos, (const u32 *)&hdr, sizeof(hdr));
+ wpos = mali_vpu_fw_write32n(host->in_data, wpos, (const u32 *)payload, size);
+
+ /*
+ * Publish the producer position only after the complete message has
+ * been written.
+ */
+ dma_wmb();
+ WRITE_ONCE(host->in_wpos, wpos);
+
+ dma_sync_single_for_device(fw->dev, host_page->dma, sizeof(*host), DMA_TO_DEVICE);
+
+ return 0;
+}
+
+static size_t mali_vpu_get_capacity_bytes(int rpos, int wpos)
+{
+ ssize_t capacity;
+
+ capacity = wpos - rpos;
+ if (capacity < 0)
+ capacity += MVE_COMM_QUEUE_SIZE_IN_WORDS;
+
+ return capacity * sizeof(u32);
+}
+
+static void mali_vpu_fw_print_debug(struct mali_vpu_fw *fw)
+{
+ struct mve_comm_area_host *host = fw->msg_host.cpu;
+ struct mve_comm_area_mve *mve = fw->msg_mve.cpu;
+ struct mve_msg_header hdr;
+ u16 rpos, wpos;
+ size_t capacity;
+
+ dma_sync_single_for_cpu(fw->dev, fw->msg_mve.dma, MALI_VPU_PAGE_SIZE,
+ DMA_FROM_DEVICE);
+
+ dev_dbg(fw->dev,
+ "dump message q. host={out_rpos=%u, in_wpos=%u}, mve={out_wpos=%u, in_rpos=%u}\n",
+ host->out_rpos, host->in_wpos, mve->out_wpos, mve->in_rpos);
+
+ rpos = READ_ONCE(host->out_rpos);
+ wpos = READ_ONCE(mve->out_wpos);
+
+ if (rpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS ||
+ wpos >= MVE_COMM_QUEUE_SIZE_IN_WORDS)
+ return;
+
+ /* Firmware publishes out_wpos after writing the message. */
+ dma_rmb();
+
+ while ((capacity = mali_vpu_get_capacity_bytes(rpos, wpos)) >= sizeof(hdr)) {
+ unsigned int pos;
+
+ pos = mali_vpu_fw_read32n(mve->out_data, rpos, (u32 *)&hdr, sizeof(hdr));
+
+ dev_dbg(fw->dev,
+ "queue={rpos=%u, wpos=%u, capacity=%zu}, msg={code=%u, size=%u}\n",
+ rpos, wpos, capacity, hdr.code, hdr.size);
+
+ capacity = mali_vpu_get_capacity_bytes(pos, wpos);
+ if (hdr.size > capacity) {
+ dev_dbg(fw->dev,
+ "size is larger than capacity. capacity=%zu, size=%u.\n",
+ capacity, hdr.size);
+ return;
+ }
+
+ rpos = (pos + DIV_ROUND_UP(hdr.size, sizeof(u32))) % MVE_COMM_QUEUE_SIZE_IN_WORDS;
+ }
+}
+
+int mali_vpu_fw_send_job(struct mali_vpu_fw *fw, u16 cores, u16 frames)
+{
+ struct mve_request_job job = {
+ .cores = cores,
+ .frames = frames,
+ .flags = 0,
+ };
+
+ return mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve,
+ MVE_REQUEST_CODE_JOB, &job, sizeof(job));
+}
+
+int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw)
+{
+ return mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve,
+ MVE_REQUEST_CODE_PING, NULL, 0);
+}
+
+int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state)
+{
+ u16 code;
+
+ switch (state) {
+ case MALI_VPU_FW_STATE_STOPPED:
+ code = MVE_REQUEST_CODE_STOP;
+ break;
+ case MALI_VPU_FW_STATE_RUNNING:
+ code = MVE_REQUEST_CODE_GO;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve, code, NULL, 0);
+}
+
+int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
+{
+ u16 code;
+ union {
+ struct mve_response_state_change state_change;
+ struct mve_response_error error;
+ } fw_msg = {};
+ size_t size = sizeof(fw_msg);
+ int ret;
+
+ if (!fw || !msg)
+ return -EINVAL;
+
+ ret = mali_vpu_fw_read_msg(fw, &fw->msg_host, &fw->msg_mve, &code,
+ &fw_msg, &size);
+ if (ret <= 0)
+ return ret; /* 0 = no msg, <0 = error */
+
+ memset(msg, 0, sizeof(*msg));
+
+ switch (code) {
+ case MVE_RESPONSE_CODE_SWITCHED_IN:
+ msg->code = MALI_VPU_FW_CODE_SWITCH_IN;
+ break;
+ case MVE_RESPONSE_CODE_SWITCHED_OUT:
+ msg->code = MALI_VPU_FW_CODE_SWITCH_OUT;
+ break;
+ case MVE_RESPONSE_CODE_JOB_DEQUEUED:
+ msg->code = MALI_VPU_FW_CODE_JOB;
+ break;
+ case MVE_RESPONSE_CODE_PONG:
+ msg->code = MALI_VPU_FW_CODE_PONG;
+ break;
+ case MVE_RESPONSE_CODE_IDLE:
+ msg->code = MALI_VPU_FW_CODE_IDLE;
+ break;
+ case MVE_RESPONSE_CODE_STATE_CHANGE: {
+ if (size != sizeof(fw_msg.state_change))
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_STATE_CHANGE;
+
+ switch (fw_msg.state_change.new_state) {
+ case MVE_STATE_STOPPED:
+ msg->data.state = MALI_VPU_FW_STATE_STOPPED;
+ break;
+ case MVE_STATE_RUNNING:
+ msg->data.state = MALI_VPU_FW_STATE_RUNNING;
+ break;
+ default:
+ return -EPROTO;
+ }
+ break;
+ }
+ case MVE_RESPONSE_CODE_ERROR: {
+ size_t message_offset = offsetof(struct mve_response_error, message);
+ size_t message_size;
+
+ if (size < message_offset + 1 || size > sizeof(fw_msg.error))
+ return -EPROTO;
+
+ message_size = size - message_offset;
+ if (fw_msg.error.message[message_size - 1] != '\0')
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_ERROR;
+
+ switch (fw_msg.error.error_code) {
+ case MVE_ERROR_ABORT:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_ABORT;
+ break;
+ case MVE_ERROR_OUT_OF_MEMORY:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_OUT_OF_MEMORY;
+ break;
+ case MVE_ERROR_ASSERT:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_ASSERT;
+ break;
+ case MVE_ERROR_UNSUPPORTED:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_UNSUPPORTED;
+ break;
+ case MVE_ERROR_INVALID_BUFFER:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_INVALID_BUFFER;
+ break;
+ case MVE_ERROR_INVALID_STATE:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_INVALID_STATE;
+ break;
+ case MVE_ERROR_WATCHDOG:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_WATCHDOG;
+ break;
+ default:
+ msg->data.error.error_code = MALI_VPU_FW_ERROR_UNKNOWN;
+ dev_warn(fw->dev, "unsupported fw error code. code=%u.\n",
+ fw_msg.error.error_code);
+ break;
+ }
+
+ strscpy(msg->data.error.message, fw_msg.error.message,
+ min(sizeof(msg->data.error.message), sizeof(fw_msg.error.message)));
+
+ break;
+ }
+ default:
+ dev_warn(fw->dev,
+ "unknown fw message code. code=%u, size=%zu.\n",
+ code, size);
+ msg->code = MALI_VPU_FW_CODE_UNKNOWN;
+ mali_vpu_fw_print_debug(fw);
+ break;
+ }
+
+ return ret;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index a40928db774f0..f26fd78101661 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -15,6 +15,85 @@ struct mali_vpu_mmu;
struct mali_vpu_fw_bin;
struct mali_vpu_mmu_pages;
+/**
+ * enum mali_vpu_fw_state - Firmware execution state.
+ * @MALI_VPU_FW_STATE_STOPPED: Firmware is STOPPED.
+ * @MALI_VPU_FW_STATE_RUNNING: Firmware is RUNNING.
+ */
+enum mali_vpu_fw_state {
+ MALI_VPU_FW_STATE_STOPPED,
+ MALI_VPU_FW_STATE_RUNNING
+};
+
+/**
+ * struct mali_vpu_fw_job - Job request.
+ * @cores: Number for cores to use.
+ * @frames: Number of frames to process before job is switched out.
+ */
+struct mali_vpu_fw_job {
+ unsigned int cores;
+ unsigned int frames;
+};
+
+/**
+ * struct mali_vpu_fw_error - Firmware error message.
+ * @error_code: What kind of error that was reported.
+ * @message: Error message string.
+ */
+struct mali_vpu_fw_error {
+ enum {
+ MALI_VPU_FW_ERROR_UNKNOWN,
+ MALI_VPU_FW_ERROR_ABORT,
+ MALI_VPU_FW_ERROR_OUT_OF_MEMORY,
+ MALI_VPU_FW_ERROR_ASSERT,
+ MALI_VPU_FW_ERROR_UNSUPPORTED,
+ MALI_VPU_FW_ERROR_INVALID_BUFFER,
+ MALI_VPU_FW_ERROR_INVALID_STATE,
+ MALI_VPU_FW_ERROR_WATCHDOG
+ } error_code;
+ char message[128];
+};
+
+/**
+ * enum mali_vpu_fw_code - Driver representation of firmware messages.
+ * @MALI_VPU_FW_CODE_ERROR: Firmware reported fatal error.
+ * @MALI_VPU_FW_CODE_JOB: Firmware dequeued a job.
+ * @MALI_VPU_FW_CODE_PING: Ping request sent to firmware.
+ * @MALI_VPU_FW_CODE_PONG: Firmware responded to a ping request.
+ * @MALI_VPU_FW_CODE_IDLE: Firmware informs the driver that it's idle.
+ * @MALI_VPU_FW_CODE_STATE_CHANGE: Firmware state-change request or response.
+ * @MALI_VPU_FW_CODE_SWITCH_IN: Firmware was booted on a Core in a session.
+ * @MALI_VPU_FW_CODE_SWITCH_OUT: The last core in a session was switched out.
+ * @MALI_VPU_FW_CODE_UNKNOWN: Unrecognized firmware message.
+ * @MALI_VPU_FW_CODE_MAX: Sentinel marking the end of the message-code enum.
+ */
+enum mali_vpu_fw_code {
+ MALI_VPU_FW_CODE_ERROR, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_JOB, /* Driver -> Firmware. */
+ MALI_VPU_FW_CODE_PING, /* Driver -> Firmware. */
+ MALI_VPU_FW_CODE_PONG, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_IDLE, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_STATE_CHANGE, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_SWITCH_IN, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_SWITCH_OUT, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_MAX
+};
+
+/**
+ * struct mali_vpu_fw_msg - Parsed firmware message.
+ * @code: Driver representation of the firmware message code.
+ * @data: Message-specific decoded payload.
+ */
+struct mali_vpu_fw_msg {
+ enum mali_vpu_fw_code code;
+ union {
+ enum mali_vpu_fw_state state;
+ struct mali_vpu_fw_job job;
+ struct mali_vpu_fw_error error;
+ } data;
+};
+
/**
* enum mali_vpu_fw_region - Regions in the firmware virtual address space.
* @MALI_VPU_FW_REGION_MSG_HOST: Host-to-firmware message queue.
@@ -104,5 +183,9 @@ struct mali_vpu_fw {
int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
struct device *dev, struct mali_vpu_mmu *mmu, u32 ncores);
void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw);
+int mali_vpu_fw_send_job(struct mali_vpu_fw *fw, u16 cores, u16 frames);
+int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
+int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
+int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg);
#endif /* __MALI_VPU_FIRMWARE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index b6d46f91287c4..3523ac3e49f1d 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -122,6 +122,151 @@
/* One struct mve_rpc_communication_area located here */
#define MVE_COMM_RPC_ADDR (0x1007F000u)
+/*
+ * One page of memory (4 kB) is used for each queue,
+ * so maximum 1024 words, but need room for some counters as well,
+ * see structs mve_comm_area_mve and mve_comm_area_host below.
+ */
+#define MVE_COMM_QUEUE_SIZE_IN_WORDS 1020
+
+/* This is the part of the message area that is written by host. */
+struct mve_comm_area_host {
+ u16 out_rpos;
+ u16 in_wpos;
+ u32 reserved[3];
+ /*
+ * Queue of messages to MVE, each block of data prefixed with
+ * a mve_msg_header
+ */
+ u32 in_data[MVE_COMM_QUEUE_SIZE_IN_WORDS];
+};
+
+/* This is the part of the message area that is written by MVE. */
+struct mve_comm_area_mve {
+ u16 out_wpos;
+ u16 in_rpos;
+ u32 reserved[3];
+ /*
+ * Queue of messages to host, each block of data prefixed with
+ * a mve_msg_header
+ */
+ u32 out_data[MVE_COMM_QUEUE_SIZE_IN_WORDS];
+};
+
+/*********************************************************************
+ *
+ * Message codes
+ *
+ *********************************************************************/
+
+/*
+ * Messages consist of one struct mve_msg_header, possibly followed
+ * by extra data.
+ */
+struct mve_msg_header {
+ u16 code;
+ /*
+ * REQUESTs are messages from the
+ * host/driver to the firmware: Code: Extra data in message:
+ */
+ #define MVE_REQUEST_CODE_GO (1001) /* no extra data */
+ #define MVE_REQUEST_CODE_STOP (1002) /* no extra data */
+ #define MVE_REQUEST_CODE_PING (1006) /* no extra data */
+ #define MVE_REQUEST_CODE_JOB (1009) /* struct mve_request_job */
+
+ /*
+ * RESPONSEs are messages from
+ * the firmware to the host:
+ */
+ /* struct mve_response_switched_in */
+ #define MVE_RESPONSE_CODE_SWITCHED_IN (2001)
+ /* struct mve_response_switched_out */
+ #define MVE_RESPONSE_CODE_SWITCHED_OUT (2002)
+ /* struct mve_response_job_dequeued */
+ #define MVE_RESPONSE_CODE_JOB_DEQUEUED (2004)
+ #define MVE_RESPONSE_CODE_PONG (2009)
+ /* struct mve_response_error */
+ #define MVE_RESPONSE_CODE_ERROR (2010)
+ /* struct mve_response_state_change */
+ #define MVE_RESPONSE_CODE_STATE_CHANGE (2011)
+ #define MVE_RESPONSE_CODE_IDLE (2013)
+
+ u16 size; /* size in bytes of trailing data, 0 if none */
+};
+
+/*********************************************************************
+ *
+ * REQUESTs are messages from the host to the firmware
+ *
+ * Some of the MVE_REQUEST_CODE_ codes are followed by one of the
+ * structs below.
+ *
+ *********************************************************************/
+
+struct mve_request_job {
+ u16 cores; /* >= 1, number of cores to use, must match request to HW scheduler */
+ u16 frames; /* number of frames to process, zero means infinite */
+ u32 flags; /* can be zero */
+ #define MVE_JOB_FLAG_DISABLE_BNDMGR (0x01)
+};
+
+/*********************************************************************
+ *
+ * RESPONSEs are messages from the firmware to the host
+ *
+ * Some of the MVE_RESPONSE_CODE_ codes are followed by one of the
+ * structs below.
+ *
+ *********************************************************************/
+
+/* Sent when firmware has booted. */
+struct mve_response_switched_in {
+ u32 core;
+};
+
+/* Sent when last core in a session has switched out. */
+struct mve_response_switched_out {
+ u32 core;
+ u32 reason;
+ u32 sub_reason;
+};
+
+/*
+ * Response confirming state transition after either GO or STOP
+ * command from host.
+ */
+struct mve_response_state_change {
+ u32 new_state;
+ #define MVE_STATE_STOPPED (0)
+ #define MVE_STATE_RUNNING (2)
+};
+
+/*
+ * Message sent when all cores in the session have dequeued a
+ * job from the firmware job queue.
+ */
+struct mve_response_job_dequeued {
+ u32 valid_job;
+};
+
+/*
+ * Fatal error message from firmware, if sent then no further
+ * operation is possible.
+ */
+struct mve_response_error {
+ u32 error_code;
+ #define MVE_ERROR_ABORT (1)
+ #define MVE_ERROR_OUT_OF_MEMORY (2)
+ #define MVE_ERROR_ASSERT (3)
+ #define MVE_ERROR_UNSUPPORTED (4)
+ #define MVE_ERROR_INVALID_BUFFER (6)
+ #define MVE_ERROR_INVALID_STATE (8)
+ #define MVE_ERROR_WATCHDOG (9)
+
+ #define MVE_MAX_ERROR_MESSAGE_SIZE (128)
+ char message[MVE_MAX_ERROR_MESSAGE_SIZE];
+};
+
#define MVE_CHROMA_FORMAT_420 (0x1)
/* format bitfield: */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 14/28] media: mali-vpu: route decoder controls to firmware
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (12 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 13/28] media: mali-vpu: add firmware message transport Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 15/28] media: mali-vpu: handle firmware RPC requests Charvi Mehta
` (13 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Translate decoder controls to firmware SET_OPTION requests and add
parsing for option confirmation and failures. Validate message payload
sizes and distinguish rejected options from successful requests.
Track outstanding requests awaiting a firmware response.
Use the firmware binary state as the single source of firmware readiness
and remove the redundant readiness boolean.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-controls.c | 40 +++--
.../platform/arm/mali-vpu/mali-vpu-dec.c | 18 +-
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 159 +++++++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 66 +++++++-
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 68 +++++++-
.../platform/arm/mali-vpu/mali-vpu-session.c | 7 +-
.../platform/arm/mali-vpu/mali-vpu-session.h | 4 +-
7 files changed, 328 insertions(+), 34 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
index d4921eddf0fcf..100f9185dbec1 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c
@@ -56,7 +56,7 @@ static int mali_vpu_apply_or_stage_ctrl(struct mali_vpu_session *sess,
bool staged;
int ret;
- staged = !sess->is_fw_ready;
+ staged = sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY;
if (!staged) {
ret = desc->set(sess, state->cid, value);
@@ -267,7 +267,7 @@ int mali_vpu_apply_staged_controls(struct mali_vpu_session *sess)
u32 i;
int ret;
- if (!sess->is_fw_ready)
+ if (sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY)
return -EINVAL;
for (i = 0; i < sess->num_ctrls_caps; i++) {
@@ -288,28 +288,36 @@ int mali_vpu_apply_staged_controls(struct mali_vpu_session *sess)
int mali_vpu_ctrl_set_nalu_format(struct mali_vpu_session *sess, u32 cid,
s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_NALU_FORMAT)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_NALU_FORMAT;
+ option.nalu_format = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
int mali_vpu_ctrl_set_intbuf_size(struct mali_vpu_session *sess, u32 cid,
s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_INTBUF_SIZE || value < 0 || value > INT_MAX)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_INTBUF_SIZE;
+ option.intbuf_size = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
int mali_vpu_ctrl_set_stream_escaping(struct mali_vpu_session *sess, u32 cid,
s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_STREAM_ESCAPING)
return -EINVAL;
@@ -319,21 +327,25 @@ int mali_vpu_ctrl_set_stream_escaping(struct mali_vpu_session *sess, u32 cid,
if (value != 0 && value != 1)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_STREAM_ESCAPING;
+ option.stream_escaping = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
int mali_vpu_ctrl_set_host_print_enable(struct mali_vpu_session *sess, u32 cid,
s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_HOST_PRINT_ENABLE)
return -EINVAL;
if (value != 0 && value != 1)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_HOST_PROTOCOL_PRINTS;
+ option.host_protocol_prints = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
index 7e78c475a4ff3..51d509abb9dd1 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
@@ -28,6 +28,8 @@ void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess,
u32 cid, s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_IGNORE_STREAM_HEADERS)
return -EINVAL;
@@ -37,14 +39,17 @@ int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess,
if (value != 0 && value != 1)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_IGNORE_STREAM_HEADERS;
+ option.ignore_stream_headers = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid,
s64 value)
{
+ struct mali_vpu_fw_set_option option;
+
if (cid != V4L2_CID_MALI_VPU_FRAME_REORDERING)
return -EINVAL;
@@ -54,7 +59,8 @@ int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid,
if (value != 0 && value != 1)
return -EINVAL;
- (void)sess;
- (void)value;
- return 0;
+ option.code = MALI_VPU_FW_SET_FRAME_REORDERING;
+ option.frame_reordering = value;
+
+ return mali_vpu_session_fw_send_set_option(sess, &option);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 4aead9a062260..91a989c098cbc 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -624,6 +624,7 @@ int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw)
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state)
{
u16 code;
+ int ret;
switch (state) {
case MALI_VPU_FW_STATE_STOPPED:
@@ -636,13 +637,135 @@ int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state
return -EINVAL;
}
- return mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve, code, NULL, 0);
+ ret = mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve, code, NULL, 0);
+ if (!ret)
+ fw->pending_response_count++;
+
+ return ret;
+}
+
+static int mali_vpu_to_mve_nalu_format(s64 nalu_format, u32 *mve_nalu_format)
+{
+ switch (nalu_format) {
+ case MALI_VPU_NALU_FORMAT_START_CODES:
+ *mve_nalu_format = MVE_OPT_NALU_FORMAT_START_CODES;
+ break;
+ case MALI_VPU_NALU_FORMAT_ONE_NALU_PER_BUFFER:
+ *mve_nalu_format = MVE_OPT_NALU_FORMAT_ONE_NALU_PER_BUFFER;
+ break;
+ case MALI_VPU_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD:
+ *mve_nalu_format = MVE_OPT_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD;
+ break;
+ case MALI_VPU_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD:
+ *mve_nalu_format = MVE_OPT_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD;
+ break;
+ case MALI_VPU_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD:
+ *mve_nalu_format = MVE_OPT_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_fw_send_option(struct mali_vpu_fw *fw,
+ struct mve_request_set_option *mve_opt,
+ size_t size)
+{
+ int ret;
+
+ ret = mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve,
+ MVE_REQUEST_CODE_SET_OPTION, mve_opt,
+ offsetof(typeof(*mve_opt), data) + size);
+ if (!ret)
+ fw->pending_response_count++;
+
+ return ret;
+}
+
+int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_set_option *option)
+{
+ int ret;
+
+ switch (option->code) {
+ case MALI_VPU_FW_SET_NALU_FORMAT: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_NALU_FORMAT;
+
+ ret = mali_vpu_to_mve_nalu_format(option->nalu_format,
+ &mve_opt.data.arg);
+ if (!ret)
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+
+ break;
+ }
+ case MALI_VPU_FW_SET_STREAM_ESCAPING: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_STREAM_ESCAPING;
+ mve_opt.data.arg = option->stream_escaping ? 1 : 0;
+
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+ break;
+ }
+ case MALI_VPU_FW_SET_HOST_PROTOCOL_PRINTS: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_HOST_PROTOCOL_PRINTS;
+ mve_opt.data.arg = option->host_protocol_prints ? 1 : 0;
+
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+ break;
+ }
+ case MALI_VPU_FW_SET_IGNORE_STREAM_HEADERS: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_IGNORE_STREAM_HEADERS;
+ mve_opt.data.arg = option->ignore_stream_headers ? 1 : 0;
+
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+ break;
+ }
+ case MALI_VPU_FW_SET_FRAME_REORDERING: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_FRAME_REORDERING;
+ mve_opt.data.arg = option->frame_reordering ? 1 : 0;
+
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+ break;
+ }
+ case MALI_VPU_FW_SET_INTBUF_SIZE: {
+ struct mve_request_set_option mve_opt;
+
+ mve_opt.index = MVE_SET_OPT_INDEX_INTBUF_SIZE;
+ mve_opt.data.arg = option->intbuf_size;
+
+ ret = mali_vpu_fw_send_option(fw, &mve_opt,
+ sizeof(mve_opt.data.arg));
+ break;
+ }
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
}
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
{
u16 code;
union {
+ struct mve_response_set_option_confirm set_option_confirm;
+ struct mve_response_set_option_fail set_option_fail;
struct mve_response_state_change state_change;
struct mve_response_error error;
} fw_msg = {};
@@ -691,6 +814,40 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
default:
return -EPROTO;
}
+
+ if (fw->pending_response_count == 0)
+ return -EPROTO;
+
+ fw->pending_response_count--;
+ break;
+ }
+ case MVE_RESPONSE_CODE_SET_OPTION_CONFIRM:
+ if (size != sizeof(fw_msg.set_option_confirm))
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_SET_OPTION;
+
+ if (fw->pending_response_count == 0)
+ return -EPROTO;
+
+ fw->pending_response_count--;
+ break;
+ case MVE_RESPONSE_CODE_SET_OPTION_FAIL: {
+ if (size != sizeof(fw_msg.set_option_fail))
+ return -EPROTO;
+
+ dev_err(fw->dev,
+ "firmware set option failed. index=%u, msg=%.*s.\n",
+ fw_msg.set_option_fail.index,
+ (int)sizeof(fw_msg.set_option_fail.message),
+ fw_msg.set_option_fail.message);
+
+ msg->code = MALI_VPU_FW_CODE_SET_OPTION_FAIL;
+
+ if (fw->pending_response_count == 0)
+ return -EPROTO;
+
+ fw->pending_response_count--;
break;
}
case MVE_RESPONSE_CODE_ERROR: {
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index f26fd78101661..e5dd67ce3397a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -6,6 +6,7 @@
#ifndef __MALI_VPU_FIRMWARE_H__
#define __MALI_VPU_FIRMWARE_H__
+#include <linux/media/arm/mali-vpu-config.h>
#include <linux/types.h>
#define MALI_VPU_FW_TEXT_BASE_ADDR 0x1000u
@@ -35,6 +36,46 @@ struct mali_vpu_fw_job {
unsigned int frames;
};
+/**
+ * enum mali_vpu_fw_set_option_code - Firmware option selectors
+ * @MALI_VPU_FW_SET_NALU_FORMAT: Set the NAL-unit format.
+ * @MALI_VPU_FW_SET_STREAM_ESCAPING: Configure stream escaping.
+ * @MALI_VPU_FW_SET_HOST_PROTOCOL_PRINTS: Configure firmware logging.
+ * @MALI_VPU_FW_SET_IGNORE_STREAM_HEADERS: Configure stream-header handling.
+ * @MALI_VPU_FW_SET_FRAME_REORDERING: Configure frame reordering.
+ * @MALI_VPU_FW_SET_INTBUF_SIZE: Set the suggested intermediate-buffer size.
+ */
+enum mali_vpu_fw_set_option_code {
+ MALI_VPU_FW_SET_NALU_FORMAT,
+ MALI_VPU_FW_SET_STREAM_ESCAPING,
+ MALI_VPU_FW_SET_HOST_PROTOCOL_PRINTS,
+ MALI_VPU_FW_SET_IGNORE_STREAM_HEADERS,
+ MALI_VPU_FW_SET_FRAME_REORDERING,
+ MALI_VPU_FW_SET_INTBUF_SIZE,
+};
+
+/**
+ * struct mali_vpu_fw_set_option - Set firmware options.
+ * @code: Code for the option to set.
+ * @nalu_format: NAL-unit encapsulation format.
+ * @stream_escaping: Whether stream escaping is enabled.
+ * @host_protocol_prints: Whether firmware protocol logging is enabled.
+ * @ignore_stream_headers: Whether firmware ignores stream headers.
+ * @frame_reordering: Whether frame reordering is enabled.
+ * @intbuf_size: Suggested intermediate-buffer size.
+ */
+struct mali_vpu_fw_set_option {
+ enum mali_vpu_fw_set_option_code code;
+ union {
+ enum mali_vpu_nalu_format nalu_format;
+ bool stream_escaping;
+ bool host_protocol_prints;
+ bool ignore_stream_headers;
+ bool frame_reordering;
+ u32 intbuf_size;
+ };
+};
+
/**
* struct mali_vpu_fw_error - Firmware error message.
* @error_code: What kind of error that was reported.
@@ -61,6 +102,8 @@ struct mali_vpu_fw_error {
* @MALI_VPU_FW_CODE_PING: Ping request sent to firmware.
* @MALI_VPU_FW_CODE_PONG: Firmware responded to a ping request.
* @MALI_VPU_FW_CODE_IDLE: Firmware informs the driver that it's idle.
+ * @MALI_VPU_FW_CODE_SET_OPTION: Firmware set option request or success response.
+ * @MALI_VPU_FW_CODE_SET_OPTION_FAIL: Firmware set option failure response.
* @MALI_VPU_FW_CODE_STATE_CHANGE: Firmware state-change request or response.
* @MALI_VPU_FW_CODE_SWITCH_IN: Firmware was booted on a Core in a session.
* @MALI_VPU_FW_CODE_SWITCH_OUT: The last core in a session was switched out.
@@ -68,15 +111,17 @@ struct mali_vpu_fw_error {
* @MALI_VPU_FW_CODE_MAX: Sentinel marking the end of the message-code enum.
*/
enum mali_vpu_fw_code {
- MALI_VPU_FW_CODE_ERROR, /* Driver <- Firmware. */
- MALI_VPU_FW_CODE_JOB, /* Driver -> Firmware. */
- MALI_VPU_FW_CODE_PING, /* Driver -> Firmware. */
- MALI_VPU_FW_CODE_PONG, /* Driver <- Firmware. */
- MALI_VPU_FW_CODE_IDLE, /* Driver <- Firmware. */
- MALI_VPU_FW_CODE_STATE_CHANGE, /* Driver <-> Firmware. */
- MALI_VPU_FW_CODE_SWITCH_IN, /* Driver <- Firmware. */
- MALI_VPU_FW_CODE_SWITCH_OUT, /* Driver <-> Firmware. */
- MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_ERROR, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_JOB, /* Driver -> Firmware. */
+ MALI_VPU_FW_CODE_PING, /* Driver -> Firmware. */
+ MALI_VPU_FW_CODE_PONG, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_IDLE, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_SET_OPTION, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_SET_OPTION_FAIL, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_STATE_CHANGE, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_SWITCH_IN, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_SWITCH_OUT, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_MAX
};
@@ -161,6 +206,7 @@ struct mali_vpu_fw_proto_page {
* @bss_pages: Private BSS pages allocated separately for each core instance.
* @shared_bss_pages: Shared BSS pages mapped into every core instance.
* @ncores: Number of firmware core-instance regions constructed.
+ * @pending_response_count: Number of requests awaiting a firmware response.
*/
struct mali_vpu_fw {
struct device *dev;
@@ -178,6 +224,7 @@ struct mali_vpu_fw {
struct mali_vpu_mmu_pages *bss_pages;
struct mali_vpu_mmu_pages *shared_bss_pages;
u32 ncores;
+ u32 pending_response_count;
};
int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
@@ -186,6 +233,7 @@ void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_job(struct mali_vpu_fw *fw, u16 cores, u16 frames);
int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
+int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw, struct mali_vpu_fw_set_option *option);
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg);
#endif /* __MALI_VPU_FIRMWARE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index 3523ac3e49f1d..c7c18bd4e1551 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -173,7 +173,10 @@ struct mve_msg_header {
#define MVE_REQUEST_CODE_STOP (1002) /* no extra data */
#define MVE_REQUEST_CODE_PING (1006) /* no extra data */
#define MVE_REQUEST_CODE_JOB (1009) /* struct mve_request_job */
-
+ #define MVE_REQUEST_CODE_SET_OPTION (1010) /* struct
+ * mve_request_set_option
+ * (variable size)
+ */
/*
* RESPONSEs are messages from
* the firmware to the host:
@@ -182,6 +185,7 @@ struct mve_msg_header {
#define MVE_RESPONSE_CODE_SWITCHED_IN (2001)
/* struct mve_response_switched_out */
#define MVE_RESPONSE_CODE_SWITCHED_OUT (2002)
+ #define MVE_RESPONSE_CODE_SET_OPTION_CONFIRM (2003)
/* struct mve_response_job_dequeued */
#define MVE_RESPONSE_CODE_JOB_DEQUEUED (2004)
#define MVE_RESPONSE_CODE_PONG (2009)
@@ -190,6 +194,8 @@ struct mve_msg_header {
/* struct mve_response_state_change */
#define MVE_RESPONSE_CODE_STATE_CHANGE (2011)
#define MVE_RESPONSE_CODE_IDLE (2013)
+ /* struct mve_response_set_option_fail */
+ #define MVE_RESPONSE_CODE_SET_OPTION_FAIL (2017)
u16 size; /* size in bytes of trailing data, 0 if none */
};
@@ -210,6 +216,42 @@ struct mve_request_job {
#define MVE_JOB_FLAG_DISABLE_BNDMGR (0x01)
};
+struct mve_request_set_option {
+ u32 index;
+ #define MVE_SET_OPT_INDEX_NALU_FORMAT (1) /* see arg,
+ * MVE_OPT_NALU_FORMAT_
+ */
+ #define MVE_SET_OPT_INDEX_STREAM_ESCAPING (2) /* arg=1 to enable (def),
+ * arg=0 to disable
+ */
+ #define MVE_SET_OPT_INDEX_HOST_PROTOCOL_PRINTS (4) /* arg=1 to enable,
+ * arg=0 to disable (def)
+ */
+ #define MVE_SET_OPT_INDEX_IGNORE_STREAM_HEADERS (7) /* decode, arg=1 to enable,
+ * arg=0 to disable (def)
+ */
+ #define MVE_SET_OPT_INDEX_FRAME_REORDERING (8) /* decode arg=1 enable(def),
+ * arg=0 to disable
+ */
+ #define MVE_SET_OPT_INDEX_INTBUF_SIZE (9) /* decode, arg = suggested
+ * limit of intermediate
+ * buffer allocation
+ */
+ union {
+ u32 arg; /* Most options only need a u32 as argument */
+ /*
+ * For option MVE_SET_OPT_INDEX_NALU_FORMAT, arg should
+ * be one of these:
+ */
+ #define MVE_OPT_NALU_FORMAT_START_CODES (1)
+ #define MVE_OPT_NALU_FORMAT_ONE_NALU_PER_BUFFER (2)
+ #define MVE_OPT_NALU_FORMAT_ONE_BYTE_LENGTH_FIELD (4)
+ #define MVE_OPT_NALU_FORMAT_TWO_BYTE_LENGTH_FIELD (8)
+ #define MVE_OPT_NALU_FORMAT_FOUR_BYTE_LENGTH_FIELD (16)
+ #define MVE_OPT_NALU_FORMAT_MBINFO (32) /* debugging only */
+ } data;
+};
+
/*********************************************************************
*
* RESPONSEs are messages from the firmware to the host
@@ -267,6 +309,30 @@ struct mve_response_error {
char message[MVE_MAX_ERROR_MESSAGE_SIZE];
};
+/*
+ * When a set-option succeeds, a confirmation message is
+ * sent, including the index-code for that particular option.
+ */
+struct mve_response_set_option_confirm {
+ u32 index; /* Same as 'index' in struct mve_request_set_option */
+};
+
+/*
+ * If a set-option request fails, this message is returned.
+ * This is not a fatal error. The set-option had no effect,
+ * and the session is still alive.
+ * For example, trying to set an option with a too large
+ * or small parameter would result in this message.
+ * The included text string is meant for development and
+ * debugging purposes only.
+ * (When a set-option succeeds the set-option-confirm
+ * message code is sent instead.)
+ */
+struct mve_response_set_option_fail {
+ u32 index; /* Same as 'index' in struct mve_request_set_option */
+ char message[MVE_MAX_ERROR_MESSAGE_SIZE];
+};
+
#define MVE_CHROMA_FORMAT_420 (0x1)
/* format bitfield: */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index c9a2c683b8dab..36197e82461be 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -68,6 +68,12 @@ mali_vpu_both_ports_unrequested(struct mali_vpu_session *sess)
!sess->port_dst.stream_requested;
}
+int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
+ struct mali_vpu_fw_set_option *opt)
+{
+ return mali_vpu_fw_send_set_option(&sess->fw, opt);
+}
+
int mali_vpu_session_init(struct mali_vpu_session *sess)
{
int ret;
@@ -91,7 +97,6 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
goto err_mutex_destroy;
}
- sess->is_fw_ready = false;
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
ret = mali_vpu_ctrls_init(sess);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index ecd048e3e5c01..f0aa98005e92a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -65,7 +65,6 @@ enum mali_vpu_session_fw_bin_state {
* @port_dst: Destination (CAPTURE) port for this session.
* @mmu: MMU handle for this session
* @vq_lock: Mutex lock to serialise per-session queue related ioctls.
- * @is_fw_ready: True if the firmware is loaded and ready to accept commands.
* @min_buffers_required: Driver-reported minimum buffers for role-specific queue.
* @ctrls_caps: Variant-provided immutable control capability table.
* @num_ctrls_caps: Number of entries in @ctrls_caps.
@@ -86,7 +85,6 @@ struct mali_vpu_session {
struct mali_vpu_port port_dst;
struct mali_vpu_mmu mmu;
struct mutex vq_lock; /* lock to serialise per-session ioctls */
- bool is_fw_ready;
u32 min_buffers_required;
const struct mali_vpu_ctrl_desc *ctrls_caps;
@@ -107,6 +105,8 @@ struct mali_vpu_session {
int mali_vpu_session_init(struct mali_vpu_session *sess);
int mali_vpu_session_cleanup(struct mali_vpu_session *sess);
int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess);
+int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
+ struct mali_vpu_fw_set_option *opt);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 15/28] media: mali-vpu: handle firmware RPC requests
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (13 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 14/28] media: mali-vpu: route decoder controls to firmware Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:51 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 16/28] media: mali-vpu: add LSID hardware support Charvi Mehta
` (12 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Handle firmware requests for debug output and memory allocation,
resizing and release. Track allocations by firmware virtual address
and release outstanding allocations during firmware teardown.
Flag resize and free requests so the caller can invalidate cached
firmware MMU translations before resuming execution.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 277 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 12 +
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 53 ++++
3 files changed, 342 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 91a989c098cbc..c11d5c6c0ec5f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -7,6 +7,7 @@
#include <linux/dma-mapping.h>
#include <linux/err.h>
#include <linux/errno.h>
+#include <linux/sizes.h>
#include <linux/string.h>
#include "mali-vpu-firmware.h"
@@ -15,6 +16,7 @@
#include "mali-vpu-protocol-def-v3.h"
#define MALI_VPU_FW_MAX_CORES 8
+#define MALI_VPU_RPC_VA_GRANULE SZ_16M
static int mali_vpu_fw_get_region(enum mali_vpu_fw_region region,
mali_vpu_mmu_va *begin, mali_vpu_mmu_va *end)
@@ -80,6 +82,14 @@ static int mali_vpu_fw_get_region(enum mali_vpu_fw_region region,
*begin = MVE_MEM_REGION_FW_INSTANCE7_ADDR_BEGIN;
*end = MVE_MEM_REGION_FW_INSTANCE7_ADDR_END;
break;
+ case MALI_VPU_FW_REGION_PROTECTED:
+ *begin = MVE_MEM_REGION_PROTECTED_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_PROTECTED_ADDR_END;
+ break;
+ case MALI_VPU_FW_REGION_FRAMEBUF:
+ *begin = MVE_MEM_REGION_FRAMEBUF_ADDR_BEGIN;
+ *end = MVE_MEM_REGION_FRAMEBUF_ADDR_END;
+ break;
default:
return -EINVAL;
}
@@ -352,6 +362,7 @@ int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *
fw->dev = dev;
fw->mmu = mmu;
fw->ncores = ncores;
+ hash_init(fw->rpc_mem);
/* Initialise firmware core and protocol regions */
ret = mali_vpu_fw_map_core_regions(fw, fw_bin);
@@ -375,11 +386,21 @@ int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *
void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw)
{
+ struct mali_vpu_mmu_pages *pages;
+ struct hlist_node *tmp;
+ unsigned int bkt;
+
if (!fw)
return;
mali_vpu_fw_unmap_protocol(fw);
mali_vpu_fw_unmap_core_regions(fw);
+
+ hash_for_each_safe(fw->rpc_mem, bkt, tmp, pages, node) {
+ hash_del(&pages->node);
+ mali_vpu_mmu_free_pages(pages);
+ }
+
memset(fw, 0, sizeof(*fw));
}
@@ -908,3 +929,259 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
return ret;
}
+
+static void mali_vpu_fw_dbg_rpc(struct mali_vpu_fw *fw, const char *direction,
+ const struct mve_rpc_communication_area *rpc)
+{
+ dev_dbg(fw->dev, "rpc %s: state=%u call_id=%u size=%u\n",
+ direction, rpc->state, rpc->call_id, rpc->size);
+}
+
+static struct mali_vpu_mmu_pages *mali_vpu_find_pages(struct mali_vpu_fw *fw,
+ mali_vpu_mmu_va va)
+{
+ struct mali_vpu_mmu_pages *pages;
+
+ hash_for_each_possible(fw->rpc_mem, pages, node, va) {
+ if (pages->va == va)
+ return pages;
+ }
+
+ return NULL;
+}
+
+static void mali_vpu_rpc_mem_resize(struct mali_vpu_fw *fw,
+ struct mve_rpc_communication_area *rpc_area)
+{
+ union mve_rpc_params *p = &rpc_area->params;
+ struct mali_vpu_mmu_pages *pages;
+ mali_vpu_mmu_va va = 0;
+ int ret;
+
+ pages = mali_vpu_find_pages(fw, p->mem_resize.ve_pointer);
+ if (pages) {
+ size_t npages;
+
+ /* Resize the allocated pages. */
+ npages = DIV_ROUND_UP(p->mem_resize.new_size, MALI_VPU_PAGE_SIZE);
+ ret = mali_vpu_mmu_resize_pages(pages, npages);
+ if (ret) {
+ dev_warn(fw->dev,
+ "failed to resize RPC mapped pages. ret=%d\n",
+ ret);
+ goto err;
+ }
+
+ va = pages->va;
+ } else {
+ dev_warn(fw->dev, "could not find pages for RPC resize. va=0x%x\n",
+ p->mem_resize.ve_pointer);
+ }
+
+ dev_dbg(fw->dev, "RPC resize memory. va=0x%x, new_size=%u\n",
+ p->mem_resize.ve_pointer, p->mem_resize.new_size);
+
+err:
+ /*
+ * The memory RPC ABI returns the resulting firmware VA. Firmware treats
+ * a zero VA as allocation failure and reports MVE_ERROR_OUT_OF_MEMORY.
+ * The RPC itself was still handled successfully.
+ */
+ rpc_area->size = sizeof(u32);
+ p->data[0] = va;
+}
+
+static void mali_vpu_rpc_mem_alloc(struct mali_vpu_fw *fw,
+ struct mve_rpc_communication_area *rpc_area)
+{
+ union mve_rpc_params *p = &rpc_area->params;
+ enum mali_vpu_fw_region region;
+ struct mali_vpu_mmu_pages *pages;
+ size_t npages, max_pages;
+ mali_vpu_mmu_va end, begin = 0;
+ int ret;
+
+ if (!p->mem_alloc.size ||
+ p->mem_alloc.size > p->mem_alloc.max_size) {
+ dev_warn(fw->dev,
+ "invalid RPC allocation:size=%u max_size=%u alignment=%u\n",
+ p->mem_alloc.size, p->mem_alloc.max_size,
+ p->mem_alloc.log2_alignment);
+ goto err;
+ }
+
+ switch (p->mem_alloc.region) {
+ case MVE_MEM_REGION_PROTECTED:
+ region = MALI_VPU_FW_REGION_PROTECTED;
+ break;
+ case MVE_MEM_REGION_OUTBUF:
+ region = MALI_VPU_FW_REGION_FRAMEBUF;
+ break;
+ default:
+ dev_err(fw->dev,
+ "unsupported RPC mem alloc region. region=%u\n",
+ p->mem_alloc.region);
+ goto err;
+ }
+
+ ret = mali_vpu_fw_get_region(region, &begin, &end);
+ if (ret)
+ goto err;
+
+ npages = DIV_ROUND_UP(p->mem_alloc.size, MALI_VPU_PAGE_SIZE);
+ max_pages = DIV_ROUND_UP(p->mem_alloc.max_size, MALI_VPU_PAGE_SIZE);
+
+ pages = mali_vpu_mmu_alloc_pages(fw->dev, npages, max_pages);
+ if (IS_ERR(pages)) {
+ begin = 0;
+ goto err;
+ }
+
+ ret = -EINVAL;
+ while (begin < end) {
+ ret = mali_vpu_mmu_map_pages(fw->mmu, begin, pages,
+ MALI_VPU_ATTR_SHARED_RW,
+ MALI_VPU_ACCESS_READ_WRITE);
+ if (!ret)
+ break;
+
+ begin += MALI_VPU_RPC_VA_GRANULE;
+ }
+
+ if (ret) {
+ dev_err(fw->dev,
+ "failed to find memory region for RPC alloc.\n");
+ mali_vpu_mmu_free_pages(pages);
+ begin = 0;
+ goto err;
+ }
+
+ hash_add(fw->rpc_mem, &pages->node, pages->va);
+ begin = pages->va;
+
+ dev_dbg(fw->dev,
+ "RPC alloc memory. size=%u, max_size=%u, region=%u, npages=%zu, begin=0x%x\n",
+ p->mem_alloc.size, p->mem_alloc.max_size, p->mem_alloc.region, npages, begin);
+
+err:
+ rpc_area->size = sizeof(u32);
+ p->data[0] = begin;
+}
+
+static void mali_vpu_rpc_mem_free(struct mali_vpu_fw *fw,
+ struct mve_rpc_communication_area *rpc_area)
+{
+ union mve_rpc_params *p = &rpc_area->params;
+ struct mali_vpu_mmu_pages *pages;
+
+ pages = mali_vpu_find_pages(fw, p->mem_free.ve_pointer);
+ if (pages) {
+ hash_del(&pages->node);
+ mali_vpu_mmu_free_pages(pages);
+ } else {
+ dev_warn(fw->dev,
+ "could not find pages for RPC free. va=0x%x\n",
+ p->mem_free.ve_pointer);
+ }
+
+ dev_dbg(fw->dev, "RPC free memory. va=0x%x\n", p->mem_free.ve_pointer);
+
+ rpc_area->size = 0;
+}
+
+int mali_vpu_fw_handle_rpc(struct mali_vpu_fw *fw, bool *flush_mmu)
+{
+ struct mve_rpc_communication_area *rpc_area = fw->rpc.cpu;
+ int ret = 0;
+
+ if (flush_mmu)
+ *flush_mmu = false;
+
+ dma_sync_single_for_cpu(fw->dev, fw->rpc.dma, sizeof(*rpc_area),
+ DMA_FROM_DEVICE);
+
+ if (READ_ONCE(rpc_area->state) == MVE_RPC_STATE_PARAM) {
+ /*
+ * Firmware publishes param only after writing the RPC request.
+ */
+ dma_rmb();
+
+ ret = 1;
+
+ mali_vpu_fw_dbg_rpc(fw, "fw->host", rpc_area);
+
+ switch (rpc_area->call_id) {
+ case MVE_RPC_FUNCTION_DEBUG_PRINTF: {
+ if (!rpc_area->size ||
+ rpc_area->size > sizeof(rpc_area->params.debug_print.string)) {
+ ret = -EPROTO;
+ rpc_area->size = 0;
+ break;
+ }
+
+ const char *s = rpc_area->params.debug_print.string;
+ size_t len = strnlen(s, rpc_area->size);
+
+ /* Strip trailing characters */
+ while (len && (s[len - 1] == '\n' || s[len - 1] == '\r'))
+ len--;
+
+ dev_info(fw->dev,
+ "firmware DEBUG Print: %.*s\n", (int)len, s);
+ break;
+ }
+ case MVE_RPC_FUNCTION_MEM_ALLOC: {
+ if (rpc_area->size != sizeof(rpc_area->params.mem_alloc)) {
+ ret = -EPROTO;
+ break;
+ }
+
+ mali_vpu_rpc_mem_alloc(fw, rpc_area);
+ break;
+ }
+ case MVE_RPC_FUNCTION_MEM_RESIZE: {
+ if (rpc_area->size != sizeof(rpc_area->params.mem_resize)) {
+ ret = -EPROTO;
+ break;
+ }
+
+ mali_vpu_rpc_mem_resize(fw, rpc_area);
+ if (flush_mmu)
+ *flush_mmu = true;
+ break;
+ }
+ case MVE_RPC_FUNCTION_MEM_FREE: {
+ if (rpc_area->size != sizeof(rpc_area->params.mem_free)) {
+ ret = -EPROTO;
+ break;
+ }
+
+ mali_vpu_rpc_mem_free(fw, rpc_area);
+ if (flush_mmu)
+ *flush_mmu = true;
+ break;
+ }
+ default:
+ dev_warn_ratelimited(fw->dev,
+ "unsupported RPC request. call_id=%u\n",
+ rpc_area->call_id);
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ /*
+ * Make sure the whole RPC message body has been written before
+ * the RPC message area is returned to the firmware.
+ */
+ dma_wmb();
+ WRITE_ONCE(rpc_area->state, MVE_RPC_STATE_RETURN);
+
+ dma_sync_single_for_device(fw->dev, fw->rpc.dma,
+ sizeof(*rpc_area), DMA_TO_DEVICE);
+
+ /* Log RPC response. */
+ mali_vpu_fw_dbg_rpc(fw, "host->fw", rpc_area);
+ }
+
+ return ret;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index e5dd67ce3397a..c03a07d994873 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -6,9 +6,11 @@
#ifndef __MALI_VPU_FIRMWARE_H__
#define __MALI_VPU_FIRMWARE_H__
+#include <linux/hashtable.h>
#include <linux/media/arm/mali-vpu-config.h>
#include <linux/types.h>
+#define MALI_VPU_FW_HTABLE_BITS 3
#define MALI_VPU_FW_TEXT_BASE_ADDR 0x1000u
struct device;
@@ -156,6 +158,8 @@ struct mali_vpu_fw_msg {
* @MALI_VPU_FW_REGION_CORE_5: Firmware instance region for core 5.
* @MALI_VPU_FW_REGION_CORE_6: Firmware instance region for core 6.
* @MALI_VPU_FW_REGION_CORE_7: Firmware instance region for core 7.
+ * @MALI_VPU_FW_REGION_PROTECTED: Protected firmware memory region.
+ * @MALI_VPU_FW_REGION_FRAMEBUF: Frame-buffer memory region.
*/
enum mali_vpu_fw_region {
MALI_VPU_FW_REGION_MSG_HOST,
@@ -175,6 +179,9 @@ enum mali_vpu_fw_region {
MALI_VPU_FW_REGION_CORE_5,
MALI_VPU_FW_REGION_CORE_6,
MALI_VPU_FW_REGION_CORE_7,
+
+ MALI_VPU_FW_REGION_PROTECTED,
+ MALI_VPU_FW_REGION_FRAMEBUF,
};
/**
@@ -207,6 +214,8 @@ struct mali_vpu_fw_proto_page {
* @shared_bss_pages: Shared BSS pages mapped into every core instance.
* @ncores: Number of firmware core-instance regions constructed.
* @pending_response_count: Number of requests awaiting a firmware response.
+ * @rpc_mem: Keeps track of RPC allocated memory. Maps MVE virtual address to
+ * 'struct mali_vpu_mmu_pages' object.
*/
struct mali_vpu_fw {
struct device *dev;
@@ -225,6 +234,8 @@ struct mali_vpu_fw {
struct mali_vpu_mmu_pages *shared_bss_pages;
u32 ncores;
u32 pending_response_count;
+
+ DECLARE_HASHTABLE(rpc_mem, MALI_VPU_FW_HTABLE_BITS);
};
int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
@@ -235,5 +246,6 @@ int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw, struct mali_vpu_fw_set_option *option);
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg);
+int mali_vpu_fw_handle_rpc(struct mali_vpu_fw *fw, bool *flush_mmu);
#endif /* __MALI_VPU_FIRMWARE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index c7c18bd4e1551..8a81ed1fd730a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -77,6 +77,16 @@
* 0x10079000 - 0x1007FFFF, see special defines further down.
*/
+/* PROTECTED virtual memory region */
+#define MVE_MEM_REGION_PROTECTED_ADDR_BEGIN (0x20000000u)
+#define MVE_MEM_REGION_PROTECTED_ADDR_END (0x6FFFFFFFu + 1)
+
+/* FRAMEBUF virtual memory region */
+#define MVE_MEM_REGION_FRAMEBUF_ADDR_BEGIN (0x70000000u)
+#define MVE_MEM_REGION_FRAMEBUF_ADDR_END (0xEFFFFFFFu + 1)
+
+/* 0xF0000000 - 0xFFFFFFFF is used internally in MVE */
+
/*****************************************************************************
*
* Communication queues between HOST/DRIVER and MVE
@@ -153,6 +163,49 @@ struct mve_comm_area_mve {
u32 out_data[MVE_COMM_QUEUE_SIZE_IN_WORDS];
};
+#define MVE_RPC_AREA_SIZE_IN_WORDS 256
+#define MVE_RPC_DATA_SIZE_IN_WORDS (MVE_RPC_AREA_SIZE_IN_WORDS - 3)
+union mve_rpc_params {
+ u32 data[MVE_RPC_DATA_SIZE_IN_WORDS];
+ struct {
+ char string[MVE_RPC_DATA_SIZE_IN_WORDS * 4];
+ } debug_print;
+ struct {
+ u32 size;
+ u32 max_size;
+ u8 region; /* Memory region selection */
+ #define MVE_MEM_REGION_PROTECTED (0)
+ #define MVE_MEM_REGION_OUTBUF (1)
+ #define MVE_MEM_REGION_FRAMEBUF (MVE_MEM_REGION_OUTBUF)
+
+ /* The newly allocated memory must be placed
+ * on (at least) a 2^(log2_alignment) boundary
+ */
+ u8 log2_alignment;
+ } mem_alloc;
+ struct {
+ u32 ve_pointer;
+ u32 new_size;
+ } mem_resize;
+ struct {
+ u32 ve_pointer;
+ } mem_free;
+};
+
+struct mve_rpc_communication_area {
+ u32 state;
+ #define MVE_RPC_STATE_FREE (0)
+ #define MVE_RPC_STATE_PARAM (1)
+ #define MVE_RPC_STATE_RETURN (2)
+ u32 call_id;
+ #define MVE_RPC_FUNCTION_DEBUG_PRINTF (1)
+ #define MVE_RPC_FUNCTION_MEM_ALLOC (2)
+ #define MVE_RPC_FUNCTION_MEM_RESIZE (3)
+ #define MVE_RPC_FUNCTION_MEM_FREE (4)
+ u32 size;
+ union mve_rpc_params params;
+};
+
/*********************************************************************
*
* Message codes
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 15/28] media: mali-vpu: handle firmware RPC requests
2026-11-19 15:37 ` [RFC PATCH 15/28] media: mali-vpu: handle firmware RPC requests Charvi Mehta
@ 2026-11-19 15:51 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:51 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] media: mali-vpu: out-of-bounds VA mapping in
mali_vpu_rpc_mem_alloc()
- [Medium] media: mali-vpu: stale RPC return payload in
mali_vpu_fw_handle_rpc()
- [Low] media: mali-vpu: log2_alignment ignored in
mali_vpu_rpc_mem_alloc()
--
commit 6596ddd7663ec38dc1e5a77c8bd517be369bf9ab
Author: Charvi Mehta <charvi.mehta@arm.com>
media: mali-vpu: handle firmware RPC requests
This commit adds handling for firmware RPC requests covering debug print
output and firmware memory allocation, resizing, and freeing, tracking
allocated pages by virtual address in a hash table.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
> index 91a989c098cbc..c11d5c6c0ec5f 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
[ ... ]
> @@ -908,3 +929,259 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
[ ... ]
> +static void mali_vpu_rpc_mem_alloc(struct mali_vpu_fw *fw,
> + struct mve_rpc_communication_area *rpc_area)
> +{
> + union mve_rpc_params *p = &rpc_area->params;
> + enum mali_vpu_fw_region region;
> + struct mali_vpu_mmu_pages *pages;
> + size_t npages, max_pages;
> + mali_vpu_mmu_va end, begin = 0;
> + int ret;
> +
> + if (!p->mem_alloc.size ||
> + p->mem_alloc.size > p->mem_alloc.max_size) {
> + dev_warn(fw->dev,
> + "invalid RPC allocation:size=%u max_size=%u alignment=%u\n",
> + p->mem_alloc.size, p->mem_alloc.max_size,
> + p->mem_alloc.log2_alignment);
> + goto err;
> + }
[Severity: Low]
Should mali_vpu_rpc_mem_alloc() validate or apply
p->mem_alloc.log2_alignment?
union mve_rpc_params documents that the allocation must be placed on at
least a 2^(log2_alignment) boundary, and the warning above logs
log2_alignment, but the placement loop always steps candidate addresses by
MALI_VPU_RPC_VA_GRANULE (SZ_16M) without rejecting or honouring larger
alignments.
> +
> + switch (p->mem_alloc.region) {
> + case MVE_MEM_REGION_PROTECTED:
> + region = MALI_VPU_FW_REGION_PROTECTED;
> + break;
> + case MVE_MEM_REGION_OUTBUF:
> + region = MALI_VPU_FW_REGION_FRAMEBUF;
> + break;
> + default:
> + dev_err(fw->dev,
> + "unsupported RPC mem alloc region. region=%u\n",
> + p->mem_alloc.region);
> + goto err;
> + }
> +
> + ret = mali_vpu_fw_get_region(region, &begin, &end);
> + if (ret)
> + goto err;
> +
> + npages = DIV_ROUND_UP(p->mem_alloc.size, MALI_VPU_PAGE_SIZE);
> + max_pages = DIV_ROUND_UP(p->mem_alloc.max_size, MALI_VPU_PAGE_SIZE);
> +
> + pages = mali_vpu_mmu_alloc_pages(fw->dev, npages, max_pages);
> + if (IS_ERR(pages)) {
> + begin = 0;
> + goto err;
> + }
> +
> + ret = -EINVAL;
> + while (begin < end) {
> + ret = mali_vpu_mmu_map_pages(fw->mmu, begin, pages,
> + MALI_VPU_ATTR_SHARED_RW,
> + MALI_VPU_ACCESS_READ_WRITE);
> + if (!ret)
> + break;
> +
> + begin += MALI_VPU_RPC_VA_GRANULE;
> + }
[Severity: Medium]
Can this loop map pages past the end of the target region?
mali_vpu_mmu_map_pages() maps and reserves pages->capacity pages starting
at begin, but the loop only checks begin < end without verifying that the
full reserved range fits within end - begin. When earlier 16 MiB granules
are occupied and begin advances near end (or when max_size exceeds the
region size), the mapping spills past end into the adjacent address
region.
In addition, p->mem_alloc.max_size and p->mem_resize.new_size in
mali_vpu_rpc_mem_resize() are passed directly to DIV_ROUND_UP(...,
MALI_VPU_PAGE_SIZE) without an upper bound check. On 32-bit kernels
MALI_VPU_PAGE_SIZE is a 32-bit unsigned long, so values above 0xfffff000
wrap DIV_ROUND_UP() to 0 (causing mali_vpu_rpc_mem_resize() to shrink the
allocation to 0 pages while returning pages->va as a success), and on
64-bit kernels an unbounded max_size can trigger a slab allocator warning
in devm_kzalloc().
[ ... ]
> +int mali_vpu_fw_handle_rpc(struct mali_vpu_fw *fw, bool *flush_mmu)
> +{
> + struct mve_rpc_communication_area *rpc_area = fw->rpc.cpu;
> + int ret = 0;
> +
> + if (flush_mmu)
> + *flush_mmu = false;
> +
> + dma_sync_single_for_cpu(fw->dev, fw->rpc.dma, sizeof(*rpc_area),
> + DMA_FROM_DEVICE);
> +
> + if (READ_ONCE(rpc_area->state) == MVE_RPC_STATE_PARAM) {
> + /*
> + * Firmware publishes param only after writing the RPC request.
> + */
> + dma_rmb();
> +
> + ret = 1;
> +
> + mali_vpu_fw_dbg_rpc(fw, "fw->host", rpc_area);
> +
> + switch (rpc_area->call_id) {
> + case MVE_RPC_FUNCTION_DEBUG_PRINTF: {
> + if (!rpc_area->size ||
> + rpc_area->size > sizeof(rpc_area->params.debug_print.string)) {
> + ret = -EPROTO;
> + rpc_area->size = 0;
> + break;
> + }
> +
> + const char *s = rpc_area->params.debug_print.string;
> + size_t len = strnlen(s, rpc_area->size);
> +
> + /* Strip trailing characters */
> + while (len && (s[len - 1] == '\n' || s[len - 1] == '\r'))
> + len--;
> +
> + dev_info(fw->dev,
> + "firmware DEBUG Print: %.*s\n", (int)len, s);
> + break;
> + }
> + case MVE_RPC_FUNCTION_MEM_ALLOC: {
> + if (rpc_area->size != sizeof(rpc_area->params.mem_alloc)) {
> + ret = -EPROTO;
> + break;
> + }
> +
> + mali_vpu_rpc_mem_alloc(fw, rpc_area);
> + break;
> + }
> + case MVE_RPC_FUNCTION_MEM_RESIZE: {
> + if (rpc_area->size != sizeof(rpc_area->params.mem_resize)) {
> + ret = -EPROTO;
> + break;
> + }
> +
> + mali_vpu_rpc_mem_resize(fw, rpc_area);
> + if (flush_mmu)
> + *flush_mmu = true;
> + break;
> + }
> + case MVE_RPC_FUNCTION_MEM_FREE: {
> + if (rpc_area->size != sizeof(rpc_area->params.mem_free)) {
> + ret = -EPROTO;
> + break;
> + }
> +
> + mali_vpu_rpc_mem_free(fw, rpc_area);
> + if (flush_mmu)
> + *flush_mmu = true;
> + break;
> + }
> + default:
> + dev_warn_ratelimited(fw->dev,
> + "unsupported RPC request. call_id=%u\n",
> + rpc_area->call_id);
> + ret = -EOPNOTSUPP;
> + break;
> + }
> +
> + /*
> + * Make sure the whole RPC message body has been written before
> + * the RPC message area is returned to the firmware.
> + */
> + dma_wmb();
> + WRITE_ONCE(rpc_area->state, MVE_RPC_STATE_RETURN);
> +
> + dma_sync_single_for_device(fw->dev, fw->rpc.dma,
> + sizeof(*rpc_area), DMA_TO_DEVICE);
[Severity: Medium]
Does returning MVE_RPC_STATE_RETURN after a size mismatch leave stale
request words in the RPC response payload?
union mve_rpc_params aliases data[0] with mem_alloc.size and
mem_resize.ve_pointer. When MVE_RPC_FUNCTION_MEM_ALLOC or
MVE_RPC_FUNCTION_MEM_RESIZE breaks out with -EPROTO, rpc_area->state is
still set to MVE_RPC_STATE_RETURN and synced back to the device with
rpc_area->params.data[0] holding the non-zero input size or old
ve_pointer, which the firmware ABI interprets as a valid allocated virtual
address rather than 0 for failure.
Also, MVE_RPC_FUNCTION_DEBUG_PRINTF sets rpc_area->size = 0 on its error
path, but leaves rpc_area->size set to the input string length on normal
completion, and the other error paths leave rpc_area->size untouched.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=15
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 16/28] media: mali-vpu: add LSID hardware support
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (14 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 15/28] media: mali-vpu: handle firmware RPC requests Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 17/28] media: mali-vpu: add single session scheduler Charvi Mehta
` (11 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add low-level support of managing the VPU local session IDs (LSIDs).
Define the per-LSID and job queue registers and provide helpers to map and
unmap sessions, manage the job queue, signal firmware, flush the session
MMU and terminate execution.
These operations provide hardware interface for the scheduler added
in a subsequent patch.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-lsid.c | 271 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-lsid.h | 55 ++++
.../arm/mali-vpu/mali-vpu-registers.h | 44 ++-
4 files changed, 370 insertions(+), 3 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 168b4bc43d54d..51d49d815f1fb 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -11,6 +11,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-controls.o \
mali-vpu-mmu.o \
mali-vpu-firmware-cache.o \
- mali-vpu-firmware.o
+ mali-vpu-firmware.o \
+ mali-vpu-lsid.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.c
new file mode 100644
index 0000000000000..42bf32b0937ff
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.c
@@ -0,0 +1,271 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/iopoll.h>
+
+#include "mali-vpu-device.h"
+#include "mali-vpu-lsid.h"
+#include "mali-vpu-registers.h"
+
+#define MALI_VPU_TERMINATE_READ_TIMEOUT_US 10000000
+#define MALI_VPU_TERMINATE_POLL_INTERVAL_US 10
+
+static u32 mali_vpu_lsid_reg(u32 lsid, u32 off)
+{
+ return MALI_VPU_LSID_BASE + (MALI_VPU_LSID_STRIDE * lsid) + off;
+}
+
+static u32 mali_vpu_read_lsid(void __iomem *regs_base, u32 lsid, u32 off)
+{
+ return mali_vpu_read_reg(regs_base, mali_vpu_lsid_reg(lsid, off));
+}
+
+static void mali_vpu_write_lsid(void __iomem *regs_base, u32 lsid, u32 off, u32 val)
+{
+ mali_vpu_write_reg(regs_base, mali_vpu_lsid_reg(lsid, off), val);
+}
+
+static bool mali_vpu_is_lsid_allocated(struct mali_vpu_lsid *lsid)
+{
+ u32 alloc = mali_vpu_read_lsid(lsid->vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_ALLOC);
+
+ return alloc != MALI_VPU_ALLOC_FREE;
+}
+
+static u32 mali_vpu_jobqueue_get_lsid(u32 jobqueue, u32 job)
+{
+ u32 shift = MALI_VPU_JOBQUEUE_JOB_STRIDE * job;
+ u32 lsid_mask = (1 << MALI_VPU_JOBQUEUE_LSID_BITS) - 1;
+
+ return (jobqueue >> shift) & lsid_mask;
+}
+
+static u32 mali_vpu_jobqueue_get_ncores_minus1(u32 jobqueue, u32 job)
+{
+ u32 shift = (MALI_VPU_JOBQUEUE_JOB_STRIDE * job) + MALI_VPU_JOBQUEUE_LSID_BITS;
+ u32 ncores_mask = (1 << MALI_VPU_JOBQUEUE_NCORES_BITS) - 1;
+
+ return (jobqueue >> shift) & ncores_mask;
+}
+
+static u32 mali_vpu_jobqueue_set_job(u32 jobqueue, u32 job_id, u32 lsid,
+ u32 ncores_minus1)
+{
+ u32 shift = MALI_VPU_JOBQUEUE_JOB_STRIDE * job_id;
+ u32 job_mask = ((1u << MALI_VPU_JOBQUEUE_JOB_STRIDE) - 1) << shift;
+ u32 job_val, lsid_mask, ncores_mask;
+
+ lsid_mask = ((1u << MALI_VPU_JOBQUEUE_LSID_BITS) - 1);
+ ncores_mask = ((1u << MALI_VPU_JOBQUEUE_NCORES_BITS) - 1);
+
+ job_val = lsid & lsid_mask;
+ job_val |= (ncores_minus1 & ncores_mask) << MALI_VPU_JOBQUEUE_LSID_BITS;
+
+ return (jobqueue & ~job_mask) | (job_val << shift);
+}
+
+int mali_vpu_lsid_terminate(struct mali_vpu_lsid *lsid)
+{
+ struct mali_vpu_dev *vpu_dev = lsid->vpu_dev;
+ u32 val;
+
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_TERMINATE, 1);
+
+ return read_poll_timeout(mali_vpu_read_lsid, val, val == 0,
+ MALI_VPU_TERMINATE_POLL_INTERVAL_US,
+ MALI_VPU_TERMINATE_READ_TIMEOUT_US, false,
+ vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_TERMINATE);
+}
+
+void mali_vpu_lsid_init(struct mali_vpu_lsid *lsid, struct mali_vpu_dev *dev, u32 id)
+{
+ lsid->vpu_dev = dev;
+ lsid->id = id;
+}
+
+int mali_vpu_lsid_map(struct mali_vpu_lsid *lsid, struct mali_vpu_lsid_pcb *pcb)
+{
+ struct mali_vpu_dev *vpu_dev = lsid->vpu_dev;
+ u32 alloc;
+ int ret;
+
+ /* Check that the LSID is not already allocated. */
+ if (mali_vpu_is_lsid_allocated(lsid)) {
+ dev_err(vpu_dev->dev,
+ "failed to map session to LSID. LSID already allocated. lsid=%u\n",
+ lsid->id);
+ return -EBUSY;
+ }
+
+ /* Allocate LSID. */
+ alloc = pcb->nprot == 0 ? MALI_VPU_ALLOC_PROTECTED : MALI_VPU_ALLOC_NON_PROTECTED;
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_ALLOC, alloc);
+
+ if (mali_vpu_read_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_NPROT) != pcb->nprot) {
+ dev_err(vpu_dev->dev,
+ "unexpected protection state for LSID %u\n", lsid->id);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_ALLOC, MALI_VPU_ALLOC_FREE);
+ return -EIO;
+ }
+
+ /* Ensure that the cores have stopped executing any session previously mapped. */
+ ret = mali_vpu_lsid_terminate(lsid);
+ if (ret) {
+ dev_err(vpu_dev->dev, "failed to terminate LSID %u: %d\n",
+ lsid->id, ret);
+ /*
+ * The caller must reset the VPU on termination timeout. The
+ * LSID cannot safely be marked free while hardware activity
+ * is uncertain.
+ */
+ return ret;
+ }
+
+ /* Configure number of cores to use and which cores to disable. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_CTRL, pcb->ctrl);
+
+ /* Configure MMU L0 entry and flush MMU tables. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_MMU_CTRL,
+ pcb->mmu_ctrl);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_FLUSH_ALL, 0);
+
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_BUSATTR_0, 0x0);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_BUSATTR_1, 0x0);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_BUSATTR_2, 0x33);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_BUSATTR_3, 0x33);
+
+ /* Restore interrupt registers. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_IRQVE, 0);
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_IRQHOST, pcb->irqhost);
+
+ /*
+ * Order the configuration writes before enabling scheduling.
+ */
+ wmb();
+
+ /* Enable LSID for scheduling. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_SCHED, 1);
+
+ return 0;
+}
+
+/*
+ * An LSID must not be unmapped if it is present in the job queue or core LSID.
+ * It is the responsibility of the scheduler to guarantee that the LSID is idle
+ * before it is unmapped.
+ */
+void mali_vpu_lsid_unmap(struct mali_vpu_lsid *lsid, struct mali_vpu_lsid_pcb *pcb)
+{
+ struct mali_vpu_dev *vpu_dev = lsid->vpu_dev;
+
+ if (!mali_vpu_is_lsid_allocated(lsid)) {
+ dev_dbg(vpu_dev->dev, "no session mapped to LSID %u\n", lsid->id);
+ return;
+ }
+
+ /* Prevent the LSID from being scheduled. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_SCHED, 0);
+
+ /* Store registers in process control block. */
+ pcb->ctrl = mali_vpu_read_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_CTRL);
+ pcb->mmu_ctrl = mali_vpu_read_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_MMU_CTRL);
+ pcb->irqhost = mali_vpu_read_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_IRQHOST);
+ pcb->nprot = mali_vpu_read_lsid(vpu_dev->regs_base, lsid->id,
+ MALI_VPU_REG_LSID_NPROT);
+
+ /* Deallocate LSID. */
+ mali_vpu_write_lsid(vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_ALLOC,
+ MALI_VPU_ALLOC_FREE);
+}
+
+int mali_vpu_lsid_jobqueue_add(struct mali_vpu_lsid *lsid, u32 ncores)
+{
+ struct mali_vpu_dev *vpu_dev = lsid->vpu_dev;
+ u32 jobqueue;
+ int job, ret = -EAGAIN;
+
+ if (!ncores || ncores > BIT(MALI_VPU_JOBQUEUE_NCORES_BITS))
+ return -EINVAL;
+
+ /* Stop the hardware scheduler while updating the jobqueue. */
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_ENABLE, 0);
+
+ jobqueue = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_JOBQUEUE);
+
+ /* Search if the LSID is already in the job queue. */
+ for (job = 0; job < MALI_VPU_JOBQUEUE_NJOBS; job++)
+ if (mali_vpu_jobqueue_get_lsid(jobqueue, job) == lsid->id) {
+ ret = 0;
+ goto enable_hw_scheduler;
+ }
+
+ /* Search for a free slot in the job queue. */
+ for (job = 0; job < MALI_VPU_JOBQUEUE_NJOBS; job++) {
+ if (mali_vpu_jobqueue_get_lsid(jobqueue, job) != MALI_VPU_JOBQUEUE_JOB_INVALID)
+ continue;
+
+ jobqueue = mali_vpu_jobqueue_set_job(jobqueue, job, lsid->id, ncores - 1);
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_JOBQUEUE, jobqueue);
+ ret = 0;
+ break;
+ }
+
+enable_hw_scheduler:
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_ENABLE, 1);
+ return ret;
+}
+
+void mali_vpu_lsid_jobqueue_remove(struct mali_vpu_lsid *lsid)
+{
+ struct mali_vpu_dev *vpu_dev = lsid->vpu_dev;
+ u32 old_jobqueue, new_jobqueue = 0;
+ u32 old_job_id, new_job_id, old_lsid, old_ncores_minus1;
+
+ /* Disable scheduling. */
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_ENABLE, 0);
+
+ old_jobqueue = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_JOBQUEUE);
+
+ /* Copy job entries that do not match the LSID to be removed. */
+ for (old_job_id = 0, new_job_id = 0; old_job_id < MALI_VPU_JOBQUEUE_NJOBS; old_job_id++) {
+ old_lsid = mali_vpu_jobqueue_get_lsid(old_jobqueue, old_job_id);
+ old_ncores_minus1 = mali_vpu_jobqueue_get_ncores_minus1(old_jobqueue, old_job_id);
+
+ if (old_lsid == MALI_VPU_JOBQUEUE_JOB_INVALID ||
+ old_lsid == lsid->id)
+ continue;
+
+ new_jobqueue = mali_vpu_jobqueue_set_job(new_jobqueue, new_job_id,
+ old_lsid,
+ old_ncores_minus1);
+ new_job_id++;
+ }
+
+ while (new_job_id < MALI_VPU_JOBQUEUE_NJOBS) {
+ new_jobqueue = mali_vpu_jobqueue_set_job(new_jobqueue, new_job_id,
+ MALI_VPU_JOBQUEUE_JOB_INVALID, 0);
+ new_job_id++;
+ }
+
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_JOBQUEUE, new_jobqueue);
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_ENABLE, 1);
+}
+
+void mali_vpu_lsid_send_irq(struct mali_vpu_lsid *lsid)
+{
+ mali_vpu_write_lsid(lsid->vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_IRQHOST, 1);
+}
+
+void mali_vpu_lsid_flush_mmu(struct mali_vpu_lsid *lsid)
+{
+ mali_vpu_write_lsid(lsid->vpu_dev->regs_base, lsid->id, MALI_VPU_REG_LSID_FLUSH_ALL, 0);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
new file mode 100644
index 0000000000000..97c32f9ac9aa0
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_LSID_H__
+#define __MALI_VPU_LSID_H__
+
+#include <linux/types.h>
+
+struct mali_vpu_dev;
+
+/*
+ * A local session identifier (LSID) represents a hardware execution
+ * context. A firmware session must be mapped to an LSID before it can be
+ * scheduled on the VPU cores.
+ */
+
+/**
+ * struct mali_vpu_lsid_pcb - LSID process control block.
+ * @ctrl: Saved LSID control register value.
+ * @mmu_ctrl: Saved LSID MMU control register value.
+ * @irqhost: Saved pending host IRQ state.
+ * @nprot: Saved non-protected/protected allocation state.
+ *
+ * This structure is used to store the register map when a session is unmapped
+ * from a LSID, so it can be restored again when the session is remapped.
+ */
+struct mali_vpu_lsid_pcb {
+ u32 ctrl;
+ u32 mmu_ctrl;
+ u32 irqhost;
+ u32 nprot;
+};
+
+/**
+ * struct mali_vpu_lsid - Hardware LSID instance.
+ * @vpu_dev: Back pointer to parent Mali VPU device.
+ * @id: Hardware LSID instance.
+ */
+struct mali_vpu_lsid {
+ struct mali_vpu_dev *vpu_dev;
+ u32 id;
+};
+
+void mali_vpu_lsid_init(struct mali_vpu_lsid *lsid, struct mali_vpu_dev *vpu_dev, u32 id);
+int mali_vpu_lsid_map(struct mali_vpu_lsid *lsid, struct mali_vpu_lsid_pcb *pcb);
+void mali_vpu_lsid_unmap(struct mali_vpu_lsid *lsid, struct mali_vpu_lsid_pcb *pcb);
+int mali_vpu_lsid_jobqueue_add(struct mali_vpu_lsid *lsid, u32 ncores);
+void mali_vpu_lsid_jobqueue_remove(struct mali_vpu_lsid *lsid);
+void mali_vpu_lsid_send_irq(struct mali_vpu_lsid *lsid);
+void mali_vpu_lsid_flush_mmu(struct mali_vpu_lsid *lsid);
+int mali_vpu_lsid_terminate(struct mali_vpu_lsid *lsid);
+
+#endif /* __MALI_VPU_LSID_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
index b7883b1ea47f3..f2386bdc4943a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
@@ -14,8 +14,32 @@
*
* Offsets are relative to the VPU MMIO base.
*/
-#define MALI_VPU_REG_VERSION 0x00000
-#define MALI_VPU_REG_NCORES 0x00008
+#define MALI_VPU_REG_VERSION 0x00000
+#define MALI_VPU_REG_ENABLE 0x00004
+#define MALI_VPU_REG_NCORES 0x00008
+#define MALI_VPU_REG_JOBQUEUE 0x00014
+
+#define MALI_VPU_LSID_BASE 0x00200
+#define MALI_VPU_LSID_STRIDE 0x40
+
+/*
+ * Per-LSID registers.
+ * Address = MALI_VPU_LSID_BASE + (MALI_VPU_LSID_STRIDE * lsid) + per-LSID offset
+ * Each LSID has its own set of these registers, which are identical in layout.
+ */
+#define MALI_VPU_REG_LSID_CTRL 0x00
+#define MALI_VPU_REG_LSID_MMU_CTRL 0x04
+#define MALI_VPU_REG_LSID_NPROT 0x08
+#define MALI_VPU_REG_LSID_ALLOC 0x0C
+#define MALI_VPU_REG_LSID_FLUSH_ALL 0x10
+#define MALI_VPU_REG_LSID_SCHED 0x14
+#define MALI_VPU_REG_LSID_TERMINATE 0x18
+#define MALI_VPU_REG_LSID_IRQVE 0x1C
+#define MALI_VPU_REG_LSID_IRQHOST 0x20
+#define MALI_VPU_REG_LSID_BUSATTR_0 0x30
+#define MALI_VPU_REG_LSID_BUSATTR_1 0x34
+#define MALI_VPU_REG_LSID_BUSATTR_2 0x38
+#define MALI_VPU_REG_LSID_BUSATTR_3 0x3C
/* Version register fields */
#define MALI_VPU_VERSION_PRODUCT_ID_MASK GENMASK(31, 16)
@@ -23,9 +47,25 @@
#define MALI_VPU_VERSION_PATCH_MASK GENMASK(7, 0)
#define MALI_VPU_PID_SUFFIX_MASK GENMASK(15, 12)
+/* JOBQUEUE register: 4 entries x 8 bits (LSID[3:0], NCORES-1[7:4]) */
+#define MALI_VPU_JOBQUEUE_LSID_BITS 4
+#define MALI_VPU_JOBQUEUE_NCORES_BITS 4
+#define MALI_VPU_JOBQUEUE_JOB_STRIDE 8
+#define MALI_VPU_JOBQUEUE_JOB_INVALID 0xf
+#define MALI_VPU_JOBQUEUE_NJOBS 4
+
+#define MALI_VPU_ALLOC_FREE 0
+#define MALI_VPU_ALLOC_NON_PROTECTED 1
+#define MALI_VPU_ALLOC_PROTECTED 2
+
static inline u32 mali_vpu_read_reg(void __iomem *regs_base, u32 off)
{
return readl(regs_base + off);
}
+static inline void mali_vpu_write_reg(void __iomem *regs_base, u32 off, u32 val)
+{
+ writel(val, regs_base + off);
+}
+
#endif /* __MALI_VPU_REGISTERS_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 17/28] media: mali-vpu: add single session scheduler
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (15 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 16/28] media: mali-vpu: add LSID hardware support Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 18/28] media: mali-vpu: handle firmware interrupts Charvi Mehta
` (10 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add a scheduler with a simple policy that assigns all available VPU cores
to a single session mapped at LSID 0. Provide operations for switching
the session in, signalling firmware, flushing its MMU context and
terminating execution.
Initialize the scheduler during probe and prepare the session hardware
context when its firmware is loaded.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
drivers/media/platform/arm/mali-vpu/Makefile | 3 +-
.../platform/arm/mali-vpu/mali-vpu-device.h | 4 +
.../platform/arm/mali-vpu/mali-vpu-lsid.h | 3 +
.../platform/arm/mali-vpu/mali-vpu-mmu.c | 6 +
.../platform/arm/mali-vpu/mali-vpu-mmu.h | 2 +
.../platform/arm/mali-vpu/mali-vpu-probe.c | 11 +-
.../arm/mali-vpu/mali-vpu-registers.h | 9 +
.../arm/mali-vpu/mali-vpu-scheduler.c | 224 ++++++++++++++++++
.../arm/mali-vpu/mali-vpu-scheduler.h | 57 +++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 14 +-
.../platform/arm/mali-vpu/mali-vpu-session.h | 6 +
.../arm/mali-vpu/mali-vpu-variant-v76.c | 2 +
.../platform/arm/mali-vpu/mali-vpu-variant.h | 2 +
13 files changed, 334 insertions(+), 9 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
diff --git a/drivers/media/platform/arm/mali-vpu/Makefile b/drivers/media/platform/arm/mali-vpu/Makefile
index 51d49d815f1fb..5860cda6055cc 100644
--- a/drivers/media/platform/arm/mali-vpu/Makefile
+++ b/drivers/media/platform/arm/mali-vpu/Makefile
@@ -12,6 +12,7 @@ mali-vpu-objs := mali-vpu-probe.o \
mali-vpu-mmu.o \
mali-vpu-firmware-cache.o \
mali-vpu-firmware.o \
- mali-vpu-lsid.o
+ mali-vpu-lsid.o \
+ mali-vpu-scheduler.o
obj-$(CONFIG_VIDEO_MALI_VPU) += mali-vpu.o
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index a35c1b901aa3f..9be36e47d0879 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -9,6 +9,8 @@
#include <linux/device.h>
#include <media/v4l2-device.h>
+#include "mali-vpu-scheduler.h"
+
#define MALI_VPU_DRV_NAME "mali-vpu"
struct clk_bulk_data;
@@ -40,6 +42,7 @@ enum mali_vpu_session_role {
* @vdev_dec: Video device structure for decoder.
* @m2m_dev: V4L2 mem2mem device structure for mali vpu.
* @fw_cache: Firmware cache for sharing or reusing firmware blobs.
+ * @sched: Mali VPU hardware scheduler state.
*/
struct mali_vpu_dev {
struct device *dev;
@@ -56,6 +59,7 @@ struct mali_vpu_dev {
struct video_device *vdev_dec;
struct v4l2_m2m_dev *m2m_dev;
struct mali_vpu_fw_cache *fw_cache;
+ struct mali_vpu_sched sched;
};
#endif /* __MALI_VPU_DEVICE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
index 97c32f9ac9aa0..57733304d4e17 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-lsid.h
@@ -9,6 +9,7 @@
#include <linux/types.h>
struct mali_vpu_dev;
+struct mali_vpu_sched_session;
/*
* A local session identifier (LSID) represents a hardware execution
@@ -36,10 +37,12 @@ struct mali_vpu_lsid_pcb {
/**
* struct mali_vpu_lsid - Hardware LSID instance.
* @vpu_dev: Back pointer to parent Mali VPU device.
+ * @sched_sess: Scheduler session currently mapped to this LSID, if any.
* @id: Hardware LSID instance.
*/
struct mali_vpu_lsid {
struct mali_vpu_dev *vpu_dev;
+ struct mali_vpu_sched_session *sched_sess;
u32 id;
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
index 00cf5251eff5e..115539fe1cb3e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c
@@ -691,3 +691,9 @@ int mali_vpu_mmu_write(struct mali_vpu_mmu_pages *pages,
return 0;
}
+
+mali_vpu_mmu_pte mali_vpu_mmu_get_root_pte(const struct mali_vpu_mmu *mmu)
+{
+ return mali_vpu_mmu_set_pte(MALI_VPU_ATTR_PRIVATE, mmu->root.dma,
+ MALI_VPU_ACCESS_READ_WRITE);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
index 6ad645a2dc6bb..4baf1daaf2983 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.h
@@ -149,4 +149,6 @@ int mali_vpu_mmu_map_dma(struct mali_vpu_mmu *mmu,
void mali_vpu_mmu_unmap_va(struct mali_vpu_mmu *mmu, mali_vpu_mmu_va va, size_t size);
int mali_vpu_mmu_write(struct mali_vpu_mmu_pages *pages, const void *data, size_t size);
+mali_vpu_mmu_pte mali_vpu_mmu_get_root_pte(const struct mali_vpu_mmu *mmu);
+
#endif /* __MALI_VPU_MMU_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 123ff4d8fc937..0c7ea9035fb50 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -208,17 +208,21 @@ static int mali_vpu_probe(struct platform_device *pdev)
goto err_rst_clk;
}
+ ret = mali_vpu_sched_init(&vpu_dev->sched, vpu_dev);
+ if (ret)
+ goto err_rst_clk;
+
vpu_dev->dma_mask = vpu_dev->variant->dma_mask;
ret = dma_set_mask_and_coherent(vpu_dev->dev, vpu_dev->dma_mask);
if (ret)
- goto err_rst_clk;
+ goto err_cleanup_sched;
dma_set_max_seg_size(vpu_dev->dev, SZ_2G);
ret = v4l2_device_register(&pdev->dev, &vpu_dev->v4l2_dev);
if (ret) {
dev_err(vpu_dev->dev, "failed to register V4L2 device\n");
- goto err_rst_clk;
+ goto err_cleanup_sched;
}
vpu_dev->m2m_dev = v4l2_m2m_init(&mali_vpu_m2m_ops);
@@ -251,6 +255,8 @@ static int mali_vpu_probe(struct platform_device *pdev)
v4l2_m2m_release(vpu_dev->m2m_dev);
err_unreg_v4l2:
v4l2_device_unregister(&vpu_dev->v4l2_dev);
+err_cleanup_sched:
+ mali_vpu_sched_cleanup(&vpu_dev->sched);
err_rst_clk:
mali_vpu_assert_reset(vpu_dev);
err_clk:
@@ -268,6 +274,7 @@ static void mali_vpu_remove(struct platform_device *pdev)
v4l2_m2m_release(vpu_dev->m2m_dev);
v4l2_device_unregister(&vpu_dev->v4l2_dev);
+ mali_vpu_sched_cleanup(&vpu_dev->sched);
mali_vpu_assert_reset(vpu_dev);
mali_vpu_disable_clk(vpu_dev);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
index f2386bdc4943a..1a3e0fcdba4c6 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
@@ -17,7 +17,10 @@
#define MALI_VPU_REG_VERSION 0x00000
#define MALI_VPU_REG_ENABLE 0x00004
#define MALI_VPU_REG_NCORES 0x00008
+#define MALI_VPU_REG_NLSID 0x0000C
#define MALI_VPU_REG_JOBQUEUE 0x00014
+#define MALI_VPU_REG_CLKFORCE 0x00024
+#define MALI_VPU_REG_RESET 0x00050
#define MALI_VPU_LSID_BASE 0x00200
#define MALI_VPU_LSID_STRIDE 0x40
@@ -54,6 +57,12 @@
#define MALI_VPU_JOBQUEUE_JOB_INVALID 0xf
#define MALI_VPU_JOBQUEUE_NJOBS 4
+/* IRQVE contains one interrupt bit per LSID. */
+#define MALI_VPU_IRQVE_NUM_BITS 32
+
+/* CTRL register fields */
+#define MALI_VPU_CTRL_MAXCORES GENMASK(11, 8)
+
#define MALI_VPU_ALLOC_FREE 0
#define MALI_VPU_ALLOC_NON_PROTECTED 1
#define MALI_VPU_ALLOC_PROTECTED 2
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
new file mode 100644
index 0000000000000..cae9ac0093be9
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/string.h>
+
+#include "mali-vpu-device.h"
+#include "mali-vpu-mmu.h"
+#include "mali-vpu-registers.h"
+#include "mali-vpu-scheduler.h"
+#include "mali-vpu-session.h"
+#include "mali-vpu-variant.h"
+
+int mali_vpu_sched_init(struct mali_vpu_sched *sched, struct mali_vpu_dev *vpu_dev)
+{
+ u32 hw_nlsid, variant_nlsid;
+ u32 lsid, max_irq_lsid;
+
+ if (!vpu_dev->variant || !vpu_dev->variant->max_lsid)
+ return -EINVAL;
+
+ hw_nlsid = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_NLSID);
+ variant_nlsid = vpu_dev->variant->max_lsid;
+
+ max_irq_lsid = min_t(u32, BITS_PER_LONG, MALI_VPU_IRQVE_NUM_BITS);
+ if (variant_nlsid > max_irq_lsid) {
+ dev_err(vpu_dev->dev,
+ "variant supports %u LSIDs, but IRQ handling supports only %u\n",
+ variant_nlsid, max_irq_lsid);
+ return -EINVAL;
+ }
+
+ if (!hw_nlsid || hw_nlsid > variant_nlsid) {
+ dev_err(vpu_dev->dev,
+ "hardware reports %u LSIDs, but variant supports only %u\n",
+ hw_nlsid, variant_nlsid);
+ return -EINVAL;
+ }
+
+ sched->nlsid = hw_nlsid;
+ sched->lsid_list = devm_kcalloc(vpu_dev->dev, sched->nlsid,
+ sizeof(*sched->lsid_list), GFP_KERNEL);
+ if (!sched->lsid_list)
+ return -ENOMEM;
+
+ sched->vpu_dev = vpu_dev;
+ mutex_init(&sched->lock);
+
+ for (lsid = 0; lsid < sched->nlsid; lsid++)
+ mali_vpu_lsid_init(&sched->lsid_list[lsid], vpu_dev, lsid);
+
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_RESET, 1);
+ mali_vpu_write_reg(vpu_dev->regs_base, MALI_VPU_REG_CLKFORCE, 0);
+
+ return 0;
+}
+
+void mali_vpu_sched_cleanup(struct mali_vpu_sched *sched)
+{
+ mutex_destroy(&sched->lock);
+}
+
+int mali_vpu_sched_session_prepare(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_session *sess)
+{
+ u32 hw_ncores, ctrl = 0;
+
+ if (sched_sess->lsid)
+ return -EBUSY;
+
+ if (!sess->mmu.root.cpu)
+ return -EINVAL;
+
+ /* Simple scheduling policy - all cores for all sessions */
+ hw_ncores = mali_vpu_read_reg(sess->vpu_dev->regs_base, MALI_VPU_REG_NCORES);
+ if (!hw_ncores ||
+ hw_ncores > BIT(MALI_VPU_JOBQUEUE_NCORES_BITS))
+ return -EINVAL;
+
+ ctrl |= FIELD_PREP(MALI_VPU_CTRL_MAXCORES, hw_ncores);
+
+ memset(sched_sess, 0, sizeof(*sched_sess));
+
+ sched_sess->core_count = hw_ncores;
+ sched_sess->pcb.ctrl = ctrl;
+ sched_sess->pcb.mmu_ctrl = mali_vpu_mmu_get_root_pte(&sess->mmu);
+ sched_sess->pcb.nprot = 1;
+
+ sess->ncores = sched_sess->core_count;
+
+ return 0;
+}
+
+static void mali_vpu_sched_reset_locked(struct mali_vpu_sched *sched)
+{
+ struct mali_vpu_lsid *lsid;
+ u32 i;
+
+ lockdep_assert_held(&sched->lock);
+
+ mali_vpu_write_reg(sched->vpu_dev->regs_base, MALI_VPU_REG_RESET, 1);
+
+ for (i = 0; i < sched->nlsid; i++) {
+ lsid = &sched->lsid_list[i];
+
+ if (lsid->sched_sess)
+ lsid->sched_sess->lsid = NULL;
+
+ lsid->sched_sess = NULL;
+ }
+
+ mali_vpu_write_reg(sched->vpu_dev->regs_base, MALI_VPU_REG_CLKFORCE, 0);
+}
+
+int mali_vpu_sched_switch_in(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched)
+{
+ struct mali_vpu_lsid *lsid;
+ struct mali_vpu_sched_session *owner;
+ bool mapped_here = false;
+ int ret = 0;
+
+ mutex_lock(&sched->lock);
+
+ if (!sched->nlsid) {
+ ret = -ENODEV;
+ goto err_unlock;
+ }
+
+ /* Allow only one active session as a simple policy */
+ lsid = &sched->lsid_list[0];
+
+ owner = lsid->sched_sess;
+ if (owner && owner != sched_sess) {
+ ret = -EBUSY;
+ goto err_unlock;
+ }
+
+ if (sched_sess->lsid && sched_sess->lsid != lsid) {
+ ret = -EINVAL;
+ goto err_unlock;
+ }
+
+ if (!sched_sess->lsid) {
+ /* Map the session to an lsid first */
+ ret = mali_vpu_lsid_map(lsid, &sched_sess->pcb);
+ if (ret) {
+ if (ret == -ETIMEDOUT)
+ mali_vpu_sched_reset_locked(sched);
+ goto err_unlock;
+ }
+
+ lsid->sched_sess = sched_sess;
+ sched_sess->lsid = lsid;
+ mapped_here = true;
+ }
+
+ /* Add the session to jobqueue */
+ ret = mali_vpu_lsid_jobqueue_add(sched_sess->lsid, sched_sess->core_count);
+ if (ret && mapped_here) {
+ mali_vpu_lsid_unmap(lsid, &sched_sess->pcb);
+ lsid->sched_sess = NULL;
+ sched_sess->lsid = NULL;
+ }
+
+err_unlock:
+ mutex_unlock(&sched->lock);
+ return ret;
+}
+
+void mali_vpu_sched_send_irq(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched)
+{
+ mutex_lock(&sched->lock);
+ if (!sched_sess->lsid)
+ sched_sess->pcb.irqhost = 1;
+ else
+ mali_vpu_lsid_send_irq(sched_sess->lsid);
+ mutex_unlock(&sched->lock);
+}
+
+void mali_vpu_sched_flush_mmu(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched)
+{
+ mutex_lock(&sched->lock);
+ if (sched_sess->lsid)
+ mali_vpu_lsid_flush_mmu(sched_sess->lsid);
+ mutex_unlock(&sched->lock);
+}
+
+int mali_vpu_sched_session_terminate(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched)
+{
+ struct mali_vpu_lsid *lsid;
+ int ret = 0;
+
+ mutex_lock(&sched->lock);
+
+ lsid = sched_sess->lsid;
+ if (!lsid)
+ goto unlock;
+
+ mali_vpu_lsid_jobqueue_remove(sched_sess->lsid);
+
+ ret = mali_vpu_lsid_terminate(sched_sess->lsid);
+ if (ret) {
+ dev_err(sched->vpu_dev->dev,
+ "failed to terminate LSID %u: %d; resetting VPU\n",
+ lsid->id, ret);
+ mali_vpu_sched_reset_locked(sched);
+ goto unlock;
+ }
+
+ mali_vpu_lsid_unmap(lsid, &sched_sess->pcb);
+ lsid->sched_sess = NULL;
+ sched_sess->lsid = NULL;
+
+unlock:
+ mutex_unlock(&sched->lock);
+
+ return ret;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
new file mode 100644
index 0000000000000..23167aef6ff62
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_SCHEDULER_H__
+#define __MALI_VPU_SCHEDULER_H__
+
+#include <linux/mutex.h>
+
+#include "mali-vpu-lsid.h"
+
+struct mali_vpu_dev;
+struct mali_vpu_session;
+
+/**
+ * struct mali_vpu_sched_session - Client session class.
+ * @lsid: Pointer to LSID the session is mapped to.
+ * @pcb: LSID process control block programmed when the session is mapped.
+ * @core_count: Number of cores assigned to this scheduled session.
+ *
+ * This struct is used to keep track of session's information needed by the scheduler.
+ */
+struct mali_vpu_sched_session {
+ struct mali_vpu_lsid *lsid;
+ struct mali_vpu_lsid_pcb pcb;
+ u32 core_count;
+};
+
+/**
+ * struct mali_vpu_sched - Scheduler class.
+ * @vpu_dev: Pointer to mali_vpu_dev structure.
+ * @lock: Serialise access to sched resources.
+ * @lsid_list: Array of LSID instances.
+ * @nlsid: Number of LSIDs.
+ */
+struct mali_vpu_sched {
+ struct mali_vpu_dev *vpu_dev;
+ struct mutex lock; /* Serialise access to sched resources. */
+ struct mali_vpu_lsid *lsid_list;
+ u32 nlsid;
+};
+
+int mali_vpu_sched_init(struct mali_vpu_sched *sched, struct mali_vpu_dev *dev);
+void mali_vpu_sched_cleanup(struct mali_vpu_sched *sched);
+int mali_vpu_sched_session_prepare(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_session *sess);
+int mali_vpu_sched_switch_in(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched);
+void mali_vpu_sched_send_irq(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched);
+void mali_vpu_sched_flush_mmu(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched);
+int mali_vpu_sched_session_terminate(struct mali_vpu_sched_session *sched_sess,
+ struct mali_vpu_sched *sched);
+
+#endif /* __MALI_VPU_SCHEDULER_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 36197e82461be..c5579edfe627f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -16,7 +16,6 @@
#include "mali-vpu-firmware.h"
#include "mali-vpu-firmware-cache.h"
#include "mali-vpu-formats.h"
-#include "mali-vpu-registers.h"
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
@@ -216,7 +215,6 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
bool same_thread)
{
struct mali_vpu_session *sess = arg;
- u32 ncores;
int ret;
if (!same_thread)
@@ -239,13 +237,17 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
sess->fw_bin = vpu_fw_bin;
/*
- * Get total number of cores on VPU element.
- * This will be scheduler decision in subsequent patches.
+ * Prepare the scheduler's saved session hardware context.
+ * Scheduler assigns the session's core count.
*/
- ncores = mali_vpu_read_reg(sess->vpu_dev->regs_base, MALI_VPU_REG_NCORES);
+ ret = mali_vpu_sched_session_prepare(&sess->sched_sess, sess);
+ if (ret) {
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ goto err_cache_put;
+ }
ret = mali_vpu_fw_construct(&sess->fw, sess->fw_bin, sess->vpu_dev->dev,
- &sess->mmu, ncores);
+ &sess->mmu, sess->ncores);
if (ret) {
dev_err(sess->vpu_dev->dev,
"failed to initialise firmware: %d\n", ret);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index f0aa98005e92a..bf5923494773b 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -20,6 +20,7 @@
struct v4l2_m2m_ctx;
struct mali_vpu_fw_bin;
struct mali_vpu_fw_event;
+struct mali_vpu_sched_session;
/**
* struct mali_vpu_port - format state of a src/dst queue in a session.
@@ -77,6 +78,8 @@ enum mali_vpu_session_fw_bin_state {
* @fw_event: Firmware binary load event callback structure for this session.
* @fw_bin_state: Session firmware binary status.
* @fw_load_done: Completion used to synchronize teardown with an in-flight firmware load.
+ * @sched_sess: Scheduler state used to map this session to an LSID.
+ * @ncores: Number of cores assigned to this session by scheduler.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
@@ -100,6 +103,9 @@ struct mali_vpu_session {
struct mali_vpu_fw_event *fw_event;
enum mali_vpu_session_fw_bin_state fw_bin_state;
struct completion fw_load_done;
+
+ struct mali_vpu_sched_session sched_sess;
+ u32 ncores;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
index 0a7481d97c717..30ff5c25417e0 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant-v76.c
@@ -18,6 +18,7 @@
#define MALI_VPU_HW_PID_V76 0x56627
#define MALI_VPU_FW_VER_MAJOR_V76 1
+#define MALI_VPU_MAX_LSID_V76 4
#define MALI_V76_MAX_FRAME_WIDTH 8192
#define MALI_V76_MAX_FRAME_HEIGHT 8192
@@ -153,6 +154,7 @@ const struct mali_vpu_variant mali_vpu_v76_variant = {
.dma_mask = DMA_BIT_MASK(40),
.expected_hw_pid = MALI_VPU_HW_PID_V76,
.fw_ver_major = MALI_VPU_FW_VER_MAJOR_V76,
+ .max_lsid = MALI_VPU_MAX_LSID_V76,
.rst_names = mali_v76_rst_names,
.num_rsts = ARRAY_SIZE(mali_v76_rst_names),
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
index 3c37cd70bd81f..ac0bc3d899877 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-variant.h
@@ -51,6 +51,7 @@ struct mali_vpu_control_caps {
* @expected_hw_pid: Expected hardware product ID read from the version register.
* @dma_mask: DMA addressing capability mask.
* @fw_ver_major: Major firmware version compatible with this variant.
+ * @max_lsid: Maximum number of hardware LSID slots in a VPU element.
* @fmt_caps: Format capability tables for encode/decode paths.
* @frmsz_caps: Frame dimension limits and defaults.
* @ctrl_caps: Control capability tables for encode/decode paths.
@@ -61,6 +62,7 @@ struct mali_vpu_variant {
u32 expected_hw_pid;
u64 dma_mask;
u32 fw_ver_major;
+ u32 max_lsid;
struct mali_vpu_format_caps fmt_caps;
struct mali_vpu_framesize_caps frmsz_caps;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 18/28] media: mali-vpu: handle firmware interrupts
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (16 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 17/28] media: mali-vpu: add single session scheduler Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 19/28] media: mali-vpu: start decoder firmware through LSID scheduling Charvi Mehta
` (9 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Firmware responses and service requests are delivered through per-LSID
interrupts. Route these interrupts to the session currently assigned
to the LSID and process them outside hard IRQ context.
This allows the driver to service firmware RPC requests and consume
responses needed to complete asynchronous firmware commands.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-device.h | 7 ++
.../platform/arm/mali-vpu/mali-vpu-probe.c | 84 +++++++++++++++++-
.../arm/mali-vpu/mali-vpu-registers.h | 1 +
.../arm/mali-vpu/mali-vpu-scheduler.c | 24 ++++++
.../arm/mali-vpu/mali-vpu-scheduler.h | 1 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 86 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.h | 3 +
7 files changed, 205 insertions(+), 1 deletion(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
index 9be36e47d0879..cacdd86fe517e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-device.h
@@ -7,6 +7,7 @@
#define __MALI_VPU_DEVICE_H__
#include <linux/device.h>
+#include <linux/workqueue.h>
#include <media/v4l2-device.h>
#include "mali-vpu-scheduler.h"
@@ -37,6 +38,9 @@ enum mali_vpu_session_role {
* @rst_count: Number of reset controls in @rst_list.
* @dma_mask: DMA mask for the device, indicating supported address width.
* @irq: IRQ number for VPU.
+ * @irqve_mask: Bitmap of pending LSID IRQs.
+ * @irq_wq: Workqueue used to process LSID IRQs outside hard IRQ context.
+ * @irq_work: Work item for deferred LSID IRQ handling.
* @variant: Structure describing variant-specific features and capabilities.
* @v4l2_dev: V4L2 device structure for mali vpu.
* @vdev_dec: Video device structure for decoder.
@@ -53,6 +57,9 @@ struct mali_vpu_dev {
unsigned int rst_count;
u64 dma_mask;
int irq;
+ unsigned long irqve_mask;
+ struct workqueue_struct *irq_wq;
+ struct work_struct irq_work;
const struct mali_vpu_variant *variant;
struct v4l2_device v4l2_dev;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
index 0c7ea9035fb50..b9f9558e76701 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-probe.c
@@ -9,6 +9,7 @@
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/io.h>
+#include <linux/interrupt.h>
#include <linux/mod_devicetable.h>
#include <linux/module.h>
#include <linux/of.h>
@@ -119,6 +120,70 @@ static int mali_vpu_get_resources(struct mali_vpu_dev *vpu_dev)
return mali_vpu_get_resets(vpu_dev);
}
+static irqreturn_t mali_vpu_isr_top(int irq, void *data)
+{
+ struct mali_vpu_dev *vpu_dev = data;
+ u32 nlsid, irqve, nlsid_bound;
+ int ret = IRQ_NONE;
+
+ /*
+ * IRQVE contains one interrupt bit per LSID. Acknowledge each asserted
+ * source in hard IRQ context and record it for deferred firmware response
+ * and RPC handling, which may sleep.
+ */
+ nlsid = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_NLSID);
+ irqve = mali_vpu_read_reg(vpu_dev->regs_base, MALI_VPU_REG_IRQVE);
+ nlsid_bound = min_t(u32, nlsid, vpu_dev->variant->max_lsid);
+
+ while (nlsid_bound-- > 0)
+ if ((irqve >> nlsid_bound) & 0x1) {
+ u32 reg_off = MALI_VPU_LSID_BASE +
+ (MALI_VPU_LSID_STRIDE * nlsid_bound) +
+ MALI_VPU_REG_LSID_IRQVE;
+ mali_vpu_write_reg(vpu_dev->regs_base, reg_off, 0);
+ set_bit(nlsid_bound, &vpu_dev->irqve_mask);
+ ret = IRQ_HANDLED;
+ }
+
+ if (ret == IRQ_HANDLED)
+ queue_work(vpu_dev->irq_wq, &vpu_dev->irq_work);
+
+ return ret;
+}
+
+static void mali_vpu_irq_bottom(struct work_struct *work)
+{
+ struct mali_vpu_dev *vpu_dev = container_of(work, struct mali_vpu_dev, irq_work);
+ u32 i;
+
+ for (i = 0; i < vpu_dev->sched.nlsid; i++) {
+ if (test_and_clear_bit(i, &vpu_dev->irqve_mask))
+ mali_vpu_sched_handle_lsid_irq(&vpu_dev->sched, i);
+ }
+}
+
+static int mali_vpu_irq_work_init(struct mali_vpu_dev *vpu_dev)
+{
+ vpu_dev->irqve_mask = 0;
+
+ vpu_dev->irq_wq = alloc_workqueue("%s-irq", WQ_UNBOUND, 1, dev_name(vpu_dev->dev));
+ if (!vpu_dev->irq_wq)
+ return -ENOMEM;
+
+ INIT_WORK(&vpu_dev->irq_work, mali_vpu_irq_bottom);
+ return 0;
+}
+
+static void mali_vpu_irq_work_cleanup(struct mali_vpu_dev *vpu_dev)
+{
+ if (!vpu_dev->irq_wq)
+ return;
+
+ cancel_work_sync(&vpu_dev->irq_work);
+ destroy_workqueue(vpu_dev->irq_wq);
+ vpu_dev->irq_wq = NULL;
+}
+
static int mali_vpu_register_video_device(struct mali_vpu_dev *vpu_dev,
enum mali_vpu_session_role role)
{
@@ -219,10 +284,19 @@ static int mali_vpu_probe(struct platform_device *pdev)
dma_set_max_seg_size(vpu_dev->dev, SZ_2G);
+ ret = mali_vpu_irq_work_init(vpu_dev);
+ if (ret)
+ goto err_cleanup_sched;
+
+ ret = devm_request_irq(&pdev->dev, vpu_dev->irq, mali_vpu_isr_top, 0, pdev->name,
+ vpu_dev);
+ if (ret)
+ goto err_cleanup_irq_work;
+
ret = v4l2_device_register(&pdev->dev, &vpu_dev->v4l2_dev);
if (ret) {
dev_err(vpu_dev->dev, "failed to register V4L2 device\n");
- goto err_cleanup_sched;
+ goto err_free_irq;
}
vpu_dev->m2m_dev = v4l2_m2m_init(&mali_vpu_m2m_ops);
@@ -255,6 +329,10 @@ static int mali_vpu_probe(struct platform_device *pdev)
v4l2_m2m_release(vpu_dev->m2m_dev);
err_unreg_v4l2:
v4l2_device_unregister(&vpu_dev->v4l2_dev);
+err_free_irq:
+ devm_free_irq(vpu_dev->dev, vpu_dev->irq, vpu_dev);
+err_cleanup_irq_work:
+ mali_vpu_irq_work_cleanup(vpu_dev);
err_cleanup_sched:
mali_vpu_sched_cleanup(&vpu_dev->sched);
err_rst_clk:
@@ -270,6 +348,10 @@ static void mali_vpu_remove(struct platform_device *pdev)
struct mali_vpu_dev *vpu_dev = platform_get_drvdata(pdev);
video_unregister_device(vpu_dev->vdev_dec);
+
+ devm_free_irq(vpu_dev->dev, vpu_dev->irq, vpu_dev);
+ mali_vpu_irq_work_cleanup(vpu_dev);
+
mali_vpu_fw_cache_cleanup(vpu_dev->fw_cache);
v4l2_m2m_release(vpu_dev->m2m_dev);
v4l2_device_unregister(&vpu_dev->v4l2_dev);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
index 1a3e0fcdba4c6..467f2cdb4c278 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-registers.h
@@ -19,6 +19,7 @@
#define MALI_VPU_REG_NCORES 0x00008
#define MALI_VPU_REG_NLSID 0x0000C
#define MALI_VPU_REG_JOBQUEUE 0x00014
+#define MALI_VPU_REG_IRQVE 0x00018
#define MALI_VPU_REG_CLKFORCE 0x00024
#define MALI_VPU_REG_RESET 0x00050
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
index cae9ac0093be9..77503f20e4b29 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.c
@@ -222,3 +222,27 @@ int mali_vpu_sched_session_terminate(struct mali_vpu_sched_session *sched_sess,
return ret;
}
+
+void mali_vpu_sched_handle_lsid_irq(struct mali_vpu_sched *sched, u32 lsid)
+{
+ struct mali_vpu_session *sess = NULL;
+ struct mali_vpu_sched_session *sched_sess;
+
+ /* Only single active session policy - always mapped to lsid 0 */
+ if (lsid != 0) {
+ dev_warn(sched->vpu_dev->dev,
+ "unexpected interrupt from LSID %u\n", lsid);
+ return;
+ }
+
+ mutex_lock(&sched->lock);
+ if (sched->nlsid) {
+ sched_sess = sched->lsid_list[lsid].sched_sess;
+ if (sched_sess)
+ sess = container_of(sched_sess, struct mali_vpu_session, sched_sess);
+ }
+ mutex_unlock(&sched->lock);
+
+ if (sess)
+ mali_vpu_session_irq(sess);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
index 23167aef6ff62..ffa93495e8b47 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-scheduler.h
@@ -53,5 +53,6 @@ void mali_vpu_sched_flush_mmu(struct mali_vpu_sched_session *sched_sess,
struct mali_vpu_sched *sched);
int mali_vpu_sched_session_terminate(struct mali_vpu_sched_session *sched_sess,
struct mali_vpu_sched *sched);
+void mali_vpu_sched_handle_lsid_irq(struct mali_vpu_sched *sched, u32 lsid);
#endif /* __MALI_VPU_SCHEDULER_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index c5579edfe627f..09ec6d14f4ad0 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -73,6 +73,56 @@ int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
return mali_vpu_fw_send_set_option(&sess->fw, opt);
}
+static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
+ const struct mali_vpu_fw_msg *fw_msg)
+{
+ switch (fw_msg->code) {
+ case MALI_VPU_FW_CODE_ERROR:
+ dev_warn(sess->vpu_dev->dev, "firmware rsp: Error. code=%u, message=%s\n",
+ fw_msg->data.error.error_code, fw_msg->data.error.message);
+ return -EIO;
+ case MALI_VPU_FW_CODE_JOB:
+ /*
+ * The session uses an infinite job and does not implement
+ * switch-out scheduling.
+ */
+ return 0;
+ case MALI_VPU_FW_CODE_PONG:
+ /* Receiving the responses completes the diagnostic ping. */
+ return 0;
+ case MALI_VPU_FW_CODE_STATE_CHANGE:
+ sess->fw_state = fw_msg->data.state;
+ return 0;
+ case MALI_VPU_FW_CODE_SWITCH_IN:
+ /* Single session policy: the scheduler already tracks the active LSID mapping. */
+ return 0;
+ case MALI_VPU_FW_CODE_SWITCH_OUT:
+ dev_warn(sess->vpu_dev->dev,
+ "Unexpected fw switch-out response for a single session policy.\n");
+ return 0;
+ case MALI_VPU_FW_CODE_SET_OPTION:
+ /*
+ * The parser validated the confirmation and decremented the
+ * pending_response_count.
+ */
+ return 0;
+ case MALI_VPU_FW_CODE_SET_OPTION_FAIL:
+ /*
+ * The parser printed the error and decremented the
+ * pending_response_count.
+ */
+ return -EINVAL;
+ case MALI_VPU_FW_CODE_IDLE:
+ dev_dbg(sess->vpu_dev->dev, "fw entered Idle/sleeping state.\n");
+ return 0;
+ case MALI_VPU_FW_CODE_UNKNOWN:
+ case MALI_VPU_FW_CODE_MAX:
+ default:
+ dev_warn(sess->vpu_dev->dev, "unknown fw msg code=%u\n", fw_msg->code);
+ return -EPROTO;
+ }
+}
+
int mali_vpu_session_init(struct mali_vpu_session *sess)
{
int ret;
@@ -120,6 +170,7 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
sess->port_src.stream_requested = false;
sess->port_dst.stream_requested = false;
init_completion(&sess->fw_load_done);
+ sess->fw_state = MALI_VPU_FW_STATE_STOPPED;
return 0;
@@ -321,3 +372,38 @@ int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess)
return 0;
}
+
+void mali_vpu_session_irq(struct mali_vpu_session *sess)
+{
+ struct mali_vpu_fw_msg msg;
+ bool flush_mmu = false;
+ int ret;
+
+ mutex_lock(&sess->vq_lock);
+
+ ret = mali_vpu_fw_handle_rpc(&sess->fw, &flush_mmu);
+ if (ret) {
+ if (flush_mmu)
+ mali_vpu_sched_flush_mmu(&sess->sched_sess, &sess->vpu_dev->sched);
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+
+ if (ret < 0)
+ goto unlock_mutex;
+ }
+
+ do {
+ ret = mali_vpu_fw_parse_msg(&sess->fw, &msg);
+ if (ret < 0)
+ goto unlock_mutex;
+
+ if (!ret)
+ break;
+
+ ret = mali_vpu_session_handle_fw_msg(sess, &msg);
+ if (ret)
+ goto unlock_mutex;
+ } while (ret >= 0);
+
+unlock_mutex:
+ mutex_unlock(&sess->vq_lock);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index bf5923494773b..e6b9de3b272cf 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -78,6 +78,7 @@ enum mali_vpu_session_fw_bin_state {
* @fw_event: Firmware binary load event callback structure for this session.
* @fw_bin_state: Session firmware binary status.
* @fw_load_done: Completion used to synchronize teardown with an in-flight firmware load.
+ * @fw_state: Last state confirmed by the firmware.
* @sched_sess: Scheduler state used to map this session to an LSID.
* @ncores: Number of cores assigned to this session by scheduler.
*/
@@ -103,6 +104,7 @@ struct mali_vpu_session {
struct mali_vpu_fw_event *fw_event;
enum mali_vpu_session_fw_bin_state fw_bin_state;
struct completion fw_load_done;
+ enum mali_vpu_fw_state fw_state;
struct mali_vpu_sched_session sched_sess;
u32 ncores;
@@ -113,6 +115,7 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess);
int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess);
int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
struct mali_vpu_fw_set_option *opt);
+void mali_vpu_session_irq(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 19/28] media: mali-vpu: start decoder firmware through LSID scheduling
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (17 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 18/28] media: mali-vpu: handle firmware interrupts Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 16:03 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 20/28] media: mali-vpu: implement vb2 buffer queue handling Charvi Mehta
` (8 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Connect decoder firmware execution to queue preparation and asynchronous
firmware loading. Start the decoder when OUTPUT requests streaming,
without requiring CAPTURE to be streaming.
Activate the LSID and wait for pending option responses before requesting
the firmware running state. Track requested and confirmed firmware states
and coordinate shutdown with firmware responses and interrupt processing.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-formats.c | 7 +
.../platform/arm/mali-vpu/mali-vpu-formats.h | 1 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 297 +++++++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-session.h | 12 +
.../platform/arm/mali-vpu/mali-vpu-vb2.c | 46 ++-
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 2 +
6 files changed, 356 insertions(+), 9 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
index 57aa1b3baf40f..072e3b4aa9473 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.c
@@ -20,6 +20,13 @@ bool mali_vpu_is_coded_q(const struct mali_vpu_session *sess, enum v4l2_buf_type
return V4L2_TYPE_IS_CAPTURE(type);
}
+bool mali_vpu_is_raw_q(const struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ if (sess->role == MALI_VPU_DECODER)
+ return V4L2_TYPE_IS_CAPTURE(type);
+ return V4L2_TYPE_IS_OUTPUT(type);
+}
+
const struct mali_vpu_format_desc *
mali_vpu_find_format(struct mali_vpu_session *sess, u32 pixfmt,
enum v4l2_buf_type type)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
index 37198325ee481..6901e295d7c33 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-formats.h
@@ -34,5 +34,6 @@ int mali_vpu_fill_pixmp_fmt(const struct mali_vpu_format_desc *fmt,
u32 width, u32 height,
struct v4l2_pix_format_mplane *pix_mp);
bool mali_vpu_is_coded_q(const struct mali_vpu_session *sess, enum v4l2_buf_type type);
+bool mali_vpu_is_raw_q(const struct mali_vpu_session *sess, enum v4l2_buf_type type);
#endif /* __MALI_VPU_FORMATS_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 09ec6d14f4ad0..6d22a5d479008 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -19,8 +19,15 @@
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
+static const unsigned int mali_vpu_fw_response_timeout_ms = 10000;
static const unsigned int mali_vpu_fw_load_timeout_ms = 10000;
+static struct mali_vpu_port *
+mali_vpu_session_get_port(struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ return V4L2_TYPE_IS_OUTPUT(type) ? &sess->port_src : &sess->port_dst;
+}
+
static int mali_vpu_set_port_fmt(struct mali_vpu_port *port,
enum v4l2_buf_type type,
u32 width, u32 height)
@@ -67,10 +74,91 @@ mali_vpu_both_ports_unrequested(struct mali_vpu_session *sess)
!sess->port_dst.stream_requested;
}
+static bool mali_vpu_session_can_stream(const struct mali_vpu_session *sess)
+{
+ /*
+ * Start a decoder with OUTPUT/src port only, so firmware can
+ * parse the bitstream and report its decoded format.
+ * An encoder requires both ports.
+ *
+ * Use the driver's preparation state because VB2 sets its streaming
+ * flag only after prepare_streaming() and initial buffer handoff.
+ */
+ if (sess->role == MALI_VPU_DECODER)
+ return sess->port_src.stream_requested;
+
+ return sess->port_src.stream_requested &&
+ sess->port_dst.stream_requested;
+}
+
+static int mali_vpu_session_fw_send_job(struct mali_vpu_session *sess, u16 frames)
+{
+ int ret;
+
+ ret = mali_vpu_fw_send_job(&sess->fw, sess->ncores, frames);
+ if (ret)
+ return ret;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+ return 0;
+}
+
+static int mali_vpu_session_fw_send_ping(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ ret = mali_vpu_fw_send_ping(&sess->fw);
+ if (ret)
+ return ret;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+ return 0;
+}
+
+static int mali_vpu_session_fw_send_state(struct mali_vpu_session *sess,
+ enum mali_vpu_fw_state new_state)
+{
+ int ret;
+
+ if (sess->fw_requested_state != sess->fw_state)
+ /* There is a pending request: is this one duplicate? */
+ return sess->fw_requested_state == new_state ? 0 : -EBUSY;
+
+ if (sess->fw_state == new_state)
+ return 0;
+
+ ret = mali_vpu_fw_send_state_change(&sess->fw, new_state);
+ if (ret)
+ return ret;
+
+ sess->fw_requested_state = new_state;
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+
+ return 0;
+}
+
int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
struct mali_vpu_fw_set_option *opt)
{
- return mali_vpu_fw_send_set_option(&sess->fw, opt);
+ int ret;
+
+ ret = mali_vpu_fw_send_set_option(&sess->fw, opt);
+ if (ret)
+ return ret;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+ return 0;
+}
+
+static int mali_vpu_session_sched_activate(struct mali_vpu_session *sess)
+{
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (sess->sched_sess.lsid)
+ return 0;
+
+ /* This maps LSID 0 and makes any staged IRQHOST visible. */
+ return mali_vpu_sched_switch_in(&sess->sched_sess, &sess->vpu_dev->sched);
}
static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
@@ -123,6 +211,102 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
}
}
+static int mali_vpu_session_wait_pending(struct mali_vpu_session *sess)
+{
+ long timeout;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ while (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
+ sess->fw.pending_response_count) {
+ mutex_unlock(&sess->vq_lock);
+
+ timeout = wait_event_timeout(sess->fw_waitq,
+ READ_ONCE(sess->fw_bin_state) !=
+ MALI_VPU_SESSION_FW_BIN_READY ||
+ !READ_ONCE(sess->fw.pending_response_count),
+ msecs_to_jiffies(mali_vpu_fw_response_timeout_ms));
+
+ mutex_lock(&sess->vq_lock);
+
+ if (!timeout && sess->fw.pending_response_count)
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_session_fw_init(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (sess->fw_requested_state != sess->fw_state) {
+ if (sess->fw_requested_state != MALI_VPU_FW_STATE_RUNNING)
+ return -EBUSY;
+
+ return mali_vpu_session_sched_activate(sess);
+ }
+
+ if (sess->fw_state == MALI_VPU_FW_STATE_RUNNING)
+ return 0;
+
+ ret = mali_vpu_session_fw_send_job(sess, 0);
+ if (ret)
+ return ret;
+
+ /*
+ * Map the LSID and allow the stopped firmware to process the
+ * queued SET_OPTION requests before it can be RUNNING.
+ */
+ ret = mali_vpu_session_sched_activate(sess);
+ if (ret)
+ return ret;
+
+ ret = mali_vpu_session_fw_send_ping(sess);
+ if (ret)
+ return ret;
+
+ ret = mali_vpu_apply_staged_controls(sess);
+ if (ret)
+ return ret;
+
+ /*
+ * SET_OPTION is only valid while stopped. Wait for pending option
+ * responses before requesting RUNNING.
+ */
+ ret = mali_vpu_session_wait_pending(sess);
+ if (ret)
+ return ret;
+
+ /* Waiting drops vq_lock, so streaming may have been withdrawn. */
+ if (!mali_vpu_session_can_stream(sess))
+ return 0;
+
+ return mali_vpu_session_fw_send_state(sess, MALI_VPU_FW_STATE_RUNNING);
+}
+
+static int mali_vpu_session_fw_stop(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ ret = mali_vpu_session_wait_pending(sess);
+ if (ret)
+ return ret;
+
+ if (sess->fw_state == MALI_VPU_FW_STATE_STOPPED)
+ return 0;
+
+ ret = mali_vpu_session_fw_send_state(sess, MALI_VPU_FW_STATE_STOPPED);
+ if (ret)
+ return ret;
+
+ return mali_vpu_session_wait_pending(sess);
+}
+
int mali_vpu_session_init(struct mali_vpu_session *sess)
{
int ret;
@@ -170,7 +354,9 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
sess->port_src.stream_requested = false;
sess->port_dst.stream_requested = false;
init_completion(&sess->fw_load_done);
+ init_waitqueue_head(&sess->fw_waitq);
sess->fw_state = MALI_VPU_FW_STATE_STOPPED;
+ sess->fw_requested_state = MALI_VPU_FW_STATE_STOPPED;
return 0;
@@ -208,6 +394,8 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
{
struct mali_vpu_dev *vpu_dev = sess->vpu_dev;
int fw_load_ret = 0;
+ int fw_stop_ret = 0;
+ int terminate_ret;
bool waiter_cancelled = false;
mutex_lock(&sess->vq_lock);
@@ -215,7 +403,9 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
sess->port_src.stream_requested = false;
sess->port_dst.stream_requested = false;
- if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_LOADING) {
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_LOADING ||
+ (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
+ !completion_done(&sess->fw_load_done))) {
mutex_unlock(&sess->vq_lock);
if (!wait_for_completion_timeout(&sess->fw_load_done,
@@ -243,6 +433,21 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
}
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY)
+ fw_stop_ret = mali_vpu_session_fw_stop(sess);
+
+ terminate_ret = mali_vpu_sched_session_terminate(&sess->sched_sess,
+ &vpu_dev->sched);
+ mutex_unlock(&sess->vq_lock);
+
+ /*
+ * The LSID is no longer routed to this session. Drain a worker that
+ * may have resolved sess before it was detached.
+ */
+ flush_work(&vpu_dev->irq_work);
+
+ mutex_lock(&sess->vq_lock);
+
mali_vpu_session_release_fw_bin(sess);
if (sess->m2m_ctx) {
@@ -258,7 +463,72 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
mali_vpu_ctrls_cleanup(sess);
mutex_destroy(&sess->vq_lock);
- return fw_load_ret;
+ if (fw_load_ret)
+ return fw_load_ret;
+
+ if (fw_stop_ret)
+ return fw_stop_ret;
+
+ return terminate_ret;
+}
+
+int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
+ enum v4l2_buf_type type)
+{
+ struct mali_vpu_port *port = mali_vpu_session_get_port(sess, type);
+ int ret;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (port->stream_requested)
+ return 0;
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
+ sess->fw_requested_state != sess->fw_state &&
+ sess->fw_requested_state != MALI_VPU_FW_STATE_RUNNING)
+ return -EBUSY;
+
+ port->stream_requested = true;
+
+ if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
+ mali_vpu_session_can_stream(sess)) {
+ ret = mali_vpu_session_fw_init(sess);
+ if (ret)
+ goto err_rollback;
+ }
+
+ return 0;
+
+err_rollback:
+ port->stream_requested = false;
+
+ return ret;
+}
+
+int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ struct mali_vpu_port *port = mali_vpu_session_get_port(sess, type);
+ int ret;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (!port->stream_requested)
+ return 0;
+
+ port->stream_requested = false;
+
+ if (sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY ||
+ !sess->sched_sess.lsid)
+ return 0;
+
+ if (mali_vpu_is_raw_q(sess, type))
+ return mali_vpu_session_wait_pending(sess);
+
+ ret = mali_vpu_session_fw_stop(sess);
+ if (ret)
+ return ret;
+
+ return mali_vpu_session_wait_pending(sess);
}
static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
@@ -308,8 +578,27 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_READY;
+ /*
+ * A synchronous callback leaves startup to session preparation.
+ * A delayed callback completes startup itself.
+ */
+ if (!same_thread && mali_vpu_session_can_stream(sess)) {
+ ret = mali_vpu_session_fw_init(sess);
+ if (ret) {
+ dev_err(sess->vpu_dev->dev,
+ "failed to start streaming: %d\n", ret);
+ sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ goto err_fw_cleanup;
+ }
+ }
+
goto out_complete;
+err_fw_cleanup:
+ mali_vpu_fw_cleanup(&sess->fw);
+ sess->fw_state = MALI_VPU_FW_STATE_STOPPED;
+ sess->fw_requested_state = MALI_VPU_FW_STATE_STOPPED;
+
err_cache_put:
mali_vpu_fw_cache_put(vpu_fw_bin);
@@ -405,5 +694,7 @@ void mali_vpu_session_irq(struct mali_vpu_session *sess)
} while (ret >= 0);
unlock_mutex:
+ wake_up(&sess->fw_waitq);
+
mutex_unlock(&sess->vq_lock);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index e6b9de3b272cf..e331965d32fd8 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -47,6 +47,11 @@ struct mali_vpu_port {
* construct the firmware address space before making the binary READY. The
* session retains its binary reference until cleanup.
*
+ * A synchronous callback leaves firmware startup to session preparation.
+ * An asynchronous callback starts firmware if the required ports still request
+ * streaming: OUTPUT for a decoder, both ports for an encoder.
+ * Binary READY and firmware RUNNING are separate states.
+ *
* Session cleanup waits for an in-flight callback before releasing the
* session-owned binary reference and callback event. On timeout, it cancels
* the pending cache waiter or waits for an already claimed callback.
@@ -78,7 +83,9 @@ enum mali_vpu_session_fw_bin_state {
* @fw_event: Firmware binary load event callback structure for this session.
* @fw_bin_state: Session firmware binary status.
* @fw_load_done: Completion used to synchronize teardown with an in-flight firmware load.
+ * @fw_waitq: Wait queue used while waiting for firmware responses.
* @fw_state: Last state confirmed by the firmware.
+ * @fw_requested_state: Last firmware state requested by the driver.
* @sched_sess: Scheduler state used to map this session to an LSID.
* @ncores: Number of cores assigned to this session by scheduler.
*/
@@ -104,7 +111,9 @@ struct mali_vpu_session {
struct mali_vpu_fw_event *fw_event;
enum mali_vpu_session_fw_bin_state fw_bin_state;
struct completion fw_load_done;
+ wait_queue_head_t fw_waitq;
enum mali_vpu_fw_state fw_state;
+ enum mali_vpu_fw_state fw_requested_state;
struct mali_vpu_sched_session sched_sess;
u32 ncores;
@@ -115,6 +124,9 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess);
int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess);
int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
struct mali_vpu_fw_set_option *opt);
+int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
+ enum v4l2_buf_type type);
+int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type);
void mali_vpu_session_irq(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
index 4262758a5be26..b9f6968e9f13f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
@@ -62,8 +62,6 @@ static void mali_vpu_vb2_buffer_queue(struct vb2_buffer *vb2_buf)
static int mali_vpu_vb2_prepare_streaming(struct vb2_queue *q)
{
struct mali_vpu_session *sess = vb2_get_drv_priv(q);
- struct mali_vpu_port *port = V4L2_TYPE_IS_OUTPUT(q->type) ?
- &sess->port_src : &sess->port_dst;
int ret;
if (mali_vpu_is_coded_q(sess, q->type)) {
@@ -72,17 +70,51 @@ static int mali_vpu_vb2_prepare_streaming(struct vb2_queue *q)
return ret;
}
- port->stream_requested = true;
+ return mali_vpu_session_prepare_streaming(sess, q->type);
+}
+
+static int mali_vpu_vb2_start_streaming(struct vb2_queue *q,
+ unsigned int count __always_unused)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(q);
+
+ v4l2_m2m_update_start_streaming_state(sess->m2m_ctx, q);
return 0;
}
+static void mali_vpu_vb2_stop_streaming(struct vb2_queue *q)
+{
+ struct mali_vpu_session *sess = vb2_get_drv_priv(q);
+ struct vb2_v4l2_buffer *v4l2_buf;
+ int ret;
+
+ ret = mali_vpu_session_streamoff(sess, q->type);
+ if (ret)
+ dev_warn(sess->vpu_dev->dev, "failed to stop streaming: %d\n", ret);
+
+ if (V4L2_TYPE_IS_OUTPUT(q->type)) {
+ while ((v4l2_buf = v4l2_m2m_src_buf_remove(sess->m2m_ctx)))
+ v4l2_m2m_buf_done(v4l2_buf, VB2_BUF_STATE_ERROR);
+ } else {
+ while ((v4l2_buf = v4l2_m2m_dst_buf_remove(sess->m2m_ctx)))
+ v4l2_m2m_buf_done(v4l2_buf, VB2_BUF_STATE_ERROR);
+ }
+
+ v4l2_m2m_update_stop_streaming_state(sess->m2m_ctx, q);
+}
+
static void mali_vpu_vb2_unprepare_streaming(struct vb2_queue *q)
{
struct mali_vpu_session *sess = vb2_get_drv_priv(q);
- struct mali_vpu_port *port = V4L2_TYPE_IS_OUTPUT(q->type) ?
- &sess->port_src : &sess->port_dst;
+ int ret;
+
+ ret = mali_vpu_session_streamoff(sess, q->type);
+ if (ret)
+ dev_warn(sess->vpu_dev->dev,
+ "failed to unprepare streaming: %d\n", ret);
- port->stream_requested = false;
+ if (!vb2_start_streaming_called(q))
+ v4l2_m2m_update_stop_streaming_state(sess->m2m_ctx, q);
}
static const struct vb2_ops mali_vpu_vb2_ops = {
@@ -92,6 +124,8 @@ static const struct vb2_ops mali_vpu_vb2_ops = {
.buf_prepare = mali_vpu_vb2_buffer_prepare,
.buf_queue = mali_vpu_vb2_buffer_queue,
.prepare_streaming = mali_vpu_vb2_prepare_streaming,
+ .start_streaming = mali_vpu_vb2_start_streaming,
+ .stop_streaming = mali_vpu_vb2_stop_streaming,
.unprepare_streaming = mali_vpu_vb2_unprepare_streaming,
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index d9196cc86cdf4..0fcd3835e52d3 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -357,4 +357,6 @@ const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
.vidioc_enum_framesizes = mali_vpu_enum_framesizes,
.vidioc_subscribe_event = mali_vpu_subscribe_event,
.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+ .vidioc_streamon = v4l2_m2m_ioctl_streamon,
+ .vidioc_streamoff = v4l2_m2m_ioctl_streamoff,
};
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 19/28] media: mali-vpu: start decoder firmware through LSID scheduling
2026-11-19 15:37 ` [RFC PATCH 19/28] media: mali-vpu: start decoder firmware through LSID scheduling Charvi Mehta
@ 2026-11-19 16:03 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 16:03 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] media: mali-vpu: vq_lock drop in
mali_vpu_session_wait_pending() races
- [Medium] media: mali-vpu: 10s hang in mali_vpu_session_cleanup() on
unmapped LSID
--
commit df41ad5c55cc78aaaf50be9e6b1f9cc272e547c5
Author: Charvi Mehta <charvi.mehta@arm.com>
media: mali-vpu: start decoder firmware through LSID scheduling
This commit connects decoder firmware execution to VB2 queue preparation and
asynchronous firmware loading, activating the LSID and tracking requested and
confirmed firmware states across stream startup and shutdown.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
> index 09ec6d14f4ad0..6d22a5d479008 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
[ ... ]
> @@ -123,6 +211,102 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
> }
> }
>
> +static int mali_vpu_session_wait_pending(struct mali_vpu_session *sess)
> +{
> + long timeout;
> +
> + lockdep_assert_held(&sess->vq_lock);
> +
> + while (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
> + sess->fw.pending_response_count) {
> + mutex_unlock(&sess->vq_lock);
> +
> + timeout = wait_event_timeout(sess->fw_waitq,
> + READ_ONCE(sess->fw_bin_state) !=
> + MALI_VPU_SESSION_FW_BIN_READY ||
> + !READ_ONCE(sess->fw.pending_response_count),
> + msecs_to_jiffies(mali_vpu_fw_response_timeout_ms));
> +
> + mutex_lock(&sess->vq_lock);
> +
> + if (!timeout && sess->fw.pending_response_count)
> + return -ETIMEDOUT;
> + }
> +
> + return 0;
> +}
> +
> +static int mali_vpu_session_fw_init(struct mali_vpu_session *sess)
> +{
> + int ret;
> +
> + lockdep_assert_held(&sess->vq_lock);
> +
> + if (sess->fw_requested_state != sess->fw_state) {
> + if (sess->fw_requested_state != MALI_VPU_FW_STATE_RUNNING)
> + return -EBUSY;
> +
> + return mali_vpu_session_sched_activate(sess);
> + }
> +
> + if (sess->fw_state == MALI_VPU_FW_STATE_RUNNING)
> + return 0;
> +
> + ret = mali_vpu_session_fw_send_job(sess, 0);
> + if (ret)
> + return ret;
> +
> + /*
> + * Map the LSID and allow the stopped firmware to process the
> + * queued SET_OPTION requests before it can be RUNNING.
> + */
> + ret = mali_vpu_session_sched_activate(sess);
> + if (ret)
> + return ret;
> +
> + ret = mali_vpu_session_fw_send_ping(sess);
> + if (ret)
> + return ret;
> +
> + ret = mali_vpu_apply_staged_controls(sess);
> + if (ret)
> + return ret;
> +
> + /*
> + * SET_OPTION is only valid while stopped. Wait for pending option
> + * responses before requesting RUNNING.
> + */
> + ret = mali_vpu_session_wait_pending(sess);
> + if (ret)
> + return ret;
> +
> + /* Waiting drops vq_lock, so streaming may have been withdrawn. */
> + if (!mali_vpu_session_can_stream(sess))
> + return 0;
> +
> + return mali_vpu_session_fw_send_state(sess, MALI_VPU_FW_STATE_RUNNING);
> +}
[Severity: High]
Can dropping sess->vq_lock in mali_vpu_session_wait_pending() race with
concurrent queue ioctls during vb2_core_streamon()?
sess->vq_lock is also assigned to vq->lock and m2m_ctx->q_lock, which
videobuf2 expects to stay held across prepare_streaming() and
start_streaming() until q->streaming is set. When staged controls are
applied, mali_vpu_session_prepare_streaming() sets port->stream_requested =
true and then drops sess->vq_lock inside mali_vpu_session_wait_pending()
while q->streaming is still 0:
CPU0 CPU1
vb2_core_streamon(q)
mali_vpu_vb2_prepare_streaming(q)
port->stream_requested = true
mali_vpu_session_fw_init()
mali_vpu_apply_staged_controls()
mali_vpu_session_wait_pending()
mutex_unlock(&sess->vq_lock)
vb2_core_streamon(q)
/* q->streaming is still 0 */
mali_vpu_vb2_prepare_streaming(q)
if (port->stream_requested)
return 0;
vb2_start_streaming(q)
mutex_lock(&sess->vq_lock)
vb2_start_streaming(q) <-- corrupts q_ctx->rdy_queue list
Because both threads run vb2_start_streaming(q), the queued buffers are
passed to mali_vpu_vb2_buffer_queue() -> v4l2_m2m_buf_queue() twice,
corrupting q_ctx->rdy_queue via double list_add_tail() and double
incrementing q->owned_by_drv_count.
Dropping sess->vq_lock here exposes three other races while waiting:
- sess->fw_requested_state is not set to MALI_VPU_FW_STATE_RUNNING until
mali_vpu_session_fw_send_state() after mali_vpu_session_wait_pending()
returns, so a concurrent streamon on the other port (or against the
asynchronous mali_vpu_session_fw_bin_ready() callback) sees
fw_requested_state == fw_state == MALI_VPU_FW_STATE_STOPPED and re-enters
mali_vpu_session_fw_init(), sending duplicate job and ping requests.
- If another thread calls VIDIOC_STREAMOFF while sess->vq_lock is dropped,
!mali_vpu_session_can_stream(sess) returns 0 to vb2_core_streamon(),
which still sets q->streaming = 1 even though port->stream_requested was
cleared and the firmware remains stopped.
- A concurrent VIDIOC_REQBUFS(0) can enter __vb2_queue_cancel() and free
buffers while vb2_core_streamon() is still in flight.
[ ... ]
> @@ -243,6 +433,21 @@ int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
> sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
> }
>
> + if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY)
> + fw_stop_ret = mali_vpu_session_fw_stop(sess);
[Severity: Medium]
Can mali_vpu_session_cleanup() hang for 10 seconds in
mali_vpu_session_fw_stop() when the firmware binary is ready but no LSID is
mapped?
Unlike mali_vpu_session_streamoff(), which returns early when
!sess->sched_sess.lsid, mali_vpu_session_cleanup() calls
mali_vpu_session_fw_stop() -> mali_vpu_session_wait_pending() without
checking sess->sched_sess.lsid. Meanwhile, mali_vpu_apply_or_stage_ctrl()
sends MVE_REQUEST_CODE_SET_OPTION immediately whenever fw_bin_state is
MALI_VPU_SESSION_FW_BIN_READY, even if sess->sched_sess.lsid is NULL:
drivers/media/platform/arm/mali-vpu/mali-vpu-controls.c:mali_vpu_apply_or_stage_ctrl() {
...
staged = sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY;
if (!staged) {
ret = desc->set(sess, state->cid, value);
if (ret)
return ret;
}
...
}
If fw_bin_state reaches MALI_VPU_SESSION_FW_BIN_READY without an active LSID
(for example, when CAPTURE is prepared while OUTPUT is stopped, or when
mali_vpu_sched_switch_in() fails with -EBUSY because another session holds
LSID 0) and userspace sets a control before closing the fd,
sess->fw.pending_response_count is incremented for an unscheduled firmware
instance and mali_vpu_session_cleanup() blocks until the 10-second
mali_vpu_fw_response_timeout_ms expires.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=19
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 20/28] media: mali-vpu: implement vb2 buffer queue handling
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (18 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 19/28] media: mali-vpu: start decoder firmware through LSID scheduling Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 16:06 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 21/28] media: mali-vpu: handle firmware buffer exchange Charvi Mehta
` (7 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Rath Sankaralingam <rath.sankaralingam@arm.com>
Implement the vb2 callbacks needed to initialise, queue and clean up
Mali VPU buffers.
Record each DMA-SG plane in an MMU pages object during buffer
initialisation. When a buffer is queued, copy its timestamp, flags and
per-plane payload information into the driver buffer, add it to the
v4l2-m2m queue and prepare it for firmware submission.
Add helpers to map and unmap buffer planes in the VPU MMU, synchronise
their cache state and configure decoded frame layouts. Defer each buffer
until its port requests streaming and the firmware is running, then submit
deferred buffers when firmware reports RUNNING. Release the per-plane MMU
bookkeeping when vb2 cleans up the buffer.
Signed-off-by: Rath Sankaralingam <rath.sankaralingam@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-buffer.c | 275 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-buffer.h | 74 +++++
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 4 +-
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 7 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 145 +++++++++
.../platform/arm/mali-vpu/mali-vpu-session.h | 6 +
.../platform/arm/mali-vpu/mali-vpu-vb2.c | 23 ++
7 files changed, 532 insertions(+), 2 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
index 7ace0489996a8..c1df3e2d4eacf 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
@@ -4,13 +4,44 @@
*/
#include <linux/errno.h>
+#include <media/videobuf2-dma-sg.h>
#include "mali-vpu-buffer.h"
#include "mali-vpu-session.h"
+static int mali_vpu_buffer_alloc_planes(struct mali_vpu_buffer *vpu_buf,
+ struct sg_table **sgt)
+{
+ u32 i;
+ int ret;
+
+ for (i = 0; i < vpu_buf->nplanes; ++i) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ plane->pages = mali_vpu_mmu_alloc_pages_sg(vpu_buf->dev, sgt[i], 0);
+ if (IS_ERR(plane->pages)) {
+ ret = PTR_ERR(plane->pages);
+ goto free_pages;
+ }
+ }
+
+ return 0;
+
+free_pages:
+ while (i--)
+ mali_vpu_mmu_free_pages(vpu_buf->planes[i].pages);
+
+ return ret;
+}
+
int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf)
{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
+ struct sg_table *sgt[MALI_VPU_BUFFER_MAX_PLANES] = { 0 };
+ u32 i;
+ int ret;
if (vb2_buf->num_planes > MALI_VPU_BUFFER_MAX_PLANES) {
dev_err(sess->vpu_dev->dev,
@@ -19,6 +50,82 @@ int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf)
return -EINVAL;
}
+ vpu_buf->dev = sess->vpu_dev->dev;
+ vpu_buf->mmu = &sess->mmu;
+ vpu_buf->nplanes = vb2_buf->num_planes;
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_NONE;
+
+ for (i = 0; i < vb2_buf->num_planes; ++i) {
+ sgt[i] = vb2_dma_sg_plane_desc(vb2_buf, i);
+ if (!sgt[i]) {
+ dev_err(sess->vpu_dev->dev,
+ "cannot fetch SG descriptor. vb=%p, plane=%u",
+ vb2_buf, i);
+ return -ENOMEM;
+ }
+ }
+
+ ret = mali_vpu_buffer_alloc_planes(vpu_buf, sgt);
+
+ return ret;
+}
+
+bool mali_vpu_is_buffer_mapped(struct mali_vpu_buffer *vpu_buf)
+{
+ return (vpu_buf->planes[0].pages && vpu_buf->planes[0].pages->va);
+}
+
+static size_t mali_vpu_buffer_get_bytesused(struct vb2_buffer *b)
+{
+ size_t size;
+ u32 i;
+
+ for (i = 0, size = 0; i < b->num_planes; i++)
+ size += b->planes[i].bytesused;
+
+ return size;
+}
+
+static void mali_vpu_buffer_update_flags_from_v4l2(struct vb2_buffer *vb2_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
+
+ u32 flags = v4l2_buf->flags;
+
+ vpu_buf->flags = 0;
+
+ if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type) && mali_vpu_buffer_get_bytesused(vb2_buf) == 0)
+ flags |= V4L2_BUF_FLAG_LAST;
+
+ if (flags & V4L2_BUF_FLAG_LAST)
+ vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
+}
+
+int mali_vpu_buffer_update_info_from_v4l2(struct vb2_buffer *vb2_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
+ u32 offset;
+ u32 i;
+ int ret;
+
+ vpu_buf->user_data = vb2_buf->timestamp;
+ vpu_buf->type = vb2_buf->type;
+
+ for (i = 0; i < vb2_buf->num_planes; i++) {
+ offset = vb2_buf->planes[i].data_offset;
+
+ /* set plane payload size and offset */
+ ret = mali_vpu_buffer_set_plane_payload(vpu_buf, i, vb2_buf->planes[i].bytesused,
+ offset);
+ if (ret)
+ return ret;
+ }
+
+ /* update flags */
+ mali_vpu_buffer_update_flags_from_v4l2(vb2_buf);
+
return 0;
}
@@ -49,3 +156,171 @@ int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf)
return 0;
}
+
+static int mali_vpu_buffer_plane_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin,
+ mali_vpu_mmu_va end, u32 plane)
+{
+ while (begin < end) {
+ struct mali_vpu_buffer_plane *p = &vpu_buf->planes[plane];
+ int ret;
+
+ ret = mali_vpu_mmu_map_pages(vpu_buf->mmu, begin, p->pages,
+ MALI_VPU_ATTR_SHARED_RW, MALI_VPU_ACCESS_READ_WRITE);
+ if (!ret) {
+ dev_dbg(vpu_buf->dev,
+ "memory map buffer. buf=%p, plane=%u, va=0x%x, size=%zu.",
+ vpu_buf, plane, p->pages->va,
+ mali_vpu_mmu_size_pages(p->pages));
+ return 0;
+ }
+
+ if (ret != -EAGAIN)
+ return ret;
+
+ begin += SZ_1M;
+ }
+
+ return -ENOMEM;
+}
+
+int mali_vpu_buffer_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin, mali_vpu_mmu_va end)
+{
+ u32 i;
+ int ret;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ if (plane->pages) {
+ ret = mali_vpu_buffer_plane_map(vpu_buf, begin, end, i);
+ if (ret) {
+ mali_vpu_buffer_unmap(vpu_buf);
+ return ret;
+ }
+ }
+ }
+ return 0;
+}
+
+void mali_vpu_buffer_unmap(struct mali_vpu_buffer *vpu_buf)
+{
+ u32 i;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ if (plane->pages && plane->pages->va != 0)
+ mali_vpu_mmu_unmap_pages(plane->pages);
+ }
+}
+
+size_t mali_vpu_buffer_plane_size(struct mali_vpu_buffer *vpu_buf, u32 plane)
+{
+ struct vb2_buffer *vb2_buf = &vpu_buf->m2m_buf.vb.vb2_buf;
+
+ if (plane >= vb2_buf->num_planes)
+ return 0;
+
+ return vb2_plane_size(vb2_buf, plane);
+}
+
+int mali_vpu_buffer_sync(struct mali_vpu_buffer *vpu_buf, enum dma_data_direction dir)
+{
+ u32 i;
+ int ret;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ if (plane->pages) {
+ ret = mali_vpu_mmu_sync_pages(plane->pages, dir);
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+int mali_vpu_buffer_set_plane_payload(struct mali_vpu_buffer *vpu_buf, u32 plane,
+ u32 bytesused, u32 offset)
+{
+ struct mali_vpu_buffer_plane *p;
+ size_t capacity;
+
+ if (plane >= vpu_buf->nplanes)
+ return -EINVAL;
+
+ capacity = mali_vpu_buffer_plane_size(vpu_buf, plane);
+
+ if (offset > capacity || bytesused > capacity || offset > bytesused) {
+ dev_err(vpu_buf->dev,
+ "invalid plane payload. Plane: %u, capacity: %zu, bytesused=%u, offset=%u.\n",
+ plane, capacity, bytesused, offset);
+ return -EINVAL;
+ }
+
+ p = &vpu_buf->planes[plane];
+
+ p->payload_size = bytesused - offset;
+ p->offset = offset;
+
+ return 0;
+}
+
+int mali_vpu_buffer_set_frame_layout(struct mali_vpu_buffer *vpu_buf)
+{
+ struct vb2_buffer *vb2_buf = &vpu_buf->m2m_buf.vb.vb2_buf;
+ struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
+ struct mali_vpu_port *port;
+ struct v4l2_pix_format_mplane *pix_mp;
+ int ret;
+ size_t size;
+ u32 i;
+
+ port = V4L2_TYPE_IS_OUTPUT(vb2_buf->vb2_queue->type) ? &sess->port_src :
+ &sess->port_dst;
+
+ pix_mp = &port->curr_fmt.fmt.pix_mp;
+
+ if (vpu_buf->nplanes != pix_mp->num_planes)
+ return -EINVAL;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ size = pix_mp->plane_fmt[i].sizeimage;
+ plane->stride = pix_mp->plane_fmt[i].bytesperline;
+ plane->offset = 0;
+
+ if (!V4L2_TYPE_IS_OUTPUT(vpu_buf->type)) {
+ ret = mali_vpu_buffer_set_plane_payload(vpu_buf, i, size, 0);
+ if (ret)
+ return ret;
+ }
+
+ if (plane->payload_size != size) {
+ dev_err(vpu_buf->dev,
+ "payload size mismatch. Plane: %u, payload_size: %u, actual: %zu.\n",
+ i, plane->payload_size, size);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+void mali_vpu_vb2_buffer_cleanup(struct vb2_buffer *vb2_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
+ u32 i;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ if (!vpu_buf->planes[i].pages)
+ continue;
+
+ mali_vpu_mmu_free_pages(vpu_buf->planes[i].pages);
+ vpu_buf->planes[i].pages = NULL;
+ }
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
index f5d67d48e5921..645dcc775cd2c 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
@@ -8,14 +8,88 @@
#include <media/v4l2-mem2mem.h>
+#include "mali-vpu-mmu.h"
+
#define MALI_VPU_BUFFER_MAX_PLANES 3
+#define MALI_VPU_BUFFER_EOS 0x00000001
+
+/**
+ * enum mali_vpu_buffer_attributes - Buffer submission state.
+ * @MALI_VPU_BUF_ATTR_NONE: Buffer is neither deferred nor queued for firmware
+ * processing.
+ * @MALI_VPU_BUF_ATTR_DEFERRED: Buffer is held by driver for later submission
+ * to firmware.
+ * @MALI_VPU_BUF_ATTR_QUEUED: Buffer has been submitted to firmware and is
+ * awaiting its return.
+ */
+enum mali_vpu_buffer_attributes {
+ MALI_VPU_BUF_ATTR_NONE,
+ MALI_VPU_BUF_ATTR_DEFERRED,
+ MALI_VPU_BUF_ATTR_QUEUED
+};
+
+/**
+ * struct mali_vpu_buffer_plane - Plane information.
+ * @pages: MMU pages object.
+ * @payload_size: Number of bytes written to this plane. For a frame buffer this
+ * value should always match the size of the plane.
+ * @offset: Byte offset from beginning of buffer to first bitstream data.
+ * @stride: Bytes per line for the plane.
+ */
+struct mali_vpu_buffer_plane {
+ struct mali_vpu_mmu_pages *pages;
+ u32 payload_size;
+ u32 offset;
+ u32 stride;
+};
+
+/**
+ * struct mali_vpu_buffer - Buffer descriptor.
+ * @m2m_buf: Embedded v4l2-mem2mem buffer.
+ * @dev: Pointer to device.
+ * @mmu: Pointer to MMU.
+ * @nplanes: Number of planes.
+ * @type: Type of the buffer (output/capture).
+ * @user_data: Userdata(timestamp) to be copied to capture buffer.
+ * @flags: Buffer Flags.
+ * @planes: Array of planes.
+ * @attr: Firmware Queuing Status Attribute.
+ */
struct mali_vpu_buffer {
struct v4l2_m2m_buffer m2m_buf;
+ struct device *dev;
+ struct mali_vpu_mmu *mmu;
+ u32 nplanes;
+ u32 type;
+ u64 user_data;
+ u32 flags;
+ struct mali_vpu_buffer_plane planes[MALI_VPU_BUFFER_MAX_PLANES];
+ enum mali_vpu_buffer_attributes attr;
};
+static inline struct mali_vpu_buffer *to_mali_vpu_buffer(struct vb2_v4l2_buffer *v4l2_buf)
+{
+ struct v4l2_m2m_buffer *m2m_buf = container_of(v4l2_buf, struct v4l2_m2m_buffer, vb);
+
+ return container_of(m2m_buf, struct mali_vpu_buffer, m2m_buf);
+}
+
int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf);
int mali_vpu_vb2_buffer_prepare(struct vb2_buffer *vb2_buf);
int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf);
+void mali_vpu_vb2_buffer_cleanup(struct vb2_buffer *vb2_buf);
+
+int mali_vpu_buffer_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin,
+ mali_vpu_mmu_va end);
+void mali_vpu_buffer_unmap(struct mali_vpu_buffer *vpu_buf);
+int mali_vpu_buffer_sync(struct mali_vpu_buffer *vpu_buf, enum dma_data_direction dir);
+bool mali_vpu_is_buffer_mapped(struct mali_vpu_buffer *vpu_buf);
+
+int mali_vpu_buffer_update_info_from_v4l2(struct vb2_buffer *vb2_buf);
+int mali_vpu_buffer_set_plane_payload(struct mali_vpu_buffer *vpu_buf, u32 plane,
+ u32 bytesused, u32 offset);
+int mali_vpu_buffer_set_frame_layout(struct mali_vpu_buffer *vpu_buf);
+size_t mali_vpu_buffer_plane_size(struct mali_vpu_buffer *vpu_buf, u32 plane);
#endif /* __MALI_VPU_BUFFER_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index c11d5c6c0ec5f..820853ca4519a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -18,8 +18,8 @@
#define MALI_VPU_FW_MAX_CORES 8
#define MALI_VPU_RPC_VA_GRANULE SZ_16M
-static int mali_vpu_fw_get_region(enum mali_vpu_fw_region region,
- mali_vpu_mmu_va *begin, mali_vpu_mmu_va *end)
+int mali_vpu_fw_get_region(enum mali_vpu_fw_region region,
+ mali_vpu_mmu_va *begin, mali_vpu_mmu_va *end)
{
switch (region) {
case MALI_VPU_FW_REGION_MSG_HOST:
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index c03a07d994873..5431959befb24 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -10,6 +10,8 @@
#include <linux/media/arm/mali-vpu-config.h>
#include <linux/types.h>
+#include "mali-vpu-mmu.h"
+
#define MALI_VPU_FW_HTABLE_BITS 3
#define MALI_VPU_FW_TEXT_BASE_ADDR 0x1000u
@@ -101,6 +103,7 @@ struct mali_vpu_fw_error {
* enum mali_vpu_fw_code - Driver representation of firmware messages.
* @MALI_VPU_FW_CODE_ERROR: Firmware reported fatal error.
* @MALI_VPU_FW_CODE_JOB: Firmware dequeued a job.
+ * @MALI_VPU_FW_CODE_BUFFER: Send and receive buffer.
* @MALI_VPU_FW_CODE_PING: Ping request sent to firmware.
* @MALI_VPU_FW_CODE_PONG: Firmware responded to a ping request.
* @MALI_VPU_FW_CODE_IDLE: Firmware informs the driver that it's idle.
@@ -115,6 +118,7 @@ struct mali_vpu_fw_error {
enum mali_vpu_fw_code {
MALI_VPU_FW_CODE_ERROR, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_JOB, /* Driver -> Firmware. */
+ MALI_VPU_FW_CODE_BUFFER, /* Driver <-> Firmware. */
MALI_VPU_FW_CODE_PING, /* Driver -> Firmware. */
MALI_VPU_FW_CODE_PONG, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_IDLE, /* Driver <- Firmware. */
@@ -138,6 +142,7 @@ struct mali_vpu_fw_msg {
enum mali_vpu_fw_state state;
struct mali_vpu_fw_job job;
struct mali_vpu_fw_error error;
+ struct mali_vpu_buffer *vpu_buf;
} data;
};
@@ -246,6 +251,8 @@ int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw, struct mali_vpu_fw_set_option *option);
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg);
+int mali_vpu_fw_get_region(enum mali_vpu_fw_region region, mali_vpu_mmu_va *begin,
+ mali_vpu_mmu_va *end);
int mali_vpu_fw_handle_rpc(struct mali_vpu_fw *fw, bool *flush_mmu);
#endif /* __MALI_VPU_FIRMWARE_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 6d22a5d479008..540104bcd38e2 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -150,6 +150,122 @@ int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
return 0;
}
+static int mali_vpu_buffer_map_fw_region(struct mali_vpu_session *sess,
+ struct mali_vpu_buffer *vpu_buf)
+{
+ mali_vpu_mmu_va begin, end;
+ enum mali_vpu_fw_region region;
+ enum v4l2_buf_type type;
+ int ret;
+
+ type = vpu_buf->m2m_buf.vb.vb2_buf.type;
+
+ if (mali_vpu_is_coded_q(sess, type))
+ region = MALI_VPU_FW_REGION_PROTECTED;
+ else
+ region = MALI_VPU_FW_REGION_FRAMEBUF;
+
+ ret = mali_vpu_fw_get_region(region, &begin, &end);
+ if (ret)
+ return ret;
+
+ return mali_vpu_buffer_map(vpu_buf, begin, end);
+}
+
+static int mali_vpu_queue_buffer(struct mali_vpu_session *sess, struct mali_vpu_buffer *vpu_buf)
+{
+ int ret;
+ enum dma_data_direction dma_dir;
+
+ dma_dir = (vpu_buf->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ?
+ DMA_TO_DEVICE : DMA_FROM_DEVICE;
+
+ /* Map MMU buffer pages if it is not already mapped */
+ if (!mali_vpu_is_buffer_mapped(vpu_buf)) {
+ ret = mali_vpu_buffer_map_fw_region(sess, vpu_buf);
+ if (ret)
+ return ret;
+ }
+
+ if (!mali_vpu_is_coded_q(sess, vpu_buf->type) && sess->role == MALI_VPU_DECODER) {
+ ret = mali_vpu_buffer_set_frame_layout(vpu_buf);
+ if (ret)
+ return ret;
+ }
+
+ /* dma sync for cache coherency */
+ ret = mali_vpu_buffer_sync(vpu_buf, dma_dir);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int mali_vpu_submit_deferred_bufs_for_queue(struct mali_vpu_session *sess,
+ enum v4l2_buf_type type)
+{
+ struct v4l2_m2m_buffer *buffer;
+ struct mali_vpu_buffer *vpu_buf;
+ int ret;
+
+ if (!mali_vpu_session_get_port(sess, type)->stream_requested ||
+ !mali_vpu_session_can_stream(sess) ||
+ sess->fw_state != MALI_VPU_FW_STATE_RUNNING)
+ return 0;
+
+ if (V4L2_TYPE_IS_OUTPUT(type)) {
+ v4l2_m2m_for_each_src_buf(sess->m2m_ctx, buffer) {
+ vpu_buf = to_mali_vpu_buffer(&buffer->vb);
+ if (vpu_buf->attr != MALI_VPU_BUF_ATTR_DEFERRED)
+ continue;
+
+ ret = mali_vpu_queue_buffer(sess, vpu_buf);
+ if (ret)
+ return ret;
+
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
+ }
+ } else if (V4L2_TYPE_IS_CAPTURE(type)) {
+ v4l2_m2m_for_each_dst_buf(sess->m2m_ctx, buffer) {
+ vpu_buf = to_mali_vpu_buffer(&buffer->vb);
+ if (vpu_buf->attr != MALI_VPU_BUF_ATTR_DEFERRED)
+ continue;
+
+ ret = mali_vpu_queue_buffer(sess, vpu_buf);
+ if (ret)
+ return ret;
+
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
+ }
+ } else {
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_submit_deferred_bufs(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ ret = mali_vpu_submit_deferred_bufs_for_queue(sess, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+ if (ret)
+ goto err_submit;
+
+ ret = mali_vpu_submit_deferred_bufs_for_queue(sess,
+ V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ if (ret)
+ goto err_submit;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+
+ return 0;
+
+err_submit:
+ dev_err(sess->vpu_dev->dev, "failed to queue deferred buffers\n");
+ return ret;
+}
+
static int mali_vpu_session_sched_activate(struct mali_vpu_session *sess)
{
lockdep_assert_held(&sess->vq_lock);
@@ -180,6 +296,9 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
return 0;
case MALI_VPU_FW_CODE_STATE_CHANGE:
sess->fw_state = fw_msg->data.state;
+
+ if (sess->fw_state == MALI_VPU_FW_STATE_RUNNING)
+ return mali_vpu_submit_deferred_bufs(sess);
return 0;
case MALI_VPU_FW_CODE_SWITCH_IN:
/* Single session policy: the scheduler already tracks the active LSID mapping. */
@@ -390,6 +509,32 @@ static void mali_vpu_session_release_fw_bin(struct mali_vpu_session *sess)
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
}
+int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
+ struct mali_vpu_port *port =
+ mali_vpu_session_get_port(sess, vb2_buf->type);
+ int ret;
+
+ if (!port->stream_requested ||
+ !mali_vpu_session_can_stream(sess) ||
+ sess->fw_state != MALI_VPU_FW_STATE_RUNNING) {
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_DEFERRED;
+ return 0;
+ }
+
+ ret = mali_vpu_queue_buffer(sess, vpu_buf);
+ if (ret)
+ return ret;
+
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+
+ return 0;
+}
+
int mali_vpu_session_cleanup(struct mali_vpu_session *sess)
{
struct mali_vpu_dev *vpu_dev = sess->vpu_dev;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index e331965d32fd8..5cabb7844131b 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -12,6 +12,7 @@
#include <media/v4l2-ctrls.h>
#include <media/v4l2-fh.h>
+#include "mali-vpu-buffer.h"
#include "mali-vpu-device.h"
#include "mali-vpu-firmware.h"
#include "mali-vpu-formats.h"
@@ -52,6 +53,10 @@ struct mali_vpu_port {
* streaming: OUTPUT for a decoder, both ports for an encoder.
* Binary READY and firmware RUNNING are separate states.
*
+ * Buffers received before the firmware confirms RUNNING are deferred. The
+ * RUNNING response submits those buffers. Buffers received afterwards are
+ * submitted directly.
+ *
* Session cleanup waits for an in-flight callback before releasing the
* session-owned binary reference and callback event. On timeout, it cancels
* the pending cache waiter or waits for an already claimed callback.
@@ -128,6 +133,7 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
enum v4l2_buf_type type);
int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type);
void mali_vpu_session_irq(struct mali_vpu_session *sess);
+int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_buf);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
index b9f6968e9f13f..41158bf4e0ecb 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
@@ -55,8 +55,30 @@ static void mali_vpu_vb2_buffer_queue(struct vb2_buffer *vb2_buf)
{
struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
+ int ret;
+
+ /* Update vb2 metadata and payload info onto mali_vpu_buffer. */
+ ret = mali_vpu_buffer_update_info_from_v4l2(vb2_buf);
+ if (ret)
+ goto err_buf_done;
v4l2_m2m_buf_queue(sess->m2m_ctx, v4l2_buf);
+
+ ret = mali_vpu_session_qbuf(sess, vb2_buf);
+ if (ret)
+ goto err_m2m_buf_remove;
+
+ return;
+
+err_m2m_buf_remove:
+ if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type))
+ v4l2_m2m_src_buf_remove_by_buf(sess->m2m_ctx, v4l2_buf);
+ else
+ v4l2_m2m_dst_buf_remove_by_buf(sess->m2m_ctx, v4l2_buf);
+
+err_buf_done:
+ v4l2_m2m_buf_done(v4l2_buf, VB2_BUF_STATE_ERROR);
+
}
static int mali_vpu_vb2_prepare_streaming(struct vb2_queue *q)
@@ -123,6 +145,7 @@ static const struct vb2_ops mali_vpu_vb2_ops = {
.buf_init = mali_vpu_vb2_buffer_init,
.buf_prepare = mali_vpu_vb2_buffer_prepare,
.buf_queue = mali_vpu_vb2_buffer_queue,
+ .buf_cleanup = mali_vpu_vb2_buffer_cleanup,
.prepare_streaming = mali_vpu_vb2_prepare_streaming,
.start_streaming = mali_vpu_vb2_start_streaming,
.stop_streaming = mali_vpu_vb2_stop_streaming,
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 20/28] media: mali-vpu: implement vb2 buffer queue handling
2026-11-19 15:37 ` [RFC PATCH 20/28] media: mali-vpu: implement vb2 buffer queue handling Charvi Mehta
@ 2026-11-19 16:06 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 16:06 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] media: mali-vpu: out-of-bounds VA mapping in
mali_vpu_buffer_plane_map()
- [High] media: mali-vpu: wrong capture DMA sync in
mali_vpu_queue_buffer()
- [High] media: mali-vpu: missing MMU flush in
mali_vpu_vb2_buffer_cleanup()
- [Medium] media: mali-vpu: stranded buffers on error in
mali_vpu_submit_deferred_bufs()
- [Medium] media: mali-vpu: unreachable 0-byte EOS check without
allow_zero_bytesused
--
commit 8e09878f27886489a304a46715a5c2d9a14beef9
Author: Rath Sankaralingam <rath.sankaralingam@arm.com>
media: mali-vpu: implement vb2 buffer queue handling
This commit implements the vb2 callbacks for initializing, queuing, and
cleaning up Mali VPU buffers, along with MMU mapping, cache synchronization,
and deferred buffer submission when the firmware enters the RUNNING state.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
> index 7ace0489996a8..c1df3e2d4eacf 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
[ ... ]
> @@ -19,6 +50,82 @@ int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf)
> return -EINVAL;
> }
>
> + vpu_buf->dev = sess->vpu_dev->dev;
[ ... ]
> +static void mali_vpu_buffer_update_flags_from_v4l2(struct vb2_buffer *vb2_buf)
> +{
> + struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
> + struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
> +
> + u32 flags = v4l2_buf->flags;
> +
> + vpu_buf->flags = 0;
> +
> + if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type) && mali_vpu_buffer_get_bytesused(vb2_buf) == 0)
> + flags |= V4L2_BUF_FLAG_LAST;
> +
> + if (flags & V4L2_BUF_FLAG_LAST)
> + vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
> +}
[Severity: Medium]
Can mali_vpu_buffer_get_bytesused(vb2_buf) == 0 ever be reached on an
OUTPUT buffer without vq->allow_zero_bytesused set?
mali_vpu_queue_init_type() does not set allow_zero_bytesused on the OUTPUT
vb2_queue, so vb2_fill_vb2_v4l2_buffer() replaces a 0-byte bytesused from
userspace with the full plane length. As a result, a 0-byte OUTPUT buffer
queued to signal EOS will instead be given a payload_size equal to the full
buffer capacity rather than setting MALI_VPU_BUFFER_EOS. Also,
mali_vpu_buffer_get_bytesused() sums b->planes[i].bytesused without
subtracting data_offset, so an empty payload with a non-zero data_offset
will not be treated as 0 bytes used either.
[ ... ]
> @@ -49,3 +156,171 @@ int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf)
>
> return 0;
> }
> +
> +static int mali_vpu_buffer_plane_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin,
> + mali_vpu_mmu_va end, u32 plane)
> +{
> + while (begin < end) {
> + struct mali_vpu_buffer_plane *p = &vpu_buf->planes[plane];
> + int ret;
> +
> + ret = mali_vpu_mmu_map_pages(vpu_buf->mmu, begin, p->pages,
> + MALI_VPU_ATTR_SHARED_RW, MALI_VPU_ACCESS_READ_WRITE);
> + if (!ret) {
> + dev_dbg(vpu_buf->dev,
> + "memory map buffer. buf=%p, plane=%u, va=0x%x, size=%zu.",
> + vpu_buf, plane, p->pages->va,
> + mali_vpu_mmu_size_pages(p->pages));
> + return 0;
> + }
> +
> + if (ret != -EAGAIN)
> + return ret;
> +
> + begin += SZ_1M;
> + }
> +
> + return -ENOMEM;
> +}
[Severity: High]
Can mali_vpu_buffer_plane_map() map a plane past the end of the firmware
virtual address region?
The loop only checks begin < end, and mali_vpu_mmu_map_pages() maps
p->pages->capacity contiguous VPU pages starting at begin without taking an
upper bound. When earlier mappings advance begin close to end (or a large
DMA-BUF is imported) so that the plane size exceeds end - begin, the
mapping spills past end into the adjacent firmware region or wraps around
the 32-bit VPU virtual address space.
[ ... ]
> +void mali_vpu_vb2_buffer_cleanup(struct vb2_buffer *vb2_buf)
> +{
> + struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
> + struct mali_vpu_buffer *vpu_buf = to_mali_vpu_buffer(v4l2_buf);
> + u32 i;
> +
> + for (i = 0; i < vpu_buf->nplanes; i++) {
> + if (!vpu_buf->planes[i].pages)
> + continue;
> +
> + mali_vpu_mmu_free_pages(vpu_buf->planes[i].pages);
> + vpu_buf->planes[i].pages = NULL;
> + }
> +}
[Severity: High]
Does unmapping buffer pages in mali_vpu_vb2_buffer_cleanup() without
calling mali_vpu_sched_flush_mmu() leave stale TLB entries in the VPU MMU?
mali_vpu_mmu_free_pages() -> mali_vpu_mmu_unmap_pages() clears the L2 PTEs
in sess->mmu, and when a new DMA-BUF is queued on the same VB2 slot (or
CAPTURE buffers are reallocated while the OUTPUT queue keeps the session
LSID active), mali_vpu_buffer_plane_map() immediately reuses the lowest
unmapped virtual addresses from the start of the region. Without an LSID
MMU flush, the hardware can continue using cached translations pointing to
the previous buffer's freed physical pages.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
> index 6d22a5d479008..540104bcd38e2 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
> @@ -150,6 +150,122 @@ int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
[ ... ]
> +static int mali_vpu_queue_buffer(struct mali_vpu_session *sess, struct mali_vpu_buffer *vpu_buf)
> +{
> + int ret;
> + enum dma_data_direction dma_dir;
> +
> + dma_dir = (vpu_buf->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ?
> + DMA_TO_DEVICE : DMA_FROM_DEVICE;
> +
> + /* Map MMU buffer pages if it is not already mapped */
> + if (!mali_vpu_is_buffer_mapped(vpu_buf)) {
> + ret = mali_vpu_buffer_map_fw_region(sess, vpu_buf);
> + if (ret)
> + return ret;
> + }
> +
> + if (!mali_vpu_is_coded_q(sess, vpu_buf->type) && sess->role == MALI_VPU_DECODER) {
> + ret = mali_vpu_buffer_set_frame_layout(vpu_buf);
> + if (ret)
> + return ret;
> + }
> +
> + /* dma sync for cache coherency */
> + ret = mali_vpu_buffer_sync(vpu_buf, dma_dir);
> + if (ret)
> + return ret;
> +
> + return 0;
> +}
[Severity: High]
Does calling mali_vpu_buffer_sync(vpu_buf, DMA_FROM_DEVICE) on a CAPTURE
buffer before VPU DMA runs sync in the wrong direction (for the CPU instead
of the device)?
drivers/media/platform/arm/mali-vpu/mali-vpu-mmu.c:mali_vpu_mmu_sync_pages() {
if (dir == DMA_FROM_DEVICE) {
for (i = 0; i < pages->count; i += MALI_VPU_PAGES_PER_LINUX_PAGE)
dma_sync_single_for_cpu(pages->dev, pages->pages[i].dma,
PAGE_SIZE, DMA_FROM_DEVICE);
}
When queuing a buffer to hardware before DMA starts, pages need to be
synced for the device (which vb2_dma_sg_prepare() already does during buffer
prepare), whereas dma_sync_single_for_cpu() transfers ownership back to the
CPU before the VPU writes the decoded frame.
> +
> +static int mali_vpu_submit_deferred_bufs_for_queue(struct mali_vpu_session *sess,
> + enum v4l2_buf_type type)
> +{
> + struct v4l2_m2m_buffer *buffer;
> + struct mali_vpu_buffer *vpu_buf;
> + int ret;
> +
> + if (!mali_vpu_session_get_port(sess, type)->stream_requested ||
> + !mali_vpu_session_can_stream(sess) ||
> + sess->fw_state != MALI_VPU_FW_STATE_RUNNING)
> + return 0;
> +
> + if (V4L2_TYPE_IS_OUTPUT(type)) {
> + v4l2_m2m_for_each_src_buf(sess->m2m_ctx, buffer) {
> + vpu_buf = to_mali_vpu_buffer(&buffer->vb);
> + if (vpu_buf->attr != MALI_VPU_BUF_ATTR_DEFERRED)
> + continue;
> +
> + ret = mali_vpu_queue_buffer(sess, vpu_buf);
> + if (ret)
> + return ret;
> +
> + vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
> + }
> + } else if (V4L2_TYPE_IS_CAPTURE(type)) {
> + v4l2_m2m_for_each_dst_buf(sess->m2m_ctx, buffer) {
> + vpu_buf = to_mali_vpu_buffer(&buffer->vb);
> + if (vpu_buf->attr != MALI_VPU_BUF_ATTR_DEFERRED)
> + continue;
> +
> + ret = mali_vpu_queue_buffer(sess, vpu_buf);
> + if (ret)
> + return ret;
> +
> + vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
> + }
> + } else {
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int mali_vpu_submit_deferred_bufs(struct mali_vpu_session *sess)
> +{
> + int ret;
> +
> + ret = mali_vpu_submit_deferred_bufs_for_queue(sess, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
> + if (ret)
> + goto err_submit;
> +
> + ret = mali_vpu_submit_deferred_bufs_for_queue(sess,
> + V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
> + if (ret)
> + goto err_submit;
> +
> + mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
> +
> + return 0;
> +
> +err_submit:
> + dev_err(sess->vpu_dev->dev, "failed to queue deferred buffers\n");
> + return ret;
> +}
[Severity: Medium]
If mali_vpu_queue_buffer() fails for a deferred buffer here, doesn't this
leave both the failed buffer and the remaining buffers in the batch
stranded?
Unlike mali_vpu_vb2_buffer_queue(), which removes a failed buffer from
sess->m2m_ctx and completes it with VB2_BUF_STATE_ERROR, an error in
mali_vpu_submit_deferred_bufs_for_queue() returns immediately while the
buffer is still on the m2m ready queue in VB2_BUF_STATE_ACTIVE. Bailing
out to err_submit also skips remaining deferred buffers and skips
mali_vpu_sched_send_irq() for any earlier buffers in the loop that were
already marked MALI_VPU_BUF_ATTR_QUEUED.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=20
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 21/28] media: mali-vpu: handle firmware buffer exchange
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (19 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 20/28] media: mali-vpu: implement vb2 buffer queue handling Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:58 ` sashiko-bot
2026-11-19 15:37 ` [RFC PATCH 22/28] media: mali-vpu: handle fatal firmware error Charvi Mehta
` (6 subsequent siblings)
27 siblings, 1 reply; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Rath Sankaralingam <rath.sankaralingam@arm.com>
Send queued bitstream and frame buffers to firmware using the v3 buffer
protocol. Populate the buffer descriptors with firmware-visible plane
addresses, allocation and payload sizes, strides, frame dimensions,
formats, flags and user data as appropriate for each buffer type.
Handle input and output buffer responses from firmware. Recover the
associated driver buffer, propagate returned user data, dimensions and
status flags to V4L2, synchronize capture buffers for the CPU, and
complete the corresponding v4l2-m2m buffer with the appropriate vb2
state.
Use non-zero buffer IDs to associate firmware responses with tracked
in-flight buffers.
Signed-off-by: Rath Sankaralingam <rath.sankaralingam@arm.com>
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-buffer.c | 100 ++++++
.../platform/arm/mali-vpu/mali-vpu-buffer.h | 14 +
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 300 ++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 10 +
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 128 ++++++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 49 +++
6 files changed, 601 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
index c1df3e2d4eacf..e404fb933195f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
@@ -53,6 +53,7 @@ int mali_vpu_vb2_buffer_init(struct vb2_buffer *vb2_buf)
vpu_buf->dev = sess->vpu_dev->dev;
vpu_buf->mmu = &sess->mmu;
vpu_buf->nplanes = vb2_buf->num_planes;
+ vpu_buf->fw_buffer_id = 0;
vpu_buf->attr = MALI_VPU_BUF_ATTR_NONE;
for (i = 0; i < vb2_buf->num_planes; ++i) {
@@ -86,6 +87,40 @@ static size_t mali_vpu_buffer_get_bytesused(struct vb2_buffer *b)
return size;
}
+static int mali_vpu_buffer_clear_bytesused(struct mali_vpu_buffer *vpu_buf)
+{
+ struct vb2_buffer *vb2_buf = &vpu_buf->m2m_buf.vb.vb2_buf;
+ u32 i;
+
+ for (i = 0; i < vb2_buf->num_planes; i++) {
+ vb2_set_plane_payload(vb2_buf, i, 0);
+ vb2_buf->planes[i].data_offset = 0;
+ }
+
+ return 0;
+}
+
+static int mali_vpu_buffer_update_bytesused_to_v4l2(struct mali_vpu_buffer *vpu_buf)
+{
+ struct vb2_buffer *vb2_buf = &vpu_buf->m2m_buf.vb.vb2_buf;
+ u32 i;
+
+ if (vpu_buf->nplanes != vb2_buf->num_planes) {
+ dev_dbg(vpu_buf->dev,
+ "mali VPU buf and vb2 buf have different num_planes. vpu_buf=%u vb2_buf=%u.\n",
+ vpu_buf->nplanes, vb2_buf->num_planes);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ vb2_buf->planes[i].bytesused =
+ vpu_buf->planes[i].payload_size + vpu_buf->planes[i].offset;
+ vb2_buf->planes[i].data_offset = vpu_buf->planes[i].offset;
+ }
+
+ return 0;
+}
+
static void mali_vpu_buffer_update_flags_from_v4l2(struct vb2_buffer *vb2_buf)
{
struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
@@ -102,6 +137,27 @@ static void mali_vpu_buffer_update_flags_from_v4l2(struct vb2_buffer *vb2_buf)
vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
}
+static void mali_vpu_buffer_update_flags_to_v4l2(struct mali_vpu_buffer *vpu_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = &vpu_buf->m2m_buf.vb;
+
+ v4l2_buf->flags &= ~(V4L2_BUF_FLAG_ERROR |
+ V4L2_BUF_FLAG_KEYFRAME |
+ V4L2_BUF_FLAG_LAST);
+
+ if (vpu_buf->flags & MALI_VPU_BUFFER_EOS ||
+ (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
+ mali_vpu_buffer_get_bytesused(&vpu_buf->m2m_buf.vb.vb2_buf) == 0))
+ v4l2_buf->flags |= V4L2_BUF_FLAG_LAST;
+
+ if (vpu_buf->flags & MALI_VPU_BUFFER_CORRUPT)
+ v4l2_buf->flags |= V4L2_BUF_FLAG_ERROR;
+
+ if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
+ vpu_buf->flags & MALI_VPU_BUFFER_REJECTED)
+ mali_vpu_buffer_clear_bytesused(vpu_buf);
+}
+
int mali_vpu_buffer_update_info_from_v4l2(struct vb2_buffer *vb2_buf)
{
struct vb2_v4l2_buffer *v4l2_buf = to_vb2_v4l2_buffer(vb2_buf);
@@ -129,6 +185,30 @@ int mali_vpu_buffer_update_info_from_v4l2(struct vb2_buffer *vb2_buf)
return 0;
}
+enum vb2_buffer_state mali_vpu_buffer_update_info_to_v4l2(struct mali_vpu_buffer *vpu_buf)
+{
+ struct vb2_v4l2_buffer *v4l2_buf = &vpu_buf->m2m_buf.vb;
+ struct vb2_buffer *vb2_buf = &v4l2_buf->vb2_buf;
+ int ret;
+
+ vb2_buf->timestamp = vpu_buf->user_data;
+
+ if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type)) {
+ ret = mali_vpu_buffer_update_bytesused_to_v4l2(vpu_buf);
+ if (ret)
+ goto error;
+ }
+
+ mali_vpu_buffer_update_flags_to_v4l2(vpu_buf);
+ if ((v4l2_buf->flags & V4L2_BUF_FLAG_ERROR) != 0)
+ goto error;
+
+ return VB2_BUF_STATE_DONE;
+
+error:
+ return VB2_BUF_STATE_ERROR;
+}
+
int mali_vpu_vb2_buffer_prepare(struct vb2_buffer *vb2_buf)
{
struct mali_vpu_session *sess = vb2_get_drv_priv(vb2_buf->vb2_queue);
@@ -157,6 +237,20 @@ int mali_vpu_vb2_buffer_out_validate(struct vb2_buffer *vb2_buf)
return 0;
}
+mali_vpu_mmu_va mali_vpu_buffer_plane_va(struct mali_vpu_buffer *vpu_buf, u32 plane)
+{
+ struct mali_vpu_buffer_plane *vpu_buf_plane;
+
+ if (plane >= vpu_buf->nplanes)
+ return 0;
+
+ vpu_buf_plane = &vpu_buf->planes[plane];
+ if (!vpu_buf_plane->pages)
+ return 0;
+
+ return vpu_buf_plane->pages->va + vpu_buf_plane->pages->offset;
+}
+
static int mali_vpu_buffer_plane_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin,
mali_vpu_mmu_va end, u32 plane)
{
@@ -283,6 +377,12 @@ int mali_vpu_buffer_set_frame_layout(struct mali_vpu_buffer *vpu_buf)
pix_mp = &port->curr_fmt.fmt.pix_mp;
+ vpu_buf->width = pix_mp->width;
+ vpu_buf->height = pix_mp->height;
+ vpu_buf->format = mali_vpu_find_format(sess, pix_mp->pixelformat, vpu_buf->type);
+ if (!vpu_buf->format)
+ return -EINVAL;
+
if (vpu_buf->nplanes != pix_mp->num_planes)
return -EINVAL;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
index 645dcc775cd2c..01dd49082fdcc 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.h
@@ -13,6 +13,8 @@
#define MALI_VPU_BUFFER_MAX_PLANES 3
#define MALI_VPU_BUFFER_EOS 0x00000001
+#define MALI_VPU_BUFFER_CORRUPT 0x00000004
+#define MALI_VPU_BUFFER_REJECTED 0x00000008
/**
* enum mali_vpu_buffer_attributes - Buffer submission state.
@@ -44,6 +46,8 @@ struct mali_vpu_buffer_plane {
u32 stride;
};
+struct mali_vpu_format_desc;
+
/**
* struct mali_vpu_buffer - Buffer descriptor.
* @m2m_buf: Embedded v4l2-mem2mem buffer.
@@ -53,6 +57,10 @@ struct mali_vpu_buffer_plane {
* @type: Type of the buffer (output/capture).
* @user_data: Userdata(timestamp) to be copied to capture buffer.
* @flags: Buffer Flags.
+ * @width: Frame width in pixels.
+ * @height: Frame height in pixels.
+ * @fw_buffer_id: Non-zero identifier while registered for firmware submission.
+ * @format: Buffer format.
* @planes: Array of planes.
* @attr: Firmware Queuing Status Attribute.
*/
@@ -64,6 +72,10 @@ struct mali_vpu_buffer {
u32 type;
u64 user_data;
u32 flags;
+ u32 width;
+ u32 height;
+ u32 fw_buffer_id;
+ const struct mali_vpu_format_desc *format;
struct mali_vpu_buffer_plane planes[MALI_VPU_BUFFER_MAX_PLANES];
enum mali_vpu_buffer_attributes attr;
};
@@ -85,11 +97,13 @@ int mali_vpu_buffer_map(struct mali_vpu_buffer *vpu_buf, mali_vpu_mmu_va begin,
void mali_vpu_buffer_unmap(struct mali_vpu_buffer *vpu_buf);
int mali_vpu_buffer_sync(struct mali_vpu_buffer *vpu_buf, enum dma_data_direction dir);
bool mali_vpu_is_buffer_mapped(struct mali_vpu_buffer *vpu_buf);
+mali_vpu_mmu_va mali_vpu_buffer_plane_va(struct mali_vpu_buffer *vpu_buf, u32 plane);
int mali_vpu_buffer_update_info_from_v4l2(struct vb2_buffer *vb2_buf);
int mali_vpu_buffer_set_plane_payload(struct mali_vpu_buffer *vpu_buf, u32 plane,
u32 bytesused, u32 offset);
int mali_vpu_buffer_set_frame_layout(struct mali_vpu_buffer *vpu_buf);
size_t mali_vpu_buffer_plane_size(struct mali_vpu_buffer *vpu_buf, u32 plane);
+enum vb2_buffer_state mali_vpu_buffer_update_info_to_v4l2(struct mali_vpu_buffer *vpu_buf);
#endif /* __MALI_VPU_BUFFER_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 820853ca4519a..4e3eaecb8ebee 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -12,6 +12,7 @@
#include "mali-vpu-firmware.h"
#include "mali-vpu-firmware-cache.h"
+#include "mali-vpu-formats.h"
#include "mali-vpu-mmu.h"
#include "mali-vpu-protocol-def-v3.h"
@@ -347,6 +348,20 @@ static int mali_vpu_fw_map_core_regions(struct mali_vpu_fw *fw,
return ret;
}
+static void mali_vpu_fw_clear_buffers(struct mali_vpu_fw *fw)
+{
+ struct mali_vpu_buffer *vpu_buf;
+ unsigned long id;
+
+ xa_for_each(&fw->inflight_buffers, id, vpu_buf) {
+ xa_erase(&fw->inflight_buffers, id);
+ vpu_buf->fw_buffer_id = 0;
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_NONE;
+ }
+
+ xa_destroy(&fw->inflight_buffers);
+}
+
int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *fw_bin,
struct device *dev, struct mali_vpu_mmu *mmu, u32 ncores)
{
@@ -362,6 +377,8 @@ int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *
fw->dev = dev;
fw->mmu = mmu;
fw->ncores = ncores;
+ xa_init_flags(&fw->inflight_buffers, XA_FLAGS_ALLOC1);
+ fw->next_buffer_id = 1;
hash_init(fw->rpc_mem);
/* Initialise firmware core and protocol regions */
@@ -393,6 +410,8 @@ void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw)
if (!fw)
return;
+ mali_vpu_fw_clear_buffers(fw);
+
mali_vpu_fw_unmap_protocol(fw);
mali_vpu_fw_unmap_core_regions(fw);
@@ -781,6 +800,153 @@ int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw,
return ret;
}
+static struct mali_vpu_buffer *
+mali_vpu_fw_resolve_returned_buffer(struct mali_vpu_fw *fw,
+ u64 host_handle,
+ enum v4l2_buf_type expected_type)
+{
+ struct mali_vpu_buffer *vpu_buf;
+ u32 id;
+
+ if (!host_handle || host_handle > UINT_MAX)
+ return ERR_PTR(-EPROTO);
+
+ id = host_handle;
+
+ vpu_buf = xa_load(&fw->inflight_buffers, id);
+ if (!vpu_buf)
+ return ERR_PTR(-EPROTO);
+
+ if (vpu_buf->fw_buffer_id != id ||
+ vpu_buf->type != expected_type ||
+ vpu_buf->attr != MALI_VPU_BUF_ATTR_QUEUED)
+ return ERR_PTR(-EPROTO);
+
+ if (xa_erase(&fw->inflight_buffers, id) != vpu_buf)
+ return ERR_PTR(-EPROTO);
+
+ vpu_buf->fw_buffer_id = 0;
+ vpu_buf->attr = MALI_VPU_BUF_ATTR_NONE;
+
+ return vpu_buf;
+}
+
+static int
+mali_vpu_convert_buffer_frame(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_msg *msg,
+ u32 buf_type,
+ struct mve_buffer_frame *mve_frame_buf)
+{
+ struct mali_vpu_buffer *vpu_buf;
+ unsigned int i;
+
+ vpu_buf = mali_vpu_fw_resolve_returned_buffer(fw,
+ mve_frame_buf->host_handle,
+ buf_type);
+ if (IS_ERR(vpu_buf))
+ return PTR_ERR(vpu_buf);
+
+ msg->code = MALI_VPU_FW_CODE_BUFFER;
+ msg->data.vpu_buf = vpu_buf;
+
+ if (buf_type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+ vpu_buf->width = mve_frame_buf->visible_frame_width;
+ vpu_buf->height = mve_frame_buf->visible_frame_height;
+
+ if (vpu_buf->width == 0 || vpu_buf->height == 0) {
+ for (i = 0; i < vpu_buf->nplanes; i++)
+ (void)mali_vpu_buffer_set_plane_payload(vpu_buf, i, 0, 0);
+ }
+ }
+
+ vpu_buf->user_data = mve_frame_buf->user_data_tag;
+ vpu_buf->flags = 0;
+
+ if (mve_frame_buf->frame_flags & MVE_BUFFER_FRAME_FLAG_EOS)
+ vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
+
+ if (mve_frame_buf->frame_flags & MVE_BUFFER_FRAME_FLAG_REJECTED)
+ vpu_buf->flags |= MALI_VPU_BUFFER_REJECTED;
+
+ if (mve_frame_buf->frame_flags & MVE_BUFFER_FRAME_FLAG_CORRUPT)
+ vpu_buf->flags |= MALI_VPU_BUFFER_CORRUPT;
+
+ return 0;
+}
+
+static int
+mali_vpu_convert_buffer_bitstream(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_msg *msg,
+ u32 buf_type,
+ struct mve_buffer_bitstream *mve_stream_buf)
+{
+ struct mali_vpu_buffer *vpu_buf =
+ mali_vpu_fw_resolve_returned_buffer(fw,
+ mve_stream_buf->host_handle,
+ buf_type);
+ if (IS_ERR(vpu_buf))
+ return PTR_ERR(vpu_buf);
+
+ msg->code = MALI_VPU_FW_CODE_BUFFER;
+ msg->data.vpu_buf = vpu_buf;
+
+ vpu_buf->user_data = mve_stream_buf->user_data_tag;
+ vpu_buf->flags = 0;
+
+ if (mve_stream_buf->bitstream_flags & MVE_BUFFER_BITSTREAM_FLAG_EOS)
+ vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
+
+ return 0;
+}
+
+static int mali_vpu_fw_get_buffer(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_proto_page *buf_host_page,
+ struct mali_vpu_fw_proto_page *buf_mve_page,
+ struct mali_vpu_fw_msg *msg, u32 buf_type)
+{
+ u16 code;
+ union {
+ struct mve_buffer_frame frame;
+ struct mve_buffer_bitstream bitstream;
+ } fw_msg;
+ size_t size = sizeof(fw_msg);
+ int ret;
+
+ ret = mali_vpu_fw_read_msg(fw, buf_host_page, buf_mve_page, &code,
+ &fw_msg, &size);
+ if (ret <= 0)
+ return ret;
+
+ switch (code) {
+ case MVE_BUFFER_CODE_FRAME:
+ if (size != sizeof(fw_msg.frame) ||
+ (!fw_msg.frame.host_handle &&
+ !(fw_msg.frame.frame_flags & MVE_BUFFER_FRAME_FLAG_EOS)))
+ return -EPROTO;
+
+ ret = mali_vpu_convert_buffer_frame(fw, msg, buf_type, &fw_msg.frame);
+ if (ret)
+ return ret;
+ break;
+ case MVE_BUFFER_CODE_BITSTREAM:
+ if (size != sizeof(fw_msg.bitstream) ||
+ (!fw_msg.bitstream.host_handle &&
+ !(fw_msg.bitstream.bitstream_flags & MVE_BUFFER_BITSTREAM_FLAG_EOS)))
+ return -EPROTO;
+
+ ret = mali_vpu_convert_buffer_bitstream(fw, msg, buf_type,
+ &fw_msg.bitstream);
+ if (ret)
+ return ret;
+ break;
+ default:
+ dev_dbg(fw->dev, "unknown buf code. code: %u.\n", code);
+ break;
+ }
+
+ return 1;
+}
+
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
{
u16 code;
@@ -791,6 +957,8 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
struct mve_response_error error;
} fw_msg = {};
size_t size = sizeof(fw_msg);
+ struct mali_vpu_fw_proto_page *buf_host_page;
+ struct mali_vpu_fw_proto_page *buf_mve_page;
int ret;
if (!fw || !msg)
@@ -871,6 +1039,20 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
fw->pending_response_count--;
break;
}
+ case MVE_RESPONSE_CODE_INPUT:
+ buf_host_page = &fw->buf_in_host;
+ buf_mve_page = &fw->buf_in_mve;
+
+ ret = mali_vpu_fw_get_buffer(fw, buf_host_page, buf_mve_page, msg,
+ V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE);
+ break;
+ case MVE_RESPONSE_CODE_OUTPUT:
+ buf_host_page = &fw->buf_out_host;
+ buf_mve_page = &fw->buf_out_mve;
+
+ ret = mali_vpu_fw_get_buffer(fw, buf_host_page, buf_mve_page, msg,
+ V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ break;
case MVE_RESPONSE_CODE_ERROR: {
size_t message_offset = offsetof(struct mve_response_error, message);
size_t message_size;
@@ -930,6 +1112,124 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
return ret;
}
+static int mali_vpu_fw_track_buffer(struct mali_vpu_fw *fw,
+ struct mali_vpu_buffer *vpu_buf)
+{
+ u32 id;
+ int ret;
+
+ if (vpu_buf->fw_buffer_id)
+ return -EBUSY;
+
+ ret = xa_alloc_cyclic(&fw->inflight_buffers, &id, vpu_buf,
+ XA_LIMIT(1, UINT_MAX),
+ &fw->next_buffer_id, GFP_KERNEL);
+ if (ret)
+ return ret;
+
+ vpu_buf->fw_buffer_id = id;
+
+ return 0;
+}
+
+static void mali_vpu_fw_untrack_buffer(struct mali_vpu_fw *fw,
+ struct mali_vpu_buffer *vpu_buf)
+{
+ void *entry;
+
+ if (!vpu_buf->fw_buffer_id)
+ return;
+
+ entry = xa_erase(&fw->inflight_buffers, vpu_buf->fw_buffer_id);
+ WARN_ON_ONCE(entry && entry != vpu_buf);
+
+ vpu_buf->fw_buffer_id = 0;
+}
+
+static int mali_vpu_fw_put_buffer_frame(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_msg *fw_msg,
+ struct mali_vpu_fw_proto_page *host_page,
+ struct mali_vpu_fw_proto_page *mve_page)
+{
+ struct mve_buffer_frame mve_frame_buf = { 0 };
+ struct mali_vpu_buffer *vpu_buf = fw_msg->data.vpu_buf;
+ struct mve_buffer_frame_planar *planar;
+ u32 i;
+
+ mve_frame_buf.host_handle = vpu_buf->fw_buffer_id;
+
+ mve_frame_buf.format = vpu_buf->format->mve_frame_fmt;
+ planar = &mve_frame_buf.data.planar;
+
+ for (i = 0; i < vpu_buf->nplanes; i++) {
+ struct mali_vpu_buffer_plane *plane = &vpu_buf->planes[i];
+
+ if (plane->stride > 0)
+ planar->plane_top[i] = mali_vpu_buffer_plane_va(vpu_buf, i);
+
+ planar->stride[i] = plane->stride;
+ planar->max_frame_width = vpu_buf->width;
+ planar->max_frame_height = vpu_buf->height;
+ }
+
+ return mali_vpu_fw_write_msg(fw, host_page, mve_page, MVE_BUFFER_CODE_FRAME,
+ &mve_frame_buf, sizeof(mve_frame_buf));
+}
+
+static int mali_vpu_fw_put_buffer_bitstream(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_msg *fw_msg,
+ struct mali_vpu_fw_proto_page *host_page,
+ struct mali_vpu_fw_proto_page *mve_page)
+{
+ struct mve_buffer_bitstream mve_stream_buf = { 0 };
+ struct mali_vpu_buffer *vpu_buf = fw_msg->data.vpu_buf;
+
+ if (V4L2_TYPE_IS_OUTPUT(vpu_buf->type))
+ mve_stream_buf.bitstream_filled_len = vpu_buf->planes[0].payload_size;
+
+ mve_stream_buf.host_handle = vpu_buf->fw_buffer_id;
+ mve_stream_buf.user_data_tag = vpu_buf->user_data;
+ mve_stream_buf.bitstream_alloc_bytes = mali_vpu_buffer_plane_size(vpu_buf, 0);
+ mve_stream_buf.bitstream_offset = vpu_buf->planes[0].offset;
+ mve_stream_buf.bitstream_buf_addr = mali_vpu_buffer_plane_va(vpu_buf, 0);
+
+ if (vpu_buf->flags & MALI_VPU_BUFFER_EOS)
+ mve_stream_buf.bitstream_flags |= MVE_BUFFER_BITSTREAM_FLAG_EOS;
+
+ return mali_vpu_fw_write_msg(fw, host_page, mve_page, MVE_BUFFER_CODE_BITSTREAM,
+ &mve_stream_buf, sizeof(mve_stream_buf));
+}
+
+int mali_vpu_fw_send_buffer(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *fw_msg)
+{
+ struct mali_vpu_buffer *vpu_buf = fw_msg->data.vpu_buf;
+ struct mali_vpu_fw_proto_page *host;
+ struct mali_vpu_fw_proto_page *mve;
+ int ret;
+
+ if (V4L2_TYPE_IS_OUTPUT(fw_msg->data.vpu_buf->type)) {
+ host = &fw->buf_in_host;
+ mve = &fw->buf_in_mve;
+ } else {
+ host = &fw->buf_out_host;
+ mve = &fw->buf_out_mve;
+ }
+
+ ret = mali_vpu_fw_track_buffer(fw, vpu_buf);
+ if (ret)
+ return ret;
+
+ if (V4L2_TYPE_IS_OUTPUT(fw_msg->data.vpu_buf->type))
+ ret = mali_vpu_fw_put_buffer_bitstream(fw, fw_msg, host, mve);
+ else
+ ret = mali_vpu_fw_put_buffer_frame(fw, fw_msg, host, mve);
+
+ if (ret)
+ mali_vpu_fw_untrack_buffer(fw, vpu_buf);
+
+ return ret;
+}
+
static void mali_vpu_fw_dbg_rpc(struct mali_vpu_fw *fw, const char *direction,
const struct mve_rpc_communication_area *rpc)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index 5431959befb24..c936730698837 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -9,7 +9,9 @@
#include <linux/hashtable.h>
#include <linux/media/arm/mali-vpu-config.h>
#include <linux/types.h>
+#include <linux/xarray.h>
+#include "mali-vpu-buffer.h"
#include "mali-vpu-mmu.h"
#define MALI_VPU_FW_HTABLE_BITS 3
@@ -19,6 +21,8 @@ struct device;
struct mali_vpu_mmu;
struct mali_vpu_fw_bin;
struct mali_vpu_mmu_pages;
+struct mve_buffer_frame;
+struct mve_buffer_bitstream;
/**
* enum mali_vpu_fw_state - Firmware execution state.
@@ -219,6 +223,9 @@ struct mali_vpu_fw_proto_page {
* @shared_bss_pages: Shared BSS pages mapped into every core instance.
* @ncores: Number of firmware core-instance regions constructed.
* @pending_response_count: Number of requests awaiting a firmware response.
+ * @inflight_buffers: XArray mapping non-zero firmware-visible buffer IDs to
+ * buffers currently owned by firmware.
+ * @next_buffer_id: Next candidate ID for cyclic buffer-ID allocation.
* @rpc_mem: Keeps track of RPC allocated memory. Maps MVE virtual address to
* 'struct mali_vpu_mmu_pages' object.
*/
@@ -239,6 +246,8 @@ struct mali_vpu_fw {
struct mali_vpu_mmu_pages *shared_bss_pages;
u32 ncores;
u32 pending_response_count;
+ struct xarray inflight_buffers;
+ u32 next_buffer_id;
DECLARE_HASHTABLE(rpc_mem, MALI_VPU_FW_HTABLE_BITS);
};
@@ -247,6 +256,7 @@ int mali_vpu_fw_construct(struct mali_vpu_fw *fw, const struct mali_vpu_fw_bin *
struct device *dev, struct mali_vpu_mmu *mmu, u32 ncores);
void mali_vpu_fw_cleanup(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_job(struct mali_vpu_fw *fw, u16 cores, u16 frames);
+int mali_vpu_fw_send_buffer(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *fw_msg);
int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw, struct mali_vpu_fw_set_option *option);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index 8a81ed1fd730a..7f3a13ec744b9 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -241,6 +241,10 @@ struct mve_msg_header {
#define MVE_RESPONSE_CODE_SET_OPTION_CONFIRM (2003)
/* struct mve_response_job_dequeued */
#define MVE_RESPONSE_CODE_JOB_DEQUEUED (2004)
+ /* no extra data, but buffer placed in buffer queue */
+ #define MVE_RESPONSE_CODE_INPUT (2005)
+ /* no extra data, but buffer placed in buffer queue */
+ #define MVE_RESPONSE_CODE_OUTPUT (2006)
#define MVE_RESPONSE_CODE_PONG (2009)
/* struct mve_response_error */
#define MVE_RESPONSE_CODE_ERROR (2010)
@@ -250,6 +254,14 @@ struct mve_msg_header {
/* struct mve_response_set_option_fail */
#define MVE_RESPONSE_CODE_SET_OPTION_FAIL (2017)
+ /* BUFFERs are sent from host to firmware,
+ * and then return at some time:
+ */
+ /* struct mve_buffer_frame */
+ #define MVE_BUFFER_CODE_FRAME (3001)
+ /* struct mve_buffer_bitstream */
+ #define MVE_BUFFER_CODE_BITSTREAM (3002)
+
u16 size; /* size in bytes of trailing data, 0 if none */
};
@@ -409,4 +421,120 @@ struct mve_response_set_option_fail {
(2 << MVE_FORMAT_BF_N) | \
(1 << MVE_FORMAT_BF_V))
+/*********************************************************************
+ *
+ * BUFFERs are sent both ways, from host to firmware and back again
+ *
+ * Each MVE_BUFFER_CODE_ code is followed by one of the structs
+ * below.
+ *
+ *********************************************************************/
+
+/* Flags in mve_buffer_frame::frame_flags:
+ * Set by whom? Meaning:
+ * DECODE:
+ * MVE_BUFFER_FRAME_FLAG_REJECTED fw Buffer was too small, host should re-allocate
+ *
+ * MVE_BUFFER_FRAME_FLAG_CORRUPT fw Frame contains visual corruption
+ *
+ * MVE_BUFFER_FRAME_FLAG_EOS fw This is the last frame in the stream.
+ */
+
+/* mve_buffer_frame_planar stores uncompressed YUV pictures.
+ * ________________________________________
+ * | ^ | | ^
+ * |<-:--visible_frame_width---->| | :
+ * | : | | :
+ * | : | | :
+ * | visible_frame_height | | max_frame_height
+ * | : | | :
+ * | : | | :
+ * |__v__________________________| | :
+ * | | :
+ * |<-------------max_frame_width---------->| :
+ * |________________________________________| v
+ *
+ */
+struct mve_buffer_frame_planar {
+ /* Y,Cb,Cr top field */
+ u32 plane_top[3];
+
+ /* Y,Cb,Cr bottom field (interlace only) */
+ u32 plane_bot[3];
+
+ /* Stride between rows, in bytes */
+ s32 stride[3];
+
+ /* Size of largest frame allowed to put in this buffer */
+ u16 max_frame_width;
+ u16 max_frame_height;
+};
+
+/* mve_buffer_frame stores PLANAR buffer information. */
+struct mve_buffer_frame {
+ /* For identification of the buffer, this is not changed by
+ * the firmware.
+ */
+ u64 host_handle;
+
+ /* For matching input buffer with output buffers, the firmware
+ * copies these values between frame buffers and bitstream buffers.
+ */
+ u64 user_data_tag;
+
+ /* Frame buffer flags, see commentary above */
+ u32 frame_flags;
+ #define MVE_BUFFER_FRAME_FLAG_REJECTED (0x00001000)
+ #define MVE_BUFFER_FRAME_FLAG_CORRUPT (0x00002000)
+ #define MVE_BUFFER_FRAME_FLAG_EOS (0x00010000)
+ /* Height (in luma samples) of visible part of frame,
+ * may be smaller than allocated frame size.
+ */
+ u16 visible_frame_height;
+
+ /* Width (in luma samples) of visible part of frame,
+ * may be smaller than allocated frame size.
+ */
+ u16 visible_frame_width;
+
+ /* Color format of buffer */
+ u16 format;
+
+ u16 reserved0; /* force 'data' to be 4-byte aligned */
+
+ union {
+ struct mve_buffer_frame_planar planar;
+ } data;
+
+ u32 reserved1; /* force size to be multiple of 8 bytes */
+};
+
+/* The bitstream buffer stores a number of bitstream bytes */
+struct mve_buffer_bitstream {
+ /* For identification of the buffer, this is not changed by
+ * the firmware.
+ */
+ u64 host_handle;
+
+ /* For matching input buffer with output buffers, the firmware
+ * copies these values between frame buffers and bitstream buffers.
+ */
+ u64 user_data_tag;
+
+ /* BufferFlags */
+ u32 bitstream_flags;
+ #define MVE_BUFFER_BITSTREAM_FLAG_EOS (0x00000001)
+
+ /* Length of allocated buffer */
+ u32 bitstream_alloc_bytes;
+ /* Byte offset from start to first byte */
+ u32 bitstream_offset;
+ /* Number of bytes in the buffer */
+ u32 bitstream_filled_len;
+ /* Pointer to buffer start */
+ u32 bitstream_buf_addr;
+ /* Pad to force 8-byte alignment */
+ u32 reserved;
+};
+
#endif /* __MALI_VPU_PROTOCOL_DEF_V3_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 540104bcd38e2..60e4b7e25eff8 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -176,6 +176,7 @@ static int mali_vpu_queue_buffer(struct mali_vpu_session *sess, struct mali_vpu_
{
int ret;
enum dma_data_direction dma_dir;
+ struct mali_vpu_fw_msg fw_msg;
dma_dir = (vpu_buf->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) ?
DMA_TO_DEVICE : DMA_FROM_DEVICE;
@@ -198,6 +199,13 @@ static int mali_vpu_queue_buffer(struct mali_vpu_session *sess, struct mali_vpu_
if (ret)
return ret;
+ /* put buf to fw q */
+ fw_msg.code = MALI_VPU_FW_CODE_BUFFER;
+ fw_msg.data.vpu_buf = vpu_buf;
+ ret = mali_vpu_fw_send_buffer(&sess->fw, &fw_msg);
+ if (ret)
+ return ret;
+
return 0;
}
@@ -277,6 +285,28 @@ static int mali_vpu_session_sched_activate(struct mali_vpu_session *sess)
return mali_vpu_sched_switch_in(&sess->sched_sess, &sess->vpu_dev->sched);
}
+static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
+ struct mali_vpu_buffer *vpu_buf,
+ enum vb2_buffer_state state)
+{
+ struct vb2_v4l2_buffer *vbuf = &vpu_buf->m2m_buf.vb;
+ struct vb2_buffer *vb2_buf = &vbuf->vb2_buf;
+
+ if (vb2_buf->state != VB2_BUF_STATE_ACTIVE) {
+ dev_dbg(sess->vpu_dev->dev,
+ "ignore returned buffer. type=%u index=%u vb2_state=%u\n",
+ vpu_buf->type, vb2_buf->index, vb2_buf->state);
+ return;
+ }
+
+ if (V4L2_TYPE_IS_OUTPUT(vpu_buf->type))
+ v4l2_m2m_src_buf_remove_by_buf(sess->m2m_ctx, vbuf);
+ else
+ v4l2_m2m_dst_buf_remove_by_buf(sess->m2m_ctx, vbuf);
+
+ v4l2_m2m_buf_done(vbuf, state);
+}
+
static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
const struct mali_vpu_fw_msg *fw_msg)
{
@@ -291,6 +321,25 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
* switch-out scheduling.
*/
return 0;
+ case MALI_VPU_FW_CODE_BUFFER: {
+ struct mali_vpu_buffer *vpu_buf = fw_msg->data.vpu_buf;
+ enum vb2_buffer_state state;
+
+ dev_dbg(sess->vpu_dev->dev,
+ "firmware rsp: Buffer. type=%u, len=[%u, %u, %u], flags=0x%08x, eos=%u",
+ vpu_buf->type, vpu_buf->planes[0].payload_size,
+ vpu_buf->planes[1].payload_size, vpu_buf->planes[2].payload_size,
+ vpu_buf->flags, (vpu_buf->flags & MALI_VPU_BUFFER_EOS) != 0);
+
+ if (vpu_buf->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+ mali_vpu_buffer_sync(vpu_buf, DMA_FROM_DEVICE);
+
+ state = mali_vpu_buffer_update_info_to_v4l2(vpu_buf);
+
+ mali_vpu_session_buffer_done(sess, vpu_buf, state);
+
+ return 0;
+ }
case MALI_VPU_FW_CODE_PONG:
/* Receiving the responses completes the diagnostic ping. */
return 0;
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [RFC PATCH 21/28] media: mali-vpu: handle firmware buffer exchange
2026-11-19 15:37 ` [RFC PATCH 21/28] media: mali-vpu: handle firmware buffer exchange Charvi Mehta
@ 2026-11-19 15:58 ` sashiko-bot
0 siblings, 0 replies; 37+ messages in thread
From: sashiko-bot @ 2026-11-19 15:58 UTC (permalink / raw)
To: Charvi Mehta, devicetree; +Cc: robh, conor+dt, media-ci
Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] media: mali-vpu: in-flight buffers not untracked on streamoff
or cleanup
- [Medium] media: mali-vpu: bogus error and early loop exit in
mali_vpu_fw_get_buffer()
- [Medium] media: mali-vpu: spurious V4L2_BUF_FLAG_LAST on rejected
capture buffers
--
commit 470c746dd3c53a73aac1ec2091f9e9c5b26e72a8
Author: Rath Sankaralingam <rath.sankaralingam@arm.com>
media: mali-vpu: handle firmware buffer exchange
This commit implements buffer submission to the firmware and handling of
returned input and output buffer responses using the v3 buffer protocol,
tracking in-flight buffers in an xarray by non-zero buffer IDs.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
> index c1df3e2d4eacf..e404fb933195f 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-buffer.c
[ ... ]
> @@ -102,6 +137,27 @@ static void mali_vpu_buffer_update_flags_from_v4l2(struct vb2_buffer *vb2_buf)
> vpu_buf->flags |= MALI_VPU_BUFFER_EOS;
> }
>
> +static void mali_vpu_buffer_update_flags_to_v4l2(struct mali_vpu_buffer *vpu_buf)
> +{
> + struct vb2_v4l2_buffer *v4l2_buf = &vpu_buf->m2m_buf.vb;
> +
> + v4l2_buf->flags &= ~(V4L2_BUF_FLAG_ERROR |
> + V4L2_BUF_FLAG_KEYFRAME |
> + V4L2_BUF_FLAG_LAST);
> +
> + if (vpu_buf->flags & MALI_VPU_BUFFER_EOS ||
> + (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
> + mali_vpu_buffer_get_bytesused(&vpu_buf->m2m_buf.vb.vb2_buf) == 0))
> + v4l2_buf->flags |= V4L2_BUF_FLAG_LAST;
> +
> + if (vpu_buf->flags & MALI_VPU_BUFFER_CORRUPT)
> + v4l2_buf->flags |= V4L2_BUF_FLAG_ERROR;
> +
> + if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
> + vpu_buf->flags & MALI_VPU_BUFFER_REJECTED)
> + mali_vpu_buffer_clear_bytesused(vpu_buf);
> +}
[Severity: Medium]
Can a rejected capture buffer be returned to userspace with
V4L2_BUF_FLAG_LAST spuriously set?
When firmware rejects a capture frame buffer with
MVE_BUFFER_FRAME_FLAG_REJECTED and zero visible dimensions,
mali_vpu_convert_buffer_frame() zeroes plane payload_size and offset, and
mali_vpu_buffer_update_bytesused_to_v4l2() propagates bytesused = 0 to
vb2_buf before mali_vpu_buffer_update_flags_to_v4l2() runs. As a result,
mali_vpu_buffer_get_bytesused() is already 0 here and sets
V4L2_BUF_FLAG_LAST even when MALI_VPU_BUFFER_EOS is not set.
> diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
> index 820853ca4519a..4e3eaecb8ebee 100644
> --- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
> +++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
[ ... ]
> @@ -347,6 +348,20 @@ static int mali_vpu_fw_map_core_regions(struct mali_vpu_fw *fw,
> return ret;
> }
>
> +static void mali_vpu_fw_clear_buffers(struct mali_vpu_fw *fw)
> +{
> + struct mali_vpu_buffer *vpu_buf;
> + unsigned long id;
> +
> + xa_for_each(&fw->inflight_buffers, id, vpu_buf) {
> + xa_erase(&fw->inflight_buffers, id);
> + vpu_buf->fw_buffer_id = 0;
> + vpu_buf->attr = MALI_VPU_BUF_ATTR_NONE;
> + }
> +
> + xa_destroy(&fw->inflight_buffers);
> +}
[Severity: High]
Can this cause a use-after-free on vpu_buf if buffers are freed after
VIDIOC_STREAMOFF?
mali_vpu_vb2_stop_streaming() completes queued buffers back to vb2 without
removing in-flight buffers from fw->inflight_buffers or clearing
vpu_buf->fw_buffer_id, and mali_vpu_vb2_buffer_cleanup() does not untrack
them either. If userspace frees the buffers after streamoff (e.g. via
VIDIOC_REQBUFS(0)), dangling vpu_buf pointers remain in
fw->inflight_buffers and are written to here during session close (or in
mali_vpu_fw_resolve_returned_buffer() if firmware returns the buffer).
If userspace instead restarts streaming and re-queues the same buffers
after streamoff, vpu_buf->fw_buffer_id is still non-zero, so
mali_vpu_fw_track_buffer() fails with -EBUSY. Also, when firmware returns
a buffer that was already completed by stop_streaming,
mali_vpu_session_handle_fw_msg() calls
mali_vpu_buffer_update_info_to_v4l2() and mutates vb2_buf metadata before
mali_vpu_session_buffer_done() checks whether vb2_buf->state is still
VB2_BUF_STATE_ACTIVE.
[ ... ]
> +static int mali_vpu_fw_get_buffer(struct mali_vpu_fw *fw,
> + struct mali_vpu_fw_proto_page *buf_host_page,
> + struct mali_vpu_fw_proto_page *buf_mve_page,
> + struct mali_vpu_fw_msg *msg, u32 buf_type)
> +{
> + u16 code;
> + union {
> + struct mve_buffer_frame frame;
> + struct mve_buffer_bitstream bitstream;
> + } fw_msg;
> + size_t size = sizeof(fw_msg);
> + int ret;
> +
> + ret = mali_vpu_fw_read_msg(fw, buf_host_page, buf_mve_page, &code,
> + &fw_msg, &size);
> + if (ret <= 0)
> + return ret;
> +
> + switch (code) {
> + case MVE_BUFFER_CODE_FRAME:
> + if (size != sizeof(fw_msg.frame) ||
> + (!fw_msg.frame.host_handle &&
> + !(fw_msg.frame.frame_flags & MVE_BUFFER_FRAME_FLAG_EOS)))
> + return -EPROTO;
> +
> + ret = mali_vpu_convert_buffer_frame(fw, msg, buf_type, &fw_msg.frame);
> + if (ret)
> + return ret;
> + break;
> + case MVE_BUFFER_CODE_BITSTREAM:
> + if (size != sizeof(fw_msg.bitstream) ||
> + (!fw_msg.bitstream.host_handle &&
> + !(fw_msg.bitstream.bitstream_flags & MVE_BUFFER_BITSTREAM_FLAG_EOS)))
> + return -EPROTO;
> +
> + ret = mali_vpu_convert_buffer_bitstream(fw, msg, buf_type,
> + &fw_msg.bitstream);
> + if (ret)
> + return ret;
> + break;
> + default:
> + dev_dbg(fw->dev, "unknown buf code. code: %u.\n", code);
> + break;
> + }
> +
> + return 1;
> +}
[Severity: Medium]
Does an unknown buffer code in mali_vpu_fw_get_buffer() leave msg->code at
0 (MALI_VPU_FW_CODE_ERROR) while returning 1?
mali_vpu_fw_parse_msg() zeroes *msg before calling mali_vpu_fw_get_buffer(),
so hitting the default branch returns 1 with msg->code ==
MALI_VPU_FW_CODE_ERROR, causing mali_vpu_session_handle_fw_msg() to treat
it as a fatal firmware error and fail with -EIO. In addition, if
mali_vpu_fw_read_msg() returns 0 because the buffer queue is unexpectedly
empty after MVE_RESPONSE_CODE_INPUT or MVE_RESPONSE_CODE_OUTPUT was read
from msg_mve, propagating 0 causes mali_vpu_session_irq() to treat msg_mve
as empty and stop draining remaining messages instead of flagging -EPROTO.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261119153758.1426673-1-charvi.mehta@arm.com?part=21
^ permalink raw reply [flat|nested] 37+ messages in thread
* [RFC PATCH 22/28] media: mali-vpu: handle fatal firmware error
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (20 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 21/28] media: mali-vpu: handle firmware buffer exchange Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 23/28] media: mali-vpu: propagate buffer metadata Charvi Mehta
` (5 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Rath Sankaralingam <rath.sankaralingam@arm.com>
Record the first fatal error encountered by a session so that failures
are propagated consistently instead of allowing firmware operations to
continue on an unusable session.
On firmware loading, initialization, message handling or response timeout
errors, stop the firmware state machine, terminate the scheduled session,
mark both vb2 queues as failed and wake any firmware waiters. Reject
subsequent buffer and streaming operations with the recorded error.
Signed-off-by: Rath Sankaralingam <rath.sankaralingam@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-session.c | 97 ++++++++++++++++---
.../platform/arm/mali-vpu/mali-vpu-session.h | 5 +
2 files changed, 90 insertions(+), 12 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 60e4b7e25eff8..9cbd116124109 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -142,6 +142,9 @@ int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
{
int ret;
+ if (sess->error)
+ return sess->error;
+
ret = mali_vpu_fw_send_set_option(&sess->fw, opt);
if (ret)
return ret;
@@ -307,6 +310,31 @@ static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
v4l2_m2m_buf_done(vbuf, state);
}
+static void mali_vpu_session_set_error(struct mali_vpu_session *sess, int error)
+{
+ struct vb2_queue *src_q;
+ struct vb2_queue *dst_q;
+
+ lockdep_assert_held(&sess->vq_lock);
+
+ if (sess->error)
+ return;
+
+ sess->error = error;
+ sess->fw_requested_state = MALI_VPU_FW_STATE_STOPPED;
+ sess->fw_state = MALI_VPU_FW_STATE_STOPPED;
+
+ mali_vpu_sched_session_terminate(&sess->sched_sess, &sess->vpu_dev->sched);
+
+ src_q = v4l2_m2m_get_src_vq(sess->m2m_ctx);
+ dst_q = v4l2_m2m_get_dst_vq(sess->m2m_ctx);
+
+ vb2_queue_error(src_q);
+ vb2_queue_error(dst_q);
+
+ wake_up(&sess->fw_waitq);
+}
+
static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
const struct mali_vpu_fw_msg *fw_msg)
{
@@ -385,11 +413,13 @@ static int mali_vpu_session_wait_pending(struct mali_vpu_session *sess)
lockdep_assert_held(&sess->vq_lock);
- while (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
+ while (!sess->error &&
+ sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
sess->fw.pending_response_count) {
mutex_unlock(&sess->vq_lock);
timeout = wait_event_timeout(sess->fw_waitq,
+ READ_ONCE(sess->error) ||
READ_ONCE(sess->fw_bin_state) !=
MALI_VPU_SESSION_FW_BIN_READY ||
!READ_ONCE(sess->fw.pending_response_count),
@@ -397,11 +427,14 @@ static int mali_vpu_session_wait_pending(struct mali_vpu_session *sess)
mutex_lock(&sess->vq_lock);
+ if (sess->error)
+ return sess->error;
+
if (!timeout && sess->fw.pending_response_count)
- return -ETIMEDOUT;
+ mali_vpu_session_set_error(sess, -ETIMEDOUT);
}
- return 0;
+ return sess->error;
}
static int mali_vpu_session_fw_init(struct mali_vpu_session *sess)
@@ -518,6 +551,7 @@ int mali_vpu_session_init(struct mali_vpu_session *sess)
goto err_mmu_cleanup;
}
+ sess->error = 0;
sess->fw_bin = NULL;
sess->port_src.stream_requested = false;
sess->port_dst.stream_requested = false;
@@ -566,6 +600,9 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
mali_vpu_session_get_port(sess, vb2_buf->type);
int ret;
+ if (sess->error)
+ return sess->error;
+
if (!port->stream_requested ||
!mali_vpu_session_can_stream(sess) ||
sess->fw_state != MALI_VPU_FW_STATE_RUNNING) {
@@ -674,6 +711,9 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
lockdep_assert_held(&sess->vq_lock);
+ if (sess->error)
+ return sess->error;
+
if (port->stream_requested)
return 0;
@@ -687,8 +727,10 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
mali_vpu_session_can_stream(sess)) {
ret = mali_vpu_session_fw_init(sess);
- if (ret)
+ if (ret) {
+ mali_vpu_session_set_error(sess, ret);
goto err_rollback;
+ }
}
return 0;
@@ -711,18 +753,31 @@ int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type
port->stream_requested = false;
+ if (sess->error)
+ return 0;
+
if (sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY ||
!sess->sched_sess.lsid)
return 0;
- if (mali_vpu_is_raw_q(sess, type))
- return mali_vpu_session_wait_pending(sess);
+ if (mali_vpu_is_raw_q(sess, type)) {
+ ret = mali_vpu_session_wait_pending(sess);
+ if (ret)
+ goto err_session;
+
+ return 0;
+ }
ret = mali_vpu_session_fw_stop(sess);
if (ret)
- return ret;
+ goto err_session;
return mali_vpu_session_wait_pending(sess);
+
+err_session:
+ mali_vpu_session_set_error(sess, ret);
+
+ return ret;
}
static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
@@ -739,13 +794,15 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
* A synchronous cache hit precedes marking the port requested.
* An asynchronous callback acquires vq_lock after preparation.
*/
- if (!same_thread && mali_vpu_both_ports_unrequested(sess)) {
+ if (sess->error ||
+ (!same_thread && mali_vpu_both_ports_unrequested(sess))) {
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
goto err_cache_put;
}
if (vpu_fw_bin->load_status) {
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ mali_vpu_session_set_error(sess, vpu_fw_bin->load_status);
goto err_cache_put;
}
@@ -758,6 +815,7 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
ret = mali_vpu_sched_session_prepare(&sess->sched_sess, sess);
if (ret) {
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ mali_vpu_session_set_error(sess, ret);
goto err_cache_put;
}
@@ -767,6 +825,7 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
dev_err(sess->vpu_dev->dev,
"failed to initialise firmware: %d\n", ret);
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ mali_vpu_session_set_error(sess, ret);
goto err_cache_put;
}
@@ -782,6 +841,7 @@ static void mali_vpu_session_fw_bin_ready(struct mali_vpu_fw_bin *vpu_fw_bin,
dev_err(sess->vpu_dev->dev,
"failed to start streaming: %d\n", ret);
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_UNINITIALIZED;
+ mali_vpu_session_set_error(sess, ret);
goto err_fw_cleanup;
}
}
@@ -833,6 +893,9 @@ int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess)
lockdep_assert_held(&sess->vq_lock);
+ if (sess->error)
+ return sess->error;
+
if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY ||
sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_LOADING)
return 0;
@@ -849,6 +912,7 @@ int mali_vpu_session_request_fw_load(struct mali_vpu_session *sess)
ret = mali_vpu_fw_cache_get(sess->vpu_dev->fw_cache, &fw_key, sess->fw_event);
if (ret) {
sess->fw_bin_state = MALI_VPU_SESSION_FW_BIN_FAILED;
+ mali_vpu_session_set_error(sess, ret);
complete_all(&sess->fw_load_done);
return ret;
}
@@ -864,31 +928,40 @@ void mali_vpu_session_irq(struct mali_vpu_session *sess)
mutex_lock(&sess->vq_lock);
+ if (sess->error)
+ goto unlock_mutex;
+
ret = mali_vpu_fw_handle_rpc(&sess->fw, &flush_mmu);
if (ret) {
if (flush_mmu)
mali_vpu_sched_flush_mmu(&sess->sched_sess, &sess->vpu_dev->sched);
mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
- if (ret < 0)
+ if (ret < 0) {
+ mali_vpu_session_set_error(sess, ret);
goto unlock_mutex;
+ }
}
do {
ret = mali_vpu_fw_parse_msg(&sess->fw, &msg);
- if (ret < 0)
+ if (ret < 0) {
+ mali_vpu_session_set_error(sess, ret);
goto unlock_mutex;
+ }
if (!ret)
break;
ret = mali_vpu_session_handle_fw_msg(sess, &msg);
- if (ret)
+ if (ret < 0) {
+ mali_vpu_session_set_error(sess, ret);
goto unlock_mutex;
+ }
} while (ret >= 0);
-unlock_mutex:
wake_up(&sess->fw_waitq);
+unlock_mutex:
mutex_unlock(&sess->vq_lock);
}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 5cabb7844131b..c6e1add34a8f1 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -60,6 +60,9 @@ struct mali_vpu_port {
* Session cleanup waits for an in-flight callback before releasing the
* session-owned binary reference and callback event. On timeout, it cancels
* the pending cache waiter or waits for an already claimed callback.
+ *
+ * Errors follow the session error-handling policy. A FAILED binary state alone
+ * does not imply that another STREAMON can recover the session.
*/
enum mali_vpu_session_fw_bin_state {
MALI_VPU_SESSION_FW_BIN_UNINITIALIZED = 0,
@@ -93,6 +96,7 @@ enum mali_vpu_session_fw_bin_state {
* @fw_requested_state: Last firmware state requested by the driver.
* @sched_sess: Scheduler state used to map this session to an LSID.
* @ncores: Number of cores assigned to this session by scheduler.
+ * @error: Fatal session error code.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
@@ -122,6 +126,7 @@ struct mali_vpu_session {
struct mali_vpu_sched_session sched_sess;
u32 ncores;
+ int error;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 23/28] media: mali-vpu: propagate buffer metadata
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (21 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 22/28] media: mali-vpu: handle fatal firmware error Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 24/28] media: mali-vpu: add decoder EOS and DRC handling Charvi Mehta
` (4 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Firmware may return compressed buffers before producing their
corresponding decoded frames and may reorder decoded output. Preserve
timestamp-associated flags and timecodes across firmware processing and
restore them on the matching CAPTURE buffers.
Report the negotiated field on CAPTURE buffers and maintain independent
sequence counters for the OUTPUT and CAPTURE queues.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-session.c | 74 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.h | 21 ++++++
.../platform/arm/mali-vpu/mali-vpu-vb2.c | 3 +
3 files changed, 98 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 9cbd116124109..ea80aacd930d8 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -3,6 +3,7 @@
* Copyright (C) 2026 ARM Limited.
*/
+#include <linux/array_size.h>
#include <linux/device.h>
#include <linux/err.h>
#include <linux/errno.h>
@@ -19,6 +20,9 @@
#include "mali-vpu-variant.h"
#include "mali-vpu-vb2.h"
+#define MALI_VPU_TS_METADATA_FLAGS \
+ (V4L2_BUF_FLAG_TIMECODE | V4L2_BUF_FLAG_TSTAMP_SRC_MASK)
+
static const unsigned int mali_vpu_fw_response_timeout_ms = 10000;
static const unsigned int mali_vpu_fw_load_timeout_ms = 10000;
@@ -288,12 +292,65 @@ static int mali_vpu_session_sched_activate(struct mali_vpu_session *sess)
return mali_vpu_sched_switch_in(&sess->sched_sess, &sess->vpu_dev->sched);
}
+static void mali_vpu_session_store_ts_metadata(struct mali_vpu_session *sess,
+ const struct vb2_v4l2_buffer *vbuf)
+{
+ struct mali_vpu_ts_metadata *metadata;
+
+ if (sess->ts_metadata_idx >= ARRAY_SIZE(sess->ts_metadata))
+ sess->ts_metadata_idx = 0;
+
+ metadata = &sess->ts_metadata[sess->ts_metadata_idx++];
+ metadata->timestamp = vbuf->vb2_buf.timestamp;
+ metadata->flags = vbuf->flags & MALI_VPU_TS_METADATA_FLAGS;
+ metadata->valid = true;
+ metadata->timecode = vbuf->timecode;
+}
+
+static void mali_vpu_session_restore_ts_metadata(struct mali_vpu_session *sess,
+ struct vb2_v4l2_buffer *vbuf)
+{
+ u64 timestamp = vbuf->vb2_buf.timestamp;
+ u32 i;
+
+ /*
+ * Matching the auxiliary metadata by timestamp is ambiguous when
+ * OUTPUT buffers have duplicate timestamps: timecode and timestamp-
+ * source flags may be restored from a different OUTPUT buffer.
+ *
+ * The timestamp itself is returned independently through firmware.
+ * Unique association tags require reliable metadata lifetime tracking.
+ */
+
+ vbuf->flags &= ~MALI_VPU_TS_METADATA_FLAGS;
+
+ for (i = 0; i < ARRAY_SIZE(sess->ts_metadata); i++) {
+ struct mali_vpu_ts_metadata *metadata = &sess->ts_metadata[i];
+
+ if (!metadata->valid || metadata->timestamp != timestamp)
+ continue;
+
+ vbuf->flags |= metadata->flags;
+ if (metadata->flags & V4L2_BUF_FLAG_TIMECODE)
+ vbuf->timecode = metadata->timecode;
+
+ return;
+ }
+}
+
+void mali_vpu_session_reset_ts_metadata(struct mali_vpu_session *sess)
+{
+ memset(sess->ts_metadata, 0, sizeof(sess->ts_metadata));
+ sess->ts_metadata_idx = 0;
+}
+
static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
struct mali_vpu_buffer *vpu_buf,
enum vb2_buffer_state state)
{
struct vb2_v4l2_buffer *vbuf = &vpu_buf->m2m_buf.vb;
struct vb2_buffer *vb2_buf = &vbuf->vb2_buf;
+ struct mali_vpu_port *port;
if (vb2_buf->state != VB2_BUF_STATE_ACTIVE) {
dev_dbg(sess->vpu_dev->dev,
@@ -302,6 +359,14 @@ static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
return;
}
+ port = V4L2_TYPE_IS_OUTPUT(vpu_buf->type) ? &sess->port_src : &sess->port_dst;
+
+ vbuf->sequence = port->sequence++;
+ if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type)) {
+ vbuf->field = port->curr_fmt.fmt.pix_mp.field;
+ mali_vpu_session_restore_ts_metadata(sess, vbuf);
+ }
+
if (V4L2_TYPE_IS_OUTPUT(vpu_buf->type))
v4l2_m2m_src_buf_remove_by_buf(sess->m2m_ctx, vbuf);
else
@@ -607,6 +672,10 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
!mali_vpu_session_can_stream(sess) ||
sess->fw_state != MALI_VPU_FW_STATE_RUNNING) {
vpu_buf->attr = MALI_VPU_BUF_ATTR_DEFERRED;
+
+ if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type))
+ mali_vpu_session_store_ts_metadata(sess, v4l2_buf);
+
return 0;
}
@@ -616,6 +685,9 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
vpu_buf->attr = MALI_VPU_BUF_ATTR_QUEUED;
+ if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type))
+ mali_vpu_session_store_ts_metadata(sess, v4l2_buf);
+
mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
return 0;
@@ -733,6 +805,8 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
}
}
+ port->sequence = 0;
+
return 0;
err_rollback:
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index c6e1add34a8f1..7565a680397a8 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -29,11 +29,27 @@ struct mali_vpu_sched_session;
* @supported_fmts: List of format descriptors supported by the queue.
* @stream_requested: Port/Queue is being prepared for streaming or has
* successfully completed preparation. Protected by vq_lock.
+ * @sequence: Sequence number assigned to the next completed buffer.
*/
struct mali_vpu_port {
struct v4l2_format curr_fmt;
struct mali_vpu_format_list supported_fmts;
bool stream_requested;
+ u32 sequence;
+};
+
+/**
+ * struct mali_vpu_ts_metadata - OUTPUT metadata propagated to CAPTURE.
+ * @timestamp: Timestamp used as the firmware association tag.
+ * @flags: Timestamp-source and timecode-presence flags.
+ * @valid: Whether this cache entry has been populated.
+ * @timecode: Timecode associated with @timestamp.
+ */
+struct mali_vpu_ts_metadata {
+ u64 timestamp;
+ u32 flags;
+ bool valid;
+ struct v4l2_timecode timecode;
};
/*
@@ -80,6 +96,8 @@ enum mali_vpu_session_fw_bin_state {
* @mmu: MMU handle for this session
* @vq_lock: Mutex lock to serialise per-session queue related ioctls.
* @min_buffers_required: Driver-reported minimum buffers for role-specific queue.
+ * @ts_metadata: OUTPUT metadata indexed by timestamp for CAPTURE propagation.
+ * @ts_metadata_idx: Next timestamp metadata slot to use.
* @ctrls_caps: Variant-provided immutable control capability table.
* @num_ctrls_caps: Number of entries in @ctrls_caps.
* @ctrls_state: Per-session control cache and staging state.
@@ -106,6 +124,8 @@ struct mali_vpu_session {
struct mali_vpu_mmu mmu;
struct mutex vq_lock; /* lock to serialise per-session ioctls */
u32 min_buffers_required;
+ struct mali_vpu_ts_metadata ts_metadata[VIDEO_MAX_FRAME];
+ u32 ts_metadata_idx;
const struct mali_vpu_ctrl_desc *ctrls_caps;
u32 num_ctrls_caps;
@@ -139,6 +159,7 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type);
void mali_vpu_session_irq(struct mali_vpu_session *sess);
int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_buf);
+void mali_vpu_session_reset_ts_metadata(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
index 41158bf4e0ecb..81a447ceeb755 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vb2.c
@@ -137,6 +137,9 @@ static void mali_vpu_vb2_unprepare_streaming(struct vb2_queue *q)
if (!vb2_start_streaming_called(q))
v4l2_m2m_update_stop_streaming_state(sess->m2m_ctx, q);
+
+ if (!sess->port_src.stream_requested && !sess->port_dst.stream_requested)
+ mali_vpu_session_reset_ts_metadata(sess);
}
static const struct vb2_ops mali_vpu_vb2_ops = {
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 24/28] media: mali-vpu: add decoder EOS and DRC handling
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (22 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 23/28] media: mali-vpu: propagate buffer metadata Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 25/28] media: mali-vpu: handle firmware buffer parameters Charvi Mehta
` (3 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Shobhit Pandey <shobhit.pandey@arm.com>
Add decoder START/STOP command support and propagate EOS through the
firmware protocol. Handle sequence and allocation parameters, queue
flush responses, dynamic resolution changes, and V4L2 EOS/
source-change events.
Track capture reconfiguration state and prevent stale pre-DRC buffers
from being submitted back to firmware.
Signed-off-by: Shobhit Pandey <shobhit.pandey@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-dec.c | 29 ++++
.../platform/arm/mali-vpu/mali-vpu-dec.h | 4 +
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 102 ++++++++++-
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 38 +++++
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 64 +++++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 158 +++++++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-session.h | 3 +
.../platform/arm/mali-vpu/mali-vpu-vidioc.c | 3 +
8 files changed, 390 insertions(+), 11 deletions(-)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
index 51d509abb9dd1..7e34c24ae050f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
@@ -23,6 +23,7 @@ void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
sess->num_ctrls_caps = variant_ctrls->num_dec_ctrls;
sess->min_buffers_required = 3; /* For CAPTURE/dst queue */
+ sess->drc_pending = false;
}
int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess,
@@ -64,3 +65,31 @@ int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid,
return mali_vpu_session_fw_send_set_option(sess, &option);
}
+
+int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd *cmd)
+{
+ struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ int ret;
+
+ ret = v4l2_m2m_ioctl_try_decoder_cmd(filp, priv, cmd);
+ if (ret)
+ return ret;
+
+ mutex_lock(&sess->vq_lock);
+
+ switch (cmd->cmd) {
+ case V4L2_DEC_CMD_STOP:
+ ret = mali_vpu_session_fw_send_eos(sess);
+ break;
+ case V4L2_DEC_CMD_START:
+ vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(sess->m2m_ctx));
+ ret = 0;
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ mutex_unlock(&sess->vq_lock);
+ return ret;
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
index e323b38475c81..a108aaaa4a959 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
@@ -8,7 +8,9 @@
#include <linux/types.h>
+struct file;
struct mali_vpu_session;
+struct v4l2_decoder_cmd;
void mali_vpu_session_init_dec(struct mali_vpu_session *sess);
@@ -16,4 +18,6 @@ void mali_vpu_session_init_dec(struct mali_vpu_session *sess);
int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess, u32 cid, s64 value);
int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid, s64 value);
+int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd *cmd);
+
#endif /* __MALI_VPU_DEC_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 4e3eaecb8ebee..55bad502cae0b 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -800,6 +800,43 @@ int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw,
return ret;
}
+int mali_vpu_fw_send_flush_src(struct mali_vpu_fw *fw)
+{
+ int ret;
+
+ ret = mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve, MVE_REQUEST_CODE_INPUT_FLUSH,
+ NULL, 0);
+ if (!ret)
+ fw->pending_response_count++;
+
+ return ret;
+}
+
+int mali_vpu_fw_send_flush_dst(struct mali_vpu_fw *fw)
+{
+ int ret;
+
+ ret = mali_vpu_fw_write_msg(fw, &fw->msg_host, &fw->msg_mve, MVE_REQUEST_CODE_OUTPUT_FLUSH,
+ NULL, 0);
+ if (!ret)
+ fw->pending_response_count++;
+
+ return ret;
+}
+
+int mali_vpu_fw_send_eos(struct mali_vpu_fw *fw)
+{
+ struct mve_buffer_bitstream bs_buffer = {
+ .host_handle = 0,
+ .bitstream_buf_addr = MVE_MEM_REGION_PROTECTED_ADDR_BEGIN,
+ .bitstream_flags = MVE_BUFFER_BITSTREAM_FLAG_EOS,
+ };
+
+ return mali_vpu_fw_write_msg(fw, &fw->buf_in_host, &fw->buf_in_mve,
+ MVE_BUFFER_CODE_BITSTREAM, &bs_buffer,
+ sizeof(bs_buffer));
+}
+
static struct mali_vpu_buffer *
mali_vpu_fw_resolve_returned_buffer(struct mali_vpu_fw *fw,
u64 host_handle,
@@ -831,15 +868,21 @@ mali_vpu_fw_resolve_returned_buffer(struct mali_vpu_fw *fw,
return vpu_buf;
}
-static int
-mali_vpu_convert_buffer_frame(struct mali_vpu_fw *fw,
- struct mali_vpu_fw_msg *msg,
- u32 buf_type,
- struct mve_buffer_frame *mve_frame_buf)
+static int mali_vpu_convert_buffer_frame(struct mali_vpu_fw *fw,
+ struct mali_vpu_fw_msg *msg,
+ u32 buf_type,
+ struct mve_buffer_frame *mve_frame_buf)
{
struct mali_vpu_buffer *vpu_buf;
unsigned int i;
+ /* NULL host handle with the EOS flag denotes a firmware-generated EOS marker. */
+ if (!mve_frame_buf->host_handle &&
+ mve_frame_buf->frame_flags & MVE_BUFFER_FRAME_FLAG_EOS) {
+ msg->code = MALI_VPU_FW_CODE_EOS;
+ return 0;
+ }
+
vpu_buf = mali_vpu_fw_resolve_returned_buffer(fw,
mve_frame_buf->host_handle,
buf_type);
@@ -880,10 +923,18 @@ mali_vpu_convert_buffer_bitstream(struct mali_vpu_fw *fw,
u32 buf_type,
struct mve_buffer_bitstream *mve_stream_buf)
{
- struct mali_vpu_buffer *vpu_buf =
- mali_vpu_fw_resolve_returned_buffer(fw,
- mve_stream_buf->host_handle,
- buf_type);
+ struct mali_vpu_buffer *vpu_buf;
+
+ /* NULL host handle with the EOS flag denotes a firmware-generated EOS marker. */
+ if (!mve_stream_buf->host_handle &&
+ mve_stream_buf->bitstream_flags & MVE_BUFFER_BITSTREAM_FLAG_EOS) {
+ msg->code = MALI_VPU_FW_CODE_EOS;
+ return 0;
+ }
+
+ vpu_buf = mali_vpu_fw_resolve_returned_buffer(fw,
+ mve_stream_buf->host_handle,
+ buf_type);
if (IS_ERR(vpu_buf))
return PTR_ERR(vpu_buf);
@@ -954,6 +1005,8 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
struct mve_response_set_option_confirm set_option_confirm;
struct mve_response_set_option_fail set_option_fail;
struct mve_response_state_change state_change;
+ struct mve_response_frame_alloc_parameters alloc_params;
+ struct mve_response_sequence_parameters seq_params;
struct mve_response_error error;
} fw_msg = {};
size_t size = sizeof(fw_msg);
@@ -1053,6 +1106,37 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
ret = mali_vpu_fw_get_buffer(fw, buf_host_page, buf_mve_page, msg,
V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
break;
+ case MVE_RESPONSE_CODE_FRAME_ALLOC_PARAM:
+ if (size != sizeof(fw_msg.alloc_params))
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_ALLOC_PARAM;
+ msg->data.alloc_params.width = fw_msg.alloc_params.planar_alloc_frame_width;
+ msg->data.alloc_params.height = fw_msg.alloc_params.planar_alloc_frame_height;
+ break;
+ case MVE_RESPONSE_CODE_SEQUENCE_PARAMETERS:
+ if (size != sizeof(fw_msg.seq_params))
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_SEQUENCE_PARAMS;
+ msg->data.seq_params.planar.buffers_min = fw_msg.seq_params.num_buffers_planar;
+ break;
+ case MVE_RESPONSE_CODE_INPUT_FLUSHED:
+ msg->code = MALI_VPU_FW_CODE_INPUT_FLUSH;
+
+ if (fw->pending_response_count == 0)
+ return -EPROTO;
+
+ fw->pending_response_count--;
+ break;
+ case MVE_RESPONSE_CODE_OUTPUT_FLUSHED:
+ msg->code = MALI_VPU_FW_CODE_OUTPUT_FLUSH;
+
+ if (fw->pending_response_count == 0)
+ return -EPROTO;
+
+ fw->pending_response_count--;
+ break;
case MVE_RESPONSE_CODE_ERROR: {
size_t message_offset = offsetof(struct mve_response_error, message);
size_t message_size;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index c936730698837..ca90cc71ac64e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -84,6 +84,29 @@ struct mali_vpu_fw_set_option {
};
};
+/**
+ * struct mali_vpu_fw_seq_param - Sequence parameters.
+ * @planar: Planar output-buffer requirements.
+ * @planar.buffers_min: Minimum number of planar buffers required.
+ */
+struct mali_vpu_fw_seq_param {
+ struct {
+ u32 buffers_min;
+ } planar;
+};
+
+/**
+ * struct mali_vpu_fw_alloc_param - Allocation parameters
+ * @height: in pixels
+ * @width: in pixels
+ *
+ * Dimensions of the decoded frame buffer
+ */
+struct mali_vpu_fw_alloc_param {
+ u32 height;
+ u32 width;
+};
+
/**
* struct mali_vpu_fw_error - Firmware error message.
* @error_code: What kind of error that was reported.
@@ -116,6 +139,11 @@ struct mali_vpu_fw_error {
* @MALI_VPU_FW_CODE_STATE_CHANGE: Firmware state-change request or response.
* @MALI_VPU_FW_CODE_SWITCH_IN: Firmware was booted on a Core in a session.
* @MALI_VPU_FW_CODE_SWITCH_OUT: The last core in a session was switched out.
+ * @MALI_VPU_FW_CODE_EOS: Firmware returned an end-of-stream marker.
+ * @MALI_VPU_FW_CODE_INPUT_FLUSH: Firmware successfully completed a OUTPUT queue flush.
+ * @MALI_VPU_FW_CODE_OUTPUT_FLUSH: Firmware successfully completed a CAPTURE queue flush.
+ * @MALI_VPU_FW_CODE_SEQUENCE_PARAMS: Firmware reported new sequence parameters.
+ * @MALI_VPU_FW_CODE_ALLOC_PARAM: Firmware reported new frame allocation parameters.
* @MALI_VPU_FW_CODE_UNKNOWN: Unrecognized firmware message.
* @MALI_VPU_FW_CODE_MAX: Sentinel marking the end of the message-code enum.
*/
@@ -131,6 +159,11 @@ enum mali_vpu_fw_code {
MALI_VPU_FW_CODE_STATE_CHANGE, /* Driver <-> Firmware. */
MALI_VPU_FW_CODE_SWITCH_IN, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_SWITCH_OUT, /* Driver <-> Firmware. */
+ MALI_VPU_FW_CODE_EOS, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_INPUT_FLUSH, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_OUTPUT_FLUSH, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_SEQUENCE_PARAMS, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_ALLOC_PARAM, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_MAX
};
@@ -147,6 +180,8 @@ struct mali_vpu_fw_msg {
struct mali_vpu_fw_job job;
struct mali_vpu_fw_error error;
struct mali_vpu_buffer *vpu_buf;
+ struct mali_vpu_fw_alloc_param alloc_params;
+ struct mali_vpu_fw_seq_param seq_params;
} data;
};
@@ -260,6 +295,9 @@ int mali_vpu_fw_send_buffer(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *fw_m
int mali_vpu_fw_send_ping(struct mali_vpu_fw *fw);
int mali_vpu_fw_send_state_change(struct mali_vpu_fw *fw, enum mali_vpu_fw_state state);
int mali_vpu_fw_send_set_option(struct mali_vpu_fw *fw, struct mali_vpu_fw_set_option *option);
+int mali_vpu_fw_send_eos(struct mali_vpu_fw *fw);
+int mali_vpu_fw_send_flush_src(struct mali_vpu_fw *fw);
+int mali_vpu_fw_send_flush_dst(struct mali_vpu_fw *fw);
int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg);
int mali_vpu_fw_get_region(enum mali_vpu_fw_region region, mali_vpu_mmu_va *begin,
mali_vpu_mmu_va *end);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index 7f3a13ec744b9..cb4533a367957 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -224,6 +224,8 @@ struct mve_msg_header {
*/
#define MVE_REQUEST_CODE_GO (1001) /* no extra data */
#define MVE_REQUEST_CODE_STOP (1002) /* no extra data */
+ #define MVE_REQUEST_CODE_INPUT_FLUSH (1003) /* no extra data */
+ #define MVE_REQUEST_CODE_OUTPUT_FLUSH (1004) /* no extra data */
#define MVE_REQUEST_CODE_PING (1006) /* no extra data */
#define MVE_REQUEST_CODE_JOB (1009) /* struct mve_request_job */
#define MVE_REQUEST_CODE_SET_OPTION (1010) /* struct
@@ -245,12 +247,18 @@ struct mve_msg_header {
#define MVE_RESPONSE_CODE_INPUT (2005)
/* no extra data, but buffer placed in buffer queue */
#define MVE_RESPONSE_CODE_OUTPUT (2006)
+ #define MVE_RESPONSE_CODE_INPUT_FLUSHED (2007)
+ #define MVE_RESPONSE_CODE_OUTPUT_FLUSHED (2008)
#define MVE_RESPONSE_CODE_PONG (2009)
/* struct mve_response_error */
#define MVE_RESPONSE_CODE_ERROR (2010)
/* struct mve_response_state_change */
#define MVE_RESPONSE_CODE_STATE_CHANGE (2011)
#define MVE_RESPONSE_CODE_IDLE (2013)
+ /* struct mve_response_frame_alloc_parameters */
+ #define MVE_RESPONSE_CODE_FRAME_ALLOC_PARAM (2014)
+ /* struct mve_response_sequence_parameters */
+ #define MVE_RESPONSE_CODE_SEQUENCE_PARAMETERS (2015)
/* struct mve_response_set_option_fail */
#define MVE_RESPONSE_CODE_SET_OPTION_FAIL (2017)
@@ -356,6 +364,62 @@ struct mve_response_job_dequeued {
u32 valid_job;
};
+/* Decode only: This message is sent from MVE to the host so that it can
+ * allocate large enough output buffers. Output buffers that are to small
+ * will be returned to the host marked as 'rejected'.
+ */
+struct mve_response_frame_alloc_parameters {
+ /* Please note that the below information is a hint
+ * for what buffers to allocate, it does not say
+ * what actual resolution an output picture has.
+ */
+
+ /* To use if allocating PLANAR YUV output buffers: */
+ u16 planar_alloc_frame_width;
+ u16 planar_alloc_frame_height;
+ u32 afbc_alloc_bytes;
+ u32 afbc_alloc_bytes_downscaled;
+ u16 afbc_width_in_superblocks;
+ u16 afbc_width_in_superblocks_downscaled;
+
+ /* For PLANAR YUV output, every plane's address need to be adjusted to get
+ * optimal AXI bursts when the pixel data is written, the values below may
+ * be used to calculate address offsets.
+ */
+ u16 cropx;
+ u16 cropy;
+
+ u32 mbinfo_alloc_bytes; /* Only for debugging */
+};
+
+/* Decode only: This message is sent from MVE to the host so that it can
+ * allocate suitable output buffers. The needed size of the buffer is sent
+ * in a separate message (above).
+ * When MVE sends the message below, it enters a waiting-state and will not
+ * make any progress until the host sends an output-flush command, upon
+ * which MVE will return all output buffers, followed by a message saying
+ * that the output has been flushed. Only then should the host start
+ * enqueueing new output buffers.
+ */
+struct mve_response_sequence_parameters {
+ /* Other stream parameters affecting buffer allocation,
+ * any change in these values will trigger a flush.
+ */
+ u8 interlace; /* 0 or 1 */
+ u8 chroma_format;
+ #define MVE_CHROMA_FORMAT_MONO (0x0)
+ #define MVE_CHROMA_FORMAT_420 (0x1)
+ #define MVE_CHROMA_FORMAT_422 (0x2)
+ #define MVE_CHROMA_FORMAT_440 (0x3)
+ #define MVE_CHROMA_FORMAT_ARGB (0x4)
+ u8 bitdepth_luma; /* 8, 9 or 10 */
+ u8 bitdepth_chroma; /* 8, 9 or 10 */
+ u8 num_buffers_planar; /* number of planar buffers needed */
+ u8 num_buffers_afbc;
+ u8 range_mapping_enabled;
+ u8 reserved0;
+};
+
/*
* Fatal error message from firmware, if sent then no further
* operation is possible.
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index ea80aacd930d8..fcfad581223be 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -9,6 +9,7 @@
#include <linux/errno.h>
#include <linux/slab.h>
#include <linux/string.h>
+#include <media/v4l2-event.h>
#include <media/v4l2-mem2mem.h>
#include "mali-vpu-session.h"
@@ -71,6 +72,14 @@ static int mali_vpu_set_port_defaults(struct mali_vpu_session *sess)
fmsz->width_default, fmsz->height_default);
}
+static bool mali_vpu_both_queues_streaming(const struct mali_vpu_session *sess)
+{
+ struct vb2_queue *src_q = v4l2_m2m_get_src_vq(sess->m2m_ctx);
+ struct vb2_queue *dst_q = v4l2_m2m_get_dst_vq(sess->m2m_ctx);
+
+ return vb2_is_streaming(src_q) && vb2_is_streaming(dst_q);
+}
+
static bool
mali_vpu_both_ports_unrequested(struct mali_vpu_session *sess)
{
@@ -157,6 +166,67 @@ int mali_vpu_session_fw_send_set_option(struct mali_vpu_session *sess,
return 0;
}
+int mali_vpu_session_fw_send_eos(struct mali_vpu_session *sess)
+{
+ int ret;
+
+ if (sess->error)
+ return sess->error;
+
+ if (!mali_vpu_both_queues_streaming(sess))
+ return 0;
+
+ ret = mali_vpu_fw_send_eos(&sess->fw);
+ if (ret)
+ return ret;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+ return 0;
+}
+
+static int mali_vpu_session_fw_send_flush(struct mali_vpu_session *sess, bool to_src)
+{
+ int ret;
+
+ if (to_src)
+ ret = mali_vpu_fw_send_flush_src(&sess->fw);
+ else
+ ret = mali_vpu_fw_send_flush_dst(&sess->fw);
+
+ if (ret)
+ return ret;
+
+ mali_vpu_sched_send_irq(&sess->sched_sess, &sess->vpu_dev->sched);
+ return 0;
+}
+
+static int mali_vpu_session_send_drc_event(struct mali_vpu_session *sess)
+{
+ const struct v4l2_event event = {
+ .type = V4L2_EVENT_SOURCE_CHANGE,
+ .u.src_change.changes = V4L2_EVENT_SRC_CH_RESOLUTION
+ };
+
+ v4l2_event_queue_fh(&sess->fh, &event);
+ return 0;
+}
+
+static int
+mali_vpu_session_update_capture_format(struct mali_vpu_session *sess,
+ const struct mali_vpu_fw_alloc_param alloc_params)
+{
+ struct v4l2_pix_format_mplane *src = &sess->port_src.curr_fmt.fmt.pix_mp;
+ struct v4l2_pix_format_mplane *dst = &sess->port_dst.curr_fmt.fmt.pix_mp;
+ int ret = 0;
+
+ src->width = alloc_params.width;
+ src->height = alloc_params.height;
+
+ ret = v4l2_fill_pixfmt_mp(dst, dst->pixelformat, alloc_params.width,
+ alloc_params.height);
+ return ret;
+}
+
static int mali_vpu_buffer_map_fw_region(struct mali_vpu_session *sess,
struct mali_vpu_buffer *vpu_buf)
{
@@ -350,6 +420,7 @@ static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
{
struct vb2_v4l2_buffer *vbuf = &vpu_buf->m2m_buf.vb;
struct vb2_buffer *vb2_buf = &vbuf->vb2_buf;
+ struct vb2_queue *q = v4l2_m2m_get_vq(sess->m2m_ctx, vb2_buf->type);
struct mali_vpu_port *port;
if (vb2_buf->state != VB2_BUF_STATE_ACTIVE) {
@@ -373,6 +444,17 @@ static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
v4l2_m2m_dst_buf_remove_by_buf(sess->m2m_ctx, vbuf);
v4l2_m2m_buf_done(vbuf, state);
+
+ if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
+ !q->last_buffer_dequeued &&
+ (vbuf->flags & V4L2_BUF_FLAG_LAST) &&
+ !sess->drc_pending) {
+ const struct v4l2_event event = {
+ .type = V4L2_EVENT_EOS,
+ };
+
+ v4l2_event_queue_fh(&sess->fh, &event);
+ }
}
static void mali_vpu_session_set_error(struct mali_vpu_session *sess, int error)
@@ -403,6 +485,7 @@ static void mali_vpu_session_set_error(struct mali_vpu_session *sess, int error)
static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
const struct mali_vpu_fw_msg *fw_msg)
{
+ int ret = 0;
switch (fw_msg->code) {
case MALI_VPU_FW_CODE_ERROR:
dev_warn(sess->vpu_dev->dev, "firmware rsp: Error. code=%u, message=%s\n",
@@ -464,6 +547,49 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
case MALI_VPU_FW_CODE_IDLE:
dev_dbg(sess->vpu_dev->dev, "fw entered Idle/sleeping state.\n");
return 0;
+ case MALI_VPU_FW_CODE_EOS:
+ /*
+ * This is driver generated dummy EOS buffer response no action
+ * required as EOS is handled via buffer flags.
+ */
+ return 0;
+ case MALI_VPU_FW_CODE_INPUT_FLUSH:
+ /*
+ * Acknowledge completion of the input flush. No additional session
+ * state update is required.
+ */
+ return 0;
+ case MALI_VPU_FW_CODE_OUTPUT_FLUSH:
+ dev_dbg(sess->vpu_dev->dev, "fw rsp: Dst queue flushed\n");
+ if (sess->role == MALI_VPU_DECODER && sess->drc_pending)
+ return mali_vpu_session_send_drc_event(sess);
+
+ return 0;
+ case MALI_VPU_FW_CODE_ALLOC_PARAM:
+ ret = mali_vpu_session_update_capture_format(sess, fw_msg->data.alloc_params);
+ if (!ret)
+ sess->drc_pending = true;
+
+ return ret;
+ case MALI_VPU_FW_CODE_SEQUENCE_PARAMS: {
+ struct vb2_queue *dst_q;
+ u32 buffers_min;
+
+ buffers_min = fw_msg->data.seq_params.planar.buffers_min;
+ dst_q = v4l2_m2m_get_dst_vq(sess->m2m_ctx);
+
+ if (!buffers_min || buffers_min > dst_q->max_num_buffers) {
+ dev_err(sess->vpu_dev->dev,
+ "invalid minimum buffer count %u, max is %u\n",
+ buffers_min, dst_q->max_num_buffers);
+ return -EPROTO;
+ }
+
+ sess->min_buffers_required = buffers_min;
+ dst_q->min_reqbufs_allocation = sess->min_buffers_required;
+
+ return mali_vpu_session_fw_send_flush(sess, false);
+ }
case MALI_VPU_FW_CODE_UNKNOWN:
case MALI_VPU_FW_CODE_MAX:
default:
@@ -668,6 +794,22 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
if (sess->error)
return sess->error;
+ /*
+ * After SEQUENCE_PARAMS and DRC fw has flushed the destination queue,
+ * but userspace can briefly requeue one of the returned buffer before
+ * it tears down the old destination buffer allocation with REQBUF(0).
+ * Do not give the requeued buffer back to fw as it belongs to pre DRC
+ * allocation which will be unmapped immediately afterwards.
+ */
+ if (sess->drc_pending && sess->role == MALI_VPU_DECODER &&
+ mali_vpu_is_raw_q(sess, vb2_buf->type) &&
+ port->stream_requested) {
+ v4l2_buf->flags |= V4L2_BUF_FLAG_LAST;
+ mali_vpu_session_buffer_done(sess, vpu_buf, VB2_BUF_STATE_DONE);
+
+ return 0;
+ }
+
if (!port->stream_requested ||
!mali_vpu_session_can_stream(sess) ||
sess->fw_state != MALI_VPU_FW_STATE_RUNNING) {
@@ -805,6 +947,9 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
}
}
+ if (sess->role == MALI_VPU_DECODER && mali_vpu_is_raw_q(sess, type))
+ sess->drc_pending = false;
+
port->sequence = 0;
return 0;
@@ -834,7 +979,8 @@ int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type
!sess->sched_sess.lsid)
return 0;
- if (mali_vpu_is_raw_q(sess, type)) {
+ if (sess->drc_pending && sess->role == MALI_VPU_DECODER &&
+ mali_vpu_is_raw_q(sess, type)) {
ret = mali_vpu_session_wait_pending(sess);
if (ret)
goto err_session;
@@ -846,7 +992,15 @@ int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type
if (ret)
goto err_session;
- return mali_vpu_session_wait_pending(sess);
+ ret = mali_vpu_session_fw_send_flush(sess, V4L2_TYPE_IS_OUTPUT(type));
+ if (ret)
+ goto err_session;
+
+ ret = mali_vpu_session_wait_pending(sess);
+ if (ret)
+ goto err_session;
+
+ return 0;
err_session:
mali_vpu_session_set_error(sess, ret);
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 7565a680397a8..45d0b6710aced 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -115,6 +115,7 @@ enum mali_vpu_session_fw_bin_state {
* @sched_sess: Scheduler state used to map this session to an LSID.
* @ncores: Number of cores assigned to this session by scheduler.
* @error: Fatal session error code.
+ * @drc_pending: Dynamic resolution change is waiting for CAPTURE reconfigure.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
@@ -147,6 +148,7 @@ struct mali_vpu_session {
struct mali_vpu_sched_session sched_sess;
u32 ncores;
int error;
+ bool drc_pending;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
@@ -160,6 +162,7 @@ int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type
void mali_vpu_session_irq(struct mali_vpu_session *sess);
int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_buf);
void mali_vpu_session_reset_ts_metadata(struct mali_vpu_session *sess);
+int mali_vpu_session_fw_send_eos(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
index 0fcd3835e52d3..f8d93c849a34f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-vidioc.c
@@ -15,6 +15,7 @@
#include <media/v4l2-ioctl.h>
#include <media/v4l2-mem2mem.h>
+#include "mali-vpu-dec.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-session.h"
#include "mali-vpu-variant.h"
@@ -359,4 +360,6 @@ const struct v4l2_ioctl_ops mali_vpu_ioctl_ops_dec = {
.vidioc_unsubscribe_event = v4l2_event_unsubscribe,
.vidioc_streamon = v4l2_m2m_ioctl_streamon,
.vidioc_streamoff = v4l2_m2m_ioctl_streamoff,
+ .vidioc_decoder_cmd = mali_vpu_decoder_cmd,
+ .vidioc_try_decoder_cmd = v4l2_m2m_ioctl_try_decoder_cmd,
};
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 25/28] media: mali-vpu: handle firmware buffer parameters
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (23 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 24/28] media: mali-vpu: add decoder EOS and DRC handling Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 26/28] media: mali-vpu: handle firmware event responses Charvi Mehta
` (2 subsequent siblings)
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Shobhit Pandey <shobhit.pandey@arm.com>
Add protocol definitions and parsing for firmware buffer-parameter
messages. Recognize colour-description and DPB-held-frame
notifications, log and ignore them when no session action is required.
Signed-off-by: Shobhit Pandey <shobhit.pandey@arm.com>
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 32 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 5 +++
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 19 +++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 9 ++++++
4 files changed, 65 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 55bad502cae0b..34d57b288a682 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -8,6 +8,7 @@
#include <linux/err.h>
#include <linux/errno.h>
#include <linux/sizes.h>
+#include <linux/stddef.h>
#include <linux/string.h>
#include "mali-vpu-firmware.h"
@@ -950,6 +951,33 @@ mali_vpu_convert_buffer_bitstream(struct mali_vpu_fw *fw,
return 0;
}
+static int mali_vpu_convert_buffer_param(struct mali_vpu_fw_msg *msg,
+ struct mve_buffer_param *p,
+ size_t size)
+{
+ if (size < sizeof(p->type))
+ return -EPROTO;
+
+ switch (p->type) {
+ case MVE_BUFFER_PARAM_TYPE_COLOUR_DESCRIPTION:
+ /* The colour-description payload is currently ignored. */
+ msg->code = MALI_VPU_FW_CODE_COLOR_DESC;
+ break;
+ case MVE_BUFFER_PARAM_TYPE_DPB_HELD_FRAMES:
+ if (size < offsetofend(struct mve_buffer_param, data.arg))
+ return -EPROTO;
+
+ msg->code = MALI_VPU_FW_CODE_DPB_HELD_FRAMES;
+ msg->data.dpb_held_frames = p->data.arg;
+ break;
+ default:
+ msg->code = MALI_VPU_FW_CODE_UNKNOWN;
+ break;
+ }
+
+ return 1;
+}
+
static int mali_vpu_fw_get_buffer(struct mali_vpu_fw *fw,
struct mali_vpu_fw_proto_page *buf_host_page,
struct mali_vpu_fw_proto_page *buf_mve_page,
@@ -1007,6 +1035,7 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
struct mve_response_state_change state_change;
struct mve_response_frame_alloc_parameters alloc_params;
struct mve_response_sequence_parameters seq_params;
+ struct mve_buffer_param buffer_param;
struct mve_response_error error;
} fw_msg = {};
size_t size = sizeof(fw_msg);
@@ -1137,6 +1166,9 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
fw->pending_response_count--;
break;
+ case MVE_BUFFER_CODE_PARAM:
+ ret = mali_vpu_convert_buffer_param(msg, &fw_msg.buffer_param, size);
+ break;
case MVE_RESPONSE_CODE_ERROR: {
size_t message_offset = offsetof(struct mve_response_error, message);
size_t message_size;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index ca90cc71ac64e..6c46f23ac8638 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -144,6 +144,8 @@ struct mali_vpu_fw_error {
* @MALI_VPU_FW_CODE_OUTPUT_FLUSH: Firmware successfully completed a CAPTURE queue flush.
* @MALI_VPU_FW_CODE_SEQUENCE_PARAMS: Firmware reported new sequence parameters.
* @MALI_VPU_FW_CODE_ALLOC_PARAM: Firmware reported new frame allocation parameters.
+ * @MALI_VPU_FW_CODE_COLOR_DESC: Firmware reported colour-description metadata.
+ * @MALI_VPU_FW_CODE_DPB_HELD_FRAMES: Firmware reported the number of frames held in the DPB.
* @MALI_VPU_FW_CODE_UNKNOWN: Unrecognized firmware message.
* @MALI_VPU_FW_CODE_MAX: Sentinel marking the end of the message-code enum.
*/
@@ -164,6 +166,8 @@ enum mali_vpu_fw_code {
MALI_VPU_FW_CODE_OUTPUT_FLUSH, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_SEQUENCE_PARAMS, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_ALLOC_PARAM, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_COLOR_DESC, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_DPB_HELD_FRAMES, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_MAX
};
@@ -182,6 +186,7 @@ struct mali_vpu_fw_msg {
struct mali_vpu_buffer *vpu_buf;
struct mali_vpu_fw_alloc_param alloc_params;
struct mali_vpu_fw_seq_param seq_params;
+ u32 dpb_held_frames;
} data;
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index cb4533a367957..76dc7bf076c8f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -269,6 +269,8 @@ struct mve_msg_header {
#define MVE_BUFFER_CODE_FRAME (3001)
/* struct mve_buffer_bitstream */
#define MVE_BUFFER_CODE_BITSTREAM (3002)
+ /* struct mve_buffer_param */
+ #define MVE_BUFFER_CODE_PARAM (3003)
u16 size; /* size in bytes of trailing data, 0 if none */
};
@@ -601,4 +603,21 @@ struct mve_buffer_bitstream {
u32 reserved;
};
+/* Parameters that are sent in the same communication channels
+ * as the buffers. A parameter applies to all subsequent buffers.
+ * Some types are only valid for decode, and some only for encode.
+ */
+struct mve_buffer_param {
+ u32 type; /* Extra data: */
+ #define MVE_BUFFER_PARAM_TYPE_COLOUR_DESCRIPTION (15)
+ /* arg, number of output buffers that are complete and held by firmware in the
+ * DPB for reordering purposes. Valid after the next frame is output.
+ */
+ #define MVE_BUFFER_PARAM_TYPE_DPB_HELD_FRAMES (19)
+
+ union {
+ u32 arg;
+ } data;
+};
+
#endif /* __MALI_VPU_PROTOCOL_DEF_V3_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index fcfad581223be..411421e175e71 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -559,6 +559,15 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
* state update is required.
*/
return 0;
+ case MALI_VPU_FW_CODE_COLOR_DESC:
+ /* Colour metadata is currently not consumed by the driver. */
+ return 0;
+ case MALI_VPU_FW_CODE_DPB_HELD_FRAMES:
+ /* DPB occupancy is informational. */
+ dev_dbg(sess->vpu_dev->dev,
+ "firmware holds %u frames in the DPB\n",
+ fw_msg->data.dpb_held_frames);
+ return 0;
case MALI_VPU_FW_CODE_OUTPUT_FLUSH:
dev_dbg(sess->vpu_dev->dev, "fw rsp: Dst queue flushed\n");
if (sess->role == MALI_VPU_DECODER && sess->drc_pending)
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 26/28] media: mali-vpu: handle firmware event responses
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (24 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 25/28] media: mali-vpu: handle firmware buffer parameters Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 27/28] media: mali-vpu: add stateful decoder state machine Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 28/28] media: MAINTAINERS: add Arm Mali-V76 VPU driver Charvi Mehta
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add protocol definitions needed to parse firmware event responses.
Report corrupt and unsupported streams, and propagate unsupported-stream
events to the session as -EOPNOTSUPP. Ignore informational events that
require no session action.
Also retain sequence bit-depth and interlace information, and reject
interlaced or non-8-bit streams that aren't supported by the driver yet.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-firmware.c | 44 ++++++++++
.../platform/arm/mali-vpu/mali-vpu-firmware.h | 13 +++
.../arm/mali-vpu/mali-vpu-protocol-def-v3.h | 87 +++++++++++++++++++
.../platform/arm/mali-vpu/mali-vpu-session.c | 19 ++++
4 files changed, 163 insertions(+)
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
index 34d57b288a682..99d191b194bfe 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.c
@@ -1036,6 +1036,7 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
struct mve_response_frame_alloc_parameters alloc_params;
struct mve_response_sequence_parameters seq_params;
struct mve_buffer_param buffer_param;
+ struct mve_response_event event;
struct mve_response_error error;
} fw_msg = {};
size_t size = sizeof(fw_msg);
@@ -1149,6 +1150,9 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
msg->code = MALI_VPU_FW_CODE_SEQUENCE_PARAMS;
msg->data.seq_params.planar.buffers_min = fw_msg.seq_params.num_buffers_planar;
+ msg->data.seq_params.bitdepth.luma = fw_msg.seq_params.bitdepth_luma;
+ msg->data.seq_params.bitdepth.chroma = fw_msg.seq_params.bitdepth_chroma;
+ msg->data.seq_params.interlaced = fw_msg.seq_params.interlace;
break;
case MVE_RESPONSE_CODE_INPUT_FLUSHED:
msg->code = MALI_VPU_FW_CODE_INPUT_FLUSH;
@@ -1169,6 +1173,46 @@ int mali_vpu_fw_parse_msg(struct mali_vpu_fw *fw, struct mali_vpu_fw_msg *msg)
case MVE_BUFFER_CODE_PARAM:
ret = mali_vpu_convert_buffer_param(msg, &fw_msg.buffer_param, size);
break;
+ case MVE_RESPONSE_CODE_EVENT: {
+ size_t message_offset = offsetof(struct mve_response_event, event_data.message);
+ size_t message_size;
+
+ msg->code = MALI_VPU_FW_CODE_EVENT;
+
+ if (size < sizeof(fw_msg.event.event_code))
+ return -EPROTO;
+
+ switch (fw_msg.event.event_code) {
+ case MVE_EVENT_ERROR_STREAM_CORRUPT:
+ if (size <= message_offset)
+ return -EPROTO;
+
+ message_size = min(size - message_offset,
+ sizeof(fw_msg.event.event_data.message));
+
+ dev_warn_ratelimited(fw->dev,
+ "firmware reports corrupt stream: %.*s\n",
+ (int)message_size,
+ fw_msg.event.event_data.message);
+
+ break;
+ case MVE_EVENT_ERROR_STREAM_NOT_SUPPORTED:
+ if (size <= message_offset)
+ return -EPROTO;
+
+ message_size = min(size - message_offset,
+ sizeof(fw_msg.event.event_data.message));
+ dev_warn_ratelimited(fw->dev,
+ "firmware reports unsupported stream: %.*s\n",
+ (int)message_size,
+ fw_msg.event.event_data.message);
+ msg->code = MALI_VPU_FW_CODE_STREAM_NOT_SUPPORTED;
+ break;
+ default:
+ break;
+ }
+ break;
+ }
case MVE_RESPONSE_CODE_ERROR: {
size_t message_offset = offsetof(struct mve_response_error, message);
size_t message_size;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
index 6c46f23ac8638..d3928fdc9133f 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-firmware.h
@@ -86,13 +86,22 @@ struct mali_vpu_fw_set_option {
/**
* struct mali_vpu_fw_seq_param - Sequence parameters.
+ * @interlaced: Whether the decoded sequence is interlaced.
* @planar: Planar output-buffer requirements.
* @planar.buffers_min: Minimum number of planar buffers required.
+ * @bitdepth: Decoded luma and chroma bitdepths.
+ * @bitdepth.luma: Decoded luma bitdepth.
+ * @bitdepth.chroma: Decoded chroma bitdepth.
*/
struct mali_vpu_fw_seq_param {
+ bool interlaced;
struct {
u32 buffers_min;
} planar;
+ struct mali_vpu_fw_bitdepth {
+ u32 chroma;
+ u32 luma;
+ } bitdepth;
};
/**
@@ -146,6 +155,8 @@ struct mali_vpu_fw_error {
* @MALI_VPU_FW_CODE_ALLOC_PARAM: Firmware reported new frame allocation parameters.
* @MALI_VPU_FW_CODE_COLOR_DESC: Firmware reported colour-description metadata.
* @MALI_VPU_FW_CODE_DPB_HELD_FRAMES: Firmware reported the number of frames held in the DPB.
+ * @MALI_VPU_FW_CODE_STREAM_NOT_SUPPORTED: Firmware rejected the coded stream.
+ * @MALI_VPU_FW_CODE_EVENT: Firmware reported an informational event.
* @MALI_VPU_FW_CODE_UNKNOWN: Unrecognized firmware message.
* @MALI_VPU_FW_CODE_MAX: Sentinel marking the end of the message-code enum.
*/
@@ -168,6 +179,8 @@ enum mali_vpu_fw_code {
MALI_VPU_FW_CODE_ALLOC_PARAM, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_COLOR_DESC, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_DPB_HELD_FRAMES, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_STREAM_NOT_SUPPORTED, /* Driver <- Firmware. */
+ MALI_VPU_FW_CODE_EVENT, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_UNKNOWN, /* Driver <- Firmware. */
MALI_VPU_FW_CODE_MAX
};
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
index 76dc7bf076c8f..3fe572b180d72 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-protocol-def-v3.h
@@ -259,6 +259,8 @@ struct mve_msg_header {
#define MVE_RESPONSE_CODE_FRAME_ALLOC_PARAM (2014)
/* struct mve_response_sequence_parameters */
#define MVE_RESPONSE_CODE_SEQUENCE_PARAMETERS (2015)
+ /* struct mve_response_event (variable size) */
+ #define MVE_RESPONSE_CODE_EVENT (2016)
/* struct mve_response_set_option_fail */
#define MVE_RESPONSE_CODE_SET_OPTION_FAIL (2017)
@@ -440,6 +442,91 @@ struct mve_response_error {
char message[MVE_MAX_ERROR_MESSAGE_SIZE];
};
+/*
+ * Per-frame processing statistics emitted when firmware profiling is
+ * enabled.
+ */
+struct mve_event_processed {
+ u8 pic_format;
+ u8 qp;
+ u8 pad0;
+ u8 pad1;
+ u32 parse_start_time; /* Timestamp, absolute time */
+ u32 parse_end_time; /* Timestamp, absolute time */
+ u32 parse_idle_time; /* Waiting for in/out buffers or available RAM */
+ u32 pipe_start_time; /* Timestamp */
+ u32 pipe_end_time; /* Timestamp, end-start = process time. Idle time while in a frame
+ * is not measured.
+ */
+ u32 pipe_idle_time; /* Always 0 in decode, */
+ u32 parser_coreid; /* Core used to parse this frame */
+ u32 pipe_coreid; /* Core used to pipe this frame */
+ u32 bitstream_bits; /* Number of bitstream bits used for this frame. */
+ u32 intermediate_buffer_size; /* Size of intermediate (mbinfo/residuals) buffer after this
+ * frame was parsed.
+ */
+ u32 total_memory_allocated; /* after the frame was parsed. Including reference frames */
+ u32 bus_read_bytes; /* bus read bytes */
+ u32 bus_write_bytes; /* bus written bytes */
+ u32 afbc_bytes;
+ u32 slice0_end_time; /* Timestamp, absolute time */
+ u32 stream_start_time; /* Timestamp, absolute stream start time */
+ u32 stream_open_time; /* Timestamp, absolute stream open time */
+};
+
+/*
+ * Encoder reference-frame metadata emitted when reference-frame exposure
+ * is enabled.
+ */
+struct mve_event_ref_frame {
+ u32 ref_width; /* Width of display area in luma pixels */
+ u32 ref_height; /* Height of display area in luma pixels */
+ u32 ref_mb_width; /* Width in macroblocks */
+ u32 ref_mb_height; /* Height in macroblocks */
+ u32 ref_left_crop; /* Left crop in luma pixels */
+ u32 ref_top_crop; /* Top crop in luma pixels */
+ u32 ref_frame_size;
+ u32 ref_display_order;
+ u32 ref_addr;
+ u16 bit_width; /* bit width of the YUV either 8 or 10 */
+ u16 tiled_headers;
+};
+
+/*
+ * Trace-buffer allocations emitted by firmware build with event tracing
+ * enabled.
+ */
+struct mve_event_trace_buffers {
+ u16 reserved;
+ u8 num_cores;
+ u8 rasc_mask;
+ #define MVE_MAX_TRACE_BUFFERS 40
+ /* this array will contain one buffer per rasc in rasc_mask per num_core */
+ struct {
+ u32 rasc_addr; /* rasc address of the buffer */
+ u32 size; /* size of the buffer in bytes */
+ } buffers[MVE_MAX_TRACE_BUFFERS];
+};
+
+/*
+ * Firmware event response. Profiling and tracing events are informational,
+ * while stream-error events may require host action.
+ */
+struct mve_response_event {
+ u32 event_code;
+ #define MVE_EVENT_ERROR_STREAM_CORRUPT (1) /* message, text string */
+ #define MVE_EVENT_ERROR_STREAM_NOT_SUPPORTED (2) /* message, text string */
+ #define MVE_EVENT_PROCESSED (3) /* struct mve_event_processed */
+ #define MVE_EVENT_REF_FRAME (4) /* struct mve_event_ref_frame */
+ #define MVE_EVENT_TRACE_BUFFERS (5)
+ union {
+ struct mve_event_processed event_processed;
+ struct mve_event_ref_frame event_ref_frame;
+ struct mve_event_trace_buffers event_trace_buffers;
+ char message[MVE_MAX_ERROR_MESSAGE_SIZE];
+ } event_data;
+};
+
/*
* When a set-option succeeds, a confirmation message is
* sent, including the index-code for that particular option.
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 411421e175e71..500d80788d0eb 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -584,6 +584,20 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
struct vb2_queue *dst_q;
u32 buffers_min;
+ if (fw_msg->data.seq_params.interlaced) {
+ dev_err(sess->vpu_dev->dev, "interlaced streams are not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (fw_msg->data.seq_params.bitdepth.luma != 8 ||
+ fw_msg->data.seq_params.bitdepth.chroma != 8) {
+ dev_err(sess->vpu_dev->dev,
+ "bitdepth luma=%u chroma=%u is not supported\n",
+ fw_msg->data.seq_params.bitdepth.luma,
+ fw_msg->data.seq_params.bitdepth.chroma);
+ return -EOPNOTSUPP;
+ }
+
buffers_min = fw_msg->data.seq_params.planar.buffers_min;
dst_q = v4l2_m2m_get_dst_vq(sess->m2m_ctx);
@@ -599,6 +613,11 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
return mali_vpu_session_fw_send_flush(sess, false);
}
+ case MALI_VPU_FW_CODE_STREAM_NOT_SUPPORTED:
+ return -EOPNOTSUPP;
+ case MALI_VPU_FW_CODE_EVENT:
+ /* No session action is required for informational events. */
+ return 0;
case MALI_VPU_FW_CODE_UNKNOWN:
case MALI_VPU_FW_CODE_MAX:
default:
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 27/28] media: mali-vpu: add stateful decoder state machine
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (25 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 26/28] media: mali-vpu: handle firmware event responses Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
2026-11-19 15:37 ` [RFC PATCH 28/28] media: MAINTAINERS: add Arm Mali-V76 VPU driver Charvi Mehta
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
From: Shobhit Pandey <shobhit.pandey@arm.com>
Replace the decoder drc_pending flag with explicit session-state
tracking during FW SOURCE CHANGE and STREAMON/STREAMOFF.
Validate decoder state transitions and update the state during queue
STREAMON/STREAMOFF, decoder START/STOP commands, EOS handling and
firmware responses.
Distinguish initial picture-size discovery from runtime resolution
changes, and delay the source-change event until the firmware has
completed the CAPTURE queue flush. Track draining until the final
CAPTURE buffer is returned with EOS.
Signed-off-by: Shobhit Pandey <shobhit.pandey@arm.com>
---
.../platform/arm/mali-vpu/mali-vpu-dec.c | 139 +++++++++++++++++-
.../platform/arm/mali-vpu/mali-vpu-dec.h | 4 +
.../platform/arm/mali-vpu/mali-vpu-session.c | 67 +++++++--
.../platform/arm/mali-vpu/mali-vpu-session.h | 6 +-
.../platform/arm/mali-vpu/mali-vpu-state.h | 136 +++++++++++++++++
5 files changed, 331 insertions(+), 21 deletions(-)
create mode 100644 drivers/media/platform/arm/mali-vpu/mali-vpu-state.h
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
index 7e34c24ae050f..6ce61de22a802 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.c
@@ -9,6 +9,7 @@
#include "mali-vpu-dec.h"
#include "mali-vpu-session.h"
#include "mali-vpu-variant.h"
+#include "mali-vpu-state.h"
void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
{
@@ -23,7 +24,7 @@ void mali_vpu_session_init_dec(struct mali_vpu_session *sess)
sess->num_ctrls_caps = variant_ctrls->num_dec_ctrls;
sess->min_buffers_required = 3; /* For CAPTURE/dst queue */
- sess->drc_pending = false;
+ sess->sess_state = MALI_VPU_STATE_INITIALIZATION;
}
int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess,
@@ -69,6 +70,7 @@ int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid,
int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd *cmd)
{
struct mali_vpu_session *sess = mali_vpu_get_session(filp);
+ struct vb2_queue *dst_q = v4l2_m2m_get_dst_vq(sess->m2m_ctx);
int ret;
ret = v4l2_m2m_ioctl_try_decoder_cmd(filp, priv, cmd);
@@ -79,11 +81,31 @@ int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd
switch (cmd->cmd) {
case V4L2_DEC_CMD_STOP:
- ret = mali_vpu_session_fw_send_eos(sess);
+ if (sess->sess_state == MALI_VPU_STATE_DRAIN) {
+ ret = -EBUSY;
+ break;
+ } else if (sess->sess_state == MALI_VPU_STATE_STREAMING) {
+ ret = mali_vpu_session_fw_send_eos(sess);
+ if (!ret)
+ ret = mali_vpu_dec_update_state(sess, MALI_VPU_STATE_DRAIN);
+ }
break;
case V4L2_DEC_CMD_START:
- vb2_clear_last_buffer_dequeued(v4l2_m2m_get_dst_vq(sess->m2m_ctx));
- ret = 0;
+ if (sess->sess_state == MALI_VPU_STATE_DRAIN &&
+ !dst_q->last_buffer_dequeued) {
+ ret = -EBUSY;
+ break;
+ }
+
+ ret = mali_vpu_submit_deferred_bufs(sess);
+ if (ret)
+ break;
+
+ ret = mali_vpu_dec_update_state(sess, MALI_VPU_STATE_STREAMING);
+ if (ret)
+ break;
+
+ vb2_clear_last_buffer_dequeued(dst_q);
break;
default:
ret = -EINVAL;
@@ -93,3 +115,112 @@ int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd
mutex_unlock(&sess->vq_lock);
return ret;
}
+
+static bool mali_vpu_dec_allow_state_change(struct mali_vpu_session *sess,
+ enum mali_vpu_sess_state new_state)
+{
+ switch (sess->sess_state) {
+ case MALI_VPU_STATE_INITIALIZATION:
+ return (new_state == MALI_VPU_STATE_RESET ||
+ new_state == MALI_VPU_STATE_STOPPED);
+ case MALI_VPU_STATE_STREAMING:
+ return (new_state == MALI_VPU_STATE_RESET ||
+ new_state == MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE ||
+ new_state == MALI_VPU_STATE_DRAIN);
+ case MALI_VPU_STATE_RESET:
+ return (new_state == MALI_VPU_STATE_INITIALIZATION ||
+ new_state == MALI_VPU_STATE_STREAMING);
+ case MALI_VPU_STATE_DRAIN:
+ return (new_state == MALI_VPU_STATE_STOPPED ||
+ new_state == MALI_VPU_STATE_RESET);
+ case MALI_VPU_STATE_STOPPED:
+ return (new_state == MALI_VPU_STATE_INITIALIZATION ||
+ new_state == MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE ||
+ new_state == MALI_VPU_STATE_STREAMING ||
+ new_state == MALI_VPU_STATE_RESET);
+ case MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE:
+ return new_state == MALI_VPU_STATE_STOPPED;
+ case MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE:
+ return new_state == MALI_VPU_STATE_STREAMING;
+ default:
+ return false;
+ }
+}
+
+int mali_vpu_dec_update_state(struct mali_vpu_session *sess, u32 new_state)
+{
+ if (sess->role != MALI_VPU_DECODER)
+ return -EINVAL;
+
+ if (sess->sess_state == new_state)
+ return 0;
+
+ if (!mali_vpu_dec_allow_state_change(sess, new_state))
+ return -EINVAL;
+
+ sess->sess_state = new_state;
+ return 0;
+}
+
+int mali_vpu_dec_state_change_streamon(struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ enum mali_vpu_sess_state new_state;
+
+ switch (sess->sess_state) {
+ case MALI_VPU_STATE_INITIALIZATION:
+ new_state = V4L2_TYPE_IS_OUTPUT(type) ?
+ MALI_VPU_STATE_STOPPED : MALI_VPU_STATE_RESET;
+ break;
+ case MALI_VPU_STATE_STOPPED:
+ case MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE:
+ if (V4L2_TYPE_IS_OUTPUT(type))
+ return 0;
+ new_state = MALI_VPU_STATE_STREAMING;
+ break;
+ case MALI_VPU_STATE_RESET:
+ if (!V4L2_TYPE_IS_OUTPUT(type))
+ return 0;
+ new_state = MALI_VPU_STATE_STREAMING;
+ break;
+ default:
+ return 0;
+ }
+
+ return mali_vpu_dec_update_state(sess, new_state);
+}
+
+int mali_vpu_dec_state_change_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type)
+{
+ enum mali_vpu_sess_state new_state;
+
+ if (V4L2_TYPE_IS_OUTPUT(type)) {
+ switch (sess->sess_state) {
+ case MALI_VPU_STATE_STOPPED:
+ new_state = MALI_VPU_STATE_INITIALIZATION;
+ break;
+ case MALI_VPU_STATE_STREAMING:
+ case MALI_VPU_STATE_DRAIN:
+ new_state = MALI_VPU_STATE_RESET;
+ break;
+ default:
+ return 0;
+ }
+ } else {
+ switch (sess->sess_state) {
+ case MALI_VPU_STATE_RESET:
+ new_state = MALI_VPU_STATE_INITIALIZATION;
+ break;
+ case MALI_VPU_STATE_STREAMING:
+ case MALI_VPU_STATE_DRAIN:
+ new_state = MALI_VPU_STATE_STOPPED;
+ break;
+ case MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE:
+ new_state = MALI_VPU_STATE_STOPPED;
+ break;
+ default:
+ return 0;
+ }
+ }
+
+ return mali_vpu_dec_update_state(sess, new_state);
+}
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
index a108aaaa4a959..fccda5b3d5b2e 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-dec.h
@@ -7,6 +7,7 @@
#define __MALI_VPU_DEC_H__
#include <linux/types.h>
+#include <linux/videodev2.h>
struct file;
struct mali_vpu_session;
@@ -19,5 +20,8 @@ int mali_vpu_ctrl_set_ignore_stream_headers(struct mali_vpu_session *sess, u32 c
int mali_vpu_ctrl_set_frame_reordering(struct mali_vpu_session *sess, u32 cid, s64 value);
int mali_vpu_decoder_cmd(struct file *filp, void *priv, struct v4l2_decoder_cmd *cmd);
+int mali_vpu_dec_update_state(struct mali_vpu_session *sess, u32 new_state);
+int mali_vpu_dec_state_change_streamon(struct mali_vpu_session *sess, enum v4l2_buf_type type);
+int mali_vpu_dec_state_change_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type);
#endif /* __MALI_VPU_DEC_H__ */
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
index 500d80788d0eb..3ff4635e682eb 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.c
@@ -329,7 +329,7 @@ static int mali_vpu_submit_deferred_bufs_for_queue(struct mali_vpu_session *sess
return 0;
}
-static int mali_vpu_submit_deferred_bufs(struct mali_vpu_session *sess)
+int mali_vpu_submit_deferred_bufs(struct mali_vpu_session *sess)
{
int ret;
@@ -448,7 +448,7 @@ static void mali_vpu_session_buffer_done(struct mali_vpu_session *sess,
if (V4L2_TYPE_IS_CAPTURE(vpu_buf->type) &&
!q->last_buffer_dequeued &&
(vbuf->flags & V4L2_BUF_FLAG_LAST) &&
- !sess->drc_pending) {
+ sess->sess_state != MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE) {
const struct v4l2_event event = {
.type = V4L2_EVENT_EOS,
};
@@ -513,7 +513,14 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
state = mali_vpu_buffer_update_info_to_v4l2(vpu_buf);
mali_vpu_session_buffer_done(sess, vpu_buf, state);
-
+ if (vpu_buf->flags & MALI_VPU_BUFFER_EOS) {
+ if (V4L2_TYPE_IS_OUTPUT(vpu_buf->type) &&
+ sess->sess_state == MALI_VPU_STATE_STREAMING) {
+ ret = mali_vpu_dec_update_state(sess, MALI_VPU_STATE_DRAIN);
+ if (ret)
+ return ret;
+ }
+ }
return 0;
}
case MALI_VPU_FW_CODE_PONG:
@@ -569,17 +576,31 @@ static int mali_vpu_session_handle_fw_msg(struct mali_vpu_session *sess,
fw_msg->data.dpb_held_frames);
return 0;
case MALI_VPU_FW_CODE_OUTPUT_FLUSH:
- dev_dbg(sess->vpu_dev->dev, "fw rsp: Dst queue flushed\n");
- if (sess->role == MALI_VPU_DECODER && sess->drc_pending)
+ if (sess->sess_state == MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE ||
+ sess->sess_state == MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE) {
return mali_vpu_session_send_drc_event(sess);
+ }
return 0;
case MALI_VPU_FW_CODE_ALLOC_PARAM:
ret = mali_vpu_session_update_capture_format(sess, fw_msg->data.alloc_params);
- if (!ret)
- sess->drc_pending = true;
+ if (ret)
+ return ret;
- return ret;
+ if (sess->sess_state == MALI_VPU_STATE_STREAMING) {
+ ret =
+ mali_vpu_dec_update_state(sess,
+ MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE);
+ if (ret)
+ return ret;
+ } else if (sess->sess_state == MALI_VPU_STATE_STOPPED) {
+ ret = mali_vpu_dec_update_state(sess,
+ MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
case MALI_VPU_FW_CODE_SEQUENCE_PARAMS: {
struct vb2_queue *dst_q;
u32 buffers_min;
@@ -829,7 +850,7 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
* Do not give the requeued buffer back to fw as it belongs to pre DRC
* allocation which will be unmapped immediately afterwards.
*/
- if (sess->drc_pending && sess->role == MALI_VPU_DECODER &&
+ if (sess->sess_state == MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE &&
mali_vpu_is_raw_q(sess, vb2_buf->type) &&
port->stream_requested) {
v4l2_buf->flags |= V4L2_BUF_FLAG_LAST;
@@ -840,7 +861,9 @@ int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_
if (!port->stream_requested ||
!mali_vpu_session_can_stream(sess) ||
- sess->fw_state != MALI_VPU_FW_STATE_RUNNING) {
+ sess->fw_state != MALI_VPU_FW_STATE_RUNNING ||
+ (V4L2_TYPE_IS_OUTPUT(vb2_buf->type) &&
+ sess->sess_state == MALI_VPU_STATE_DRAIN)) {
vpu_buf->attr = MALI_VPU_BUF_ATTR_DEFERRED;
if (V4L2_TYPE_IS_OUTPUT(vb2_buf->type))
@@ -949,6 +972,7 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
enum v4l2_buf_type type)
{
struct mali_vpu_port *port = mali_vpu_session_get_port(sess, type);
+ enum mali_vpu_sess_state old_state;
int ret;
lockdep_assert_held(&sess->vq_lock);
@@ -964,6 +988,12 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
sess->fw_requested_state != MALI_VPU_FW_STATE_RUNNING)
return -EBUSY;
+ old_state = sess->sess_state;
+
+ ret = mali_vpu_dec_state_change_streamon(sess, type);
+ if (ret)
+ return ret;
+
port->stream_requested = true;
if (sess->fw_bin_state == MALI_VPU_SESSION_FW_BIN_READY &&
@@ -975,15 +1005,14 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
}
}
- if (sess->role == MALI_VPU_DECODER && mali_vpu_is_raw_q(sess, type))
- sess->drc_pending = false;
-
port->sequence = 0;
return 0;
err_rollback:
+
port->stream_requested = false;
+ sess->sess_state = old_state;
return ret;
}
@@ -991,6 +1020,7 @@ int mali_vpu_session_prepare_streaming(struct mali_vpu_session *sess,
int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type type)
{
struct mali_vpu_port *port = mali_vpu_session_get_port(sess, type);
+ bool drc_capture;
int ret;
lockdep_assert_held(&sess->vq_lock);
@@ -1003,12 +1033,19 @@ int mali_vpu_session_streamoff(struct mali_vpu_session *sess, enum v4l2_buf_type
if (sess->error)
return 0;
+ drc_capture =
+ sess->sess_state == MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE &&
+ mali_vpu_is_raw_q(sess, type);
+
+ ret = mali_vpu_dec_state_change_streamoff(sess, type);
+ if (ret)
+ goto err_session;
+
if (sess->fw_bin_state != MALI_VPU_SESSION_FW_BIN_READY ||
!sess->sched_sess.lsid)
return 0;
- if (sess->drc_pending && sess->role == MALI_VPU_DECODER &&
- mali_vpu_is_raw_q(sess, type)) {
+ if (drc_capture) {
ret = mali_vpu_session_wait_pending(sess);
if (ret)
goto err_session;
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
index 45d0b6710aced..b37ed569ffb7a 100644
--- a/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-session.h
@@ -17,6 +17,7 @@
#include "mali-vpu-firmware.h"
#include "mali-vpu-formats.h"
#include "mali-vpu-mmu.h"
+#include "mali-vpu-state.h"
struct v4l2_m2m_ctx;
struct mali_vpu_fw_bin;
@@ -115,7 +116,7 @@ enum mali_vpu_session_fw_bin_state {
* @sched_sess: Scheduler state used to map this session to an LSID.
* @ncores: Number of cores assigned to this session by scheduler.
* @error: Fatal session error code.
- * @drc_pending: Dynamic resolution change is waiting for CAPTURE reconfigure.
+ * @sess_state: Session main state.
*/
struct mali_vpu_session {
struct mali_vpu_dev *vpu_dev;
@@ -148,7 +149,7 @@ struct mali_vpu_session {
struct mali_vpu_sched_session sched_sess;
u32 ncores;
int error;
- bool drc_pending;
+ enum mali_vpu_sess_state sess_state;
};
int mali_vpu_session_init(struct mali_vpu_session *sess);
@@ -163,6 +164,7 @@ void mali_vpu_session_irq(struct mali_vpu_session *sess);
int mali_vpu_session_qbuf(struct mali_vpu_session *sess, struct vb2_buffer *vb2_buf);
void mali_vpu_session_reset_ts_metadata(struct mali_vpu_session *sess);
int mali_vpu_session_fw_send_eos(struct mali_vpu_session *sess);
+int mali_vpu_submit_deferred_bufs(struct mali_vpu_session *sess);
static inline struct mali_vpu_session *mali_vpu_get_session(struct file *filp)
{
diff --git a/drivers/media/platform/arm/mali-vpu/mali-vpu-state.h b/drivers/media/platform/arm/mali-vpu/mali-vpu-state.h
new file mode 100644
index 0000000000000..4602c468a079d
--- /dev/null
+++ b/drivers/media/platform/arm/mali-vpu/mali-vpu-state.h
@@ -0,0 +1,136 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Limited.
+ */
+
+#ifndef __MALI_VPU_STATE_H__
+#define __MALI_VPU_STATE_H__
+
+/*
+ * Decoder session state machine
+ *
+ * open
+ * |
+ * |
+ * v
+ * +----------------+ -> -> -> StreamOff(Output) -> -> -> +------------+
+ * | INITIALIZATION |<------------------------------------------------------>| RESET/SEEK |
+ * +----------------+ <- <- StreamOff(Capture)/REQBUF(OUTPUT, 0) <- <- +------------+
+ * ^ ^
+ * | capture format |
+ * | established StreamOff(Output)|
+ * | |
+ * | +---------------+ |
+ * +--------------------->| |-----------------------------------+
+ * +---------------------------------->| CAPTURE_SETUP/| |
+ * | | STOPPED | |
+ * | +--------------------->| | |
+ * | |capture format +---------------+ |
+ * | | established | ^ |
+ * | +-------------------+ | | |
+ * | | RESOLUTION_CHANGE | | | |
+ * | +-------------------+ | | |
+ * | ^ | | StreamOff(Capture) +-------+ |
+ * | | | +-----------------------| DRAIN | |
+ * | | | +-------+ |
+ * | | | ^ |
+ * | | |START_CMD/ | |
+ * | | |StreamOn | |
+ * | | |(Capture) | |
+ * | | Stream | | |
+ * | |Resolution | | |
+ * | | Change | | |
+ * | | | | |
+ * | | | | |
+ * | | | |STOP_CMD/ |
+ * | | | |EOS Mark |
+ * | | | | |
+ * | | v | |
+ * | | +---------------+ | |
+ * | +----------------------| STREAMING |------------------------+ |
+ * | +---------------+ |
+ * | ^ ^ |
+ * | | | -> -> StreamOff(Output) -> -> |
+ * +--------------------------------------+ +------------------------------------------+
+ * <- <- StreamOn(Output) <- <-
+ */
+
+/*
+ * INITIALIZATION -> STOPPED OUTPUT STREAMON happens first. Firmware may parse
+ * source headers while CAPTURE is not configured.
+ *
+ * STOPPED -> INITIALIZATION OUTPUT STREAMOFF while CAPTURE is stopped.
+ * Now neither queue is streaming.
+ *
+ * INITIALIZATION -> RESET CAPTURE STREAMON happens first, as exercised by the
+ * compliance blocking test. Firmware should not start
+ * decoding yet.
+ *
+ * RESET -> INITIALIZATION CAPTURE STREAMOFF while OUTPUT is already stopped.
+ * Now neither queue is streaming.
+ *
+ * RESET -> STREAMING OUTPUT STREAMON after CAPTURE was started first.
+ * Both queues are now streaming.
+ *
+ * STREAMING -> RESET OUTPUT STREAMOFF first; CAPTURE remains streaming.
+ *
+ * STREAMING -> DYNAMIC_RESOLUTION_CHANGE Firmware reports changed stream parameters while
+ * actively decoding. The driver flushes CAPTURE and
+ * sends V4L2_EVENT_SOURCE_CHANGE.
+ *
+ * DYNAMIC_RESOLUTION_CHANGE -> STOPPED Userspace performs CAPTURE STREAMOFF following the
+ * source-change event. OUTPUT remains streaming.
+ *
+ * STREAMING -> DRAIN V4L2_DEC_CMD_STOP is received or implicit EOS is
+ * detected on OUTPUT/source.
+ *
+ * DRAIN -> STOPPED Firmware returns the final CAPTURE buffer with
+ * LAST/EOS, completing the drain.
+ *
+ * DRAIN -> RESET OUTPUT STREAMOFF occurs during draining;
+ * CAPTURE remains streaming.
+ *
+ * STOPPED -> STREAMING Userspace sends V4L2_DEC_CMD_START to continue
+ * decoding while both queues remain streaming./
+ * CAPTURE STREAMON after OUTPUT was started first.
+ * Both queues are now streaming.
+ *
+ * STOPPED -> FIRST_SOURCE_CHANGE Firmware reports first RESOLUTION CHANGE in sequence
+ * occurs during before CAPTURE is configured; it is
+ * not runtime DRC. The driver flushes CAPTURE and
+ * sends V4L2_EVENT_SOURCE_CHANGE.
+ *
+ * FIRST_SOURCE_CHANGE -> STREAMING Userspace on receiving V4L2_EVENT_SOURCE_CHANGE for
+ * FIRST_SOURCE_CHANGE will allocate CAPTURE BUFFERS
+ * and call streamOn which will move the state to
+ * STREAMING as both queues are now streaming.
+ *
+ * STOPPED -> RESET OUTPUT STREAMOFF after drain completion; CAPTURE
+ * remains streaming. TO BE VERIFIED
+ */
+
+/**
+ * enum mali_vpu_sess_state - VPU session streaming states
+ *
+ * @MALI_VPU_STATE_INITIALIZATION: Session is initialized with no queues streaming.
+ * @MALI_VPU_STATE_STREAMING: Session both queues streaming.
+ * @MALI_VPU_STATE_STOPPED: For decoder session CAPTURE is not streaming.
+ * @MALI_VPU_STATE_DRAIN: For decoder session all CAPTURE buffers returned
+ * @MALI_VPU_STATE_RESET: For decoder session OUTPUT is not streaming.
+ * @MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE: Decoder only for mid stream resolution change
+ * @MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE: Decoder only for first source change when CAPTURE
+ * not streaming.
+ */
+enum mali_vpu_sess_state {
+ /* Common states for encoder and decoder */
+ MALI_VPU_STATE_INITIALIZATION = 0,
+ MALI_VPU_STATE_STREAMING,
+ MALI_VPU_STATE_STOPPED,
+ MALI_VPU_STATE_DRAIN,
+ MALI_VPU_STATE_RESET,
+ /* Decoder only states */
+ MALI_VPU_DEC_STATE_DYNAMIC_RESOLUTION_CHANGE,
+ MALI_VPU_DEC_STATE_FIRST_SOURCE_CHANGE,
+};
+
+#endif /* __MALI_VPU_STATE_H__ */
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread* [RFC PATCH 28/28] media: MAINTAINERS: add Arm Mali-V76 VPU driver
2026-11-19 15:37 [RFC PATCH 00/28] media: add arm Mali-V76 video decoder driver Charvi Mehta
` (26 preceding siblings ...)
2026-11-19 15:37 ` [RFC PATCH 27/28] media: mali-vpu: add stateful decoder state machine Charvi Mehta
@ 2026-11-19 15:37 ` Charvi Mehta
27 siblings, 0 replies; 37+ messages in thread
From: Charvi Mehta @ 2026-11-19 15:37 UTC (permalink / raw)
To: linux-media, devicetree, linux-kernel
Cc: Charvi Mehta, Mauro Carvalho Chehab, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Philipp Zabel,
Vincenzo Frascino, Rathinavel Sankaralingam, Shobhit Pandey
Add maintenance contacts and file patterns for Arm Mali-V76 VPU driver.
Signed-off-by: Charvi Mehta <charvi.mehta@arm.com>
---
MAINTAINERS | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index c08aade46f61d..3150c3bef7d54 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2409,6 +2409,17 @@ F: Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst
F: drivers/media/platform/arm/mali-c55/
F: include/uapi/linux/media/arm/mali-c55-config.h
+ARM MALI-V76 VPU DRIVER
+M: Charvi Mehta <charvi.mehta@arm.com>
+M: Vincenzo Frascino <vincenzo.frascino@arm.com>
+R: Rathinavel Sankaralingam <rath.sankaralingam@arm.com>
+R: Shobhit Pandey <shobhit.pandey@arm.com>
+L: linux-media@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/media/arm,mali-v76.yaml
+F: drivers/media/platform/arm/mali-vpu/
+F: include/uapi/linux/media/arm/mali-vpu-config.h
+
ARM MALI PANTHOR DRM DRIVER
M: Boris Brezillon <boris.brezillon@collabora.com>
M: Steven Price <steven.price@arm.com>
--
2.43.0
^ permalink raw reply related [flat|nested] 37+ messages in thread