* [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
@ 2025-08-06 15:11 Tao Tang
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
` (12 more replies)
0 siblings, 13 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
Hi all,
This patch series introduces initial support for emulating the Arm SMMUv3
Secure State.
As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
emulation is a notable missing piece in QEMU. While the FVP model has
some support, its limited PCIe capabilities make it challenging for
complex use cases. The ability to properly manage device DMA from a
secure context is a critical prerequisite for enabling device assignment
(passthrough) for confidential computing solutions like Arm CCA and
related research such as virtCCA [2]. This series aims to build that
foundational support in QEMU.
This work is being proposed as an RFC. It introduces a significant amount
of new logic, including the core concept of modeling parallel secure and
non-secure contexts within a single SMMUv3 device. I am seeking feedback
on the overall approach, the core refactoring, and the implementation
details before proceeding further.
The series begins by implementing the components of the secure programming
interface, then progressively refactors the core SMMU logic to handle
secure and non-secure contexts in parallel.
Secure Interface Implementation: The initial patches add the
secure-side registers, implement their read/write logic, and enable
the secure command and event queues. This includes the S_INIT
mechanism and the new secure TLB invalidation commands.
Core Logic Refactoring: The next set of patches makes the core SMMU
functions security-state aware. This involves plumbing an is_secure
context flag through the main code paths and adding logic to route
SMMU-originated memory accesses to the correct (Secure or Non-secure)
address space.
Cache Isolation: With the core logic now aware of security states,
the following patches refactor the configuration and translation
lookup caches. The cache keys are modified to include the security
context, ensuring that secure and non-secure entries for the same
device or address are properly isolated and preventing aliasing.
Framework Integration: The final patch connects the SMMU's internal
security context to the generic QEMU IOMMU framework by using the
iommu_index to represent the architectural SEC_SID.
To validate this work, I performed the following tests:
Non-Secure Regression: To ensure that existing functionality remains
intact, I ran a nested virtualization test. A TCG guest was created on
the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned.
Command line of TCG VM is below:
qemu-system-aarch64 \
-machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
-cpu max -smp 1 -m 4080M \
-accel tcg,thread=single,tb-size=512 \
-kernel Image \
-append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \
-device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
-device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
-drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-qmp unix:/tmp/qmp-sock12,server=on,wait=off \
-netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
-device virtio-net-device,netdev=eth0 \
-drive if=none,file=nvme.img,format=raw,id=nvme0 \
-device nvme,drive=nvme0,serial=deadbeef \
-d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic
Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
re-assigned to it via VFIO.
Command line of KVM VM inside TCG VM is below:
sudo qemu-system-aarch64 \
-enable-kvm -m 1024 -cpu host -M virt \
-machine virt,gic-version=3 \
-cpu max -append "nokaslr" -smp 1 \
-monitor stdio \
-kernel 5.15.Image \
-initrd rootfs.cpio.gz \
-display vnc=:22,id=primary \
-device vfio-pci,host=00:01.0
The KVM guest was able to perform I/O on the device
correctly, confirming that the non-secure path is not broken.
Secure Register/Command Interface: I set up an OP-TEE + Hafnium
environment. Hafnium's smmuv3_driver_init function was used to test
the secure register I/O and command queue functionality (excluding
translation). As Hafnium assumes larger queue and StreamID sizes than
are practical without TTST support, I temporarily patched Hafnium to
use smaller values, allowing its driver to initialize the emulated
secure SMMU successfully.
Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
support QEMU, and no secure device assignment feature exists yet, I
created a custom platform device to test the secure translation flow.
To trigger the translation logic, I initiated MMIO writes to this
device from within Hafnium. The device's MMIO callback handler then
performed DMA accesses via its IOMMU region, exercising the secure
translation path. While SMMUv3 is typically used for PCIe on
physical SoCs, the architecture allows its use with platform devices
via a stream-id binding in the device tree. The test harness
required some non-standard modifications to decouple the SMMU from
its tight integration with PCIe. The code for this test device is
available for review at [3]. README.md with detailed instructions is
also provided.
I've attempted to follow all of the guidance in the "Submitting a Patch"
guide, but as this is my first series of this scale, I apologize if I
missed anything and welcome all feedback.
Thanks,
Tang
[1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
[2] https://arxiv.org/abs/2306.11011
[3] https://github.com/hnusdr/qemu
Tao Tang (11):
hw/arm/smmuv3: Introduce secure registers and commands
hw/arm/smmuv3: Implement read/write logic for secure registers
hw/arm/smmuv3: Implement S_INIT for secure initialization
hw/arm/smmuv3: Enable command processing for the Secure state
hw/arm/smmuv3: Support secure event queue and error handling
hw/arm/smmuv3: Plumb security state through core functions
hw/arm/smmuv3: Add separate address space for secure SMMU accesses
hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
hw/arm/smmuv3: Make the configuration cache security-state aware
hw/arm/smmuv3: Differentiate secure TLB entries via keying
hw/arm/smmuv3: Use iommu_index to represent SEC_SID
hw/arm/smmu-common.c | 74 ++-
hw/arm/smmuv3-internal.h | 128 +++++-
hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++-----
hw/arm/trace-events | 7 +-
hw/arm/virt.c | 5 +
include/hw/arm/smmu-common.h | 23 +-
include/hw/arm/smmuv3.h | 27 ++
7 files changed, 968 insertions(+), 140 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 47+ messages in thread
* [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-11 10:22 ` Philippe Mathieu-Daudé
2025-08-18 21:21 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers Tao Tang
` (11 subsequent siblings)
12 siblings, 2 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
The Arm SMMUv3 architecture defines a set of registers and commands for
managing secure transactions and context.
This patch introduces the definitions for these secure registers and
commands within the SMMUv3 device model internal header.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3-internal.h | 57 ++++++++++++++++++++++++++++++++++++++++
include/hw/arm/smmuv3.h | 23 ++++++++++++++++
2 files changed, 80 insertions(+)
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index b6b7399347..483aaa915e 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -179,6 +179,63 @@ REG32(EVENTQ_IRQ_CFG2, 0xbc)
#define A_IDREGS 0xfd0
+/* Secure registers */
+#define SMMU_SECURE_BASE_OFFSET 0x8000
+REG32(S_IDR0, 0x8000)
+REG32(S_IDR1, 0x8004)
+ FIELD(S_IDR1, S_SIDSIZE, 0 , 6)
+ FIELD(S_IDR1, SEL2, 29, 1)
+ FIELD(S_IDR1, SECURE_IMPL, 31, 1)
+
+REG32(S_IDR2, 0x8008)
+REG32(S_IDR3, 0x800c)
+REG32(S_IDR4, 0x8010)
+
+REG32(S_CR0, 0x8020)
+ FIELD(S_CR0, SMMUEN, 0, 1)
+ FIELD(S_CR0, EVENTQEN, 2, 1)
+ FIELD(S_CR0, CMDQEN, 3, 1)
+
+REG32(S_CR0ACK, 0x8024)
+REG32(S_CR1, 0x8028)
+REG32(S_CR2, 0x802c)
+
+REG32(S_INIT, 0x803c)
+ FIELD(S_INIT, INV_ALL, 0, 1)
+
+REG32(S_GBPA, 0x8044)
+ FIELD(S_GBPA, ABORT, 20, 1)
+ FIELD(S_GBPA, UPDATE, 31, 1)
+
+REG32(S_IRQ_CTRL, 0x8050)
+ FIELD(S_IRQ_CTRL, GERROR_IRQEN, 0, 1)
+ FIELD(S_IRQ_CTRL, EVENTQ_IRQEN, 2, 1)
+
+REG32(S_IRQ_CTRLACK, 0x8054)
+
+REG32(S_GERROR, 0x8060)
+ FIELD(S_GERROR, CMDQ_ERR, 0, 1)
+
+REG32(S_GERRORN, 0x8064)
+REG64(S_GERROR_IRQ_CFG0, 0x8068)
+REG32(S_GERROR_IRQ_CFG1, 0x8070)
+REG32(S_GERROR_IRQ_CFG2, 0x8074)
+REG64(S_STRTAB_BASE, 0x8080)
+REG32(S_STRTAB_BASE_CFG, 0x8088)
+ FIELD(S_STRTAB_BASE_CFG, LOG2SIZE, 0, 6)
+ FIELD(S_STRTAB_BASE_CFG, SPLIT, 6, 5)
+ FIELD(S_STRTAB_BASE_CFG, FMT, 16, 2)
+
+REG64(S_CMDQ_BASE, 0x8090)
+REG32(S_CMDQ_PROD, 0x8098)
+REG32(S_CMDQ_CONS, 0x809c)
+REG64(S_EVENTQ_BASE, 0x80a0)
+REG32(S_EVENTQ_PROD, 0x80a8)
+REG32(S_EVENTQ_CONS, 0x80ac)
+REG64(S_EVENTQ_IRQ_CFG0, 0x80b0)
+REG32(S_EVENTQ_IRQ_CFG1, 0x80b8)
+REG32(S_EVENTQ_IRQ_CFG2, 0x80bc)
+
static inline int smmu_enabled(SMMUv3State *s)
{
return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
index d183a62766..72ad042514 100644
--- a/include/hw/arm/smmuv3.h
+++ b/include/hw/arm/smmuv3.h
@@ -63,6 +63,29 @@ struct SMMUv3State {
qemu_irq irq[4];
QemuMutex mutex;
char *stage;
+
+ /* Secure state */
+ uint32_t secure_idr[5];
+ uint32_t secure_cr[3];
+ uint32_t secure_cr0ack;
+ uint32_t secure_init;
+ uint32_t secure_gbpa;
+ uint32_t secure_irq_ctrl;
+ uint32_t secure_gerror;
+ uint32_t secure_gerrorn;
+ uint64_t secure_gerror_irq_cfg0;
+ uint32_t secure_gerror_irq_cfg1;
+ uint32_t secure_gerror_irq_cfg2;
+ uint64_t secure_strtab_base;
+ uint32_t secure_strtab_base_cfg;
+ uint8_t secure_sid_split;
+ uint32_t secure_features;
+
+ uint64_t secure_eventq_irq_cfg0;
+ uint32_t secure_eventq_irq_cfg1;
+ uint32_t secure_eventq_irq_cfg2;
+
+ SMMUQueue secure_eventq, secure_cmdq;
};
typedef enum {
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 21:53 ` Pierrick Bouvier
2025-08-18 21:24 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization Tao Tang
` (10 subsequent siblings)
12 siblings, 2 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
This patch builds upon the previous introduction of secure register
definitions by providing the functional implementation for their access.
The availability of the secure programming interface is now correctly
gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
secure functionality is enabled, the I/O handlers (smmuv3_read and
smmuv3_write) will correctly dispatch accesses to the secure
register space.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3-internal.h | 5 +
hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
2 files changed, 456 insertions(+)
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index 483aaa915e..1a8b1cb204 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -122,6 +122,11 @@ REG32(CR0, 0x20)
#define SMMU_CR0_RESERVED 0xFFFFFC20
+/*
+ * BIT1 and BIT4 are RES0 in SMMU_S_CRO
+ */
+#define SMMU_S_CR0_RESERVED 0xFFFFFC12
+
REG32(CR0ACK, 0x24)
REG32(CR1, 0x28)
REG32(CR2, 0x2c)
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index ab67972353..619180d204 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
s->gerrorn = 0;
s->statusr = 0;
s->gbpa = SMMU_GBPA_RESET_VAL;
+
+ /* Initialize secure state */
+ memset(s->secure_idr, 0, sizeof(s->secure_idr));
+ /* Secure EL2 and Secure stage 2 support */
+ s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
+ /* Secure state implemented */
+ s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
+ SECURE_IMPL, 1);
+ s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
+ S_SIDSIZE, SMMU_IDR1_SIDSIZE);
+
+ s->secure_gbpa = SMMU_GBPA_RESET_VAL;
}
static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
@@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
}
}
+/* Check if the SMMU hardware itself implements secure state features */
+static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
+{
+ return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
+}
+
static int smmuv3_cmdq_consume(SMMUv3State *s)
{
SMMUState *bs = ARM_SMMU(s);
@@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
return 0;
}
+/* Helper function for secure register write validation */
+static bool smmu_validate_secure_write(MemTxAttrs attrs, bool secure_impl,
+ hwaddr offset, const char *reg_name)
+{
+ if (!attrs.secure || !secure_impl) {
+ const char *reason = !attrs.secure ?
+ "Non-secure write attempt" :
+ "SMMU didn't implement Security State";
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
+ __func__, reason, offset, reg_name);
+ return false;
+ }
+ return true;
+}
+
+/* Helper function for secure register read validation */
+static bool smmu_validate_secure_read(MemTxAttrs attrs, bool secure_impl,
+ hwaddr offset, const char *reg_name,
+ uint64_t *data)
+{
+ if (!attrs.secure || !secure_impl) {
+ const char *reason = !attrs.secure ?
+ "Non-secure read attempt" :
+ "SMMU didn't implement Security State";
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
+ __func__, reason, offset, reg_name);
+ *data = 0; /* RAZ */
+ return false;
+ }
+ return true;
+}
+
+/* Macro for secure write validation - returns early if validation fails */
+#define SMMU_CHECK_SECURE_WRITE(reg_name) \
+ do { \
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
+ reg_name)) { \
+ return MEMTX_OK; \
+ } \
+ } while (0)
+
+/* Macro for attrs.secure only validation */
+#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
+ do { \
+ if (!attrs.secure) { \
+ qemu_log_mask(LOG_GUEST_ERROR, \
+ "%s: Non-secure write attempt at offset " \
+ "0x%" PRIx64 " (%s, WI)\n", \
+ __func__, offset, reg_name); \
+ return MEMTX_OK; \
+ } \
+ } while (0)
+
+/* Macro for secure read validation - returns RAZ if validation fails */
+#define SMMU_CHECK_SECURE_READ(reg_name) \
+ do { \
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
+ reg_name, data)) { \
+ return MEMTX_OK; \
+ } \
+ } while (0)
+
+/* Macro for attrs.secure only validation (read) */
+#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
+ do { \
+ if (!attrs.secure) { \
+ qemu_log_mask(LOG_GUEST_ERROR, \
+ "%s: Non-secure read attempt at offset " \
+ "0x%" PRIx64 " (%s, RAZ)\n", \
+ __func__, offset, reg_name); \
+ *data = 0; \
+ return MEMTX_OK; \
+ } \
+ } while (0)
+
static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
uint64_t data, MemTxAttrs attrs)
{
+ bool secure_impl = false;
+ if (offset >= SMMU_SECURE_BASE_OFFSET) {
+ secure_impl = smmu_hw_secure_implemented(s);
+ }
+
switch (offset) {
case A_GERROR_IRQ_CFG0:
s->gerror_irq_cfg0 = data;
@@ -1535,6 +1635,41 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
case A_EVENTQ_IRQ_CFG0:
s->eventq_irq_cfg0 = data;
return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0:
+ /* No need to check secure_impl here */
+ SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
+ s->secure_gerror_irq_cfg0 = data;
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE:
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset,
+ "S_STRTAB_BASE")) {
+ return MEMTX_OK;
+ }
+ s->secure_strtab_base = data;
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE:
+ SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
+ s->secure_cmdq.base = data;
+ s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
+ if (s->secure_cmdq.log2size > SMMU_CMDQS) {
+ s->secure_cmdq.log2size = SMMU_CMDQS;
+ }
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
+ s->secure_eventq.base = data;
+ s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
+ if (s->secure_eventq.log2size > SMMU_EVENTQS) {
+ s->secure_eventq.log2size = SMMU_EVENTQS;
+ }
+ return MEMTX_OK;
+ case A_S_EVENTQ_IRQ_CFG0:
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset,
+ "S_EVENTQ_IRQ_CFG0")) {
+ return MEMTX_OK;
+ }
+ s->secure_eventq_irq_cfg0 = data;
+ return MEMTX_OK;
default:
qemu_log_mask(LOG_UNIMP,
"%s Unexpected 64-bit access to 0x%"PRIx64" (WI)\n",
@@ -1546,6 +1681,11 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
uint64_t data, MemTxAttrs attrs)
{
+ bool secure_impl = false;
+ if (offset >= SMMU_SECURE_BASE_OFFSET) {
+ secure_impl = smmu_hw_secure_implemented(s);
+ }
+
switch (offset) {
case A_CR0:
s->cr[0] = data;
@@ -1650,6 +1790,137 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
case A_EVENTQ_IRQ_CFG2:
s->eventq_irq_cfg2 = data;
return MEMTX_OK;
+ case A_S_CR0:
+ SMMU_CHECK_SECURE_WRITE("S_CR0");
+ s->secure_cr[0] = data;
+ /* clear reserved bits */
+ s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
+ smmuv3_cmdq_consume(s);
+ return MEMTX_OK;
+ case A_S_CR1:
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset,
+ "S_CR1")) {
+ return MEMTX_OK;
+ }
+ s->secure_cr[1] = data;
+ return MEMTX_OK;
+ case A_S_CR2:
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset,
+ "S_CR2")) {
+ return MEMTX_OK;
+ }
+ s->secure_cr[2] = data;
+ return MEMTX_OK;
+ case A_S_IRQ_CTRL:
+ if (!smmu_validate_secure_write(attrs, secure_impl, offset,
+ "S_IRQ_CTRL")) {
+ return MEMTX_OK;
+ }
+ s->secure_irq_ctrl = data;
+ return MEMTX_OK;
+ case A_S_GERRORN:
+ SMMU_CHECK_SECURE_WRITE("S_GERRORN");
+ smmuv3_write_gerrorn(s, data);
+ smmuv3_cmdq_consume(s);
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0:
+ SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
+ s->secure_gerror_irq_cfg0 = data;
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0 + 4:
+ SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
+ s->secure_gerror_irq_cfg0 = deposit64(s->secure_gerror_irq_cfg0,
+ 32, 32, data);
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG1:
+ SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG1");
+ s->secure_gerror_irq_cfg1 = data;
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG2:
+ SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG2");
+ s->secure_gerror_irq_cfg2 = data;
+ return MEMTX_OK;
+ case A_S_GBPA:
+ SMMU_CHECK_SECURE_WRITE("S_GBPA");
+ if (data & R_S_GBPA_UPDATE_MASK) {
+ s->secure_gbpa = data & ~R_S_GBPA_UPDATE_MASK;
+ }
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE:
+ SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
+ s->secure_strtab_base = deposit64(s->secure_strtab_base, 0, 32, data);
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE + 4:
+ SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
+ s->secure_strtab_base = deposit64(s->secure_strtab_base, 32, 32, data);
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE_CFG:
+ SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE_CFG");
+ s->secure_strtab_base_cfg = data;
+ if (FIELD_EX32(data, S_STRTAB_BASE_CFG, FMT) == 1) {
+ s->secure_sid_split = FIELD_EX32(data, S_STRTAB_BASE_CFG, SPLIT);
+ s->secure_features |= SMMU_FEATURE_2LVL_STE;
+ }
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE:
+ SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
+ s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 0, 32, data);
+ s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
+ if (s->secure_cmdq.log2size > SMMU_CMDQS) {
+ s->secure_cmdq.log2size = SMMU_CMDQS;
+ }
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE + 4:
+ SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
+ s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 32, 32, data);
+ return MEMTX_OK;
+ case A_S_CMDQ_PROD:
+ SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
+ s->secure_cmdq.prod = data;
+ smmuv3_cmdq_consume(s);
+ return MEMTX_OK;
+ case A_S_CMDQ_CONS:
+ SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
+ s->secure_cmdq.cons = data;
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
+ s->secure_eventq.base = deposit64(s->secure_eventq.base, 0, 32, data);
+ s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
+ if (s->secure_eventq.log2size > SMMU_EVENTQS) {
+ s->secure_eventq.log2size = SMMU_EVENTQS;
+ }
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE + 4:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
+ s->secure_eventq.base = deposit64(s->secure_eventq.base, 32, 32, data);
+ return MEMTX_OK;
+ case A_S_EVENTQ_PROD:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_PROD");
+ s->secure_eventq.prod = data;
+ return MEMTX_OK;
+ case A_S_EVENTQ_CONS:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_CONS");
+ s->secure_eventq.cons = data;
+ return MEMTX_OK;
+ case A_S_EVENTQ_IRQ_CFG0:
+ SMMU_CHECK_SECURE_WRITE("S_EVENTQ_IRQ_CFG0");
+ s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
+ 0, 32, data);
+ return MEMTX_OK;
+ case A_S_EVENTQ_IRQ_CFG0 + 4:
+ SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG0");
+ s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
+ 32, 32, data);
+ return MEMTX_OK;
+ case A_S_EVENTQ_IRQ_CFG1:
+ SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG1");
+ s->secure_eventq_irq_cfg1 = data;
+ return MEMTX_OK;
+ case A_S_EVENTQ_IRQ_CFG2:
+ SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
+ s->secure_eventq_irq_cfg2 = data;
+ return MEMTX_OK;
default:
qemu_log_mask(LOG_UNIMP,
"%s Unexpected 32-bit access to 0x%"PRIx64" (WI)\n",
@@ -1687,6 +1958,11 @@ static MemTxResult smmu_write_mmio(void *opaque, hwaddr offset, uint64_t data,
static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
uint64_t *data, MemTxAttrs attrs)
{
+ bool secure_impl = false;
+ if (offset >= SMMU_SECURE_BASE_OFFSET) {
+ secure_impl = smmu_hw_secure_implemented(s);
+ }
+
switch (offset) {
case A_GERROR_IRQ_CFG0:
*data = s->gerror_irq_cfg0;
@@ -1700,6 +1976,31 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
case A_EVENTQ_BASE:
*data = s->eventq.base;
return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GERROR_IRQ_CFG0", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gerror_irq_cfg0;
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE:
+ SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
+ *data = s->secure_strtab_base;
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CMDQ_BASE", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cmdq.base;
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_EVENTQ_BASE", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_eventq.base;
+ return MEMTX_OK;
default:
*data = 0;
qemu_log_mask(LOG_UNIMP,
@@ -1712,6 +2013,11 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
uint64_t *data, MemTxAttrs attrs)
{
+ bool secure_impl = false;
+ if (offset >= SMMU_SECURE_BASE_OFFSET) {
+ secure_impl = smmu_hw_secure_implemented(s);
+ }
+
switch (offset) {
case A_IDREGS ... A_IDREGS + 0x2f:
*data = smmuv3_idreg(offset - A_IDREGS);
@@ -1798,6 +2104,151 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
case A_EVENTQ_CONS:
*data = s->eventq.cons;
return MEMTX_OK;
+ case A_S_IDR0 ... A_S_IDR4:
+ int idr_idx = (offset - A_S_IDR0) / 4;
+ g_assert(idr_idx >= 0 && idr_idx <= 4);
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ g_strdup_printf("S_IDR%d", idr_idx),
+ data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_idr[idr_idx];
+ return MEMTX_OK;
+ case A_S_CR0:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CR0", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cr[0];
+ return MEMTX_OK;
+ case A_S_CR0ACK:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CR0ACK", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cr0ack;
+ return MEMTX_OK;
+ case A_S_CR1:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CR1", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cr[1];
+ return MEMTX_OK;
+ case A_S_CR2:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CR2", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cr[2];
+ return MEMTX_OK;
+ case A_S_GBPA:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GBPA", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gbpa;
+ return MEMTX_OK;
+ case A_S_IRQ_CTRL:
+ case A_S_IRQ_CTRLACK:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_IRQ_CTRL", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_irq_ctrl;
+ return MEMTX_OK;
+ case A_S_GERROR:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GERROR", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gerror;
+ return MEMTX_OK;
+ case A_S_GERRORN:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GERRORN", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gerrorn;
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0:
+ SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0");
+ *data = extract64(s->secure_gerror_irq_cfg0, 0, 32);
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG0 + 4:
+ SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0+4");
+ *data = extract64(s->secure_gerror_irq_cfg0, 32, 32);
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG1:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GERROR_IRQ_CFG1", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gerror_irq_cfg1;
+ return MEMTX_OK;
+ case A_S_GERROR_IRQ_CFG2:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_GERROR_IRQ_CFG2", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_gerror_irq_cfg2;
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE:
+ SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
+ *data = extract64(s->secure_strtab_base, 0, 32);
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE + 4:
+ SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE+4");
+ *data = extract64(s->secure_strtab_base, 32, 32);
+ return MEMTX_OK;
+ case A_S_STRTAB_BASE_CFG:
+ SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE_CFG");
+ *data = s->secure_strtab_base_cfg;
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE:
+ SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
+ *data = extract64(s->secure_cmdq.base, 0, 32);
+ return MEMTX_OK;
+ case A_S_CMDQ_BASE + 4:
+ SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
+ *data = extract64(s->secure_cmdq.base, 32, 32);
+ return MEMTX_OK;
+ case A_S_CMDQ_PROD:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CMDQ_PROD", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cmdq.prod;
+ return MEMTX_OK;
+ case A_S_CMDQ_CONS:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_CMDQ_CONS", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_cmdq.cons;
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE:
+ SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
+ *data = extract64(s->secure_eventq.base, 0, 32);
+ return MEMTX_OK;
+ case A_S_EVENTQ_BASE + 4:
+ SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
+ *data = extract64(s->secure_eventq.base, 32, 32);
+ return MEMTX_OK;
+ case A_S_EVENTQ_PROD:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_EVENTQ_PROD", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_eventq.prod;
+ return MEMTX_OK;
+ case A_S_EVENTQ_CONS:
+ if (!smmu_validate_secure_read(attrs, secure_impl, offset,
+ "S_EVENTQ_CONS", data)) {
+ return MEMTX_OK;
+ }
+ *data = s->secure_eventq.cons;
+ return MEMTX_OK;
default:
*data = 0;
qemu_log_mask(LOG_UNIMP,
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
2025-08-06 15:11 ` [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-18 21:26 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state Tao Tang
` (9 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
This patch implements the S_INIT register, a secure-only register
with no non-secure counterpart. It provides a simple mechanism for
secure software to perform a global invalidation of all SMMU
configuration and translation caches.
This is typically the final step in a SMMU's probe sequence, marking
the end of initialization for the SMMU's secure interface.
With this and the previous change, a guest that is aware of the SMMUv3
secure extensions can probe the device's capabilities and perform basic
configuration of the secure interface, as is done by secure partition
managers like Hafnium in its smmuv3_driver_init function.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3.c | 29 +++++++++++++++++++++++++++++
hw/arm/trace-events | 1 +
2 files changed, 30 insertions(+)
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 619180d204..0ea9d897af 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -920,6 +920,21 @@ static void smmuv3_flush_config(SMMUDevice *sdev)
g_hash_table_remove(bc->configs, sdev);
}
+static void smmuv3_invalidate_all_caches(SMMUv3State *s)
+{
+ trace_smmuv3_invalidate_all_caches();
+ SMMUState *bs = &s->smmu_state;
+
+ /* Clear all cached configs including STE and CD*/
+ if (bs->configs) {
+ g_hash_table_remove_all(bs->configs);
+ }
+
+ /* Invalidate all SMMU IOTLB entries */
+ smmu_inv_notifiers_all(&s->smmu_state);
+ smmu_iotlb_inv_all(bs);
+}
+
/* Do translation with TLB lookup. */
static SMMUTranslationStatus smmuv3_do_translate(SMMUv3State *s, hwaddr addr,
SMMUTransCfg *cfg,
@@ -1921,6 +1936,16 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
s->secure_eventq_irq_cfg2 = data;
return MEMTX_OK;
+ case A_S_INIT:
+ SMMU_CHECK_SECURE_WRITE("S_INIT");
+ if (data & R_S_INIT_INV_ALL_MASK) {
+ /* write S_INIT and poll*/
+ s->secure_init = data & R_S_INIT_INV_ALL_MASK;
+ smmuv3_invalidate_all_caches(s);
+ }
+ /* initialization is completed and set to 0 to terminate the polling */
+ s->secure_init = 0;
+ return MEMTX_OK;
default:
qemu_log_mask(LOG_UNIMP,
"%s Unexpected 32-bit access to 0x%"PRIx64" (WI)\n",
@@ -2249,6 +2274,10 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
}
*data = s->secure_eventq.cons;
return MEMTX_OK;
+ case A_S_INIT:
+ SMMU_CHECK_SECURE_READ("S_INIT");
+ *data = s->secure_init;
+ return MEMTX_OK;
default:
*data = 0;
qemu_log_mask(LOG_UNIMP,
diff --git a/hw/arm/trace-events b/hw/arm/trace-events
index f3386bd7ae..019129ea43 100644
--- a/hw/arm/trace-events
+++ b/hw/arm/trace-events
@@ -64,6 +64,7 @@ smmuv3_cmdq_tlbi_s12_vmid(int vmid) "vmid=%d"
smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s"
smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s"
smmuv3_inv_notifiers_iova(const char *name, int asid, int vmid, uint64_t iova, uint8_t tg, uint64_t num_pages, int stage) "iommu mr=%s asid=%d vmid=%d iova=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64" stage=%d"
+smmuv3_invalidate_all_caches(void) "Invalidate all SMMU caches and TLBs"
smmu_reset_exit(void) ""
# strongarm.c
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (2 preceding siblings ...)
2025-08-06 15:11 ` [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 21:55 ` Pierrick Bouvier
2025-08-06 15:11 ` [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling Tao Tang
` (8 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
This patch enables the secure command queue, providing a dedicated
interface for secure software to issue commands to the SMMU. Based on
the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
entries directly from the Secure PA space so that we need to pass the
memory transaction attributes when reading the command queue.
This provides a parallel command mechanism to the non-secure world.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3-internal.h | 8 ++++--
hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
hw/arm/trace-events | 2 +-
3 files changed, 41 insertions(+), 24 deletions(-)
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index 1a8b1cb204..5b2ca00832 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
}
-static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
+static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
{
- return FIELD_EX32(s->cr[0], CR0, CMDQEN);
+ if (is_secure) {
+ return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
+ } else {
+ return FIELD_EX32(s->cr[0], CR0, CMDQEN);
+ }
}
static inline bool smmuv3_eventq_enabled(SMMUv3State *s)
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 0ea9d897af..0590f0f482 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -105,14 +105,17 @@ static void smmuv3_write_gerrorn(SMMUv3State *s, uint32_t new_gerrorn)
trace_smmuv3_write_gerrorn(toggled & pending, s->gerrorn);
}
-static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd)
+static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool is_secure)
{
dma_addr_t addr = Q_CONS_ENTRY(q);
MemTxResult ret;
int i;
+ MemTxAttrs attrs = is_secure ?
+ (MemTxAttrs) { .secure = 1 } :
+ (MemTxAttrs) { .unspecified = true };
ret = dma_memory_read(&address_space_memory, addr, cmd, sizeof(Cmd),
- MEMTXATTRS_UNSPECIFIED);
+ attrs);
if (ret != MEMTX_OK) {
return ret;
}
@@ -1311,14 +1314,14 @@ static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
}
-static int smmuv3_cmdq_consume(SMMUv3State *s)
+static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
{
SMMUState *bs = ARM_SMMU(s);
SMMUCmdError cmd_error = SMMU_CERROR_NONE;
- SMMUQueue *q = &s->cmdq;
+ SMMUQueue *q = is_secure ? &s->secure_cmdq : &s->cmdq;
SMMUCommandType type = 0;
- if (!smmuv3_cmdq_enabled(s)) {
+ if (!smmuv3_cmdq_enabled(s, is_secure)) {
return 0;
}
/*
@@ -1329,17 +1332,20 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
*/
while (!smmuv3_q_empty(q)) {
- uint32_t pending = s->gerror ^ s->gerrorn;
+ uint32_t pending = is_secure ? s->secure_gerror ^ s->secure_gerrorn :
+ s->gerror ^ s->gerrorn;
Cmd cmd;
trace_smmuv3_cmdq_consume(Q_PROD(q), Q_CONS(q),
- Q_PROD_WRAP(q), Q_CONS_WRAP(q));
+ Q_PROD_WRAP(q), Q_CONS_WRAP(q),
+ is_secure);
- if (FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
+ if (is_secure ? FIELD_EX32(pending, S_GERROR, CMDQ_ERR) :
+ FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
break;
}
- if (queue_read(q, &cmd) != MEMTX_OK) {
+ if (queue_read(q, &cmd, is_secure) != MEMTX_OK) {
cmd_error = SMMU_CERROR_ABT;
break;
}
@@ -1364,8 +1370,11 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
SMMUDevice *sdev = smmu_find_sdev(bs, sid);
if (CMD_SSEC(&cmd)) {
- cmd_error = SMMU_CERROR_ILL;
- break;
+ if (!is_secure) {
+ /* Secure Stream with NON-Secure command */
+ cmd_error = SMMU_CERROR_ILL;
+ break;
+ }
}
if (!sdev) {
@@ -1384,8 +1393,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
SMMUSIDRange sid_range;
if (CMD_SSEC(&cmd)) {
- cmd_error = SMMU_CERROR_ILL;
- break;
+ if (!is_secure) {
+ cmd_error = SMMU_CERROR_ILL;
+ break;
+ }
}
mask = (1ULL << (range + 1)) - 1;
@@ -1403,8 +1414,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
SMMUDevice *sdev = smmu_find_sdev(bs, sid);
if (CMD_SSEC(&cmd)) {
- cmd_error = SMMU_CERROR_ILL;
- break;
+ if (!is_secure) {
+ cmd_error = SMMU_CERROR_ILL;
+ break;
+ }
}
if (!sdev) {
@@ -1706,7 +1719,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
s->cr[0] = data;
s->cr0ack = data & ~SMMU_CR0_RESERVED;
/* in case the command queue has been enabled */
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, false);
return MEMTX_OK;
case A_CR1:
s->cr[1] = data;
@@ -1723,7 +1736,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
* By acknowledging the CMDQ_ERR, SW may notify cmds can
* be processed again
*/
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, false);
return MEMTX_OK;
case A_GERROR_IRQ_CFG0: /* 64b */
s->gerror_irq_cfg0 = deposit64(s->gerror_irq_cfg0, 0, 32, data);
@@ -1772,7 +1785,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
return MEMTX_OK;
case A_CMDQ_PROD:
s->cmdq.prod = data;
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, false);
return MEMTX_OK;
case A_CMDQ_CONS:
s->cmdq.cons = data;
@@ -1810,7 +1823,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
s->secure_cr[0] = data;
/* clear reserved bits */
s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, true);
return MEMTX_OK;
case A_S_CR1:
if (!smmu_validate_secure_write(attrs, secure_impl, offset,
@@ -1836,7 +1849,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
case A_S_GERRORN:
SMMU_CHECK_SECURE_WRITE("S_GERRORN");
smmuv3_write_gerrorn(s, data);
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, true);
return MEMTX_OK;
case A_S_GERROR_IRQ_CFG0:
SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
@@ -1892,7 +1905,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
case A_S_CMDQ_PROD:
SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
s->secure_cmdq.prod = data;
- smmuv3_cmdq_consume(s);
+ smmuv3_cmdq_consume(s, true);
return MEMTX_OK;
case A_S_CMDQ_CONS:
SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
diff --git a/hw/arm/trace-events b/hw/arm/trace-events
index 019129ea43..7d967226ff 100644
--- a/hw/arm/trace-events
+++ b/hw/arm/trace-events
@@ -35,7 +35,7 @@ smmuv3_trigger_irq(int irq) "irq=%d"
smmuv3_write_gerror(uint32_t toggled, uint32_t gerror) "toggled=0x%x, new GERROR=0x%x"
smmuv3_write_gerrorn(uint32_t acked, uint32_t gerrorn) "acked=0x%x, new GERRORN=0x%x"
smmuv3_unhandled_cmd(uint32_t type) "Unhandled command type=%d"
-smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod=%d cons=%d prod.wrap=%d cons.wrap=%d"
+smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap, bool is_secure_cmdq) "prod=%d cons=%d prod.wrap=%d cons.wrap=%d is_secure_cmdq=%d"
smmuv3_cmdq_opcode(const char *opcode) "<--- %s"
smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod:%d, cons:%d, prod_wrap:%d, cons_wrap:%d "
smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (3 preceding siblings ...)
2025-08-06 15:11 ` [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-11 10:41 ` Philippe Mathieu-Daudé
2025-08-06 15:11 ` [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions Tao Tang
` (7 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
Following the implementation of the secure command queue, this commit
introduces the infrastructure for reporting faults and events back to
secure software.
The secure event queue is now enabled, serving as the primary mechanism
for the SMMU to report translation faults and other architected events.
For more critical failures, such as an abort on an event queue write,
the SMMU_S_GERROR registers are also added. Finally, SMMU_S_IRQ_CTRL
is wired up to control interrupt notifications for both the event
queue and these global errors.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3-internal.h | 38 ++++++++++++++-----
hw/arm/smmuv3.c | 82 +++++++++++++++++++++++++---------------
hw/arm/trace-events | 2 +-
3 files changed, 81 insertions(+), 41 deletions(-)
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index 5b2ca00832..720d21652c 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -234,6 +234,8 @@ REG32(S_STRTAB_BASE_CFG, 0x8088)
REG64(S_CMDQ_BASE, 0x8090)
REG32(S_CMDQ_PROD, 0x8098)
REG32(S_CMDQ_CONS, 0x809c)
+ FIELD(S_CMDQ_CONS, ERR, 24, 7)
+
REG64(S_EVENTQ_BASE, 0x80a0)
REG32(S_EVENTQ_PROD, 0x80a8)
REG32(S_EVENTQ_CONS, 0x80ac)
@@ -269,14 +271,22 @@ static inline uint32_t smmuv3_idreg(int regoffset)
return smmuv3_ids[regoffset / 4];
}
-static inline bool smmuv3_eventq_irq_enabled(SMMUv3State *s)
+static inline bool smmuv3_eventq_irq_enabled(SMMUv3State *s, bool is_secure)
{
- return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, EVENTQ_IRQEN);
+ if (is_secure) {
+ return FIELD_EX32(s->secure_irq_ctrl, S_IRQ_CTRL, EVENTQ_IRQEN);
+ } else {
+ return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, EVENTQ_IRQEN);
+ }
}
-static inline bool smmuv3_gerror_irq_enabled(SMMUv3State *s)
+static inline bool smmuv3_gerror_irq_enabled(SMMUv3State *s, bool is_secure)
{
- return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, GERROR_IRQEN);
+ if (is_secure) {
+ return FIELD_EX32(s->secure_irq_ctrl, S_IRQ_CTRL, GERROR_IRQEN);
+ } else {
+ return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, GERROR_IRQEN);
+ }
}
/* Queue Handling */
@@ -328,14 +338,24 @@ static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
}
}
-static inline bool smmuv3_eventq_enabled(SMMUv3State *s)
+static inline bool smmuv3_eventq_enabled(SMMUv3State *s, bool is_secure)
{
- return FIELD_EX32(s->cr[0], CR0, EVENTQEN);
+ if (is_secure) {
+ return FIELD_EX32(s->secure_cr[0], S_CR0, EVENTQEN);
+ } else {
+ return FIELD_EX32(s->cr[0], CR0, EVENTQEN);
+ }
}
-static inline void smmu_write_cmdq_err(SMMUv3State *s, uint32_t err_type)
+static inline void smmu_write_cmdq_err(SMMUv3State *s, uint32_t err_type,
+ bool is_secure)
{
- s->cmdq.cons = FIELD_DP32(s->cmdq.cons, CMDQ_CONS, ERR, err_type);
+ if (is_secure) {
+ s->secure_cmdq.cons = FIELD_DP32(s->secure_cmdq.cons, S_CMDQ_CONS,
+ ERR, err_type);
+ } else {
+ s->cmdq.cons = FIELD_DP32(s->cmdq.cons, CMDQ_CONS, ERR, err_type);
+ }
}
/* Commands */
@@ -589,7 +609,7 @@ typedef struct SMMUEventInfo {
(x)->word[6] = (uint32_t)(addr & 0xffffffff); \
} while (0)
-void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *event);
+void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *event, bool is_secure);
/* Configuration Data */
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 0590f0f482..1f05cc983b 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -48,14 +48,14 @@
* @gerror_mask: mask of gerrors to toggle (relevant if @irq is GERROR)
*/
static void smmuv3_trigger_irq(SMMUv3State *s, SMMUIrq irq,
- uint32_t gerror_mask)
+ uint32_t gerror_mask, bool is_secure)
{
bool pulse = false;
switch (irq) {
case SMMU_IRQ_EVTQ:
- pulse = smmuv3_eventq_irq_enabled(s);
+ pulse = smmuv3_eventq_irq_enabled(s, is_secure);
break;
case SMMU_IRQ_PRIQ:
qemu_log_mask(LOG_UNIMP, "PRI not yet supported\n");
@@ -65,17 +65,23 @@ static void smmuv3_trigger_irq(SMMUv3State *s, SMMUIrq irq,
break;
case SMMU_IRQ_GERROR:
{
- uint32_t pending = s->gerror ^ s->gerrorn;
+ uint32_t pending = is_secure ? s->secure_gerror ^ s->secure_gerrorn :
+ s->gerror ^ s->gerrorn;
uint32_t new_gerrors = ~pending & gerror_mask;
if (!new_gerrors) {
/* only toggle non pending errors */
return;
}
- s->gerror ^= new_gerrors;
- trace_smmuv3_write_gerror(new_gerrors, s->gerror);
+ if (is_secure) {
+ s->secure_gerror ^= new_gerrors;
+ } else {
+ s->gerror ^= new_gerrors;
+ }
+ trace_smmuv3_write_gerror(new_gerrors, is_secure ? s->secure_gerror :
+ s->gerror);
- pulse = smmuv3_gerror_irq_enabled(s);
+ pulse = smmuv3_gerror_irq_enabled(s, is_secure);
break;
}
}
@@ -85,24 +91,32 @@ static void smmuv3_trigger_irq(SMMUv3State *s, SMMUIrq irq,
}
}
-static void smmuv3_write_gerrorn(SMMUv3State *s, uint32_t new_gerrorn)
+static void smmuv3_write_gerrorn(SMMUv3State *s, uint32_t new_gerrorn,
+ bool is_secure)
{
- uint32_t pending = s->gerror ^ s->gerrorn;
- uint32_t toggled = s->gerrorn ^ new_gerrorn;
+ uint32_t pending = is_secure ? s->secure_gerror ^ s->secure_gerrorn :
+ s->gerror ^ s->gerrorn;
+ uint32_t toggled = is_secure ? s->secure_gerrorn ^ new_gerrorn :
+ s->gerrorn ^ new_gerrorn;
if (toggled & ~pending) {
qemu_log_mask(LOG_GUEST_ERROR,
- "guest toggles non pending errors = 0x%x\n",
- toggled & ~pending);
+ "guest toggles non pending errors = 0x%x is_secure=%d\n",
+ toggled & ~pending, is_secure);
}
/*
* We do not raise any error in case guest toggles bits corresponding
* to not active IRQs (CONSTRAINED UNPREDICTABLE)
*/
- s->gerrorn = new_gerrorn;
+ if (is_secure) {
+ s->secure_gerrorn = new_gerrorn;
+ } else {
+ s->gerrorn = new_gerrorn;
+ }
- trace_smmuv3_write_gerrorn(toggled & pending, s->gerrorn);
+ trace_smmuv3_write_gerrorn(toggled & pending,
+ is_secure ? s->secure_gerrorn : s->gerrorn);
}
static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool is_secure)
@@ -125,18 +139,21 @@ static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool is_secure)
return ret;
}
-static MemTxResult queue_write(SMMUQueue *q, Evt *evt_in)
+static MemTxResult queue_write(SMMUQueue *q, Evt *evt_in, bool is_secure)
{
dma_addr_t addr = Q_PROD_ENTRY(q);
MemTxResult ret;
Evt evt = *evt_in;
int i;
+ MemTxAttrs attrs = is_secure ?
+ (MemTxAttrs) { .secure = 1 } :
+ (MemTxAttrs) { .unspecified = true };
for (i = 0; i < ARRAY_SIZE(evt.word); i++) {
cpu_to_le32s(&evt.word[i]);
}
ret = dma_memory_write(&address_space_memory, addr, &evt, sizeof(Evt),
- MEMTXATTRS_UNSPECIFIED);
+ attrs);
if (ret != MEMTX_OK) {
return ret;
}
@@ -145,12 +162,12 @@ static MemTxResult queue_write(SMMUQueue *q, Evt *evt_in)
return MEMTX_OK;
}
-static MemTxResult smmuv3_write_eventq(SMMUv3State *s, Evt *evt)
+static MemTxResult smmuv3_write_eventq(SMMUv3State *s, Evt *evt, bool is_secure)
{
- SMMUQueue *q = &s->eventq;
+ SMMUQueue *q = is_secure ? &s->secure_eventq : &s->eventq;
MemTxResult r;
- if (!smmuv3_eventq_enabled(s)) {
+ if (!smmuv3_eventq_enabled(s, is_secure)) {
return MEMTX_ERROR;
}
@@ -158,23 +175,23 @@ static MemTxResult smmuv3_write_eventq(SMMUv3State *s, Evt *evt)
return MEMTX_ERROR;
}
- r = queue_write(q, evt);
+ r = queue_write(q, evt, is_secure);
if (r != MEMTX_OK) {
return r;
}
if (!smmuv3_q_empty(q)) {
- smmuv3_trigger_irq(s, SMMU_IRQ_EVTQ, 0);
+ smmuv3_trigger_irq(s, SMMU_IRQ_EVTQ, 0, is_secure);
}
return MEMTX_OK;
}
-void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *info)
+void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *info, bool is_secure)
{
Evt evt = {};
MemTxResult r;
- if (!smmuv3_eventq_enabled(s)) {
+ if (!smmuv3_eventq_enabled(s, is_secure)) {
return;
}
@@ -252,10 +269,12 @@ void smmuv3_record_event(SMMUv3State *s, SMMUEventInfo *info)
g_assert_not_reached();
}
- trace_smmuv3_record_event(smmu_event_string(info->type), info->sid);
- r = smmuv3_write_eventq(s, &evt);
+ trace_smmuv3_record_event(smmu_event_string(info->type),
+ info->sid, is_secure);
+ r = smmuv3_write_eventq(s, &evt, is_secure);
if (r != MEMTX_OK) {
- smmuv3_trigger_irq(s, SMMU_IRQ_GERROR, R_GERROR_EVENTQ_ABT_ERR_MASK);
+ smmuv3_trigger_irq(s, SMMU_IRQ_GERROR, R_GERROR_EVENTQ_ABT_ERR_MASK,
+ is_secure);
}
info->recorded = true;
}
@@ -1148,7 +1167,7 @@ epilogue:
qemu_log_mask(LOG_GUEST_ERROR,
"%s translation failed for iova=0x%"PRIx64" (%s)\n",
mr->parent_obj.name, addr, smmu_event_string(event.type));
- smmuv3_record_event(s, &event);
+ smmuv3_record_event(s, &event, false);
break;
}
@@ -1358,7 +1377,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
switch (type) {
case SMMU_CMD_SYNC:
if (CMD_SYNC_CS(&cmd) & CMD_SYNC_SIG_IRQ) {
- smmuv3_trigger_irq(s, SMMU_IRQ_CMD_SYNC, 0);
+ smmuv3_trigger_irq(s, SMMU_IRQ_CMD_SYNC, 0, is_secure);
}
break;
case SMMU_CMD_PREFETCH_CONFIG:
@@ -1544,8 +1563,9 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
if (cmd_error) {
trace_smmuv3_cmdq_consume_error(smmu_cmd_string(type), cmd_error);
- smmu_write_cmdq_err(s, cmd_error);
- smmuv3_trigger_irq(s, SMMU_IRQ_GERROR, R_GERROR_CMDQ_ERR_MASK);
+ smmu_write_cmdq_err(s, cmd_error, is_secure);
+ smmuv3_trigger_irq(s, SMMU_IRQ_GERROR,
+ R_GERROR_CMDQ_ERR_MASK, is_secure);
}
trace_smmuv3_cmdq_consume_out(Q_PROD(q), Q_CONS(q),
@@ -1731,7 +1751,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
s->irq_ctrl = data;
return MEMTX_OK;
case A_GERRORN:
- smmuv3_write_gerrorn(s, data);
+ smmuv3_write_gerrorn(s, data, false);
/*
* By acknowledging the CMDQ_ERR, SW may notify cmds can
* be processed again
@@ -1848,7 +1868,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
return MEMTX_OK;
case A_S_GERRORN:
SMMU_CHECK_SECURE_WRITE("S_GERRORN");
- smmuv3_write_gerrorn(s, data);
+ smmuv3_write_gerrorn(s, data, true);
smmuv3_cmdq_consume(s, true);
return MEMTX_OK;
case A_S_GERROR_IRQ_CFG0:
diff --git a/hw/arm/trace-events b/hw/arm/trace-events
index 7d967226ff..7bb0bd0cc5 100644
--- a/hw/arm/trace-events
+++ b/hw/arm/trace-events
@@ -40,7 +40,7 @@ smmuv3_cmdq_opcode(const char *opcode) "<--- %s"
smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod:%d, cons:%d, prod_wrap:%d, cons_wrap:%d "
smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
smmuv3_write_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
-smmuv3_record_event(const char *type, uint32_t sid) "%s sid=0x%x"
+smmuv3_record_event(const char *type, uint32_t sid, bool is_secure) "%s sid=0x%x is_secure=%d"
smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split) "sid=0x%x features:0x%x, sid_split:0x%x"
smmuv3_find_ste_2lvl(uint64_t strtab_base, uint64_t l1ptr, int l1_ste_offset, uint64_t l2ptr, int l2_ste_offset, int max_l2_ste) "strtab_base:0x%"PRIx64" l1ptr:0x%"PRIx64" l1_off:0x%x, l2ptr:0x%"PRIx64" l2_off:0x%x max_l2_ste:%d"
smmuv3_get_ste(uint64_t addr) "STE addr: 0x%"PRIx64
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (4 preceding siblings ...)
2025-08-06 15:11 ` [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-18 21:28 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 07/11] hw/arm/smmuv3: Add separate address space for secure SMMU accesses Tao Tang
` (6 subsequent siblings)
12 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
To support parallel processing of secure and non-secure streams, the
SMMUv3 model needs to differentiate between the two contexts throughout
its core logic. This commit is the foundational step to make the code
security-state aware.
An is_secure flag, which will be used in subsequent patches to represent
the transaction's security state, is now plumbed through the main
processing paths.
This change is purely preparatory and introduces no functional changes
for the existing non-secure path. All current call sites are updated
to pass is_secure = false.
This refactoring paves the way for upcoming patches that will introduce
separate TLB entries for secure transactions and enable a fully
parallel secure/non-secure SMMU model.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmu-common.c | 11 ++-
hw/arm/smmuv3-internal.h | 16 +++-
hw/arm/smmuv3.c | 160 ++++++++++++++++++++++++-----------
hw/arm/trace-events | 2 +-
include/hw/arm/smmu-common.h | 1 +
include/hw/arm/smmuv3.h | 4 +
6 files changed, 138 insertions(+), 56 deletions(-)
diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
index 0dcaf2f589..28d6d1bc7f 100644
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -332,13 +332,16 @@ void smmu_iotlb_inv_vmid_s1(SMMUState *s, int vmid)
* @base_addr[@index]
*/
static int get_pte(dma_addr_t baseaddr, uint32_t index, uint64_t *pte,
- SMMUPTWEventInfo *info)
+ SMMUPTWEventInfo *info, bool is_secure)
{
int ret;
dma_addr_t addr = baseaddr + index * sizeof(*pte);
+ MemTxAttrs attrs = is_secure ?
+ (MemTxAttrs) { .secure = 1 } :
+ (MemTxAttrs) { .unspecified = true };
/* TODO: guarantee 64-bit single-copy atomicity */
- ret = ldq_le_dma(&address_space_memory, addr, pte, MEMTXATTRS_UNSPECIFIED);
+ ret = ldq_le_dma(&address_space_memory, addr, pte, attrs);
if (ret != MEMTX_OK) {
info->type = SMMU_PTW_ERR_WALK_EABT;
@@ -485,7 +488,7 @@ static int smmu_ptw_64_s1(SMMUState *bs, SMMUTransCfg *cfg,
dma_addr_t pte_addr = baseaddr + offset * sizeof(pte);
uint8_t ap;
- if (get_pte(baseaddr, offset, &pte, info)) {
+ if (get_pte(baseaddr, offset, &pte, info, cfg->secure)) {
goto error;
}
trace_smmu_ptw_level(stage, level, iova, subpage_size,
@@ -621,7 +624,7 @@ static int smmu_ptw_64_s2(SMMUTransCfg *cfg,
dma_addr_t pte_addr = baseaddr + offset * sizeof(pte);
uint8_t s2ap;
- if (get_pte(baseaddr, offset, &pte, info)) {
+ if (get_pte(baseaddr, offset, &pte, info, cfg->secure)) {
goto error;
}
trace_smmu_ptw_level(stage, level, ipa, subpage_size,
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index 720d21652c..852186cea4 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -243,9 +243,13 @@ REG64(S_EVENTQ_IRQ_CFG0, 0x80b0)
REG32(S_EVENTQ_IRQ_CFG1, 0x80b8)
REG32(S_EVENTQ_IRQ_CFG2, 0x80bc)
-static inline int smmu_enabled(SMMUv3State *s)
+static inline int smmu_enabled(SMMUv3State *s, bool is_secure)
{
- return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
+ if (is_secure) {
+ return FIELD_EX32(s->secure_cr[0], S_CR0, SMMUEN);
+ } else {
+ return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
+ }
}
/* Command Queue Entry */
@@ -661,6 +665,10 @@ typedef struct CD {
#define STE_S2S(x) extract32((x)->word[5], 25, 1)
#define STE_S2R(x) extract32((x)->word[5], 26, 1)
+#define STE_S_S2T0SZ(x) extract32((x)->word[9], 0 , 6)
+#define STE_S_S2SL0(x) extract32((x)->word[9], 6 , 2)
+#define STE_S_S2TG(x) extract32((x)->word[9], 14, 2)
+
#define STE_CTXPTR(x) \
((extract64((x)->word[1], 0, 16) << 32) | \
((x)->word[0] & 0xffffffc0))
@@ -669,6 +677,10 @@ typedef struct CD {
((extract64((x)->word[7], 0, 16) << 32) | \
((x)->word[6] & 0xfffffff0))
+#define STE_S_S2TTB(x) \
+ ((extract64((x)->word[13], 0, 16) << 32) | \
+ ((x)->word[12] & 0xfffffff0))
+
static inline int oas2bits(int oas_field)
{
switch (oas_field) {
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 1f05cc983b..bcf06679e1 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -354,14 +354,14 @@ static void smmuv3_init_regs(SMMUv3State *s)
}
static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
- SMMUEventInfo *event)
+ SMMUEventInfo *event, MemTxAttrs attrs)
{
int ret, i;
trace_smmuv3_get_ste(addr);
/* TODO: guarantee 64-bit single-copy atomicity */
ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
- MEMTXATTRS_UNSPECIFIED);
+ attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Cannot fetch pte at address=0x%"PRIx64"\n", addr);
@@ -390,6 +390,9 @@ static int smmu_get_cd(SMMUv3State *s, STE *ste, SMMUTransCfg *cfg,
int ret, i;
SMMUTranslationStatus status;
SMMUTLBEntry *entry;
+ MemTxAttrs attrs = cfg->secure ?
+ (MemTxAttrs) { .secure = 1 } :
+ (MemTxAttrs) { .unspecified = true };
trace_smmuv3_get_cd(addr);
@@ -407,7 +410,7 @@ static int smmu_get_cd(SMMUv3State *s, STE *ste, SMMUTransCfg *cfg,
/* TODO: guarantee 64-bit single-copy atomicity */
ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
- MEMTXATTRS_UNSPECIFIED);
+ attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Cannot fetch pte at address=0x%"PRIx64"\n", addr);
@@ -472,7 +475,8 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
g_assert_not_reached();
}
- switch (STE_S2TG(ste)) {
+ uint32_t s2tg = cfg->secure ? STE_S_S2TG(ste) : STE_S2TG(ste);
+ switch (s2tg) {
case 0x0: /* 4KB */
cfg->s2cfg.granule_sz = 12;
break;
@@ -484,13 +488,13 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
break;
default:
qemu_log_mask(LOG_GUEST_ERROR,
- "SMMUv3 bad STE S2TG: %x\n", STE_S2TG(ste));
+ "SMMUv3 bad STE S2TG: %x\n", s2tg);
goto bad_ste;
}
- cfg->s2cfg.vttb = STE_S2TTB(ste);
+ cfg->s2cfg.vttb = cfg->secure ? STE_S_S2TTB(ste) : STE_S2TTB(ste);
- cfg->s2cfg.sl0 = STE_S2SL0(ste);
+ cfg->s2cfg.sl0 = cfg->secure ? STE_S_S2SL0(ste) : STE_S2SL0(ste);
/* FEAT_TTST not supported. */
if (cfg->s2cfg.sl0 == 0x3) {
qemu_log_mask(LOG_UNIMP, "SMMUv3 S2SL0 = 0x3 has no meaning!\n");
@@ -519,7 +523,7 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
goto bad_ste;
}
- cfg->s2cfg.tsz = STE_S2T0SZ(ste);
+ cfg->s2cfg.tsz = cfg->secure ? STE_S_S2T0SZ(ste) : STE_S2T0SZ(ste);
if (!s2t0sz_valid(cfg)) {
qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 bad STE S2T0SZ = %d\n",
@@ -599,21 +603,52 @@ static int decode_ste(SMMUv3State *s, SMMUTransCfg *cfg,
if (cfg->aborted || cfg->bypassed) {
return 0;
}
+ bool is_secure = cfg->secure;
/*
* If a stage is enabled in SW while not advertised, throw bad ste
* according to user manual(IHI0070E) "5.2 Stream Table Entry".
*/
- if (!STAGE1_SUPPORTED(s) && STE_CFG_S1_ENABLED(config)) {
- qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 S1 used but not supported.\n");
- goto bad_ste;
- }
- if (!STAGE2_SUPPORTED(s) && STE_CFG_S2_ENABLED(config)) {
- qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 S2 used but not supported.\n");
- goto bad_ste;
+ if (!is_secure) {
+ if (!STAGE1_SUPPORTED(s) && STE_CFG_S1_ENABLED(config)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "SMMUv3 S1 used but not supported.\n");
+ goto bad_ste;
+ }
+ if (!STAGE2_SUPPORTED(s) && STE_CFG_S2_ENABLED(config)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "SMMUv3 S2 used but not supported.\n");
+ goto bad_ste;
+ }
+ } else {
+ /*
+ * As described in user manual(IHI0070G.b) "3.10.2 Support for Secure
+ * state" , the SMMU supports stage 1 translation and might support
+ * stage 2 translation.
+ */
+ if (!SECURE_IMPLEMENTED(s) && STE_CFG_S1_ENABLED(config)) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "SMMUv3 S1 used but not supported in secure state.\n");
+ goto bad_ste;
+ }
+
+ /*
+ * IHI0070G.b "6.3.53 SMMU_S_IDR1": SEL2 == 0 if SMMU_IDR0.S1P == 0 or
+ * if SMMU_IDR0.S2P == 0.
+ */
+ if (STE_CFG_S2_ENABLED(config)) {
+ if (!SECURE_S2_SUPPORTED(s) ||
+ (SECURE_S2_SUPPORTED(s) &&
+ (!STAGE1_SUPPORTED(s) || !STAGE2_SUPPORTED(s)))) {
+ qemu_log_mask(LOG_GUEST_ERROR,
+ "SMMUv3 S2 used but not supported in secure state.\n");
+ goto bad_ste;
+ }
+ }
}
- if (STAGE2_SUPPORTED(s)) {
+ if ((!is_secure && STAGE2_SUPPORTED(s)) ||
+ (is_secure && SECURE_S2_SUPPORTED(s))) {
/* VMID is considered even if s2 is disabled. */
cfg->s2cfg.vmid = STE_S2VMID(ste);
} else {
@@ -659,20 +694,29 @@ bad_ste:
* @sid: stream ID
* @ste: returned stream table entry
* @event: handle to an event info
+ * @is_secure: true if the translation is for a secure domain
*
* Supports linear and 2-level stream table
* Return 0 on success, -EINVAL otherwise
*/
static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
- SMMUEventInfo *event)
+ SMMUEventInfo *event, bool is_secure)
{
- dma_addr_t addr, strtab_base;
+ dma_addr_t addr;
uint32_t log2size;
int strtab_size_shift;
int ret;
+ uint32_t features = is_secure ? s->secure_features : s->features;
+ dma_addr_t strtab_base = is_secure ? s->secure_strtab_base : s->strtab_base;
+ uint8_t sid_split = is_secure ? s->secure_sid_split : s->sid_split;
+ MemTxAttrs attrs = is_secure ?
+ (MemTxAttrs) { .secure = 1 } :
+ (MemTxAttrs) { .unspecified = true };
- trace_smmuv3_find_ste(sid, s->features, s->sid_split);
- log2size = FIELD_EX32(s->strtab_base_cfg, STRTAB_BASE_CFG, LOG2SIZE);
+ trace_smmuv3_find_ste(sid, features, sid_split, is_secure);
+ log2size = is_secure
+ ? FIELD_EX32(s->secure_strtab_base_cfg, S_STRTAB_BASE_CFG, LOG2SIZE)
+ : FIELD_EX32(s->strtab_base_cfg, STRTAB_BASE_CFG, LOG2SIZE);
/*
* Check SID range against both guest-configured and implementation limits
*/
@@ -680,7 +724,7 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
event->type = SMMU_EVT_C_BAD_STREAMID;
return -EINVAL;
}
- if (s->features & SMMU_FEATURE_2LVL_STE) {
+ if (features & SMMU_FEATURE_2LVL_STE) {
int l1_ste_offset, l2_ste_offset, max_l2_ste, span, i;
dma_addr_t l1ptr, l2ptr;
STEDesc l1std;
@@ -689,15 +733,15 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
* Align strtab base address to table size. For this purpose, assume it
* is not bounded by SMMU_IDR1_SIDSIZE.
*/
- strtab_size_shift = MAX(5, (int)log2size - s->sid_split - 1 + 3);
- strtab_base = s->strtab_base & SMMU_BASE_ADDR_MASK &
+ strtab_size_shift = MAX(5, (int)log2size - sid_split - 1 + 3);
+ strtab_base = strtab_base & SMMU_BASE_ADDR_MASK &
~MAKE_64BIT_MASK(0, strtab_size_shift);
- l1_ste_offset = sid >> s->sid_split;
- l2_ste_offset = sid & ((1 << s->sid_split) - 1);
+ l1_ste_offset = sid >> sid_split;
+ l2_ste_offset = sid & ((1 << sid_split) - 1);
l1ptr = (dma_addr_t)(strtab_base + l1_ste_offset * sizeof(l1std));
/* TODO: guarantee 64-bit single-copy atomicity */
ret = dma_memory_read(&address_space_memory, l1ptr, &l1std,
- sizeof(l1std), MEMTXATTRS_UNSPECIFIED);
+ sizeof(l1std), attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Could not read L1PTR at 0X%"PRIx64"\n", l1ptr);
@@ -722,7 +766,7 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
}
max_l2_ste = (1 << span) - 1;
l2ptr = l1std_l2ptr(&l1std);
- trace_smmuv3_find_ste_2lvl(s->strtab_base, l1ptr, l1_ste_offset,
+ trace_smmuv3_find_ste_2lvl(strtab_base, l1ptr, l1_ste_offset,
l2ptr, l2_ste_offset, max_l2_ste);
if (l2_ste_offset > max_l2_ste) {
qemu_log_mask(LOG_GUEST_ERROR,
@@ -734,12 +778,12 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
addr = l2ptr + l2_ste_offset * sizeof(*ste);
} else {
strtab_size_shift = log2size + 5;
- strtab_base = s->strtab_base & SMMU_BASE_ADDR_MASK &
+ strtab_base = strtab_base & SMMU_BASE_ADDR_MASK &
~MAKE_64BIT_MASK(0, strtab_size_shift);
addr = strtab_base + sid * sizeof(*ste);
}
- if (smmu_get_ste(s, addr, ste, event)) {
+ if (smmu_get_ste(s, addr, ste, event, attrs)) {
return -EINVAL;
}
@@ -868,7 +912,7 @@ static int smmuv3_decode_config(IOMMUMemoryRegion *mr, SMMUTransCfg *cfg,
/* ASID defaults to -1 (if s1 is not supported). */
cfg->asid = -1;
- ret = smmu_find_ste(s, sid, &ste, event);
+ ret = smmu_find_ste(s, sid, &ste, event, cfg->secure);
if (ret) {
return ret;
}
@@ -897,12 +941,14 @@ static int smmuv3_decode_config(IOMMUMemoryRegion *mr, SMMUTransCfg *cfg,
*
* @sdev: SMMUDevice handle
* @event: output event info
+ * @is_secure: true if the translation is for a secure domain
*
* The configuration cache contains data resulting from both STE and CD
* decoding under the form of an SMMUTransCfg struct. The hash table is indexed
* by the SMMUDevice handle.
*/
-static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event)
+static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event,
+ bool is_secure)
{
SMMUv3State *s = sdev->smmu;
SMMUState *bc = &s->smmu_state;
@@ -922,6 +968,7 @@ static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event)
100 * sdev->cfg_cache_hits /
(sdev->cfg_cache_hits + sdev->cfg_cache_misses));
cfg = g_new0(SMMUTransCfg, 1);
+ cfg->secure = is_secure;
if (!smmuv3_decode_config(&sdev->iommu, cfg, event)) {
g_hash_table_insert(bc->configs, sdev, cfg);
@@ -1103,19 +1150,25 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
.perm = IOMMU_NONE,
};
SMMUTLBEntry *cached_entry = NULL;
+ /* We don't support secure translation for now */
+ bool is_secure = false;
qemu_mutex_lock(&s->mutex);
- if (!smmu_enabled(s)) {
- if (FIELD_EX32(s->gbpa, GBPA, ABORT)) {
- status = SMMU_TRANS_ABORT;
+ if (!smmu_enabled(s, is_secure)) {
+ bool abort_flag;
+
+ if (is_secure) {
+ abort_flag = FIELD_EX32(s->secure_gbpa, S_GBPA, ABORT);
} else {
- status = SMMU_TRANS_DISABLE;
+ abort_flag = FIELD_EX32(s->gbpa, GBPA, ABORT);
}
+
+ status = abort_flag ? SMMU_TRANS_ABORT : SMMU_TRANS_DISABLE;
goto epilogue;
}
- cfg = smmuv3_get_config(sdev, &event);
+ cfg = smmuv3_get_config(sdev, &event, is_secure);
if (!cfg) {
status = SMMU_TRANS_ERROR;
goto epilogue;
@@ -1167,7 +1220,7 @@ epilogue:
qemu_log_mask(LOG_GUEST_ERROR,
"%s translation failed for iova=0x%"PRIx64" (%s)\n",
mr->parent_obj.name, addr, smmu_event_string(event.type));
- smmuv3_record_event(s, &event, false);
+ smmuv3_record_event(s, &event, cfg->secure);
break;
}
@@ -1186,16 +1239,18 @@ epilogue:
* @tg: translation granule (if communicated through range invalidation)
* @num_pages: number of @granule sized pages (if tg != 0), otherwise 1
* @stage: Which stage(1 or 2) is used
+ * @is_secure: true if the translation is for a secure domain
*/
static void smmuv3_notify_iova(IOMMUMemoryRegion *mr,
IOMMUNotifier *n,
int asid, int vmid,
dma_addr_t iova, uint8_t tg,
- uint64_t num_pages, int stage)
+ uint64_t num_pages, int stage,
+ bool is_secure)
{
SMMUDevice *sdev = container_of(mr, SMMUDevice, iommu);
SMMUEventInfo eventinfo = {.inval_ste_allowed = true};
- SMMUTransCfg *cfg = smmuv3_get_config(sdev, &eventinfo);
+ SMMUTransCfg *cfg = smmuv3_get_config(sdev, &eventinfo, is_secure);
IOMMUTLBEvent event;
uint8_t granule;
@@ -1251,7 +1306,8 @@ static void smmuv3_notify_iova(IOMMUMemoryRegion *mr,
/* invalidate an asid/vmid/iova range tuple in all mr's */
static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, int vmid,
dma_addr_t iova, uint8_t tg,
- uint64_t num_pages, int stage)
+ uint64_t num_pages, int stage,
+ bool is_secure)
{
SMMUDevice *sdev;
@@ -1263,12 +1319,14 @@ static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, int vmid,
iova, tg, num_pages, stage);
IOMMU_NOTIFIER_FOREACH(n, mr) {
- smmuv3_notify_iova(mr, n, asid, vmid, iova, tg, num_pages, stage);
+ smmuv3_notify_iova(mr, n, asid, vmid, iova, tg,
+ num_pages, stage, is_secure);
}
}
}
-static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
+static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage,
+ bool is_secure)
{
dma_addr_t end, addr = CMD_ADDR(cmd);
uint8_t type = CMD_TYPE(cmd);
@@ -1284,7 +1342,8 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
SMMUv3State *smmuv3 = ARM_SMMUV3(s);
/* Only consider VMID if stage-2 is supported. */
- if (STAGE2_SUPPORTED(smmuv3)) {
+ if (STAGE2_SUPPORTED(smmuv3) ||
+ (SECURE_IMPLEMENTED(smmuv3) && SECURE_S2_SUPPORTED(smmuv3))) {
vmid = CMD_VMID(cmd);
}
@@ -1294,7 +1353,7 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
if (!tg) {
trace_smmuv3_range_inval(vmid, asid, addr, tg, 1, ttl, leaf, stage);
- smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, 1, stage);
+ smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, 1, stage, is_secure);
if (stage == SMMU_STAGE_1) {
smmu_iotlb_inv_iova(s, asid, vmid, addr, tg, 1, ttl);
} else {
@@ -1317,7 +1376,8 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
num_pages = (mask + 1) >> granule;
trace_smmuv3_range_inval(vmid, asid, addr, tg, num_pages,
ttl, leaf, stage);
- smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, num_pages, stage);
+ smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg,
+ num_pages, stage, is_secure);
if (stage == SMMU_STAGE_1) {
smmu_iotlb_inv_iova(s, asid, vmid, addr, tg, num_pages, ttl);
} else {
@@ -1461,7 +1521,8 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
* VMID is only matched when stage 2 is supported, otherwise set it
* to -1 as the value used for stage-1 only VMIDs.
*/
- if (STAGE2_SUPPORTED(s)) {
+ if (STAGE2_SUPPORTED(s) ||
+ (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
vmid = CMD_VMID(&cmd);
}
@@ -1483,7 +1544,8 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
* If stage-2 is supported, invalidate for this VMID only, otherwise
* invalidate the whole thing.
*/
- if (STAGE2_SUPPORTED(s)) {
+ if (STAGE2_SUPPORTED(s) ||
+ (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
vmid = CMD_VMID(&cmd);
trace_smmuv3_cmdq_tlbi_nh(vmid);
smmu_iotlb_inv_vmid_s1(bs, vmid);
@@ -1502,7 +1564,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
cmd_error = SMMU_CERROR_ILL;
break;
}
- smmuv3_range_inval(bs, &cmd, SMMU_STAGE_1);
+ smmuv3_range_inval(bs, &cmd, SMMU_STAGE_1, false);
break;
case SMMU_CMD_TLBI_S12_VMALL:
{
@@ -1527,7 +1589,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
* As currently only either s1 or s2 are supported
* we can reuse same function for s2.
*/
- smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2);
+ smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2, false);
break;
case SMMU_CMD_TLBI_EL3_ALL:
case SMMU_CMD_TLBI_EL3_VA:
diff --git a/hw/arm/trace-events b/hw/arm/trace-events
index 7bb0bd0cc5..92c87f0b9e 100644
--- a/hw/arm/trace-events
+++ b/hw/arm/trace-events
@@ -41,7 +41,7 @@ smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t
smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
smmuv3_write_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
smmuv3_record_event(const char *type, uint32_t sid, bool is_secure) "%s sid=0x%x is_secure=%d"
-smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split) "sid=0x%x features:0x%x, sid_split:0x%x"
+smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split, bool is_secure) "sid=0x%x features:0x%x, sid_split:0x%x is_secure=%d"
smmuv3_find_ste_2lvl(uint64_t strtab_base, uint64_t l1ptr, int l1_ste_offset, uint64_t l2ptr, int l2_ste_offset, int max_l2_ste) "strtab_base:0x%"PRIx64" l1ptr:0x%"PRIx64" l1_off:0x%x, l2ptr:0x%"PRIx64" l2_off:0x%x max_l2_ste:%d"
smmuv3_get_ste(uint64_t addr) "STE addr: 0x%"PRIx64
smmuv3_translate_disable(const char *n, uint16_t sid, uint64_t addr, bool is_write) "%s sid=0x%x bypass (smmu disabled) iova:0x%"PRIx64" is_write=%d"
diff --git a/include/hw/arm/smmu-common.h b/include/hw/arm/smmu-common.h
index e5e2d09294..5d15a1212b 100644
--- a/include/hw/arm/smmu-common.h
+++ b/include/hw/arm/smmu-common.h
@@ -116,6 +116,7 @@ typedef struct SMMUTransCfg {
SMMUTransTableInfo tt[2];
/* Used by stage-2 only. */
struct SMMUS2Cfg s2cfg;
+ bool secure;
} SMMUTransCfg;
typedef struct SMMUDevice {
diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
index 72ad042514..43c7289a43 100644
--- a/include/hw/arm/smmuv3.h
+++ b/include/hw/arm/smmuv3.h
@@ -110,4 +110,8 @@ OBJECT_DECLARE_TYPE(SMMUv3State, SMMUv3Class, ARM_SMMUV3)
#define STAGE1_SUPPORTED(s) FIELD_EX32(s->idr[0], IDR0, S1P)
#define STAGE2_SUPPORTED(s) FIELD_EX32(s->idr[0], IDR0, S2P)
+#define SECURE_IMPLEMENTED(s) \
+ FIELD_DP32(s->secure_idr[1], S_IDR1, SECURE_IMPL, 1)
+#define SECURE_S2_SUPPORTED(s) \
+ FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1)
#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 07/11] hw/arm/smmuv3: Add separate address space for secure SMMU accesses
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (5 preceding siblings ...)
2025-08-06 15:11 ` [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 15:11 ` [RFC 08/11] hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations Tao Tang
` (5 subsequent siblings)
12 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
According to the Arm architecture, SMMU-originated memory accesses,
such as fetching commands or writing events for a secure stream, must
target the Secure Physical Address (PA) space. The existing model sends
all DMA to the global address_space_memory.
This patch introduces the infrastructure to differentiate between secure
and non-secure memory accesses. A weak global symbol,
arm_secure_address_space, is added, which can be provided by the
machine model to represent the Secure PA space.
A new helper, smmu_get_address_space(), selects the target address
space based on the is_secure context. All internal DMA calls
(dma_memory_read/write) are updated to use this helper. Additionally,
the attrs.secure bit is set on transactions targeting the secure
address space.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmu-common.c | 10 +++++++++-
hw/arm/smmuv3.c | 24 ++++++++++++------------
hw/arm/virt.c | 5 +++++
include/hw/arm/smmu-common.h | 13 +++++++++++++
4 files changed, 39 insertions(+), 13 deletions(-)
diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
index 28d6d1bc7f..0877248a88 100644
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -29,6 +29,14 @@
#include "hw/arm/smmu-common.h"
#include "smmu-internal.h"
+/* Global state for secure address space availability */
+bool arm_secure_as_available;
+
+void smmu_enable_secure_address_space(void)
+{
+ arm_secure_as_available = true;
+}
+
/* IOTLB Management */
static guint smmu_iotlb_key_hash(gconstpointer v)
@@ -341,7 +349,7 @@ static int get_pte(dma_addr_t baseaddr, uint32_t index, uint64_t *pte,
(MemTxAttrs) { .unspecified = true };
/* TODO: guarantee 64-bit single-copy atomicity */
- ret = ldq_le_dma(&address_space_memory, addr, pte, attrs);
+ ret = ldq_le_dma(smmu_get_address_space(is_secure), addr, pte, attrs);
if (ret != MEMTX_OK) {
info->type = SMMU_PTW_ERR_WALK_EABT;
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index bcf06679e1..69b19754f1 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -128,8 +128,8 @@ static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool is_secure)
(MemTxAttrs) { .secure = 1 } :
(MemTxAttrs) { .unspecified = true };
- ret = dma_memory_read(&address_space_memory, addr, cmd, sizeof(Cmd),
- attrs);
+ ret = dma_memory_read(smmu_get_address_space(is_secure), addr, cmd,
+ sizeof(Cmd), attrs);
if (ret != MEMTX_OK) {
return ret;
}
@@ -152,8 +152,8 @@ static MemTxResult queue_write(SMMUQueue *q, Evt *evt_in, bool is_secure)
for (i = 0; i < ARRAY_SIZE(evt.word); i++) {
cpu_to_le32s(&evt.word[i]);
}
- ret = dma_memory_write(&address_space_memory, addr, &evt, sizeof(Evt),
- attrs);
+ ret = dma_memory_write(smmu_get_address_space(is_secure), addr, &evt,
+ sizeof(Evt), attrs);
if (ret != MEMTX_OK) {
return ret;
}
@@ -360,8 +360,8 @@ static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
trace_smmuv3_get_ste(addr);
/* TODO: guarantee 64-bit single-copy atomicity */
- ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
- attrs);
+ ret = dma_memory_read(smmu_get_address_space(attrs.secure), addr, buf,
+ sizeof(*buf), attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Cannot fetch pte at address=0x%"PRIx64"\n", addr);
@@ -409,8 +409,8 @@ static int smmu_get_cd(SMMUv3State *s, STE *ste, SMMUTransCfg *cfg,
}
/* TODO: guarantee 64-bit single-copy atomicity */
- ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
- attrs);
+ ret = dma_memory_read(smmu_get_address_space(cfg->secure), addr, buf,
+ sizeof(*buf), attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Cannot fetch pte at address=0x%"PRIx64"\n", addr);
@@ -740,8 +740,8 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
l2_ste_offset = sid & ((1 << sid_split) - 1);
l1ptr = (dma_addr_t)(strtab_base + l1_ste_offset * sizeof(l1std));
/* TODO: guarantee 64-bit single-copy atomicity */
- ret = dma_memory_read(&address_space_memory, l1ptr, &l1std,
- sizeof(l1std), attrs);
+ ret = dma_memory_read(smmu_get_address_space(is_secure), l1ptr,
+ &l1std, sizeof(l1std), attrs);
if (ret != MEMTX_OK) {
qemu_log_mask(LOG_GUEST_ERROR,
"Could not read L1PTR at 0X%"PRIx64"\n", l1ptr);
@@ -1143,7 +1143,7 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
SMMUTranslationStatus status;
SMMUTransCfg *cfg = NULL;
IOMMUTLBEntry entry = {
- .target_as = &address_space_memory,
+ .target_as = smmu_get_address_space(false),
.iova = addr,
.translated_addr = addr,
.addr_mask = ~(hwaddr)0,
@@ -1295,7 +1295,7 @@ static void smmuv3_notify_iova(IOMMUMemoryRegion *mr,
}
event.type = IOMMU_NOTIFIER_UNMAP;
- event.entry.target_as = &address_space_memory;
+ event.entry.target_as = smmu_get_address_space(is_secure);
event.entry.iova = iova;
event.entry.addr_mask = num_pages * (1 << granule) - 1;
event.entry.perm = IOMMU_NONE;
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index ef6be3660f..bb40f133f2 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -92,6 +92,8 @@
#include "hw/cxl/cxl_host.h"
#include "qemu/guest-random.h"
+AddressSpace arm_secure_address_space;
+
static GlobalProperty arm_virt_compat[] = {
{ TYPE_VIRTIO_IOMMU_PCI, "aw-bits", "48" },
};
@@ -2212,6 +2214,9 @@ static void machvirt_init(MachineState *machine)
memory_region_init(secure_sysmem, OBJECT(machine), "secure-memory",
UINT64_MAX);
memory_region_add_subregion_overlap(secure_sysmem, 0, sysmem, -1);
+ address_space_init(&arm_secure_address_space, secure_sysmem,
+ "secure-memory-space");
+ smmu_enable_secure_address_space();
}
firmware_loaded = virt_firmware_init(vms, sysmem,
diff --git a/include/hw/arm/smmu-common.h b/include/hw/arm/smmu-common.h
index 5d15a1212b..597c5ef6c9 100644
--- a/include/hw/arm/smmu-common.h
+++ b/include/hw/arm/smmu-common.h
@@ -23,6 +23,19 @@
#include "hw/pci/pci.h"
#include "qom/object.h"
+extern AddressSpace __attribute__((weak)) arm_secure_address_space;
+extern bool arm_secure_as_available;
+
+void smmu_enable_secure_address_space(void);
+
+static inline AddressSpace *smmu_get_address_space(bool is_secure)
+{
+ if (is_secure && arm_secure_as_available) {
+ return &arm_secure_address_space;
+ }
+ return &address_space_memory;
+}
+
#define SMMU_PCI_BUS_MAX 256
#define SMMU_PCI_DEVFN_MAX 256
#define SMMU_PCI_DEVFN(sid) (sid & 0xFF)
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 08/11] hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (6 preceding siblings ...)
2025-08-06 15:11 ` [RFC 07/11] hw/arm/smmuv3: Add separate address space for secure SMMU accesses Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 15:11 ` [RFC 09/11] hw/arm/smmuv3: Make the configuration cache security-state aware Tao Tang
` (4 subsequent siblings)
12 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
This change expands the secure command queue's capabilities by adding
two commands for managing secure stage 2 translation caches, as defined
by the Arm SMMUv3 architecture.
The following commands are now processed by the secure command queue:
- CMD_TLBI_S_S2_IPA: Invalidates secure stage 2 TLB entries by IPA for
a given secure VMID.
- CMD_TLBI_S_S12_VMALL: The secure equivalent of VMALLS12E1, this
invalidates all stage 1 and stage 2 entries for a specific secure
VMID.
The command handler verifies that these commands are issued only via the
secure queue and that secure stage 2 functionality is supported by the
model, raising an illegal command error otherwise.
This will be followed by a refactoring of the Configuration and
Translation lookup caches, paving the final path for enabling the
end-to-end processing of secure transactions.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmuv3-internal.h | 4 ++++
hw/arm/smmuv3.c | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
index 852186cea4..82821cbbcc 100644
--- a/hw/arm/smmuv3-internal.h
+++ b/hw/arm/smmuv3-internal.h
@@ -391,6 +391,8 @@ typedef enum SMMUCommandType {
SMMU_CMD_RESUME = 0x44,
SMMU_CMD_STALL_TERM,
SMMU_CMD_SYNC,
+ SMMU_CMD_TLBI_S_S12_VMALL = 0x58,
+ SMMU_CMD_TLBI_S_S2_IPA = 0x5a,
} SMMUCommandType;
static const char *cmd_stringify[] = {
@@ -419,6 +421,8 @@ static const char *cmd_stringify[] = {
[SMMU_CMD_RESUME] = "SMMU_CMD_RESUME",
[SMMU_CMD_STALL_TERM] = "SMMU_CMD_STALL_TERM",
[SMMU_CMD_SYNC] = "SMMU_CMD_SYNC",
+ [SMMU_CMD_TLBI_S_S12_VMALL] = "SMMU_CMD_TLBI_S_S12_VMALL",
+ [SMMU_CMD_TLBI_S_S2_IPA] = "SMMU_CMD_TLBI_S_S2_IPA",
};
static inline const char *smmu_cmd_string(SMMUCommandType type)
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 69b19754f1..5f28e27503 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -1591,6 +1591,26 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
*/
smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2, false);
break;
+ case SMMU_CMD_TLBI_S_S2_IPA:
+ if (!is_secure || !STAGE2_SUPPORTED(s) ||
+ (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
+ cmd_error = SMMU_CERROR_ILL;
+ break;
+ }
+ smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2, true);
+ break;
+ case SMMU_CMD_TLBI_S_S12_VMALL:
+ if (!is_secure || !STAGE2_SUPPORTED(s) ||
+ (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
+ cmd_error = SMMU_CERROR_ILL;
+ break;
+ }
+
+ int vmid = CMD_VMID(&cmd);
+ trace_smmuv3_cmdq_tlbi_s12_vmid(vmid);
+ smmu_inv_notifiers_all(&s->smmu_state);
+ smmu_iotlb_inv_vmid(bs, vmid);
+ break;
case SMMU_CMD_TLBI_EL3_ALL:
case SMMU_CMD_TLBI_EL3_VA:
case SMMU_CMD_TLBI_EL2_ALL:
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 09/11] hw/arm/smmuv3: Make the configuration cache security-state aware
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (7 preceding siblings ...)
2025-08-06 15:11 ` [RFC 08/11] hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 15:11 ` [RFC 10/11] hw/arm/smmuv3: Differentiate secure TLB entries via keying Tao Tang
` (3 subsequent siblings)
12 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
Previously, the configuration cache was keyed only by the SMMU device
(sdev). This is insufficient for a model where a single device can
issue both secure and non-secure transactions, as it could lead to
cache lookups returning the wrong configuration (e.g., a secure
transaction hitting a non-secure cache entry).
This commit refactors the configuration cache management. The cache key
is now a composite of the device and its security state
(SMMUDevice *sdev, bool is_secure).
This ensures that lookups correctly differentiate between security
states for the same physical device, guaranteeing that a transaction
always retrieves the appropriate cached configuration. This is a critical
step towards enabling full parallel processing of secure and non-secure
streams.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmu-common.c | 27 +++++++++++++++++++++++++--
hw/arm/smmuv3.c | 13 ++++++++++---
include/hw/arm/smmu-common.h | 6 ++++++
3 files changed, 41 insertions(+), 5 deletions(-)
diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
index 0877248a88..80f94a85e7 100644
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -37,6 +37,26 @@ void smmu_enable_secure_address_space(void)
arm_secure_as_available = true;
}
+/* Configuration Cache Management */
+static guint smmu_config_key_hash(gconstpointer key)
+{
+ const SMMUConfigKey *k = key;
+ return g_direct_hash(k->sdev) ^ (k->is_secure ? 1 : 0);
+}
+
+static gboolean smmu_config_key_equal(gconstpointer a, gconstpointer b)
+{
+ const SMMUConfigKey *ka = a;
+ const SMMUConfigKey *kb = b;
+ return ka->sdev == kb->sdev && ka->is_secure == kb->is_secure;
+}
+
+SMMUConfigKey smmu_get_config_key(SMMUDevice *sdev, bool is_secure)
+{
+ SMMUConfigKey key = {.sdev = sdev, .is_secure = is_secure};
+ return key;
+}
+
/* IOTLB Management */
static guint smmu_iotlb_key_hash(gconstpointer v)
@@ -236,7 +256,8 @@ static gboolean smmu_hash_remove_by_vmid_ipa(gpointer key, gpointer value,
static gboolean
smmu_hash_remove_by_sid_range(gpointer key, gpointer value, gpointer user_data)
{
- SMMUDevice *sdev = (SMMUDevice *)key;
+ SMMUConfigKey *config_key = (SMMUConfigKey *)key;
+ SMMUDevice *sdev = config_key->sdev;
uint32_t sid = smmu_get_sid(sdev);
SMMUSIDRange *sid_range = (SMMUSIDRange *)user_data;
@@ -943,7 +964,9 @@ static void smmu_base_realize(DeviceState *dev, Error **errp)
error_propagate(errp, local_err);
return;
}
- s->configs = g_hash_table_new_full(NULL, NULL, NULL, g_free);
+ s->configs = g_hash_table_new_full(smmu_config_key_hash,
+ smmu_config_key_equal,
+ g_free, g_free);
s->iotlb = g_hash_table_new_full(smmu_iotlb_key_hash, smmu_iotlb_key_equal,
g_free, g_free);
s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);
diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 5f28e27503..972cbc872f 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -953,8 +953,9 @@ static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event,
SMMUv3State *s = sdev->smmu;
SMMUState *bc = &s->smmu_state;
SMMUTransCfg *cfg;
+ SMMUConfigKey lookup_key = smmu_get_config_key(sdev, is_secure);
- cfg = g_hash_table_lookup(bc->configs, sdev);
+ cfg = g_hash_table_lookup(bc->configs, &lookup_key);
if (cfg) {
sdev->cfg_cache_hits++;
trace_smmuv3_config_cache_hit(smmu_get_sid(sdev),
@@ -971,7 +972,9 @@ static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event,
cfg->secure = is_secure;
if (!smmuv3_decode_config(&sdev->iommu, cfg, event)) {
- g_hash_table_insert(bc->configs, sdev, cfg);
+ SMMUConfigKey *persistent_key = g_new(SMMUConfigKey, 1);
+ *persistent_key = smmu_get_config_key(sdev, is_secure);
+ g_hash_table_insert(bc->configs, persistent_key, cfg);
} else {
g_free(cfg);
cfg = NULL;
@@ -984,9 +987,13 @@ static void smmuv3_flush_config(SMMUDevice *sdev)
{
SMMUv3State *s = sdev->smmu;
SMMUState *bc = &s->smmu_state;
+ SMMUConfigKey key_secure = smmu_get_config_key(sdev, true);
+ SMMUConfigKey key_nonsecure = smmu_get_config_key(sdev, false);
trace_smmu_config_cache_inv(smmu_get_sid(sdev));
- g_hash_table_remove(bc->configs, sdev);
+ /* Remove both secure and non-secure configurations for this device */
+ g_hash_table_remove(bc->configs, &key_secure);
+ g_hash_table_remove(bc->configs, &key_nonsecure);
}
static void smmuv3_invalidate_all_caches(SMMUv3State *s)
diff --git a/include/hw/arm/smmu-common.h b/include/hw/arm/smmu-common.h
index 597c5ef6c9..e07a9c35e7 100644
--- a/include/hw/arm/smmu-common.h
+++ b/include/hw/arm/smmu-common.h
@@ -156,6 +156,11 @@ typedef struct SMMUIOTLBKey {
uint8_t level;
} SMMUIOTLBKey;
+typedef struct SMMUConfigKey {
+ SMMUDevice *sdev;
+ bool is_secure;
+} SMMUConfigKey;
+
typedef struct SMMUSIDRange {
uint32_t start;
uint32_t end;
@@ -230,6 +235,7 @@ SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *entry);
SMMUIOTLBKey smmu_get_iotlb_key(int asid, int vmid, uint64_t iova,
uint8_t tg, uint8_t level);
+SMMUConfigKey smmu_get_config_key(SMMUDevice *sdev, bool is_secure);
void smmu_iotlb_inv_all(SMMUState *s);
void smmu_iotlb_inv_asid_vmid(SMMUState *s, int asid, int vmid);
void smmu_iotlb_inv_vmid(SMMUState *s, int vmid);
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* [RFC 10/11] hw/arm/smmuv3: Differentiate secure TLB entries via keying
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (8 preceding siblings ...)
2025-08-06 15:11 ` [RFC 09/11] hw/arm/smmuv3: Make the configuration cache security-state aware Tao Tang
@ 2025-08-06 15:11 ` Tao Tang
2025-08-06 21:11 ` [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Pierrick Bouvier
` (2 subsequent siblings)
12 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-06 15:11 UTC (permalink / raw)
To: qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi, Tao Tang
To prevent aliasing between secure and non-secure translations for the
same address space, the IOTLB lookup key must incorporate the security
state of the transaction. Without this, a secure lookup could incorrectly
hit a non-secure TLB entry and vice versa.
A secure parameter has been added to smmu_get_iotlb_key. This flag
is now part of the key generation, ensuring that secure and non-secure
TLB entries are treated as distinct entities within the cache.
This change is the final major refactoring required to correctly handle
secure transactions, preventing cache pollution between the two security
worlds.
Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
---
hw/arm/smmu-common.c | 28 ++++++++++++++++++----------
include/hw/arm/smmu-common.h | 3 ++-
2 files changed, 20 insertions(+), 11 deletions(-)
diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
index 80f94a85e7..5abec77a24 100644
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -66,7 +66,7 @@ static guint smmu_iotlb_key_hash(gconstpointer v)
/* Jenkins hash */
a = b = c = JHASH_INITVAL + sizeof(*key);
- a += key->asid + key->vmid + key->level + key->tg;
+ a += key->asid + key->vmid + key->level + key->tg + key->secure;
b += extract64(key->iova, 0, 32);
c += extract64(key->iova, 32, 32);
@@ -82,14 +82,14 @@ static gboolean smmu_iotlb_key_equal(gconstpointer v1, gconstpointer v2)
return (k1->asid == k2->asid) && (k1->iova == k2->iova) &&
(k1->level == k2->level) && (k1->tg == k2->tg) &&
- (k1->vmid == k2->vmid);
+ (k1->vmid == k2->vmid) && (k1->secure == k2->secure);
}
SMMUIOTLBKey smmu_get_iotlb_key(int asid, int vmid, uint64_t iova,
- uint8_t tg, uint8_t level)
+ uint8_t tg, uint8_t level, bool secure)
{
SMMUIOTLBKey key = {.asid = asid, .vmid = vmid, .iova = iova,
- .tg = tg, .level = level};
+ .tg = tg, .level = level, .secure = secure};
return key;
}
@@ -111,7 +111,7 @@ static SMMUTLBEntry *smmu_iotlb_lookup_all_levels(SMMUState *bs,
SMMUIOTLBKey key;
key = smmu_get_iotlb_key(cfg->asid, cfg->s2cfg.vmid,
- iova & ~mask, tg, level);
+ iova & ~mask, tg, level, cfg->secure);
entry = g_hash_table_lookup(bs->iotlb, &key);
if (entry) {
break;
@@ -175,7 +175,7 @@ void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *new)
}
*key = smmu_get_iotlb_key(cfg->asid, cfg->s2cfg.vmid, new->entry.iova,
- tg, new->level);
+ tg, new->level, cfg->secure);
trace_smmu_iotlb_insert(cfg->asid, cfg->s2cfg.vmid, new->entry.iova,
tg, new->level);
g_hash_table_insert(bs->iotlb, key, new);
@@ -282,9 +282,13 @@ void smmu_iotlb_inv_iova(SMMUState *s, int asid, int vmid, dma_addr_t iova,
uint8_t granule = tg ? tg * 2 + 10 : 12;
if (ttl && (num_pages == 1) && (asid >= 0)) {
- SMMUIOTLBKey key = smmu_get_iotlb_key(asid, vmid, iova, tg, ttl);
+ SMMUIOTLBKey key_nonsecure = smmu_get_iotlb_key(asid, vmid, iova,
+ tg, ttl, false);
+ SMMUIOTLBKey key_secure = smmu_get_iotlb_key(asid, vmid, iova,
+ tg, ttl, true);
- if (g_hash_table_remove(s->iotlb, &key)) {
+ if (g_hash_table_remove(s->iotlb, &key_nonsecure) ||
+ g_hash_table_remove(s->iotlb, &key_secure)) {
return;
}
/*
@@ -314,9 +318,13 @@ void smmu_iotlb_inv_ipa(SMMUState *s, int vmid, dma_addr_t ipa, uint8_t tg,
int asid = -1;
if (ttl && (num_pages == 1)) {
- SMMUIOTLBKey key = smmu_get_iotlb_key(asid, vmid, ipa, tg, ttl);
+ SMMUIOTLBKey key_nonsecure = smmu_get_iotlb_key(asid, vmid, ipa,
+ tg, ttl, false);
+ SMMUIOTLBKey key_secure = smmu_get_iotlb_key(asid, vmid, ipa,
+ tg, ttl, true);
- if (g_hash_table_remove(s->iotlb, &key)) {
+ if (g_hash_table_remove(s->iotlb, &key_nonsecure) ||
+ g_hash_table_remove(s->iotlb, &key_secure)) {
return;
}
}
diff --git a/include/hw/arm/smmu-common.h b/include/hw/arm/smmu-common.h
index e07a9c35e7..968c2eecbe 100644
--- a/include/hw/arm/smmu-common.h
+++ b/include/hw/arm/smmu-common.h
@@ -154,6 +154,7 @@ typedef struct SMMUIOTLBKey {
int vmid;
uint8_t tg;
uint8_t level;
+ bool secure;
} SMMUIOTLBKey;
typedef struct SMMUConfigKey {
@@ -234,7 +235,7 @@ SMMUTLBEntry *smmu_iotlb_lookup(SMMUState *bs, SMMUTransCfg *cfg,
SMMUTransTableInfo *tt, hwaddr iova);
void smmu_iotlb_insert(SMMUState *bs, SMMUTransCfg *cfg, SMMUTLBEntry *entry);
SMMUIOTLBKey smmu_get_iotlb_key(int asid, int vmid, uint64_t iova,
- uint8_t tg, uint8_t level);
+ uint8_t tg, uint8_t level, bool secure);
SMMUConfigKey smmu_get_config_key(SMMUDevice *sdev, bool is_secure);
void smmu_iotlb_inv_all(SMMUState *s);
void smmu_iotlb_inv_asid_vmid(SMMUState *s, int asid, int vmid);
--
2.34.1
^ permalink raw reply related [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (9 preceding siblings ...)
2025-08-06 15:11 ` [RFC 10/11] hw/arm/smmuv3: Differentiate secure TLB entries via keying Tao Tang
@ 2025-08-06 21:11 ` Pierrick Bouvier
2025-08-06 21:28 ` Pierrick Bouvier
2025-08-18 21:52 ` Mostafa Saleh
12 siblings, 0 replies; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-06 21:11 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, Mostafa Saleh,
Jean-Philippe Brucker
Hi Tao,
sharing this with Jean-Philippe and Mostafa, who might be interested to
review.
On 8/6/25 8:11 AM, Tao Tang wrote:
> Hi all,
>
> This patch series introduces initial support for emulating the Arm SMMUv3
> Secure State.
>
> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
> emulation is a notable missing piece in QEMU. While the FVP model has
> some support, its limited PCIe capabilities make it challenging for
> complex use cases. The ability to properly manage device DMA from a
> secure context is a critical prerequisite for enabling device assignment
> (passthrough) for confidential computing solutions like Arm CCA and
> related research such as virtCCA [2]. This series aims to build that
> foundational support in QEMU.
>
> This work is being proposed as an RFC. It introduces a significant amount
> of new logic, including the core concept of modeling parallel secure and
> non-secure contexts within a single SMMUv3 device. I am seeking feedback
> on the overall approach, the core refactoring, and the implementation
> details before proceeding further.
>
> The series begins by implementing the components of the secure programming
> interface, then progressively refactors the core SMMU logic to handle
> secure and non-secure contexts in parallel.
>
> Secure Interface Implementation: The initial patches add the
> secure-side registers, implement their read/write logic, and enable
> the secure command and event queues. This includes the S_INIT
> mechanism and the new secure TLB invalidation commands.
>
> Core Logic Refactoring: The next set of patches makes the core SMMU
> functions security-state aware. This involves plumbing an is_secure
> context flag through the main code paths and adding logic to route
> SMMU-originated memory accesses to the correct (Secure or Non-secure)
> address space.
>
> Cache Isolation: With the core logic now aware of security states,
> the following patches refactor the configuration and translation
> lookup caches. The cache keys are modified to include the security
> context, ensuring that secure and non-secure entries for the same
> device or address are properly isolated and preventing aliasing.
>
> Framework Integration: The final patch connects the SMMU's internal
> security context to the generic QEMU IOMMU framework by using the
> iommu_index to represent the architectural SEC_SID.
>
> To validate this work, I performed the following tests:
>
> Non-Secure Regression: To ensure that existing functionality remains
> intact, I ran a nested virtualization test. A TCG guest was created on
> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned.
> Command line of TCG VM is below:
>
> qemu-system-aarch64 \
> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
> -cpu max -smp 1 -m 4080M \
> -accel tcg,thread=single,tb-size=512 \
> -kernel Image \
> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \
> -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
> -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
> -device virtio-blk-device,drive=hd0 \
> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
> -device virtio-net-device,netdev=eth0 \
> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
> -device nvme,drive=nvme0,serial=deadbeef \
> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic
>
> Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
> re-assigned to it via VFIO.
> Command line of KVM VM inside TCG VM is below:
>
> sudo qemu-system-aarch64 \
> -enable-kvm -m 1024 -cpu host -M virt \
> -machine virt,gic-version=3 \
> -cpu max -append "nokaslr" -smp 1 \
> -monitor stdio \
> -kernel 5.15.Image \
> -initrd rootfs.cpio.gz \
> -display vnc=:22,id=primary \
> -device vfio-pci,host=00:01.0
>
> The KVM guest was able to perform I/O on the device
> correctly, confirming that the non-secure path is not broken.
>
> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
> environment. Hafnium's smmuv3_driver_init function was used to test
> the secure register I/O and command queue functionality (excluding
> translation). As Hafnium assumes larger queue and StreamID sizes than
> are practical without TTST support, I temporarily patched Hafnium to
> use smaller values, allowing its driver to initialize the emulated
> secure SMMU successfully.
>
> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
> support QEMU, and no secure device assignment feature exists yet, I
> created a custom platform device to test the secure translation flow.
> To trigger the translation logic, I initiated MMIO writes to this
> device from within Hafnium. The device's MMIO callback handler then
> performed DMA accesses via its IOMMU region, exercising the secure
> translation path. While SMMUv3 is typically used for PCIe on
> physical SoCs, the architecture allows its use with platform devices
> via a stream-id binding in the device tree. The test harness
> required some non-standard modifications to decouple the SMMU from
> its tight integration with PCIe. The code for this test device is
> available for review at [3]. README.md with detailed instructions is
> also provided.
>
> I've attempted to follow all of the guidance in the "Submitting a Patch"
> guide, but as this is my first series of this scale, I apologize if I
> missed anything and welcome all feedback.
>
> Thanks,
> Tang
>
> [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
> [2] https://arxiv.org/abs/2306.11011
> [3] https://github.com/hnusdr/qemu
>
> Tao Tang (11):
> hw/arm/smmuv3: Introduce secure registers and commands
> hw/arm/smmuv3: Implement read/write logic for secure registers
> hw/arm/smmuv3: Implement S_INIT for secure initialization
> hw/arm/smmuv3: Enable command processing for the Secure state
> hw/arm/smmuv3: Support secure event queue and error handling
> hw/arm/smmuv3: Plumb security state through core functions
> hw/arm/smmuv3: Add separate address space for secure SMMU accesses
> hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
> hw/arm/smmuv3: Make the configuration cache security-state aware
> hw/arm/smmuv3: Differentiate secure TLB entries via keying
> hw/arm/smmuv3: Use iommu_index to represent SEC_SID
>
> hw/arm/smmu-common.c | 74 ++-
> hw/arm/smmuv3-internal.h | 128 +++++-
> hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++-----
> hw/arm/trace-events | 7 +-
> hw/arm/virt.c | 5 +
> include/hw/arm/smmu-common.h | 23 +-
> include/hw/arm/smmuv3.h | 27 ++
> 7 files changed, 968 insertions(+), 140 deletions(-)
>
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (10 preceding siblings ...)
2025-08-06 21:11 ` [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Pierrick Bouvier
@ 2025-08-06 21:28 ` Pierrick Bouvier
2025-08-10 16:11 ` Tao Tang
2025-08-18 21:52 ` Mostafa Saleh
12 siblings, 1 reply; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-06 21:28 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
On 8/6/25 8:11 AM, Tao Tang wrote:
> Hi all,
>
> This patch series introduces initial support for emulating the Arm SMMUv3
> Secure State.
>
> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
> emulation is a notable missing piece in QEMU. While the FVP model has
> some support, its limited PCIe capabilities make it challenging for
> complex use cases. The ability to properly manage device DMA from a
> secure context is a critical prerequisite for enabling device assignment
> (passthrough) for confidential computing solutions like Arm CCA and
> related research such as virtCCA [2]. This series aims to build that
> foundational support in QEMU.
>
Thanks for posting this series, it's definitely an important piece
missing for emulating newer SMMU versions.
> This work is being proposed as an RFC. It introduces a significant amount
> of new logic, including the core concept of modeling parallel secure and
> non-secure contexts within a single SMMUv3 device. I am seeking feedback
> on the overall approach, the core refactoring, and the implementation
> details before proceeding further.
>
> The series begins by implementing the components of the secure programming
> interface, then progressively refactors the core SMMU logic to handle
> secure and non-secure contexts in parallel.
>
> Secure Interface Implementation: The initial patches add the
> secure-side registers, implement their read/write logic, and enable
> the secure command and event queues. This includes the S_INIT
> mechanism and the new secure TLB invalidation commands.
>
> Core Logic Refactoring: The next set of patches makes the core SMMU
> functions security-state aware. This involves plumbing an is_secure
> context flag through the main code paths and adding logic to route
> SMMU-originated memory accesses to the correct (Secure or Non-secure)
> address space.
>
> Cache Isolation: With the core logic now aware of security states,
> the following patches refactor the configuration and translation
> lookup caches. The cache keys are modified to include the security
> context, ensuring that secure and non-secure entries for the same
> device or address are properly isolated and preventing aliasing.
>
> Framework Integration: The final patch connects the SMMU's internal
> security context to the generic QEMU IOMMU framework by using the
> iommu_index to represent the architectural SEC_SID.
>
> To validate this work, I performed the following tests:
>
> Non-Secure Regression: To ensure that existing functionality remains
> intact, I ran a nested virtualization test. A TCG guest was created on
> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned.
> Command line of TCG VM is below:
>
> qemu-system-aarch64 \
> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
> -cpu max -smp 1 -m 4080M \
> -accel tcg,thread=single,tb-size=512 \
> -kernel Image \
> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \
> -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
> -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
> -device virtio-blk-device,drive=hd0 \
> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
> -device virtio-net-device,netdev=eth0 \
> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
> -device nvme,drive=nvme0,serial=deadbeef \
> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic
>
> Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
> re-assigned to it via VFIO.
> Command line of KVM VM inside TCG VM is below:
>
> sudo qemu-system-aarch64 \
> -enable-kvm -m 1024 -cpu host -M virt \
> -machine virt,gic-version=3 \
> -cpu max -append "nokaslr" -smp 1 \
> -monitor stdio \
> -kernel 5.15.Image \
> -initrd rootfs.cpio.gz \
> -display vnc=:22,id=primary \
> -device vfio-pci,host=00:01.0
>
> The KVM guest was able to perform I/O on the device
> correctly, confirming that the non-secure path is not broken.
>
> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
> environment. Hafnium's smmuv3_driver_init function was used to test
> the secure register I/O and command queue functionality (excluding
> translation). As Hafnium assumes larger queue and StreamID sizes than
> are practical without TTST support, I temporarily patched Hafnium to
> use smaller values, allowing its driver to initialize the emulated
> secure SMMU successfully.
>
Would that be possible to share your changes, and build instructions for
this? While working on SMMU emulation, we finally left this on the side
due to lack of a software stack being able to use secure SMMU, as we
were not aware that Hafnium + op-tee could make use of it.
> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
> support QEMU, and no secure device assignment feature exists yet, I
> created a custom platform device to test the secure translation flow.
> To trigger the translation logic, I initiated MMIO writes to this
> device from within Hafnium. The device's MMIO callback handler then
> performed DMA accesses via its IOMMU region, exercising the secure
> translation path. While SMMUv3 is typically used for PCIe on
> physical SoCs, the architecture allows its use with platform devices
> via a stream-id binding in the device tree. The test harness
> required some non-standard modifications to decouple the SMMU from
> its tight integration with PCIe. The code for this test device is
> available for review at [3]. README.md with detailed instructions is
> also provided.
>
I am not sure about the current policy in QEMU for test oriented
devices, but it would be really useful to have something similar
upstream (Note: it's out of the scope of this series).
One challenge working with SMMU emulation is that reproducing setups and
triggering specific code paths is hard to achieve, due to the indirect
use of SMMU feature (through DMA) and the complex software stack usually
involved.
Having something upstream available to work on SMMU emulation, at least
on device side, would be a great addition.
Eric, Peter, is this something that would be acceptable to merge?
> I've attempted to follow all of the guidance in the "Submitting a Patch"
> guide, but as this is my first series of this scale, I apologize if I
> missed anything and welcome all feedback.
>
> Thanks,
> Tang
>
> [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
> [2] https://arxiv.org/abs/2306.11011
I was not aware of it, thanks for sharing this excellent paper.
> [3] https://github.com/hnusdr/qemu
>
> Tao Tang (11):
> hw/arm/smmuv3: Introduce secure registers and commands
> hw/arm/smmuv3: Implement read/write logic for secure registers
> hw/arm/smmuv3: Implement S_INIT for secure initialization
> hw/arm/smmuv3: Enable command processing for the Secure state
> hw/arm/smmuv3: Support secure event queue and error handling
> hw/arm/smmuv3: Plumb security state through core functions
> hw/arm/smmuv3: Add separate address space for secure SMMU accesses
> hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
> hw/arm/smmuv3: Make the configuration cache security-state aware
> hw/arm/smmuv3: Differentiate secure TLB entries via keying
> hw/arm/smmuv3: Use iommu_index to represent SEC_SID
>
> hw/arm/smmu-common.c | 74 ++-
> hw/arm/smmuv3-internal.h | 128 +++++-
> hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++-----
> hw/arm/trace-events | 7 +-
> hw/arm/virt.c | 5 +
> include/hw/arm/smmu-common.h | 23 +-
> include/hw/arm/smmuv3.h | 27 ++
> 7 files changed, 968 insertions(+), 140 deletions(-)
>
> --
> 2.34.1
>
>
Regards,
Pierrick
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-06 15:11 ` [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers Tao Tang
@ 2025-08-06 21:53 ` Pierrick Bouvier
2025-08-10 16:54 ` Tao Tang
2025-08-18 21:24 ` Mostafa Saleh
1 sibling, 1 reply; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-06 21:53 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
On 8/6/25 8:11 AM, Tao Tang wrote:
> This patch builds upon the previous introduction of secure register
> definitions by providing the functional implementation for their access.
>
> The availability of the secure programming interface is now correctly
> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
> secure functionality is enabled, the I/O handlers (smmuv3_read and
> smmuv3_write) will correctly dispatch accesses to the secure
> register space.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 5 +
> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 456 insertions(+)
>
> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> index 483aaa915e..1a8b1cb204 100644
> --- a/hw/arm/smmuv3-internal.h
> +++ b/hw/arm/smmuv3-internal.h
> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>
> #define SMMU_CR0_RESERVED 0xFFFFFC20
>
> +/*
> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
> + */
> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
> +
> REG32(CR0ACK, 0x24)
> REG32(CR1, 0x28)
> REG32(CR2, 0x2c)
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index ab67972353..619180d204 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
> s->gerrorn = 0;
> s->statusr = 0;
> s->gbpa = SMMU_GBPA_RESET_VAL;
> +
> + /* Initialize secure state */
> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
> + /* Secure EL2 and Secure stage 2 support */
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
> + /* Secure state implemented */
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> + SECURE_IMPL, 1);
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> + S_SIDSIZE, SMMU_IDR1_SIDSIZE);
> +
> + s->secure_gbpa = SMMU_GBPA_RESET_VAL;
> }
>
Should we wait for the end of the series to enable this support, as not
everything is implemented yet?
> static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
> @@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> }
> }
>
> +/* Check if the SMMU hardware itself implements secure state features */
> +static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
> +{
> + return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
> +}
> +
> static int smmuv3_cmdq_consume(SMMUv3State *s)
> {
> SMMUState *bs = ARM_SMMU(s);
> @@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> return 0;
> }
>
> +/* Helper function for secure register write validation */
> +static bool smmu_validate_secure_write(MemTxAttrs attrs, bool secure_impl,
> + hwaddr offset, const char *reg_name)
> +{
> + if (!attrs.secure || !secure_impl) {
> + const char *reason = !attrs.secure ?
> + "Non-secure write attempt" :
> + "SMMU didn't implement Security State";
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
> + __func__, reason, offset, reg_name);
> + return false;
> + }
> + return true;
> +}
> +
> +/* Helper function for secure register read validation */
> +static bool smmu_validate_secure_read(MemTxAttrs attrs, bool secure_impl,
> + hwaddr offset, const char *reg_name,
> + uint64_t *data)
> +{
> + if (!attrs.secure || !secure_impl) {
> + const char *reason = !attrs.secure ?
> + "Non-secure read attempt" :
> + "SMMU didn't implement Security State";
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
> + __func__, reason, offset, reg_name);
> + *data = 0; /* RAZ */
> + return false;
> + }
> + return true;
> +}
> +
> +/* Macro for secure write validation - returns early if validation fails */
> +#define SMMU_CHECK_SECURE_WRITE(reg_name) \
> + do { \
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
> + reg_name)) { \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
> +/* Macro for attrs.secure only validation */
> +#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
> + do { \
> + if (!attrs.secure) { \
> + qemu_log_mask(LOG_GUEST_ERROR, \
> + "%s: Non-secure write attempt at offset " \
> + "0x%" PRIx64 " (%s, WI)\n", \
> + __func__, offset, reg_name); \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
> +/* Macro for secure read validation - returns RAZ if validation fails */
> +#define SMMU_CHECK_SECURE_READ(reg_name) \
> + do { \
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
> + reg_name, data)) { \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
For this, and previous macros, shouldn't we return MEMTX_ERROR instead?
> +/* Macro for attrs.secure only validation (read) */
> +#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
> + do { \
> + if (!attrs.secure) { \
> + qemu_log_mask(LOG_GUEST_ERROR, \
> + "%s: Non-secure read attempt at offset " \
> + "0x%" PRIx64 " (%s, RAZ)\n", \
> + __func__, offset, reg_name); \
> + *data = 0; \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
A simple style detail, but it's more readable to keep all backslashes on
same column (after the longest line).
#define SMMU_CHECK_SECURE_READ(reg_name) \
do { \
if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
reg_name, data)) { \
return MEMTX_OK; \
} \
} while (0)
> static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> uint64_t data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_GERROR_IRQ_CFG0:
> s->gerror_irq_cfg0 = data;
> @@ -1535,6 +1635,41 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_IRQ_CFG0:
> s->eventq_irq_cfg0 = data;
> return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + /* No need to check secure_impl here */
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = data;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_STRTAB_BASE")) {
> + return MEMTX_OK;
> + }
> + s->secure_strtab_base = data;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = data;
> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
> + s->secure_cmdq.log2size = SMMU_CMDQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = data;
> + s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
> + s->secure_eventq.log2size = SMMU_EVENTQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_EVENTQ_IRQ_CFG0")) {
> + return MEMTX_OK;
> + }
> + s->secure_eventq_irq_cfg0 = data;
> + return MEMTX_OK;
> default:
> qemu_log_mask(LOG_UNIMP,
> "%s Unexpected 64-bit access to 0x%"PRIx64" (WI)\n",
> @@ -1546,6 +1681,11 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> uint64_t data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_CR0:
> s->cr[0] = data;
> @@ -1650,6 +1790,137 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_IRQ_CFG2:
> s->eventq_irq_cfg2 = data;
> return MEMTX_OK;
> + case A_S_CR0:
> + SMMU_CHECK_SECURE_WRITE("S_CR0");
> + s->secure_cr[0] = data;
> + /* clear reserved bits */
> + s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_CR1:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_CR1")) {
> + return MEMTX_OK;
> + }
> + s->secure_cr[1] = data;
> + return MEMTX_OK;
> + case A_S_CR2:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_CR2")) {
> + return MEMTX_OK;
> + }
> + s->secure_cr[2] = data;
> + return MEMTX_OK;
> + case A_S_IRQ_CTRL:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_IRQ_CTRL")) {
> + return MEMTX_OK;
> + }
> + s->secure_irq_ctrl = data;
> + return MEMTX_OK;
> + case A_S_GERRORN:
> + SMMU_CHECK_SECURE_WRITE("S_GERRORN");
> + smmuv3_write_gerrorn(s, data);
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = data;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0 + 4:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = deposit64(s->secure_gerror_irq_cfg0,
> + 32, 32, data);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG1:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG1");
> + s->secure_gerror_irq_cfg1 = data;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG2:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG2");
> + s->secure_gerror_irq_cfg2 = data;
> + return MEMTX_OK;
> + case A_S_GBPA:
> + SMMU_CHECK_SECURE_WRITE("S_GBPA");
> + if (data & R_S_GBPA_UPDATE_MASK) {
> + s->secure_gbpa = data & ~R_S_GBPA_UPDATE_MASK;
> + }
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 0, 32, data);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE_CFG:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE_CFG");
> + s->secure_strtab_base_cfg = data;
> + if (FIELD_EX32(data, S_STRTAB_BASE_CFG, FMT) == 1) {
> + s->secure_sid_split = FIELD_EX32(data, S_STRTAB_BASE_CFG, SPLIT);
> + s->secure_features |= SMMU_FEATURE_2LVL_STE;
> + }
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 0, 32, data);
> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
> + s->secure_cmdq.log2size = SMMU_CMDQS;
> + }
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_CMDQ_PROD:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
> + s->secure_cmdq.prod = data;
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_CMDQ_CONS:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
> + s->secure_cmdq.cons = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 0, 32, data);
> + s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
> + s->secure_eventq.log2size = SMMU_EVENTQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_PROD:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_PROD");
> + s->secure_eventq.prod = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_CONS:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_CONS");
> + s->secure_eventq.cons = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_IRQ_CFG0");
> + s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
> + 0, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0 + 4:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG0");
> + s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
> + 32, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG1:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG1");
> + s->secure_eventq_irq_cfg1 = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG2:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
> + s->secure_eventq_irq_cfg2 = data;
> + return MEMTX_OK;
> default:
> qemu_log_mask(LOG_UNIMP,
> "%s Unexpected 32-bit access to 0x%"PRIx64" (WI)\n",
> @@ -1687,6 +1958,11 @@ static MemTxResult smmu_write_mmio(void *opaque, hwaddr offset, uint64_t data,
> static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> uint64_t *data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_GERROR_IRQ_CFG0:
> *data = s->gerror_irq_cfg0;
> @@ -1700,6 +1976,31 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_BASE:
> *data = s->eventq.base;
> return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG0", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg0;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
> + *data = s->secure_strtab_base;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_BASE", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.base;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_BASE", data)) {
> + return MEMTX_OK;
> + }
Why we don't reuse SMMU_CHECK_SECURE_READ for this and all other similar
if below?
> + *data = s->secure_eventq.base;
> + return MEMTX_OK;
> default:
> *data = 0;
> qemu_log_mask(LOG_UNIMP,
> @@ -1712,6 +2013,11 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
> uint64_t *data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_IDREGS ... A_IDREGS + 0x2f:
> *data = smmuv3_idreg(offset - A_IDREGS);
> @@ -1798,6 +2104,151 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_CONS:
> *data = s->eventq.cons;
> return MEMTX_OK;
> + case A_S_IDR0 ... A_S_IDR4:
> + int idr_idx = (offset - A_S_IDR0) / 4;
> + g_assert(idr_idx >= 0 && idr_idx <= 4);
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + g_strdup_printf("S_IDR%d", idr_idx),
> + data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_idr[idr_idx];
> + return MEMTX_OK;
> + case A_S_CR0:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR0", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[0];
> + return MEMTX_OK;
> + case A_S_CR0ACK:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR0ACK", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr0ack;
> + return MEMTX_OK;
> + case A_S_CR1:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR1", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[1];
> + return MEMTX_OK;
> + case A_S_CR2:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR2", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[2];
> + return MEMTX_OK;
> + case A_S_GBPA:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GBPA", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gbpa;
> + return MEMTX_OK;
> + case A_S_IRQ_CTRL:
> + case A_S_IRQ_CTRLACK:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_IRQ_CTRL", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_irq_ctrl;
> + return MEMTX_OK;
> + case A_S_GERROR:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror;
> + return MEMTX_OK;
> + case A_S_GERRORN:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERRORN", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerrorn;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0");
> + *data = extract64(s->secure_gerror_irq_cfg0, 0, 32);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0 + 4:
> + SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0+4");
> + *data = extract64(s->secure_gerror_irq_cfg0, 32, 32);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG1:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG1", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg1;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG2:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG2", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg2;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
> + *data = extract64(s->secure_strtab_base, 0, 32);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE + 4:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE+4");
> + *data = extract64(s->secure_strtab_base, 32, 32);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE_CFG:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE_CFG");
> + *data = s->secure_strtab_base_cfg;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
> + *data = extract64(s->secure_cmdq.base, 0, 32);
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE + 4:
> + SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
> + *data = extract64(s->secure_cmdq.base, 32, 32);
> + return MEMTX_OK;
> + case A_S_CMDQ_PROD:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_PROD", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.prod;
> + return MEMTX_OK;
> + case A_S_CMDQ_CONS:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_CONS", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.cons;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
> + *data = extract64(s->secure_eventq.base, 0, 32);
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE + 4:
> + SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
> + *data = extract64(s->secure_eventq.base, 32, 32);
> + return MEMTX_OK;
> + case A_S_EVENTQ_PROD:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_PROD", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_eventq.prod;
> + return MEMTX_OK;
> + case A_S_EVENTQ_CONS:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_CONS", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_eventq.cons;
> + return MEMTX_OK;
> default:
> *data = 0;
> qemu_log_mask(LOG_UNIMP,
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-06 15:11 ` [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state Tao Tang
@ 2025-08-06 21:55 ` Pierrick Bouvier
2025-08-10 16:59 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-06 21:55 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
On 8/6/25 8:11 AM, Tao Tang wrote:
> This patch enables the secure command queue, providing a dedicated
> interface for secure software to issue commands to the SMMU. Based on
> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
> entries directly from the Secure PA space so that we need to pass the
> memory transaction attributes when reading the command queue.
>
> This provides a parallel command mechanism to the non-secure world.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 8 ++++--
> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
> hw/arm/trace-events | 2 +-
> 3 files changed, 41 insertions(+), 24 deletions(-)
>
> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> index 1a8b1cb204..5b2ca00832 100644
> --- a/hw/arm/smmuv3-internal.h
> +++ b/hw/arm/smmuv3-internal.h
> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
> }
>
> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
> {
> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
> + if (is_secure) {
> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
> + } else {
> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
> + }
> }
>
> static inline bool smmuv3_eventq_enabled(SMMUv3State *s)
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index 0ea9d897af..0590f0f482 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -105,14 +105,17 @@ static void smmuv3_write_gerrorn(SMMUv3State *s, uint32_t new_gerrorn)
> trace_smmuv3_write_gerrorn(toggled & pending, s->gerrorn);
> }
>
> -static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd)
> +static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool is_secure)
> {
> dma_addr_t addr = Q_CONS_ENTRY(q);
> MemTxResult ret;
> int i;
> + MemTxAttrs attrs = is_secure ?
> + (MemTxAttrs) { .secure = 1 } :
> + (MemTxAttrs) { .unspecified = true };
>
> ret = dma_memory_read(&address_space_memory, addr, cmd, sizeof(Cmd),
> - MEMTXATTRS_UNSPECIFIED);
> + attrs);
> if (ret != MEMTX_OK) {
> return ret;
> }
> @@ -1311,14 +1314,14 @@ static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
> return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
> }
>
> -static int smmuv3_cmdq_consume(SMMUv3State *s)
> +static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
> {
> SMMUState *bs = ARM_SMMU(s);
> SMMUCmdError cmd_error = SMMU_CERROR_NONE;
> - SMMUQueue *q = &s->cmdq;
> + SMMUQueue *q = is_secure ? &s->secure_cmdq : &s->cmdq;
> SMMUCommandType type = 0;
>
> - if (!smmuv3_cmdq_enabled(s)) {
> + if (!smmuv3_cmdq_enabled(s, is_secure)) {
> return 0;
> }
> /*
> @@ -1329,17 +1332,20 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> */
>
> while (!smmuv3_q_empty(q)) {
> - uint32_t pending = s->gerror ^ s->gerrorn;
> + uint32_t pending = is_secure ? s->secure_gerror ^ s->secure_gerrorn :
> + s->gerror ^ s->gerrorn;
> Cmd cmd;
>
> trace_smmuv3_cmdq_consume(Q_PROD(q), Q_CONS(q),
> - Q_PROD_WRAP(q), Q_CONS_WRAP(q));
> + Q_PROD_WRAP(q), Q_CONS_WRAP(q),
> + is_secure);
>
> - if (FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
> + if (is_secure ? FIELD_EX32(pending, S_GERROR, CMDQ_ERR) :
> + FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
> break;
> }
>
> - if (queue_read(q, &cmd) != MEMTX_OK) {
> + if (queue_read(q, &cmd, is_secure) != MEMTX_OK) {
> cmd_error = SMMU_CERROR_ABT;
> break;
> }
> @@ -1364,8 +1370,11 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> SMMUDevice *sdev = smmu_find_sdev(bs, sid);
>
> if (CMD_SSEC(&cmd)) {
> - cmd_error = SMMU_CERROR_ILL;
> - break;
> + if (!is_secure) {
> + /* Secure Stream with NON-Secure command */
> + cmd_error = SMMU_CERROR_ILL;
> + break;
> + }
> }
>
> if (!sdev) {
> @@ -1384,8 +1393,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> SMMUSIDRange sid_range;
>
> if (CMD_SSEC(&cmd)) {
> - cmd_error = SMMU_CERROR_ILL;
> - break;
> + if (!is_secure) {
> + cmd_error = SMMU_CERROR_ILL;
> + break;
> + }
> }
>
> mask = (1ULL << (range + 1)) - 1;
> @@ -1403,8 +1414,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> SMMUDevice *sdev = smmu_find_sdev(bs, sid);
>
> if (CMD_SSEC(&cmd)) {
> - cmd_error = SMMU_CERROR_ILL;
> - break;
> + if (!is_secure) {
> + cmd_error = SMMU_CERROR_ILL;
> + break;
> + }
> }
>
> if (!sdev) {
> @@ -1706,7 +1719,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> s->cr[0] = data;
> s->cr0ack = data & ~SMMU_CR0_RESERVED;
> /* in case the command queue has been enabled */
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, false);
> return MEMTX_OK;
> case A_CR1:
> s->cr[1] = data;
> @@ -1723,7 +1736,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> * By acknowledging the CMDQ_ERR, SW may notify cmds can
> * be processed again
> */
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, false);
> return MEMTX_OK;
> case A_GERROR_IRQ_CFG0: /* 64b */
> s->gerror_irq_cfg0 = deposit64(s->gerror_irq_cfg0, 0, 32, data);
> @@ -1772,7 +1785,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> return MEMTX_OK;
> case A_CMDQ_PROD:
> s->cmdq.prod = data;
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, false);
> return MEMTX_OK;
> case A_CMDQ_CONS:
> s->cmdq.cons = data;
> @@ -1810,7 +1823,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> s->secure_cr[0] = data;
> /* clear reserved bits */
> s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, true);
> return MEMTX_OK;
> case A_S_CR1:
> if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> @@ -1836,7 +1849,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> case A_S_GERRORN:
> SMMU_CHECK_SECURE_WRITE("S_GERRORN");
> smmuv3_write_gerrorn(s, data);
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, true);
> return MEMTX_OK;
> case A_S_GERROR_IRQ_CFG0:
> SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> @@ -1892,7 +1905,7 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> case A_S_CMDQ_PROD:
> SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
> s->secure_cmdq.prod = data;
> - smmuv3_cmdq_consume(s);
> + smmuv3_cmdq_consume(s, true);
> return MEMTX_OK;
> case A_S_CMDQ_CONS:
> SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
> diff --git a/hw/arm/trace-events b/hw/arm/trace-events
> index 019129ea43..7d967226ff 100644
> --- a/hw/arm/trace-events
> +++ b/hw/arm/trace-events
> @@ -35,7 +35,7 @@ smmuv3_trigger_irq(int irq) "irq=%d"
> smmuv3_write_gerror(uint32_t toggled, uint32_t gerror) "toggled=0x%x, new GERROR=0x%x"
> smmuv3_write_gerrorn(uint32_t acked, uint32_t gerrorn) "acked=0x%x, new GERRORN=0x%x"
> smmuv3_unhandled_cmd(uint32_t type) "Unhandled command type=%d"
> -smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod=%d cons=%d prod.wrap=%d cons.wrap=%d"
> +smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap, bool is_secure_cmdq) "prod=%d cons=%d prod.wrap=%d cons.wrap=%d is_secure_cmdq=%d"
> smmuv3_cmdq_opcode(const char *opcode) "<--- %s"
> smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t cons_wrap) "prod:%d, cons:%d, prod_wrap:%d, cons_wrap:%d "
> smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
This looks like a reasonable and readable approach to support secure and
non secure accesses.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-06 21:28 ` Pierrick Bouvier
@ 2025-08-10 16:11 ` Tao Tang
2025-08-11 10:26 ` Philippe Mathieu-Daudé
2025-08-12 18:04 ` Pierrick Bouvier
0 siblings, 2 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-10 16:11 UTC (permalink / raw)
To: Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 2025/8/7 05:28, Pierrick Bouvier wrote:
> On 8/6/25 8:11 AM, Tao Tang wrote:
>> Hi all,
>>
>> This patch series introduces initial support for emulating the Arm
>> SMMUv3
>> Secure State.
>>
>> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
>> emulation is a notable missing piece in QEMU. While the FVP model has
>> some support, its limited PCIe capabilities make it challenging for
>> complex use cases. The ability to properly manage device DMA from a
>> secure context is a critical prerequisite for enabling device assignment
>> (passthrough) for confidential computing solutions like Arm CCA and
>> related research such as virtCCA [2]. This series aims to build that
>> foundational support in QEMU.
>>
>
> Thanks for posting this series, it's definitely an important piece
> missing for emulating newer SMMU versions.
>
>> This work is being proposed as an RFC. It introduces a significant
>> amount
>> of new logic, including the core concept of modeling parallel secure and
>> non-secure contexts within a single SMMUv3 device. I am seeking feedback
>> on the overall approach, the core refactoring, and the implementation
>> details before proceeding further.
>>
>> The series begins by implementing the components of the secure
>> programming
>> interface, then progressively refactors the core SMMU logic to handle
>> secure and non-secure contexts in parallel.
>>
>> Secure Interface Implementation: The initial patches add the
>> secure-side registers, implement their read/write logic, and enable
>> the secure command and event queues. This includes the S_INIT
>> mechanism and the new secure TLB invalidation commands.
>>
>> Core Logic Refactoring: The next set of patches makes the core SMMU
>> functions security-state aware. This involves plumbing an is_secure
>> context flag through the main code paths and adding logic to route
>> SMMU-originated memory accesses to the correct (Secure or Non-secure)
>> address space.
>>
>> Cache Isolation: With the core logic now aware of security states,
>> the following patches refactor the configuration and translation
>> lookup caches. The cache keys are modified to include the security
>> context, ensuring that secure and non-secure entries for the same
>> device or address are properly isolated and preventing aliasing.
>>
>> Framework Integration: The final patch connects the SMMU's internal
>> security context to the generic QEMU IOMMU framework by using the
>> iommu_index to represent the architectural SEC_SID.
>>
>> To validate this work, I performed the following tests:
>>
>> Non-Secure Regression: To ensure that existing functionality remains
>> intact, I ran a nested virtualization test. A TCG guest was created on
>> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device
>> assigned.
>> Command line of TCG VM is below:
>>
>> qemu-system-aarch64 \
>> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
>> -cpu max -smp 1 -m 4080M \
>> -accel tcg,thread=single,tb-size=512 \
>> -kernel Image \
>> -append 'nokaslr root=/dev/vda rw rootfstype=ext4
>> iommu.passthrough=on' \
>> -device
>> pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
>> -device
>> pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
>> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
>> -device virtio-blk-device,drive=hd0 \
>> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
>> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
>> -device virtio-net-device,netdev=eth0 \
>> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
>> -device nvme,drive=nvme0,serial=deadbeef \
>> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log
>> -nographic
>>
>> Inside this TCG VM, a KVM guest was launched, and the same NVMe
>> device was
>> re-assigned to it via VFIO.
>> Command line of KVM VM inside TCG VM is below:
>>
>> sudo qemu-system-aarch64 \
>> -enable-kvm -m 1024 -cpu host -M virt \
>> -machine virt,gic-version=3 \
>> -cpu max -append "nokaslr" -smp 1 \
>> -monitor stdio \
>> -kernel 5.15.Image \
>> -initrd rootfs.cpio.gz \
>> -display vnc=:22,id=primary \
>> -device vfio-pci,host=00:01.0
>>
>> The KVM guest was able to perform I/O on the device
>> correctly, confirming that the non-secure path is not broken.
>>
>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
>> environment. Hafnium's smmuv3_driver_init function was used to test
>> the secure register I/O and command queue functionality (excluding
>> translation). As Hafnium assumes larger queue and StreamID sizes than
>> are practical without TTST support, I temporarily patched Hafnium to
>> use smaller values, allowing its driver to initialize the emulated
>> secure SMMU successfully.
>>
>
> Would that be possible to share your changes, and build instructions
> for this? While working on SMMU emulation, we finally left this on the
> side due to lack of a software stack being able to use secure SMMU, as
> we were not aware that Hafnium + op-tee could make use of it.
>
Hi Pierrick,
Thanks for your interest! I'm very happy to share my work on this. I've
documented the setup process, including our code modifications and the
step-by-step build instructions in this link:
https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG
The core point of these changes is to enable the SMMUv3 feature in
Hafnium. This leads to numerous read/write operations on SMMUv3 secure
registers and various queue manipulations within the smmuv3_driver_init
function in Hafnium.
However, it's important to note that this initialization process itself
does not initiate any DMA memory access that would trigger the
smmuv3_translate flow.
Even so, we've devised a method to test the subsequent Secure
Translation Path by leveraging the smmuv3-test platform device. This
approach allows us to verify the entire SMMUv3 flow, from initialization
to translation.
>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>> support QEMU, and no secure device assignment feature exists yet, I
>> created a custom platform device to test the secure translation flow.
>> To trigger the translation logic, I initiated MMIO writes to this
>> device from within Hafnium. The device's MMIO callback handler then
>> performed DMA accesses via its IOMMU region, exercising the secure
>> translation path. While SMMUv3 is typically used for PCIe on
>> physical SoCs, the architecture allows its use with platform devices
>> via a stream-id binding in the device tree. The test harness
>> required some non-standard modifications to decouple the SMMU from
>> its tight integration with PCIe. The code for this test device is
>> available for review at [3]. README.md with detailed instructions is
>> also provided.
>>
>
> I am not sure about the current policy in QEMU for test oriented
> devices, but it would be really useful to have something similar
> upstream (Note: it's out of the scope of this series).
> One challenge working with SMMU emulation is that reproducing setups
> and triggering specific code paths is hard to achieve, due to the
> indirect use of SMMU feature (through DMA) and the complex software
> stack usually involved.
> Having something upstream available to work on SMMU emulation, at
> least on device side, would be a great addition.
>
> Eric, Peter, is this something that would be acceptable to merge?
>
Looking ahead, my plan is to refactor the smmuv3-test platform device.
The goal is to make it self-contained within QEMU, removing the current
dependency on Hafnium to trigger its operations. I plan to submit this
as a separate RFC patch series in the next few days.
However, the current version at [3] is already sufficient to test the
code logic introduced in this RFC.
>> I've attempted to follow all of the guidance in the "Submitting a Patch"
>> guide, but as this is my first series of this scale, I apologize if I
>> missed anything and welcome all feedback.
>>
>> Thanks,
>> Tang
>>
>> [1]
>> https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
>> [2] https://arxiv.org/abs/2306.11011
>
> I was not aware of it, thanks for sharing this excellent paper.
>
>> [3] https://github.com/hnusdr/qemu
>>
>> Tao Tang (11):
>> hw/arm/smmuv3: Introduce secure registers and commands
>> hw/arm/smmuv3: Implement read/write logic for secure registers
>> hw/arm/smmuv3: Implement S_INIT for secure initialization
>> hw/arm/smmuv3: Enable command processing for the Secure state
>> hw/arm/smmuv3: Support secure event queue and error handling
>> hw/arm/smmuv3: Plumb security state through core functions
>> hw/arm/smmuv3: Add separate address space for secure SMMU accesses
>> hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
>> hw/arm/smmuv3: Make the configuration cache security-state aware
>> hw/arm/smmuv3: Differentiate secure TLB entries via keying
>> hw/arm/smmuv3: Use iommu_index to represent SEC_SID
>>
>> hw/arm/smmu-common.c | 74 ++-
>> hw/arm/smmuv3-internal.h | 128 +++++-
>> hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++-----
>> hw/arm/trace-events | 7 +-
>> hw/arm/virt.c | 5 +
>> include/hw/arm/smmu-common.h | 23 +-
>> include/hw/arm/smmuv3.h | 27 ++
>> 7 files changed, 968 insertions(+), 140 deletions(-)
>>
>> --
>> 2.34.1
>>
>>
>
> Regards,
> Pierrick
--
Best,
Tang
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-06 21:53 ` Pierrick Bouvier
@ 2025-08-10 16:54 ` Tao Tang
2025-08-12 17:12 ` Pierrick Bouvier
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-10 16:54 UTC (permalink / raw)
To: Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 2025/8/7 05:53, Pierrick Bouvier wrote:
> On 8/6/25 8:11 AM, Tao Tang wrote:
>> This patch builds upon the previous introduction of secure register
>> definitions by providing the functional implementation for their access.
>>
>> The availability of the secure programming interface is now correctly
>> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
>> secure functionality is enabled, the I/O handlers (smmuv3_read and
>> smmuv3_write) will correctly dispatch accesses to the secure
>> register space.
>>
>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>> ---
>> hw/arm/smmuv3-internal.h | 5 +
>> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 456 insertions(+)
>>
>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>> index 483aaa915e..1a8b1cb204 100644
>> --- a/hw/arm/smmuv3-internal.h
>> +++ b/hw/arm/smmuv3-internal.h
>> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>> #define SMMU_CR0_RESERVED 0xFFFFFC20
>> +/*
>> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
>> + */
>> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
>> +
>> REG32(CR0ACK, 0x24)
>> REG32(CR1, 0x28)
>> REG32(CR2, 0x2c)
>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>> index ab67972353..619180d204 100644
>> --- a/hw/arm/smmuv3.c
>> +++ b/hw/arm/smmuv3.c
>> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
>> s->gerrorn = 0;
>> s->statusr = 0;
>> s->gbpa = SMMU_GBPA_RESET_VAL;
>> +
>> + /* Initialize secure state */
>> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
>> + /* Secure EL2 and Secure stage 2 support */
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
>> + /* Secure state implemented */
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
>> + SECURE_IMPL, 1);
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
>> + S_SIDSIZE, SMMU_IDR1_SIDSIZE);
>> +
>> + s->secure_gbpa = SMMU_GBPA_RESET_VAL;
>> }
>>
>
> Should we wait for the end of the series to enable this support, as
> not everything is implemented yet?
>
Hi Pierrick,
Thansk for your suggestion! I will move all of this code into the final
patch. Then I will check if there are other codes with similar
situations that also need to be moved into the final patch.
>> static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
>> @@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s,
>> Cmd *cmd, SMMUStage stage)
>> }
>> }
>> +/* Check if the SMMU hardware itself implements secure state
>> features */
>> +static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
>> +{
>> + return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
>> +}
>> +
>> static int smmuv3_cmdq_consume(SMMUv3State *s)
>> {
>> SMMUState *bs = ARM_SMMU(s);
>> @@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> return 0;
>> }
>> +/* Helper function for secure register write validation */
>> +static bool smmu_validate_secure_write(MemTxAttrs attrs, bool
>> secure_impl,
>> + hwaddr offset, const char
>> *reg_name)
>> +{
>> + if (!attrs.secure || !secure_impl) {
>> + const char *reason = !attrs.secure ?
>> + "Non-secure write attempt" :
>> + "SMMU didn't implement Security State";
>> + qemu_log_mask(LOG_GUEST_ERROR,
>> + "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
>> + __func__, reason, offset, reg_name);
>> + return false;
>> + }
>> + return true;
>> +}
>> +
>> +/* Helper function for secure register read validation */
>> +static bool smmu_validate_secure_read(MemTxAttrs attrs, bool
>> secure_impl,
>> + hwaddr offset, const char
>> *reg_name,
>> + uint64_t *data)
>> +{
>> + if (!attrs.secure || !secure_impl) {
>> + const char *reason = !attrs.secure ?
>> + "Non-secure read attempt" :
>> + "SMMU didn't implement Security State";
>> + qemu_log_mask(LOG_GUEST_ERROR,
>> + "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
>> + __func__, reason, offset, reg_name);
>> + *data = 0; /* RAZ */
>> + return false;
>> + }
>> + return true;
>> +}
>> +
>> +/* Macro for secure write validation - returns early if validation
>> fails */
>> +#define SMMU_CHECK_SECURE_WRITE(reg_name) \
>> + do { \
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
>> + reg_name)) { \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>> +/* Macro for attrs.secure only validation */
>> +#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
>> + do { \
>> + if (!attrs.secure) { \
>> + qemu_log_mask(LOG_GUEST_ERROR, \
>> + "%s: Non-secure write attempt at offset " \
>> + "0x%" PRIx64 " (%s, WI)\n", \
>> + __func__, offset, reg_name); \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>> +/* Macro for secure read validation - returns RAZ if validation
>> fails */
>> +#define SMMU_CHECK_SECURE_READ(reg_name) \
>> + do { \
>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
>> + reg_name, data)) { \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>
> For this, and previous macros, shouldn't we return MEMTX_ERROR instead?
>
According to ARM IHI 0070 G.b page 87/:
/
/When SMMU_S_IDR1.SECURE_IMPL == 0, /
/ - The SMMU does not support the Secure state/
/ - SMMU_S_* registers are RAZ/WI to all accesses./
RAZ/WI is indeed a deterministic hardware behavior, and I think it is
architecturally distinct from conditions like a Terminate or a Fault.
While the software might not get a "real" value from a register (it gets
zeros) or its write might have no effect, the hardware access itself
completes without any protocol-level error.
This is my current understanding, but I'm keen to hear if anyone has a
different perspective or sees it differently.
>> +/* Macro for attrs.secure only validation (read) */
>> +#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
>> + do { \
>> + if (!attrs.secure) { \
>> + qemu_log_mask(LOG_GUEST_ERROR, \
>> + "%s: Non-secure read attempt at offset " \
>> + "0x%" PRIx64 " (%s, RAZ)\n", \
>> + __func__, offset, reg_name); \
>> + *data = 0; \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>
> A simple style detail, but it's more readable to keep all backslashes
> on same column (after the longest line).
>
> #define SMMU_CHECK_SECURE_READ(reg_name) \
> do { \
> if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
> reg_name, data)) { \
> return MEMTX_OK; \
> } \
> } while (0)
>
That's a good point, it's much more readable that way. Thanks for the
suggestion, I'll check all the code and fix it in the next version.
>> static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
>> uint64_t data, MemTxAttrs attrs)
>> {
>> + bool secure_impl = false;
>> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
>> + secure_impl = smmu_hw_secure_implemented(s);
>> + }
>> +
>> switch (offset) {
>> case A_GERROR_IRQ_CFG0:
>> s->gerror_irq_cfg0 = data;
>> @@ -1535,6 +1635,41 @@ static MemTxResult smmu_writell(SMMUv3State
>> *s, hwaddr offset,
>> case A_EVENTQ_IRQ_CFG0:
>> s->eventq_irq_cfg0 = data;
>> return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG0:
>> + /* No need to check secure_impl here */
>> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
>> + s->secure_gerror_irq_cfg0 = data;
>> + return MEMTX_OK;
>> + case A_S_STRTAB_BASE:
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> + "S_STRTAB_BASE")) {
>> + return MEMTX_OK;
>> + }
>> + s->secure_strtab_base = data;
>> + return MEMTX_OK;
>> + case A_S_CMDQ_BASE:
>> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
>> + s->secure_cmdq.base = data;
>> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
>> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
>> + s->secure_cmdq.log2size = SMMU_CMDQS;
>> + }
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_BASE:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
>> + s->secure_eventq.base = data;
>> + s->secure_eventq.log2size = extract64(s->secure_eventq.base,
>> 0, 5);
>> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
>> + s->secure_eventq.log2size = SMMU_EVENTQS;
>> + }
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_IRQ_CFG0:
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> + "S_EVENTQ_IRQ_CFG0")) {
>> + return MEMTX_OK;
>> + }
>> + s->secure_eventq_irq_cfg0 = data;
>> + return MEMTX_OK;
>> default:
>> qemu_log_mask(LOG_UNIMP,
>> "%s Unexpected 64-bit access to 0x%"PRIx64"
>> (WI)\n",
>> @@ -1546,6 +1681,11 @@ static MemTxResult smmu_writell(SMMUv3State
>> *s, hwaddr offset,
>> static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
>> uint64_t data, MemTxAttrs attrs)
>> {
>> + bool secure_impl = false;
>> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
>> + secure_impl = smmu_hw_secure_implemented(s);
>> + }
>> +
>> switch (offset) {
>> case A_CR0:
>> s->cr[0] = data;
>> @@ -1650,6 +1790,137 @@ static MemTxResult smmu_writel(SMMUv3State
>> *s, hwaddr offset,
>> case A_EVENTQ_IRQ_CFG2:
>> s->eventq_irq_cfg2 = data;
>> return MEMTX_OK;
>> + case A_S_CR0:
>> + SMMU_CHECK_SECURE_WRITE("S_CR0");
>> + s->secure_cr[0] = data;
>> + /* clear reserved bits */
>> + s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
>> + smmuv3_cmdq_consume(s);
>> + return MEMTX_OK;
>> + case A_S_CR1:
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> + "S_CR1")) {
>> + return MEMTX_OK;
>> + }
>> + s->secure_cr[1] = data;
>> + return MEMTX_OK;
>> + case A_S_CR2:
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> + "S_CR2")) {
>> + return MEMTX_OK;
>> + }
>> + s->secure_cr[2] = data;
>> + return MEMTX_OK;
>> + case A_S_IRQ_CTRL:
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> + "S_IRQ_CTRL")) {
>> + return MEMTX_OK;
>> + }
>> + s->secure_irq_ctrl = data;
>> + return MEMTX_OK;
>> + case A_S_GERRORN:
>> + SMMU_CHECK_SECURE_WRITE("S_GERRORN");
>> + smmuv3_write_gerrorn(s, data);
>> + smmuv3_cmdq_consume(s);
>> + return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG0:
>> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
>> + s->secure_gerror_irq_cfg0 = data;
>> + return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG0 + 4:
>> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
>> + s->secure_gerror_irq_cfg0 =
>> deposit64(s->secure_gerror_irq_cfg0,
>> + 32, 32, data);
>> + return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG1:
>> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG1");
>> + s->secure_gerror_irq_cfg1 = data;
>> + return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG2:
>> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG2");
>> + s->secure_gerror_irq_cfg2 = data;
>> + return MEMTX_OK;
>> + case A_S_GBPA:
>> + SMMU_CHECK_SECURE_WRITE("S_GBPA");
>> + if (data & R_S_GBPA_UPDATE_MASK) {
>> + s->secure_gbpa = data & ~R_S_GBPA_UPDATE_MASK;
>> + }
>> + return MEMTX_OK;
>> + case A_S_STRTAB_BASE:
>> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
>> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 0,
>> 32, data);
>> + return MEMTX_OK;
>> + case A_S_STRTAB_BASE + 4:
>> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
>> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 32,
>> 32, data);
>> + return MEMTX_OK;
>> + case A_S_STRTAB_BASE_CFG:
>> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE_CFG");
>> + s->secure_strtab_base_cfg = data;
>> + if (FIELD_EX32(data, S_STRTAB_BASE_CFG, FMT) == 1) {
>> + s->secure_sid_split = FIELD_EX32(data,
>> S_STRTAB_BASE_CFG, SPLIT);
>> + s->secure_features |= SMMU_FEATURE_2LVL_STE;
>> + }
>> + return MEMTX_OK;
>> + case A_S_CMDQ_BASE:
>> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
>> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 0, 32,
>> data);
>> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
>> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
>> + s->secure_cmdq.log2size = SMMU_CMDQS;
>> + }
>> + return MEMTX_OK;
>> + case A_S_CMDQ_BASE + 4:
>> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
>> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 32, 32,
>> data);
>> + return MEMTX_OK;
>> + case A_S_CMDQ_PROD:
>> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
>> + s->secure_cmdq.prod = data;
>> + smmuv3_cmdq_consume(s);
>> + return MEMTX_OK;
>> + case A_S_CMDQ_CONS:
>> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
>> + s->secure_cmdq.cons = data;
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_BASE:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
>> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 0,
>> 32, data);
>> + s->secure_eventq.log2size = extract64(s->secure_eventq.base,
>> 0, 5);
>> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
>> + s->secure_eventq.log2size = SMMU_EVENTQS;
>> + }
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_BASE + 4:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
>> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 32,
>> 32, data);
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_PROD:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_PROD");
>> + s->secure_eventq.prod = data;
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_CONS:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_CONS");
>> + s->secure_eventq.cons = data;
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_IRQ_CFG0:
>> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_IRQ_CFG0");
>> + s->secure_eventq_irq_cfg0 =
>> deposit64(s->secure_eventq_irq_cfg0,
>> + 0, 32, data);
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_IRQ_CFG0 + 4:
>> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG0");
>> + s->secure_eventq_irq_cfg0 =
>> deposit64(s->secure_eventq_irq_cfg0,
>> + 32, 32, data);
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_IRQ_CFG1:
>> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG1");
>> + s->secure_eventq_irq_cfg1 = data;
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_IRQ_CFG2:
>> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
>> + s->secure_eventq_irq_cfg2 = data;
>> + return MEMTX_OK;
>> default:
>> qemu_log_mask(LOG_UNIMP,
>> "%s Unexpected 32-bit access to 0x%"PRIx64"
>> (WI)\n",
>> @@ -1687,6 +1958,11 @@ static MemTxResult smmu_write_mmio(void
>> *opaque, hwaddr offset, uint64_t data,
>> static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
>> uint64_t *data, MemTxAttrs attrs)
>> {
>> + bool secure_impl = false;
>> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
>> + secure_impl = smmu_hw_secure_implemented(s);
>> + }
>> +
>> switch (offset) {
>> case A_GERROR_IRQ_CFG0:
>> *data = s->gerror_irq_cfg0;
>> @@ -1700,6 +1976,31 @@ static MemTxResult smmu_readll(SMMUv3State *s,
>> hwaddr offset,
>> case A_EVENTQ_BASE:
>> *data = s->eventq.base;
>> return MEMTX_OK;
>> + case A_S_GERROR_IRQ_CFG0:
>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
>> + "S_GERROR_IRQ_CFG0", data)) {
>> + return MEMTX_OK;
>> + }
>> + *data = s->secure_gerror_irq_cfg0;
>> + return MEMTX_OK;
>> + case A_S_STRTAB_BASE:
>> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
>> + *data = s->secure_strtab_base;
>> + return MEMTX_OK;
>> + case A_S_CMDQ_BASE:
>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
>> + "S_CMDQ_BASE", data)) {
>> + return MEMTX_OK;
>> + }
>> + *data = s->secure_cmdq.base;
>> + return MEMTX_OK;
>> + case A_S_EVENTQ_BASE:
>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
>> + "S_EVENTQ_BASE", data)) {
>> + return MEMTX_OK;
>> + }
>
> Why we don't reuse SMMU_CHECK_SECURE_READ for this and all other
> similar if below?
My apologies, I seem to have missed these when doing the macro
replacement. I will be sure to fix this in the next version. Thank you
for your feedback.
Thanks,
Tang
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-06 21:55 ` Pierrick Bouvier
@ 2025-08-10 16:59 ` Tao Tang
2025-08-11 10:34 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-10 16:59 UTC (permalink / raw)
To: Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 2025/8/7 05:55, Pierrick Bouvier wrote:
> On 8/6/25 8:11 AM, Tao Tang wrote:
>> This patch enables the secure command queue, providing a dedicated
>> interface for secure software to issue commands to the SMMU. Based on
>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
>> entries directly from the Secure PA space so that we need to pass the
>> memory transaction attributes when reading the command queue.
>>
>> This provides a parallel command mechanism to the non-secure world.
>>
>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>> ---
>> hw/arm/smmuv3-internal.h | 8 ++++--
>> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
>> hw/arm/trace-events | 2 +-
>> 3 files changed, 41 insertions(+), 24 deletions(-)
>>
>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>> index 1a8b1cb204..5b2ca00832 100644
>> --- a/hw/arm/smmuv3-internal.h
>> +++ b/hw/arm/smmuv3-internal.h
>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
>> }
>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
>> {
>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>> + if (is_secure) {
>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
>> + } else {
>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>> + }
>> }
>> static inline bool smmuv3_eventq_enabled(SMMUv3State *s)
>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>> index 0ea9d897af..0590f0f482 100644
>> --- a/hw/arm/smmuv3.c
>> +++ b/hw/arm/smmuv3.c
>> @@ -105,14 +105,17 @@ static void smmuv3_write_gerrorn(SMMUv3State
>> *s, uint32_t new_gerrorn)
>> trace_smmuv3_write_gerrorn(toggled & pending, s->gerrorn);
>> }
>> -static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd)
>> +static inline MemTxResult queue_read(SMMUQueue *q, Cmd *cmd, bool
>> is_secure)
>> {
>> dma_addr_t addr = Q_CONS_ENTRY(q);
>> MemTxResult ret;
>> int i;
>> + MemTxAttrs attrs = is_secure ?
>> + (MemTxAttrs) { .secure = 1 } :
>> + (MemTxAttrs) { .unspecified = true };
>> ret = dma_memory_read(&address_space_memory, addr, cmd,
>> sizeof(Cmd),
>> - MEMTXATTRS_UNSPECIFIED);
>> + attrs);
>> if (ret != MEMTX_OK) {
>> return ret;
>> }
>> @@ -1311,14 +1314,14 @@ static inline bool
>> smmu_hw_secure_implemented(SMMUv3State *s)
>> return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
>> }
>> -static int smmuv3_cmdq_consume(SMMUv3State *s)
>> +static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
>> {
>> SMMUState *bs = ARM_SMMU(s);
>> SMMUCmdError cmd_error = SMMU_CERROR_NONE;
>> - SMMUQueue *q = &s->cmdq;
>> + SMMUQueue *q = is_secure ? &s->secure_cmdq : &s->cmdq;
>> SMMUCommandType type = 0;
>> - if (!smmuv3_cmdq_enabled(s)) {
>> + if (!smmuv3_cmdq_enabled(s, is_secure)) {
>> return 0;
>> }
>> /*
>> @@ -1329,17 +1332,20 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> */
>> while (!smmuv3_q_empty(q)) {
>> - uint32_t pending = s->gerror ^ s->gerrorn;
>> + uint32_t pending = is_secure ? s->secure_gerror ^
>> s->secure_gerrorn :
>> + s->gerror ^ s->gerrorn;
>> Cmd cmd;
>> trace_smmuv3_cmdq_consume(Q_PROD(q), Q_CONS(q),
>> - Q_PROD_WRAP(q), Q_CONS_WRAP(q));
>> + Q_PROD_WRAP(q), Q_CONS_WRAP(q),
>> + is_secure);
>> - if (FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
>> + if (is_secure ? FIELD_EX32(pending, S_GERROR, CMDQ_ERR) :
>> + FIELD_EX32(pending, GERROR, CMDQ_ERR)) {
>> break;
>> }
>> - if (queue_read(q, &cmd) != MEMTX_OK) {
>> + if (queue_read(q, &cmd, is_secure) != MEMTX_OK) {
>> cmd_error = SMMU_CERROR_ABT;
>> break;
>> }
>> @@ -1364,8 +1370,11 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> SMMUDevice *sdev = smmu_find_sdev(bs, sid);
>> if (CMD_SSEC(&cmd)) {
>> - cmd_error = SMMU_CERROR_ILL;
>> - break;
>> + if (!is_secure) {
>> + /* Secure Stream with NON-Secure command */
>> + cmd_error = SMMU_CERROR_ILL;
>> + break;
>> + }
>> }
>> if (!sdev) {
>> @@ -1384,8 +1393,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> SMMUSIDRange sid_range;
>> if (CMD_SSEC(&cmd)) {
>> - cmd_error = SMMU_CERROR_ILL;
>> - break;
>> + if (!is_secure) {
>> + cmd_error = SMMU_CERROR_ILL;
>> + break;
>> + }
>> }
>> mask = (1ULL << (range + 1)) - 1;
>> @@ -1403,8 +1414,10 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> SMMUDevice *sdev = smmu_find_sdev(bs, sid);
>> if (CMD_SSEC(&cmd)) {
>> - cmd_error = SMMU_CERROR_ILL;
>> - break;
>> + if (!is_secure) {
>> + cmd_error = SMMU_CERROR_ILL;
>> + break;
>> + }
>> }
>> if (!sdev) {
>> @@ -1706,7 +1719,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> s->cr[0] = data;
>> s->cr0ack = data & ~SMMU_CR0_RESERVED;
>> /* in case the command queue has been enabled */
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, false);
>> return MEMTX_OK;
>> case A_CR1:
>> s->cr[1] = data;
>> @@ -1723,7 +1736,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> * By acknowledging the CMDQ_ERR, SW may notify cmds can
>> * be processed again
>> */
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, false);
>> return MEMTX_OK;
>> case A_GERROR_IRQ_CFG0: /* 64b */
>> s->gerror_irq_cfg0 = deposit64(s->gerror_irq_cfg0, 0, 32,
>> data);
>> @@ -1772,7 +1785,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> return MEMTX_OK;
>> case A_CMDQ_PROD:
>> s->cmdq.prod = data;
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, false);
>> return MEMTX_OK;
>> case A_CMDQ_CONS:
>> s->cmdq.cons = data;
>> @@ -1810,7 +1823,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> s->secure_cr[0] = data;
>> /* clear reserved bits */
>> s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, true);
>> return MEMTX_OK;
>> case A_S_CR1:
>> if (!smmu_validate_secure_write(attrs, secure_impl, offset,
>> @@ -1836,7 +1849,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> case A_S_GERRORN:
>> SMMU_CHECK_SECURE_WRITE("S_GERRORN");
>> smmuv3_write_gerrorn(s, data);
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, true);
>> return MEMTX_OK;
>> case A_S_GERROR_IRQ_CFG0:
>> SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
>> @@ -1892,7 +1905,7 @@ static MemTxResult smmu_writel(SMMUv3State *s,
>> hwaddr offset,
>> case A_S_CMDQ_PROD:
>> SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
>> s->secure_cmdq.prod = data;
>> - smmuv3_cmdq_consume(s);
>> + smmuv3_cmdq_consume(s, true);
>> return MEMTX_OK;
>> case A_S_CMDQ_CONS:
>> SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
>> diff --git a/hw/arm/trace-events b/hw/arm/trace-events
>> index 019129ea43..7d967226ff 100644
>> --- a/hw/arm/trace-events
>> +++ b/hw/arm/trace-events
>> @@ -35,7 +35,7 @@ smmuv3_trigger_irq(int irq) "irq=%d"
>> smmuv3_write_gerror(uint32_t toggled, uint32_t gerror)
>> "toggled=0x%x, new GERROR=0x%x"
>> smmuv3_write_gerrorn(uint32_t acked, uint32_t gerrorn) "acked=0x%x,
>> new GERRORN=0x%x"
>> smmuv3_unhandled_cmd(uint32_t type) "Unhandled command type=%d"
>> -smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap,
>> uint8_t cons_wrap) "prod=%d cons=%d prod.wrap=%d cons.wrap=%d"
>> +smmuv3_cmdq_consume(uint32_t prod, uint32_t cons, uint8_t prod_wrap,
>> uint8_t cons_wrap, bool is_secure_cmdq) "prod=%d cons=%d prod.wrap=%d
>> cons.wrap=%d is_secure_cmdq=%d"
>> smmuv3_cmdq_opcode(const char *opcode) "<--- %s"
>> smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t
>> prod_wrap, uint8_t cons_wrap) "prod:%d, cons:%d, prod_wrap:%d,
>> cons_wrap:%d "
>> smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error)
>> "Error on %s command execution: %d"
>
> This looks like a reasonable and readable approach to support secure
> and non secure accesses.
Hi Pierrick,
Thank you so much for taking the time to review and for the very
positive feedback.
I'm very relieved to hear you find the approach "reasonable and
readable". I was hoping that explicitly passing the parameter was the
right way to avoid issues with global state or code duplication, and
your confirmation is the best encouragement I could ask for.
Thanks again!
Best,
Tang
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
@ 2025-08-11 10:22 ` Philippe Mathieu-Daudé
2025-08-11 10:43 ` Philippe Mathieu-Daudé
2025-08-18 21:21 ` Mostafa Saleh
1 sibling, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-11 10:22 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
Hi,
On 6/8/25 17:11, Tao Tang wrote:
> The Arm SMMUv3 architecture defines a set of registers and commands for
> managing secure transactions and context.
>
> This patch introduces the definitions for these secure registers and
> commands within the SMMUv3 device model internal header.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 57 ++++++++++++++++++++++++++++++++++++++++
> include/hw/arm/smmuv3.h | 23 ++++++++++++++++
> 2 files changed, 80 insertions(+)
> diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
> index d183a62766..72ad042514 100644
> --- a/include/hw/arm/smmuv3.h
> +++ b/include/hw/arm/smmuv3.h
> @@ -63,6 +63,29 @@ struct SMMUv3State {
> qemu_irq irq[4];
> QemuMutex mutex;
> char *stage;
> +
> + /* Secure state */
> + uint32_t secure_idr[5];
> + uint32_t secure_cr[3];
> + uint32_t secure_cr0ack;
> + uint32_t secure_init;
> + uint32_t secure_gbpa;
> + uint32_t secure_irq_ctrl;
> + uint32_t secure_gerror;
> + uint32_t secure_gerrorn;
> + uint64_t secure_gerror_irq_cfg0;
> + uint32_t secure_gerror_irq_cfg1;
> + uint32_t secure_gerror_irq_cfg2;
> + uint64_t secure_strtab_base;
> + uint32_t secure_strtab_base_cfg;
> + uint8_t secure_sid_split;
> + uint32_t secure_features;
> +
> + uint64_t secure_eventq_irq_cfg0;
> + uint32_t secure_eventq_irq_cfg1;
> + uint32_t secure_eventq_irq_cfg2;
> +
> + SMMUQueue secure_eventq, secure_cmdq;
Note, we could also add these fields as
struct {
uint32_t idr[5];
...
} secure;
With some IDEs it allows to only expand which set you are
interested in when debugging.
I then since it is mostly the same banked set, I wonder why we
don't extract the state and bank it:
struct {
uint32_t idr[5];
...
} state[REG_NUM_BANKS];
I haven't looked at the rest, but this might simplify the
implementation.
Then maybe we can use the ARMASIdx enum as index.
> };Shouldn't we add a subsection for these new fields in vmstate_smmuv3?
(If using banked state, then this is greatly simplified IMHO).
Regards,
Phil.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-10 16:11 ` Tao Tang
@ 2025-08-11 10:26 ` Philippe Mathieu-Daudé
2025-08-12 17:50 ` Pierrick Bouvier
2025-08-12 18:04 ` Pierrick Bouvier
1 sibling, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-11 10:26 UTC (permalink / raw)
To: Tao Tang, Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 10/8/25 18:11, Tao Tang wrote:
>
> On 2025/8/7 05:28, Pierrick Bouvier wrote:
>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>>> support QEMU, and no secure device assignment feature exists yet, I
>>> created a custom platform device to test the secure translation flow.
>>> To trigger the translation logic, I initiated MMIO writes to this
>>> device from within Hafnium. The device's MMIO callback handler then
>>> performed DMA accesses via its IOMMU region, exercising the secure
>>> translation path. While SMMUv3 is typically used for PCIe on
>>> physical SoCs, the architecture allows its use with platform devices
>>> via a stream-id binding in the device tree. The test harness
>>> required some non-standard modifications to decouple the SMMU from
>>> its tight integration with PCIe. The code for this test device is
>>> available for review at [3]. README.md with detailed instructions is
>>> also provided.
>>>
>>
>> I am not sure about the current policy in QEMU for test oriented
>> devices, but it would be really useful to have something similar
>> upstream (Note: it's out of the scope of this series).
>> One challenge working with SMMU emulation is that reproducing setups
>> and triggering specific code paths is hard to achieve, due to the
>> indirect use of SMMU feature (through DMA) and the complex software
>> stack usually involved.
>> Having something upstream available to work on SMMU emulation, at
>> least on device side, would be a great addition.
>>
>> Eric, Peter, is this something that would be acceptable to merge?
This shouldn't be an issue, we already have some:
$ git ls-files|fgrep testdev
chardev/testdev.c
docs/specs/pci-testdev.rst
hw/hyperv/hyperv_testdev.c
hw/misc/pc-testdev.c
hw/misc/pci-testdev.c
hw/tricore/tricore_testdevice.c
> Looking ahead, my plan is to refactor the smmuv3-test platform device.
> The goal is to make it self-contained within QEMU, removing the current
> dependency on Hafnium to trigger its operations. I plan to submit this
> as a separate RFC patch series in the next few days.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-10 16:59 ` Tao Tang
@ 2025-08-11 10:34 ` Philippe Mathieu-Daudé
2025-08-12 17:27 ` Pierrick Bouvier
2025-08-12 18:42 ` Peter Maydell
0 siblings, 2 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-11 10:34 UTC (permalink / raw)
To: Tao Tang, Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
Hi,
On 10/8/25 18:59, Tao Tang wrote:
> On 2025/8/7 05:55, Pierrick Bouvier wrote:
>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>> This patch enables the secure command queue, providing a dedicated
>>> interface for secure software to issue commands to the SMMU. Based on
>>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
>>> entries directly from the Secure PA space so that we need to pass the
>>> memory transaction attributes when reading the command queue.
>>>
>>> This provides a parallel command mechanism to the non-secure world.
>>>
>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>> ---
>>> hw/arm/smmuv3-internal.h | 8 ++++--
>>> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
>>> hw/arm/trace-events | 2 +-
>>> 3 files changed, 41 insertions(+), 24 deletions(-)
>>>
>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>> index 1a8b1cb204..5b2ca00832 100644
>>> --- a/hw/arm/smmuv3-internal.h
>>> +++ b/hw/arm/smmuv3-internal.h
>>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
>>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
>>> }
>>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
>>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
>>> {
>>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>> + if (is_secure) {
>>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
>>> + } else {
>>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>> + }
>>> }
>> This looks like a reasonable and readable approach to support secure
>> and non secure accesses.
>
> Hi Pierrick,
>
> Thank you so much for taking the time to review and for the very
> positive feedback.
>
> I'm very relieved to hear you find the approach "reasonable and
> readable". I was hoping that explicitly passing the parameter was the
> right way to avoid issues with global state or code duplication, and
> your confirmation is the best encouragement I could ask for.
An alternative (also suggested in patch #1) is to use index for banked
registers.
For example we use M_REG_NS with Cortex-M cores (target/arm/cpu-qom.h):
/* For M profile, some registers are banked secure vs non-secure;
* these are represented as a 2-element array where the first
* element is the non-secure copy and the second is the secure copy.
* When the CPU does not have implement the security extension then
* only the first element is used.
* This means that the copy for the current security state can be
* accessed via env->registerfield[env->v7m.secure] (whether the
* security extension is implemented or not).
*/
enum {
M_REG_NS = 0,
M_REG_S = 1,
M_REG_NUM_BANKS = 2,
};
And generally for address spaces (target/arm/cpu.h):
typedef enum ARMASIdx {
ARMASIdx_NS = 0,
ARMASIdx_S = 1,
ARMASIdx_TagNS = 2,
ARMASIdx_TagS = 3,
} ARMASIdx;
(not sure this one is appropriate here).
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling
2025-08-06 15:11 ` [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling Tao Tang
@ 2025-08-11 10:41 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-11 10:41 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
On 6/8/25 17:11, Tao Tang wrote:
> Following the implementation of the secure command queue, this commit
> introduces the infrastructure for reporting faults and events back to
> secure software.
>
> The secure event queue is now enabled, serving as the primary mechanism
> for the SMMU to report translation faults and other architected events.
>
> For more critical failures, such as an abort on an event queue write,
> the SMMU_S_GERROR registers are also added. Finally, SMMU_S_IRQ_CTRL
> is wired up to control interrupt notifications for both the event
> queue and these global errors.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 38 ++++++++++++++-----
> hw/arm/smmuv3.c | 82 +++++++++++++++++++++++++---------------
> hw/arm/trace-events | 2 +-
> 3 files changed, 81 insertions(+), 41 deletions(-)
> -static inline bool smmuv3_eventq_irq_enabled(SMMUv3State *s)
> +static inline bool smmuv3_eventq_irq_enabled(SMMUv3State *s, bool is_secure)
> {
> - return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, EVENTQ_IRQEN);
> + if (is_secure) {
> + return FIELD_EX32(s->secure_irq_ctrl, S_IRQ_CTRL, EVENTQ_IRQEN);
> + } else {
> + return FIELD_EX32(s->irq_ctrl, IRQ_CTRL, EVENTQ_IRQEN);
> + }
> }
Or using bank as suggested in patch #1:
static inline bool smmuv3_eventq_irq_enabled(SMMUv3State *s,
unsigned idx)
{
return FIELD_EX32(s->bank[idx].irq_ctrl, IRQ_CTRL, EVENTQ_IRQEN);
}
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands
2025-08-11 10:22 ` Philippe Mathieu-Daudé
@ 2025-08-11 10:43 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-11 10:43 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel; +Cc: Eric Auger, Peter Maydell, Chen Baozi
On 11/8/25 12:22, Philippe Mathieu-Daudé wrote:
> Hi,
>
> On 6/8/25 17:11, Tao Tang wrote:
>> The Arm SMMUv3 architecture defines a set of registers and commands for
>> managing secure transactions and context.
>>
>> This patch introduces the definitions for these secure registers and
>> commands within the SMMUv3 device model internal header.
>>
>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>> ---
>> hw/arm/smmuv3-internal.h | 57 ++++++++++++++++++++++++++++++++++++++++
>> include/hw/arm/smmuv3.h | 23 ++++++++++++++++
>> 2 files changed, 80 insertions(+)
>
>
>> diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
>> index d183a62766..72ad042514 100644
>> --- a/include/hw/arm/smmuv3.h
>> +++ b/include/hw/arm/smmuv3.h
>> @@ -63,6 +63,29 @@ struct SMMUv3State {
>> qemu_irq irq[4];
>> QemuMutex mutex;
>> char *stage;
>> +
>> + /* Secure state */
>> + uint32_t secure_idr[5];
>> + uint32_t secure_cr[3];
>> + uint32_t secure_cr0ack;
>> + uint32_t secure_init;
>> + uint32_t secure_gbpa;
>> + uint32_t secure_irq_ctrl;
>> + uint32_t secure_gerror;
>> + uint32_t secure_gerrorn;
>> + uint64_t secure_gerror_irq_cfg0;
>> + uint32_t secure_gerror_irq_cfg1;
>> + uint32_t secure_gerror_irq_cfg2;
>> + uint64_t secure_strtab_base;
>> + uint32_t secure_strtab_base_cfg;
>> + uint8_t secure_sid_split;
>> + uint32_t secure_features;
>> +
>> + uint64_t secure_eventq_irq_cfg0;
>> + uint32_t secure_eventq_irq_cfg1;
>> + uint32_t secure_eventq_irq_cfg2;
>> +
>> + SMMUQueue secure_eventq, secure_cmdq;
>
> Note, we could also add these fields as
>
> struct {
> uint32_t idr[5];
> ...
>
> } secure;
>
> With some IDEs it allows to only expand which set you are
> interested in when debugging.
>
> I then since it is mostly the same banked set, I wonder why we
> don't extract the state and bank it:
>
> struct {
> uint32_t idr[5];
> ...
>
> } state[REG_NUM_BANKS];
>
> I haven't looked at the rest, but this might simplify the
> implementation.
>
> Then maybe we can use the ARMASIdx enum as index.
After looking at the rest of the series, ARMASIdx seems appropriate.
I'd use:
struct {
uint32_t idr[6];
...
} bank[2]; /* ARMASIdx_NS and ARMASIdx_S */
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-10 16:54 ` Tao Tang
@ 2025-08-12 17:12 ` Pierrick Bouvier
0 siblings, 0 replies; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-12 17:12 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 8/10/25 9:54 AM, Tao Tang wrote:
>>> +/* Macro for secure read validation - returns RAZ if validation
>>> fails */
>>> +#define SMMU_CHECK_SECURE_READ(reg_name) \
>>> + do { \
>>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
>>> + reg_name, data)) { \
>>> + return MEMTX_OK; \
>>> + } \
>>> + } while (0)
>>> +
>>
>> For this, and previous macros, shouldn't we return MEMTX_ERROR instead?
>>
>
> According to ARM IHI 0070 G.b page 87/:
> /
>
> /When SMMU_S_IDR1.SECURE_IMPL == 0, /
>
> / - The SMMU does not support the Secure state/
>
> / - SMMU_S_* registers are RAZ/WI to all accesses./
>
> RAZ/WI is indeed a deterministic hardware behavior, and I think it is
> architecturally distinct from conditions like a Terminate or a Fault.
> While the software might not get a "real" value from a register (it gets
> zeros) or its write might have no effect, the hardware access itself
> completes without any protocol-level error.
>
> This is my current understanding, but I'm keen to hear if anyone has a
> different perspective or sees it differently.
>
>
This looks like the expected behaviour indeed.
Thanks for sharing the reference.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-11 10:34 ` Philippe Mathieu-Daudé
@ 2025-08-12 17:27 ` Pierrick Bouvier
2025-08-12 17:39 ` Philippe Mathieu-Daudé
2025-08-12 18:42 ` Peter Maydell
1 sibling, 1 reply; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-12 17:27 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 8/11/25 3:34 AM, Philippe Mathieu-Daudé wrote:
> Hi,
>
> On 10/8/25 18:59, Tao Tang wrote:
>> On 2025/8/7 05:55, Pierrick Bouvier wrote:
>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>>> This patch enables the secure command queue, providing a dedicated
>>>> interface for secure software to issue commands to the SMMU. Based on
>>>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
>>>> entries directly from the Secure PA space so that we need to pass the
>>>> memory transaction attributes when reading the command queue.
>>>>
>>>> This provides a parallel command mechanism to the non-secure world.
>>>>
>>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>>> ---
>>>> hw/arm/smmuv3-internal.h | 8 ++++--
>>>> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
>>>> hw/arm/trace-events | 2 +-
>>>> 3 files changed, 41 insertions(+), 24 deletions(-)
>>>>
>>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>>> index 1a8b1cb204..5b2ca00832 100644
>>>> --- a/hw/arm/smmuv3-internal.h
>>>> +++ b/hw/arm/smmuv3-internal.h
>>>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
>>>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
>>>> }
>>>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
>>>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
>>>> {
>>>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>> + if (is_secure) {
>>>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
>>>> + } else {
>>>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>> + }
>>>> }
>
>
>>> This looks like a reasonable and readable approach to support secure
>>> and non secure accesses.
>>
>> Hi Pierrick,
>>
>> Thank you so much for taking the time to review and for the very
>> positive feedback.
>>
>> I'm very relieved to hear you find the approach "reasonable and
>> readable". I was hoping that explicitly passing the parameter was the
>> right way to avoid issues with global state or code duplication, and
>> your confirmation is the best encouragement I could ask for.
>
> An alternative (also suggested in patch #1) is to use index for banked
> registers.
>
> For example we use M_REG_NS with Cortex-M cores (target/arm/cpu-qom.h):
>
> /* For M profile, some registers are banked secure vs non-secure;
> * these are represented as a 2-element array where the first
> * element is the non-secure copy and the second is the secure copy.
> * When the CPU does not have implement the security extension then
> * only the first element is used.
> * This means that the copy for the current security state can be
> * accessed via env->registerfield[env->v7m.secure] (whether the
> * security extension is implemented or not).
> */
> enum {
> M_REG_NS = 0,
> M_REG_S = 1,
> M_REG_NUM_BANKS = 2,
> };
>
> And generally for address spaces (target/arm/cpu.h):
>
> typedef enum ARMASIdx {
> ARMASIdx_NS = 0,
> ARMASIdx_S = 1,
> ARMASIdx_TagNS = 2,
> ARMASIdx_TagS = 3,
> } ARMASIdx;
>
> (not sure this one is appropriate here).
>
This could be useful, especially to not duplicate register definitions.
However, for now, we only have two indexes (secure vs non-secure) and
it's not clear if we'll need something more than that in a close future.
I think the current approach (a simple bool) is quite readable, and the
boilerplate is limited, especially versus changing all existing code to
support the new Idx approach.
As well, it can always be refactored easily later, when we'll need it.
That said, I will let the maintainers decide which approach they want to
see implemented, to avoid Tao jumping between different people opinions.
Regards,
Pierrick
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-12 17:27 ` Pierrick Bouvier
@ 2025-08-12 17:39 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2025-08-12 17:39 UTC (permalink / raw)
To: Pierrick Bouvier, Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 12/8/25 19:27, Pierrick Bouvier wrote:
> On 8/11/25 3:34 AM, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> On 10/8/25 18:59, Tao Tang wrote:
>>> On 2025/8/7 05:55, Pierrick Bouvier wrote:
>>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>>>> This patch enables the secure command queue, providing a dedicated
>>>>> interface for secure software to issue commands to the SMMU. Based on
>>>>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
>>>>> entries directly from the Secure PA space so that we need to pass the
>>>>> memory transaction attributes when reading the command queue.
>>>>>
>>>>> This provides a parallel command mechanism to the non-secure world.
>>>>>
>>>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>>>> ---
>>>>> hw/arm/smmuv3-internal.h | 8 ++++--
>>>>> hw/arm/smmuv3.c | 55 ++++++++++++++++++++++++
>>>>> +---------------
>>>>> hw/arm/trace-events | 2 +-
>>>>> 3 files changed, 41 insertions(+), 24 deletions(-)
>>>>>
>>>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>>>> index 1a8b1cb204..5b2ca00832 100644
>>>>> --- a/hw/arm/smmuv3-internal.h
>>>>> +++ b/hw/arm/smmuv3-internal.h
>>>>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
>>>>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
>>>>> }
>>>>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
>>>>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool
>>>>> is_secure)
>>>>> {
>>>>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>>> + if (is_secure) {
>>>>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
>>>>> + } else {
>>>>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>>> + }
>>>>> }
>>
>>
>>>> This looks like a reasonable and readable approach to support secure
>>>> and non secure accesses.
>>>
>>> Hi Pierrick,
>>>
>>> Thank you so much for taking the time to review and for the very
>>> positive feedback.
>>>
>>> I'm very relieved to hear you find the approach "reasonable and
>>> readable". I was hoping that explicitly passing the parameter was the
>>> right way to avoid issues with global state or code duplication, and
>>> your confirmation is the best encouragement I could ask for.
>>
>> An alternative (also suggested in patch #1) is to use index for banked
>> registers.
>>
>> For example we use M_REG_NS with Cortex-M cores (target/arm/cpu-qom.h):
>>
>> /* For M profile, some registers are banked secure vs non-secure;
>> * these are represented as a 2-element array where the first
>> * element is the non-secure copy and the second is the secure
>> copy.
>> * When the CPU does not have implement the security extension then
>> * only the first element is used.
>> * This means that the copy for the current security state can be
>> * accessed via env->registerfield[env->v7m.secure] (whether the
>> * security extension is implemented or not).
>> */
>> enum {
>> M_REG_NS = 0,
>> M_REG_S = 1,
>> M_REG_NUM_BANKS = 2,
>> };
>>
>> And generally for address spaces (target/arm/cpu.h):
>>
>> typedef enum ARMASIdx {
>> ARMASIdx_NS = 0,
>> ARMASIdx_S = 1,
>> ARMASIdx_TagNS = 2,
>> ARMASIdx_TagS = 3,
>> } ARMASIdx;
>>
>> (not sure this one is appropriate here).
>>
>
> This could be useful, especially to not duplicate register definitions.
>
> However, for now, we only have two indexes (secure vs non-secure) and
> it's not clear if we'll need something more than that in a close future.
> I think the current approach (a simple bool) is quite readable, and the
> boilerplate is limited, especially versus changing all existing code to
> support the new Idx approach.
> As well, it can always be refactored easily later, when we'll need it.
>
> That said, I will let the maintainers decide which approach they want to
> see implemented, to avoid Tao jumping between different people opinions.
Sure, no problem.
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-11 10:26 ` Philippe Mathieu-Daudé
@ 2025-08-12 17:50 ` Pierrick Bouvier
0 siblings, 0 replies; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-12 17:50 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 8/11/25 3:26 AM, Philippe Mathieu-Daudé wrote:
> On 10/8/25 18:11, Tao Tang wrote:
>>
>> On 2025/8/7 05:28, Pierrick Bouvier wrote:
>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>
>
>>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>>>> support QEMU, and no secure device assignment feature exists yet, I
>>>> created a custom platform device to test the secure translation flow.
>>>> To trigger the translation logic, I initiated MMIO writes to this
>>>> device from within Hafnium. The device's MMIO callback handler then
>>>> performed DMA accesses via its IOMMU region, exercising the secure
>>>> translation path. While SMMUv3 is typically used for PCIe on
>>>> physical SoCs, the architecture allows its use with platform devices
>>>> via a stream-id binding in the device tree. The test harness
>>>> required some non-standard modifications to decouple the SMMU from
>>>> its tight integration with PCIe. The code for this test device is
>>>> available for review at [3]. README.md with detailed instructions is
>>>> also provided.
>>>>
>>>
>>> I am not sure about the current policy in QEMU for test oriented
>>> devices, but it would be really useful to have something similar
>>> upstream (Note: it's out of the scope of this series).
>>> One challenge working with SMMU emulation is that reproducing setups
>>> and triggering specific code paths is hard to achieve, due to the
>>> indirect use of SMMU feature (through DMA) and the complex software
>>> stack usually involved.
>>> Having something upstream available to work on SMMU emulation, at
>>> least on device side, would be a great addition.
>>>
>>> Eric, Peter, is this something that would be acceptable to merge?
>
>
> This shouldn't be an issue, we already have some:
>
> $ git ls-files|fgrep testdev
> chardev/testdev.c
> docs/specs/pci-testdev.rst
> hw/hyperv/hyperv_testdev.c
> hw/misc/pc-testdev.c
> hw/misc/pci-testdev.c
> hw/tricore/tricore_testdevice.c
>
Looks good indeed, and we have the TEST_DEVICES category for that.
>
>> Looking ahead, my plan is to refactor the smmuv3-test platform device.
>> The goal is to make it self-contained within QEMU, removing the current
>> dependency on Hafnium to trigger its operations. I plan to submit this
>> as a separate RFC patch series in the next few days.
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-10 16:11 ` Tao Tang
2025-08-11 10:26 ` Philippe Mathieu-Daudé
@ 2025-08-12 18:04 ` Pierrick Bouvier
2025-08-15 5:49 ` Tao Tang
1 sibling, 1 reply; 47+ messages in thread
From: Pierrick Bouvier @ 2025-08-12 18:04 UTC (permalink / raw)
To: Tao Tang, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe
On 8/10/25 9:11 AM, Tao Tang wrote:
>
> On 2025/8/7 05:28, Pierrick Bouvier wrote:
>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>> Hi all,
>>>
>>> This patch series introduces initial support for emulating the Arm
>>> SMMUv3
>>> Secure State.
>>>
>>> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
>>> emulation is a notable missing piece in QEMU. While the FVP model has
>>> some support, its limited PCIe capabilities make it challenging for
>>> complex use cases. The ability to properly manage device DMA from a
>>> secure context is a critical prerequisite for enabling device assignment
>>> (passthrough) for confidential computing solutions like Arm CCA and
>>> related research such as virtCCA [2]. This series aims to build that
>>> foundational support in QEMU.
>>>
>>
>> Thanks for posting this series, it's definitely an important piece
>> missing for emulating newer SMMU versions.
>>
>>> This work is being proposed as an RFC. It introduces a significant
>>> amount
>>> of new logic, including the core concept of modeling parallel secure and
>>> non-secure contexts within a single SMMUv3 device. I am seeking feedback
>>> on the overall approach, the core refactoring, and the implementation
>>> details before proceeding further.
>>>
>>> The series begins by implementing the components of the secure
>>> programming
>>> interface, then progressively refactors the core SMMU logic to handle
>>> secure and non-secure contexts in parallel.
>>>
>>> Secure Interface Implementation: The initial patches add the
>>> secure-side registers, implement their read/write logic, and enable
>>> the secure command and event queues. This includes the S_INIT
>>> mechanism and the new secure TLB invalidation commands.
>>>
>>> Core Logic Refactoring: The next set of patches makes the core SMMU
>>> functions security-state aware. This involves plumbing an is_secure
>>> context flag through the main code paths and adding logic to route
>>> SMMU-originated memory accesses to the correct (Secure or Non-secure)
>>> address space.
>>>
>>> Cache Isolation: With the core logic now aware of security states,
>>> the following patches refactor the configuration and translation
>>> lookup caches. The cache keys are modified to include the security
>>> context, ensuring that secure and non-secure entries for the same
>>> device or address are properly isolated and preventing aliasing.
>>>
>>> Framework Integration: The final patch connects the SMMU's internal
>>> security context to the generic QEMU IOMMU framework by using the
>>> iommu_index to represent the architectural SEC_SID.
>>>
>>> To validate this work, I performed the following tests:
>>>
>>> Non-Secure Regression: To ensure that existing functionality remains
>>> intact, I ran a nested virtualization test. A TCG guest was created on
>>> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device
>>> assigned.
>>> Command line of TCG VM is below:
>>>
>>> qemu-system-aarch64 \
>>> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
>>> -cpu max -smp 1 -m 4080M \
>>> -accel tcg,thread=single,tb-size=512 \
>>> -kernel Image \
>>> -append 'nokaslr root=/dev/vda rw rootfstype=ext4
>>> iommu.passthrough=on' \
>>> -device
>>> pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
>>> -device
>>> pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
>>> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
>>> -device virtio-blk-device,drive=hd0 \
>>> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
>>> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
>>> -device virtio-net-device,netdev=eth0 \
>>> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
>>> -device nvme,drive=nvme0,serial=deadbeef \
>>> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log
>>> -nographic
>>>
>>> Inside this TCG VM, a KVM guest was launched, and the same NVMe
>>> device was
>>> re-assigned to it via VFIO.
>>> Command line of KVM VM inside TCG VM is below:
>>>
>>> sudo qemu-system-aarch64 \
>>> -enable-kvm -m 1024 -cpu host -M virt \
>>> -machine virt,gic-version=3 \
>>> -cpu max -append "nokaslr" -smp 1 \
>>> -monitor stdio \
>>> -kernel 5.15.Image \
>>> -initrd rootfs.cpio.gz \
>>> -display vnc=:22,id=primary \
>>> -device vfio-pci,host=00:01.0
>>>
>>> The KVM guest was able to perform I/O on the device
>>> correctly, confirming that the non-secure path is not broken.
>>>
>>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
>>> environment. Hafnium's smmuv3_driver_init function was used to test
>>> the secure register I/O and command queue functionality (excluding
>>> translation). As Hafnium assumes larger queue and StreamID sizes than
>>> are practical without TTST support, I temporarily patched Hafnium to
>>> use smaller values, allowing its driver to initialize the emulated
>>> secure SMMU successfully.
>>>
>>
>> Would that be possible to share your changes, and build instructions
>> for this? While working on SMMU emulation, we finally left this on the
>> side due to lack of a software stack being able to use secure SMMU, as
>> we were not aware that Hafnium + op-tee could make use of it.
>>
> Hi Pierrick,
>
> Thanks for your interest! I'm very happy to share my work on this. I've
> documented the setup process, including our code modifications and the
> step-by-step build instructions in this link:
>
> https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG
>
Thanks for taking the time to assemble all this in a comprehensible
post, I'll give it a try when I have some spare time.
>
> The core point of these changes is to enable the SMMUv3 feature in
> Hafnium. This leads to numerous read/write operations on SMMUv3 secure
> registers and various queue manipulations within the smmuv3_driver_init
> function in Hafnium.
>
> However, it's important to note that this initialization process itself
> does not initiate any DMA memory access that would trigger the
> smmuv3_translate flow.
>
I understand the difference. It can be tricky to generate specific
translation scenarios, which is where a custom test device can really help.
> Even so, we've devised a method to test the subsequent Secure
> Translation Path by leveraging the smmuv3-test platform device. This
> approach allows us to verify the entire SMMUv3 flow, from initialization
> to translation.
>
Does it rely on a custom driver integration into an existing firmware or
the kernel?
>
>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>>> support QEMU, and no secure device assignment feature exists yet, I
>>> created a custom platform device to test the secure translation flow.
>>> To trigger the translation logic, I initiated MMIO writes to this
>>> device from within Hafnium. The device's MMIO callback handler then
>>> performed DMA accesses via its IOMMU region, exercising the secure
>>> translation path. While SMMUv3 is typically used for PCIe on
>>> physical SoCs, the architecture allows its use with platform devices
>>> via a stream-id binding in the device tree. The test harness
>>> required some non-standard modifications to decouple the SMMU from
>>> its tight integration with PCIe. The code for this test device is
>>> available for review at [3]. README.md with detailed instructions is
>>> also provided.
>>>
>>
>> I am not sure about the current policy in QEMU for test oriented
>> devices, but it would be really useful to have something similar
>> upstream (Note: it's out of the scope of this series).
>> One challenge working with SMMU emulation is that reproducing setups
>> and triggering specific code paths is hard to achieve, due to the
>> indirect use of SMMU feature (through DMA) and the complex software
>> stack usually involved.
>> Having something upstream available to work on SMMU emulation, at
>> least on device side, would be a great addition.
>>
>> Eric, Peter, is this something that would be acceptable to merge?
>>
>
> Looking ahead, my plan is to refactor the smmuv3-test platform device.
> The goal is to make it self-contained within QEMU, removing the current
> dependency on Hafnium to trigger its operations. I plan to submit this
> as a separate RFC patch series in the next few days.
>
This is very welcome. Once this is in place, it would be great to add a
new test to make sure things don't regress, and from where we can iterate.
By self-contained within QEMU, do you mean a QTest based test?
Regards,
Pierrick
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-11 10:34 ` Philippe Mathieu-Daudé
2025-08-12 17:27 ` Pierrick Bouvier
@ 2025-08-12 18:42 ` Peter Maydell
2025-08-15 6:02 ` Tao Tang
1 sibling, 1 reply; 47+ messages in thread
From: Peter Maydell @ 2025-08-12 18:42 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Tao Tang, Pierrick Bouvier, qemu-arm, qemu-devel, Eric Auger,
Chen Baozi, smostafa, jean-philippe
On Mon, 11 Aug 2025 at 11:34, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> Hi,
>
> On 10/8/25 18:59, Tao Tang wrote:
> > On 2025/8/7 05:55, Pierrick Bouvier wrote:
> >> On 8/6/25 8:11 AM, Tao Tang wrote:
> >>> This patch enables the secure command queue, providing a dedicated
> >>> interface for secure software to issue commands to the SMMU. Based on
> >>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
> >>> entries directly from the Secure PA space so that we need to pass the
> >>> memory transaction attributes when reading the command queue.
> >>>
> >>> This provides a parallel command mechanism to the non-secure world.
> >>>
> >>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> >>> ---
> >>> hw/arm/smmuv3-internal.h | 8 ++++--
> >>> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
> >>> hw/arm/trace-events | 2 +-
> >>> 3 files changed, 41 insertions(+), 24 deletions(-)
> >>>
> >>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> >>> index 1a8b1cb204..5b2ca00832 100644
> >>> --- a/hw/arm/smmuv3-internal.h
> >>> +++ b/hw/arm/smmuv3-internal.h
> >>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
> >>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
> >>> }
> >>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
> >>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
> >>> {
> >>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
> >>> + if (is_secure) {
> >>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
> >>> + } else {
> >>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
> >>> + }
> >>> }
>
>
> >> This looks like a reasonable and readable approach to support secure
> >> and non secure accesses.
> >
> > Hi Pierrick,
> >
> > Thank you so much for taking the time to review and for the very
> > positive feedback.
> >
> > I'm very relieved to hear you find the approach "reasonable and
> > readable". I was hoping that explicitly passing the parameter was the
> > right way to avoid issues with global state or code duplication, and
> > your confirmation is the best encouragement I could ask for.
>
> An alternative (also suggested in patch #1) is to use index for banked
> registers.
This is often how we handle S/NS banked registers, yes.
Other examples:
include/hw/intc/arm_gicv3_common.h: the S/NS banked registers
ICC_CTLR_EL1, GICD_STATUSR, GICR_STATUSR are implemented as
uint32_t icc_ctlr_el1[2] etc. (The GICv3 also has some
group-related registers which is why it also has 3-element arrays)
The banked AArch32 system registers in cpu.h are a bit
more complicated, and there we have some macros
A32_BANKED_REG_GET/SET which access the right fields.
This lets us define the mapping between the NS and S
registers and the AArch32 ELx views. (I'm not sure I'd
go down this path again if we did it now: our AArch32
design here pre-dates AArch64. The AArch64 views use
an array that they index by EL.)
Sometimes when there are only a few registers that need
to be banked S/NS we give them names, as with the
GICv2 where we have separate apr[][] and nsapr[][] arrays.
I think I would say that it depends mostly on how many
registers there are, and whether you wind up wanting to do
a lot of "the logic is the same for both S and NS, it
just operates on the other register in the bank". It
can make the code a lot easier to read in that case if
you have functions that take the is_secure parameter
and just use it all the way through to index into the
register array. Even in the simple example above of
smmuv3_cmdq_enabled(), you can see that
return FIELD_EX32(s->cr[is_secure][0], CR0, CMDQEN);
is more succinct than the if-else version.
I have not looked closely enough the SMMUv3 code or spec to
be certain which side of the line it falls, but a glance
at the register overview does suggest that the design
has registers for all of normal, secure, realm and root.
Using an array will probably make our lives easier
if/when we have to implement the RME support in the SMMU.
thanks
-- PMM
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-12 18:04 ` Pierrick Bouvier
@ 2025-08-15 5:49 ` Tao Tang
2025-09-30 4:04 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-15 5:49 UTC (permalink / raw)
To: Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi, smostafa, jean-philippe,
Philippe Mathieu-Daudé
On 2025/8/13 02:04, Pierrick Bouvier wrote:
> On 8/10/25 9:11 AM, Tao Tang wrote:
>>
>> On 2025/8/7 05:28, Pierrick Bouvier wrote:
>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>>> Hi all,
>>>>
>>>> This patch series introduces initial support for emulating the Arm
>>>> SMMUv3
>>>> Secure State.
>>>> ------------------------------<snip>------------------------------
>>>>
>>>>
>>>> ------------------------------<snip>------------------------------
>>>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
>>>> environment. Hafnium's smmuv3_driver_init function was used to test
>>>> the secure register I/O and command queue functionality (excluding
>>>> translation). As Hafnium assumes larger queue and StreamID sizes than
>>>> are practical without TTST support, I temporarily patched Hafnium to
>>>> use smaller values, allowing its driver to initialize the emulated
>>>> secure SMMU successfully.
>>>>
>>>
>>> Would that be possible to share your changes, and build instructions
>>> for this? While working on SMMU emulation, we finally left this on the
>>> side due to lack of a software stack being able to use secure SMMU, as
>>> we were not aware that Hafnium + op-tee could make use of it.
>>>
>> Hi Pierrick,
>>
>> Thanks for your interest! I'm very happy to share my work on this. I've
>> documented the setup process, including our code modifications and the
>> step-by-step build instructions in this link:
>>
>> https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG
>>
>
> Thanks for taking the time to assemble all this in a comprehensible
> post, I'll give it a try when I have some spare time.
Hi Pierrick,
You're welcome, and please feel free to let me know if you run into any
issues.
>
>>
>> The core point of these changes is to enable the SMMUv3 feature in
>> Hafnium. This leads to numerous read/write operations on SMMUv3 secure
>> registers and various queue manipulations within the smmuv3_driver_init
>> function in Hafnium.
>>
>> However, it's important to note that this initialization process itself
>> does not initiate any DMA memory access that would trigger the
>> smmuv3_translate flow.
>>
>
> I understand the difference. It can be tricky to generate specific
> translation scenarios, which is where a custom test device can really
> help.
>
>> Even so, we've devised a method to test the subsequent Secure
>> Translation Path by leveraging the smmuv3-test platform device. This
>> approach allows us to verify the entire SMMUv3 flow, from initialization
>> to translation.
>>
>
> Does it rely on a custom driver integration into an existing firmware
> or the kernel?
>
>>
>>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>>>> support QEMU, and no secure device assignment feature exists yet, I
>>>> created a custom platform device to test the secure translation flow.
>>>> To trigger the translation logic, I initiated MMIO writes to this
>>>> device from within Hafnium. The device's MMIO callback handler then
>>>> performed DMA accesses via its IOMMU region, exercising the secure
>>>> translation path. While SMMUv3 is typically used for PCIe on
>>>> physical SoCs, the architecture allows its use with platform devices
>>>> via a stream-id binding in the device tree. The test harness
>>>> required some non-standard modifications to decouple the SMMU from
>>>> its tight integration with PCIe. The code for this test device is
>>>> available for review at [3]. README.md with detailed instructions is
>>>> also provided.
>>>>
>>>
>>> I am not sure about the current policy in QEMU for test oriented
>>> devices, but it would be really useful to have something similar
>>> upstream (Note: it's out of the scope of this series).
>>> One challenge working with SMMU emulation is that reproducing setups
>>> and triggering specific code paths is hard to achieve, due to the
>>> indirect use of SMMU feature (through DMA) and the complex software
>>> stack usually involved.
>>> Having something upstream available to work on SMMU emulation, at
>>> least on device side, would be a great addition.
>>>
>>> Eric, Peter, is this something that would be acceptable to merge?
>>>
>>
>> Looking ahead, my plan is to refactor the smmuv3-test platform device.
>> The goal is to make it self-contained within QEMU, removing the current
>> dependency on Hafnium to trigger its operations. I plan to submit this
>> as a separate RFC patch series in the next few days.
>>
>
> This is very welcome. Once this is in place, it would be great to add
> a new test to make sure things don't regress, and from where we can
> iterate.
> By self-contained within QEMU, do you mean a QTest based test?
>
> Regards,
> Pierrick
Thanks for the follow-up and the great questions.
To answer your question about the custom test driver: yes, the current
implementation of the smmuv3-test device relies on integration with
Hafnium to be triggered.
The test flow is initiated when Hafnium performs an mmio_write32 to the
smmuv3-test device's MMIO space. This triggers the device's read/write
callback in QEMU. Inside this callback, I use address_space_write/read
to first populate the necessary SMMU structures (STEs, CDs, PTEs) in
guest secure memory, and then perform another address_space_write/read
to an IOMMU-protected region of smmuv3-test. It is this final access
that exercises the full secure translation path. So for now, it is
indeed dependent on the Hafnium firmware for debugging.
That brings me to your next point about making the test
"self-contained". My goal is exactly to remove this dependency. I'm
currently exploring ways to achieve this within QEMU—for instance, by
using QMP commands or another monitor interface to replace the
mmio_write32 action that Hafnium currently provides. This is what I
meant by "self-contained": allowing us to test the entire SMMUv3
translation flow relying only on the QEMU emulator itself, without any
specific guest firmware or kernel.
This leads perfectly to your question about QTest. I'm not very familiar
with it, so your suggestion is very helpful. Is QTest the standard or
required framework for implementing this kind of self-contained test
device? I would appreciate any guidance or pointers you could provide on
this topic.
Thanks again for the valuable discussion.
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-12 18:42 ` Peter Maydell
@ 2025-08-15 6:02 ` Tao Tang
2025-08-15 14:53 ` Peter Maydell
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-15 6:02 UTC (permalink / raw)
To: Peter Maydell, Philippe Mathieu-Daudé
Cc: Pierrick Bouvier, qemu-arm, qemu-devel, Eric Auger, Chen Baozi,
smostafa, jean-philippe
On 2025/8/13 02:42, Peter Maydell wrote:
> On Mon, 11 Aug 2025 at 11:34, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>> Hi,
>>
>> On 10/8/25 18:59, Tao Tang wrote:
>>> On 2025/8/7 05:55, Pierrick Bouvier wrote:
>>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>>>> This patch enables the secure command queue, providing a dedicated
>>>>> interface for secure software to issue commands to the SMMU. Based on
>>>>> the SMMU_S_CMDQ_BASE configuration, the SMMU now fetches command
>>>>> entries directly from the Secure PA space so that we need to pass the
>>>>> memory transaction attributes when reading the command queue.
>>>>>
>>>>> This provides a parallel command mechanism to the non-secure world.
>>>>>
>>>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>>>> ---
>>>>> hw/arm/smmuv3-internal.h | 8 ++++--
>>>>> hw/arm/smmuv3.c | 55 +++++++++++++++++++++++++---------------
>>>>> hw/arm/trace-events | 2 +-
>>>>> 3 files changed, 41 insertions(+), 24 deletions(-)
>>>>>
>>>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>>>> index 1a8b1cb204..5b2ca00832 100644
>>>>> --- a/hw/arm/smmuv3-internal.h
>>>>> +++ b/hw/arm/smmuv3-internal.h
>>>>> @@ -319,9 +319,13 @@ static inline void queue_cons_incr(SMMUQueue *q)
>>>>> q->cons = deposit32(q->cons, 0, q->log2size + 1, q->cons + 1);
>>>>> }
>>>>> -static inline bool smmuv3_cmdq_enabled(SMMUv3State *s)
>>>>> +static inline bool smmuv3_cmdq_enabled(SMMUv3State *s, bool is_secure)
>>>>> {
>>>>> - return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>>> + if (is_secure) {
>>>>> + return FIELD_EX32(s->secure_cr[0], S_CR0, CMDQEN);
>>>>> + } else {
>>>>> + return FIELD_EX32(s->cr[0], CR0, CMDQEN);
>>>>> + }
>>>>> }
>>
>>>> This looks like a reasonable and readable approach to support secure
>>>> and non secure accesses.
>>> Hi Pierrick,
>>>
>>> Thank you so much for taking the time to review and for the very
>>> positive feedback.
>>>
>>> I'm very relieved to hear you find the approach "reasonable and
>>> readable". I was hoping that explicitly passing the parameter was the
>>> right way to avoid issues with global state or code duplication, and
>>> your confirmation is the best encouragement I could ask for.
>> An alternative (also suggested in patch #1) is to use index for banked
>> registers.
> This is often how we handle S/NS banked registers, yes.
> Other examples:
>
> include/hw/intc/arm_gicv3_common.h: the S/NS banked registers
> ICC_CTLR_EL1, GICD_STATUSR, GICR_STATUSR are implemented as
> uint32_t icc_ctlr_el1[2] etc. (The GICv3 also has some
> group-related registers which is why it also has 3-element arrays)
>
> The banked AArch32 system registers in cpu.h are a bit
> more complicated, and there we have some macros
> A32_BANKED_REG_GET/SET which access the right fields.
> This lets us define the mapping between the NS and S
> registers and the AArch32 ELx views. (I'm not sure I'd
> go down this path again if we did it now: our AArch32
> design here pre-dates AArch64. The AArch64 views use
> an array that they index by EL.)
>
> Sometimes when there are only a few registers that need
> to be banked S/NS we give them names, as with the
> GICv2 where we have separate apr[][] and nsapr[][] arrays.
>
> I think I would say that it depends mostly on how many
> registers there are, and whether you wind up wanting to do
> a lot of "the logic is the same for both S and NS, it
> just operates on the other register in the bank". It
> can make the code a lot easier to read in that case if
> you have functions that take the is_secure parameter
> and just use it all the way through to index into the
> register array. Even in the simple example above of
> smmuv3_cmdq_enabled(), you can see that
>
> return FIELD_EX32(s->cr[is_secure][0], CR0, CMDQEN);
>
> is more succinct than the if-else version.
>
> I have not looked closely enough the SMMUv3 code or spec to
> be certain which side of the line it falls, but a glance
> at the register overview does suggest that the design
> has registers for all of normal, secure, realm and root.
> Using an array will probably make our lives easier
> if/when we have to implement the RME support in the SMMU.
>
> thanks
> -- PMM
Hi Peter, and all,
Thank you all for the very helpful feedback and discussion on this
series. I will start refactoring the code to adopt the register banking
approach as suggested.
While working on the design, I have some questions regarding the best
way to index these banks, especially looking ahead to RME support, and I
would appreciate your thoughts on them.
Here is my understanding of the current situation:
According to the ARM SMMUv3 specification (IHI 0070 G.b), the SEC_SID is
the ground truth for determining the security context of a transaction.
For an RME-capable SMMU, its encoding is:
SEC_SID=0 <-> Non-secure
SEC_SID=1 <-> Secure
SEC_SID=2 <-> Realm
SEC_SID=3 <-> Reserved
To accurately model the SMMU, our smmuv3_translate function must
ultimately use this SEC_SID value. Taking the translate callback
signature into consideration:
IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
IOMMUAccessFlags flags, int iommu_idx);
My understanding is that we can use the iommu_idx parameter to carry the
SEC_SID value for each transaction. The SMMU model would then use this
index to select the correct programming interface, which in turn
corresponds to the bank index for its internal registers (e.g.,
s->bank[iommu_idx]).
The dilemma arises when choosing the enum for this bank index.
Philippe's suggestion to use ARMASIdx is great for the current S/NS
case. For future RME support, one might consider ARMSecuritySpace.
However, as Peter’s feedback highlighted, the SMMU architecture includes
four states. The problem is that the values of ARMSecuritySpace do not
match the SEC_SID encoding at all, so it cannot be used directly(e.g., 0
for Secure, 1 for Non-secure BUT for SEC_SID, 0 for NON-secure, 1 for
Secure ) .
This leads me to two questions on the best path forward:
1. How should the SEC_SID be propagated from the device? The only thing
that is certain is that the hardware transaction will carry a SEC_SID
value that is consistent with the manual. My initial thought was to
extend MemTxAttrs.secure to carry the four states, but I am unsure if
this is the best approach.
typedef struct MemTxAttrs {
/*
* ARM/AMBA: TrustZone Secure access
* x86: System Management Mode access
*/
unsigned int secure:1;
/*
* ARM: ArmSecuritySpace. This partially overlaps secure, but it is
* easier to have both fields to assist code that does not understand
* ARMv9 RME, or no specific knowledge of ARM at all (e.g. pflash).
*/
unsigned int space:2;
...
}
2. What should we use as the internal bank index within the SMMUv3
model? I see two potential options:
a) Use the existing ARMSecuritySpace as the internal index. This would
require conversions from the SEC_SID carried by each transaction.
b) Define a new, SMMU-specific enum that perfectly matches the SEC_SID
hardware specification (e.g., enum SmmuSecSid { NS=0, S=1, REALM=2,
ROOT=3 }). We would use this new enum as our bank index internally.
I am leaning towards option (2b) as it seems to offer better
encapsulation and fidelity to the hardware specification, but I would be
very grateful to hear your opinions and guidance on this architectural
choice.
Thank you very much!
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-15 6:02 ` Tao Tang
@ 2025-08-15 14:53 ` Peter Maydell
2025-08-17 3:46 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Peter Maydell @ 2025-08-15 14:53 UTC (permalink / raw)
To: Tao Tang
Cc: Philippe Mathieu-Daudé, Pierrick Bouvier, qemu-arm,
qemu-devel, Eric Auger, Chen Baozi, smostafa, jean-philippe
On Fri, 15 Aug 2025 at 07:02, Tao Tang <tangtao1634@phytium.com.cn> wrote:
> This leads me to two questions on the best path forward:
>
> 1. How should the SEC_SID be propagated from the device? The only thing
> that is certain is that the hardware transaction will carry a SEC_SID
> value that is consistent with the manual. My initial thought was to
> extend MemTxAttrs.secure to carry the four states, but I am unsure if
> this is the best approach.
>
> typedef struct MemTxAttrs {
> /*
> * ARM/AMBA: TrustZone Secure access
> * x86: System Management Mode access
> */
> unsigned int secure:1;
> /*
> * ARM: ArmSecuritySpace. This partially overlaps secure, but it is
> * easier to have both fields to assist code that does not understand
> * ARMv9 RME, or no specific knowledge of ARM at all (e.g. pflash).
> */
> unsigned int space:2;
The SEC_SID is the information in the transaction that tells
us whether the device that initiated the transaction is
NonSecure, Secure, or Realm, right? I think we can
straightforwardly use 'space' for this. (The mapping between
SEC_SID and space can be written "space = sec_sid ^ 1" but
that is perhaps being unnecessarily clever.)
"secure" is (as the comment notes) for the benefit of code
that doesn't understand Realm. It should be initialized to
arm_space_is_secure(space). (Currently that function is in
target/arm/cpu.h, which is an awkward place for it: we
should probably move it to somewhere that lets code that's
not tied to the CPU use it.)
> To accurately model the SMMU, our smmuv3_translate function must
> ultimately use this SEC_SID value. Taking the translate callback
> signature into consideration:
>
> IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
> IOMMUAccessFlags flags, int iommu_idx);
>
> My understanding is that we can use the iommu_idx parameter to carry the
> SEC_SID value for each transaction.
Yes. You also will need to implement the attrs_to_index method,
which is what converts from the attrs in the MemTxAttrs to
the iommu_idx that the translate method gets passed.
(We do actually have hw/misc/tz-mpc.c that uses the IOMMU
APIs to handle Secure vs NonSecure transactions differently,
though it's such a long way from a real SMMU that I don't
know that it's a very useful thing to look at.)
> 2. What should we use as the internal bank index within the SMMUv3
> model? I see two potential options:
> a) Use the existing ARMSecuritySpace as the internal index. This would
> require conversions from the SEC_SID carried by each transaction.
> b) Define a new, SMMU-specific enum that perfectly matches the SEC_SID
> hardware specification (e.g., enum SmmuSecSid { NS=0, S=1, REALM=2,
> ROOT=3 }). We would use this new enum as our bank index internally.
>
> I am leaning towards option (2b) as it seems to offer better
> encapsulation and fidelity to the hardware specification, but I would be
> very grateful to hear your opinions and guidance on this architectural
> choice.
Yes, I like 2b here. This is what I am doing for the GICv5 --
there the architecture specifies various "interrupt domains",
and there's an obvious encoding from those to 0..3, which is
what I use to pick the register bank to use. The MemTxAttrs
don't quite line up with the domain index, and they are something
we need to figure out less often than "which register bank", so
it's OK for that to be something we do via a function or
whatever.
For the GICv5 in fact I have opted to create the MemTxAttrs
ahead of time at the point where the guest enables things
and we know what they should be (roughly, the device state
caches various bits of info about the configured state
in a struct, and one of those fields is a MemTxAttrs; so
you can pass cfg[domain]->txattrs as your attrs rather than
computing it on the spot every time -- especially since you
need to initialize both .space and .secure this seemed
nicer to me).
thanks
-- PMM
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state
2025-08-15 14:53 ` Peter Maydell
@ 2025-08-17 3:46 ` Tao Tang
0 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-17 3:46 UTC (permalink / raw)
To: Peter Maydell
Cc: Philippe Mathieu-Daudé, Pierrick Bouvier, qemu-arm,
qemu-devel, Eric Auger, Chen Baozi, smostafa, jean-philippe
On 2025/8/15 22:53, Peter Maydell wrote:
> On Fri, 15 Aug 2025 at 07:02, Tao Tang <tangtao1634@phytium.com.cn> wrote:
>> This leads me to two questions on the best path forward:
>>
>> 1. How should the SEC_SID be propagated from the device? The only thing
>> that is certain is that the hardware transaction will carry a SEC_SID
>> value that is consistent with the manual. My initial thought was to
>> extend MemTxAttrs.secure to carry the four states, but I am unsure if
>> this is the best approach.
>>
> The SEC_SID is the information in the transaction that tells
> us whether the device that initiated the transaction is
> NonSecure, Secure, or Realm, right? I think we can
> straightforwardly use 'space' for this. (The mapping between
> SEC_SID and space can be written "space = sec_sid ^ 1" but
> that is perhaps being unnecessarily clever.)
>
> "secure" is (as the comment notes) for the benefit of code
> that doesn't understand Realm. It should be initialized to
> arm_space_is_secure(space). (Currently that function is in
> target/arm/cpu.h, which is an awkward place for it: we
> should probably move it to somewhere that lets code that's
> not tied to the CPU use it.)
>
>> To accurately model the SMMU, our smmuv3_translate function must
>> ultimately use this SEC_SID value. Taking the translate callback
>> signature into consideration:
>>
>> IOMMUTLBEntry (*translate)(IOMMUMemoryRegion *iommu, hwaddr addr,
>> IOMMUAccessFlags flags, int iommu_idx);
>>
>> My understanding is that we can use the iommu_idx parameter to carry the
>> SEC_SID value for each transaction.
> Yes. You also will need to implement the attrs_to_index method,
> which is what converts from the attrs in the MemTxAttrs to
> the iommu_idx that the translate method gets passed.
>
> (We do actually have hw/misc/tz-mpc.c that uses the IOMMU
> APIs to handle Secure vs NonSecure transactions differently,
> though it's such a long way from a real SMMU that I don't
> know that it's a very useful thing to look at.)
>
>> 2. What should we use as the internal bank index within the SMMUv3
>> model? I see two potential options:
>> a) Use the existing ARMSecuritySpace as the internal index. This would
>> require conversions from the SEC_SID carried by each transaction.
>> b) Define a new, SMMU-specific enum that perfectly matches the SEC_SID
>> hardware specification (e.g., enum SmmuSecSid { NS=0, S=1, REALM=2,
>> ROOT=3 }). We would use this new enum as our bank index internally.
>>
>> I am leaning towards option (2b) as it seems to offer better
>> encapsulation and fidelity to the hardware specification, but I would be
>> very grateful to hear your opinions and guidance on this architectural
>> choice.
> Yes, I like 2b here. This is what I am doing for the GICv5 --
> there the architecture specifies various "interrupt domains",
> and there's an obvious encoding from those to 0..3, which is
> what I use to pick the register bank to use. The MemTxAttrs
> don't quite line up with the domain index, and they are something
> we need to figure out less often than "which register bank", so
> it's OK for that to be something we do via a function or
> whatever.
>
> For the GICv5 in fact I have opted to create the MemTxAttrs
> ahead of time at the point where the guest enables things
> and we know what they should be (roughly, the device state
> caches various bits of info about the configured state
> in a struct, and one of those fields is a MemTxAttrs; so
> you can pass cfg[domain]->txattrs as your attrs rather than
> computing it on the spot every time -- especially since you
> need to initialize both .space and .secure this seemed
> nicer to me).
>
> thanks
> -- PMM
Hi Peter,
Thanks for the very clear and actionable feedback.
I will refactor the code in V2 according to the plan you laid out. The
approach will be to use MemTxAttrs.space to carry the security state,
and I will be sure to set the legacy .secure field correctly for
compatibility as you pointed out. This will then be converted via the
attrs_to_index method, leading to the use of a new spec-compliant enum
for the internal bank index (your preferred option 2b).
Your GICv5 analogy was very helpful for clarifying the design, and I'll
keep the MemTxAttrs caching tip in mind as an optimization method.
Thanks again for the excellent guidance.
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
2025-08-11 10:22 ` Philippe Mathieu-Daudé
@ 2025-08-18 21:21 ` Mostafa Saleh
1 sibling, 0 replies; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-18 21:21 UTC (permalink / raw)
To: Tao Tang; +Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi
On Wed, Aug 06, 2025 at 11:11:24PM +0800, Tao Tang wrote:
> The Arm SMMUv3 architecture defines a set of registers and commands for
> managing secure transactions and context.
>
> This patch introduces the definitions for these secure registers and
> commands within the SMMUv3 device model internal header.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 57 ++++++++++++++++++++++++++++++++++++++++
> include/hw/arm/smmuv3.h | 23 ++++++++++++++++
> 2 files changed, 80 insertions(+)
>
> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> index b6b7399347..483aaa915e 100644
> --- a/hw/arm/smmuv3-internal.h
> +++ b/hw/arm/smmuv3-internal.h
> @@ -179,6 +179,63 @@ REG32(EVENTQ_IRQ_CFG2, 0xbc)
>
> #define A_IDREGS 0xfd0
>
> +/* Secure registers */
> +#define SMMU_SECURE_BASE_OFFSET 0x8000
> +REG32(S_IDR0, 0x8000)
> +REG32(S_IDR1, 0x8004)
> + FIELD(S_IDR1, S_SIDSIZE, 0 , 6)
> + FIELD(S_IDR1, SEL2, 29, 1)
> + FIELD(S_IDR1, SECURE_IMPL, 31, 1)
> +
> +REG32(S_IDR2, 0x8008)
> +REG32(S_IDR3, 0x800c)
> +REG32(S_IDR4, 0x8010)
> +
> +REG32(S_CR0, 0x8020)
> + FIELD(S_CR0, SMMUEN, 0, 1)
> + FIELD(S_CR0, EVENTQEN, 2, 1)
> + FIELD(S_CR0, CMDQEN, 3, 1)
> +
> +REG32(S_CR0ACK, 0x8024)
> +REG32(S_CR1, 0x8028)
> +REG32(S_CR2, 0x802c)
> +
> +REG32(S_INIT, 0x803c)
> + FIELD(S_INIT, INV_ALL, 0, 1)
> +
> +REG32(S_GBPA, 0x8044)
> + FIELD(S_GBPA, ABORT, 20, 1)
> + FIELD(S_GBPA, UPDATE, 31, 1)
> +
> +REG32(S_IRQ_CTRL, 0x8050)
> + FIELD(S_IRQ_CTRL, GERROR_IRQEN, 0, 1)
> + FIELD(S_IRQ_CTRL, EVENTQ_IRQEN, 2, 1)
> +
> +REG32(S_IRQ_CTRLACK, 0x8054)
> +
> +REG32(S_GERROR, 0x8060)
> + FIELD(S_GERROR, CMDQ_ERR, 0, 1)
> +
> +REG32(S_GERRORN, 0x8064)
> +REG64(S_GERROR_IRQ_CFG0, 0x8068)
> +REG32(S_GERROR_IRQ_CFG1, 0x8070)
> +REG32(S_GERROR_IRQ_CFG2, 0x8074)
> +REG64(S_STRTAB_BASE, 0x8080)
> +REG32(S_STRTAB_BASE_CFG, 0x8088)
> + FIELD(S_STRTAB_BASE_CFG, LOG2SIZE, 0, 6)
> + FIELD(S_STRTAB_BASE_CFG, SPLIT, 6, 5)
> + FIELD(S_STRTAB_BASE_CFG, FMT, 16, 2)
> +
> +REG64(S_CMDQ_BASE, 0x8090)
> +REG32(S_CMDQ_PROD, 0x8098)
> +REG32(S_CMDQ_CONS, 0x809c)
> +REG64(S_EVENTQ_BASE, 0x80a0)
> +REG32(S_EVENTQ_PROD, 0x80a8)
> +REG32(S_EVENTQ_CONS, 0x80ac)
> +REG64(S_EVENTQ_IRQ_CFG0, 0x80b0)
> +REG32(S_EVENTQ_IRQ_CFG1, 0x80b8)
> +REG32(S_EVENTQ_IRQ_CFG2, 0x80bc)
> +
> static inline int smmu_enabled(SMMUv3State *s)
> {
> return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
> diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
> index d183a62766..72ad042514 100644
> --- a/include/hw/arm/smmuv3.h
> +++ b/include/hw/arm/smmuv3.h
> @@ -63,6 +63,29 @@ struct SMMUv3State {
> qemu_irq irq[4];
> QemuMutex mutex;
> char *stage;
> +
> + /* Secure state */
> + uint32_t secure_idr[5];
> + uint32_t secure_cr[3];
> + uint32_t secure_cr0ack;
> + uint32_t secure_init;
> + uint32_t secure_gbpa;
> + uint32_t secure_irq_ctrl;
> + uint32_t secure_gerror;
> + uint32_t secure_gerrorn;
> + uint64_t secure_gerror_irq_cfg0;
> + uint32_t secure_gerror_irq_cfg1;
> + uint32_t secure_gerror_irq_cfg2;
> + uint64_t secure_strtab_base;
> + uint32_t secure_strtab_base_cfg;
> + uint8_t secure_sid_split;
> + uint32_t secure_features;
> +
> + uint64_t secure_eventq_irq_cfg0;
> + uint32_t secure_eventq_irq_cfg1;
> + uint32_t secure_eventq_irq_cfg2;
> +
> + SMMUQueue secure_eventq, secure_cmdq;
> };
As Philippe mentioned, this would be better the secure state is separated
in another instance of the struct, that seems it would reduce a lot of the
duplication later around the logic of MMIO and queues... in the next
patches.
Thanks,
Mostafa
>
> typedef enum {
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-06 15:11 ` [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers Tao Tang
2025-08-06 21:53 ` Pierrick Bouvier
@ 2025-08-18 21:24 ` Mostafa Saleh
2025-08-20 15:21 ` Tao Tang
1 sibling, 1 reply; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-18 21:24 UTC (permalink / raw)
To: Tao Tang; +Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi
On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
> This patch builds upon the previous introduction of secure register
> definitions by providing the functional implementation for their access.
>
> The availability of the secure programming interface is now correctly
> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
> secure functionality is enabled, the I/O handlers (smmuv3_read and
> smmuv3_write) will correctly dispatch accesses to the secure
> register space.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3-internal.h | 5 +
> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 456 insertions(+)
>
> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> index 483aaa915e..1a8b1cb204 100644
> --- a/hw/arm/smmuv3-internal.h
> +++ b/hw/arm/smmuv3-internal.h
> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>
> #define SMMU_CR0_RESERVED 0xFFFFFC20
>
> +/*
> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
> + */
> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
> +
> REG32(CR0ACK, 0x24)
> REG32(CR1, 0x28)
> REG32(CR2, 0x2c)
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index ab67972353..619180d204 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
> s->gerrorn = 0;
> s->statusr = 0;
> s->gbpa = SMMU_GBPA_RESET_VAL;
> +
> + /* Initialize secure state */
> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
> + /* Secure EL2 and Secure stage 2 support */
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
Which is not implemented in this series.
> + /* Secure state implemented */
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> + SECURE_IMPL, 1);
> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> + S_SIDSIZE, SMMU_IDR1_SIDSIZE);
> +
> + s->secure_gbpa = SMMU_GBPA_RESET_VAL;
> }
>
> static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
> @@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> }
> }
>
> +/* Check if the SMMU hardware itself implements secure state features */
> +static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
> +{
> + return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
> +}
> +
I see that the secure SMMU support is unconditional. So, is this always true?
Also, how that looks with migration?
> static int smmuv3_cmdq_consume(SMMUv3State *s)
> {
> SMMUState *bs = ARM_SMMU(s);
> @@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> return 0;
> }
>
> +/* Helper function for secure register write validation */
> +static bool smmu_validate_secure_write(MemTxAttrs attrs, bool secure_impl,
> + hwaddr offset, const char *reg_name)
> +{
> + if (!attrs.secure || !secure_impl) {
> + const char *reason = !attrs.secure ?
> + "Non-secure write attempt" :
> + "SMMU didn't implement Security State";
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
> + __func__, reason, offset, reg_name);
> + return false;
> + }
> + return true;
> +}
> +
> +/* Helper function for secure register read validation */
> +static bool smmu_validate_secure_read(MemTxAttrs attrs, bool secure_impl,
> + hwaddr offset, const char *reg_name,
> + uint64_t *data)
> +{
> + if (!attrs.secure || !secure_impl) {
> + const char *reason = !attrs.secure ?
> + "Non-secure read attempt" :
> + "SMMU didn't implement Security State";
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
> + __func__, reason, offset, reg_name);
> + *data = 0; /* RAZ */
> + return false;
> + }
> + return true;
> +}
> +
> +/* Macro for secure write validation - returns early if validation fails */
> +#define SMMU_CHECK_SECURE_WRITE(reg_name) \
> + do { \
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
> + reg_name)) { \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
> +/* Macro for attrs.secure only validation */
> +#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
> + do { \
> + if (!attrs.secure) { \
> + qemu_log_mask(LOG_GUEST_ERROR, \
> + "%s: Non-secure write attempt at offset " \
> + "0x%" PRIx64 " (%s, WI)\n", \
> + __func__, offset, reg_name); \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
> +/* Macro for secure read validation - returns RAZ if validation fails */
> +#define SMMU_CHECK_SECURE_READ(reg_name) \
> + do { \
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
> + reg_name, data)) { \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
> +/* Macro for attrs.secure only validation (read) */
> +#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
> + do { \
> + if (!attrs.secure) { \
> + qemu_log_mask(LOG_GUEST_ERROR, \
> + "%s: Non-secure read attempt at offset " \
> + "0x%" PRIx64 " (%s, RAZ)\n", \
> + __func__, offset, reg_name); \
> + *data = 0; \
> + return MEMTX_OK; \
> + } \
> + } while (0)
> +
>
Can’t we just have one check? If the access > SMMU_SECURE_BASE_OFFSET, just
check the security state?
And then based on banking, many of those switches will be common with
non secure cases.
Thanks,
Mostafa
> static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> uint64_t data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_GERROR_IRQ_CFG0:
> s->gerror_irq_cfg0 = data;
> @@ -1535,6 +1635,41 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_IRQ_CFG0:
> s->eventq_irq_cfg0 = data;
> return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + /* No need to check secure_impl here */
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = data;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_STRTAB_BASE")) {
> + return MEMTX_OK;
> + }
> + s->secure_strtab_base = data;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = data;
> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
> + s->secure_cmdq.log2size = SMMU_CMDQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = data;
> + s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
> + s->secure_eventq.log2size = SMMU_EVENTQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_EVENTQ_IRQ_CFG0")) {
> + return MEMTX_OK;
> + }
> + s->secure_eventq_irq_cfg0 = data;
> + return MEMTX_OK;
> default:
> qemu_log_mask(LOG_UNIMP,
> "%s Unexpected 64-bit access to 0x%"PRIx64" (WI)\n",
> @@ -1546,6 +1681,11 @@ static MemTxResult smmu_writell(SMMUv3State *s, hwaddr offset,
> static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> uint64_t data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_CR0:
> s->cr[0] = data;
> @@ -1650,6 +1790,137 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_IRQ_CFG2:
> s->eventq_irq_cfg2 = data;
> return MEMTX_OK;
> + case A_S_CR0:
> + SMMU_CHECK_SECURE_WRITE("S_CR0");
> + s->secure_cr[0] = data;
> + /* clear reserved bits */
> + s->secure_cr0ack = data & ~SMMU_S_CR0_RESERVED;
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_CR1:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_CR1")) {
> + return MEMTX_OK;
> + }
> + s->secure_cr[1] = data;
> + return MEMTX_OK;
> + case A_S_CR2:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_CR2")) {
> + return MEMTX_OK;
> + }
> + s->secure_cr[2] = data;
> + return MEMTX_OK;
> + case A_S_IRQ_CTRL:
> + if (!smmu_validate_secure_write(attrs, secure_impl, offset,
> + "S_IRQ_CTRL")) {
> + return MEMTX_OK;
> + }
> + s->secure_irq_ctrl = data;
> + return MEMTX_OK;
> + case A_S_GERRORN:
> + SMMU_CHECK_SECURE_WRITE("S_GERRORN");
> + smmuv3_write_gerrorn(s, data);
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = data;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0 + 4:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG0");
> + s->secure_gerror_irq_cfg0 = deposit64(s->secure_gerror_irq_cfg0,
> + 32, 32, data);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG1:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG1");
> + s->secure_gerror_irq_cfg1 = data;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG2:
> + SMMU_CHECK_ATTRS_SECURE("S_GERROR_IRQ_CFG2");
> + s->secure_gerror_irq_cfg2 = data;
> + return MEMTX_OK;
> + case A_S_GBPA:
> + SMMU_CHECK_SECURE_WRITE("S_GBPA");
> + if (data & R_S_GBPA_UPDATE_MASK) {
> + s->secure_gbpa = data & ~R_S_GBPA_UPDATE_MASK;
> + }
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 0, 32, data);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE");
> + s->secure_strtab_base = deposit64(s->secure_strtab_base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE_CFG:
> + SMMU_CHECK_SECURE_WRITE("S_STRTAB_BASE_CFG");
> + s->secure_strtab_base_cfg = data;
> + if (FIELD_EX32(data, S_STRTAB_BASE_CFG, FMT) == 1) {
> + s->secure_sid_split = FIELD_EX32(data, S_STRTAB_BASE_CFG, SPLIT);
> + s->secure_features |= SMMU_FEATURE_2LVL_STE;
> + }
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 0, 32, data);
> + s->secure_cmdq.log2size = extract64(s->secure_cmdq.base, 0, 5);
> + if (s->secure_cmdq.log2size > SMMU_CMDQS) {
> + s->secure_cmdq.log2size = SMMU_CMDQS;
> + }
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_BASE");
> + s->secure_cmdq.base = deposit64(s->secure_cmdq.base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_CMDQ_PROD:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_PROD");
> + s->secure_cmdq.prod = data;
> + smmuv3_cmdq_consume(s);
> + return MEMTX_OK;
> + case A_S_CMDQ_CONS:
> + SMMU_CHECK_SECURE_WRITE("S_CMDQ_CONS");
> + s->secure_cmdq.cons = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 0, 32, data);
> + s->secure_eventq.log2size = extract64(s->secure_eventq.base, 0, 5);
> + if (s->secure_eventq.log2size > SMMU_EVENTQS) {
> + s->secure_eventq.log2size = SMMU_EVENTQS;
> + }
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE + 4:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_BASE");
> + s->secure_eventq.base = deposit64(s->secure_eventq.base, 32, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_PROD:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_PROD");
> + s->secure_eventq.prod = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_CONS:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_CONS");
> + s->secure_eventq.cons = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0:
> + SMMU_CHECK_SECURE_WRITE("S_EVENTQ_IRQ_CFG0");
> + s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
> + 0, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG0 + 4:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG0");
> + s->secure_eventq_irq_cfg0 = deposit64(s->secure_eventq_irq_cfg0,
> + 32, 32, data);
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG1:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG1");
> + s->secure_eventq_irq_cfg1 = data;
> + return MEMTX_OK;
> + case A_S_EVENTQ_IRQ_CFG2:
> + SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
> + s->secure_eventq_irq_cfg2 = data;
> + return MEMTX_OK;
> default:
> qemu_log_mask(LOG_UNIMP,
> "%s Unexpected 32-bit access to 0x%"PRIx64" (WI)\n",
> @@ -1687,6 +1958,11 @@ static MemTxResult smmu_write_mmio(void *opaque, hwaddr offset, uint64_t data,
> static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> uint64_t *data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_GERROR_IRQ_CFG0:
> *data = s->gerror_irq_cfg0;
> @@ -1700,6 +1976,31 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_BASE:
> *data = s->eventq.base;
> return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG0", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg0;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
> + *data = s->secure_strtab_base;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_BASE", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.base;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_BASE", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_eventq.base;
> + return MEMTX_OK;
> default:
> *data = 0;
> qemu_log_mask(LOG_UNIMP,
> @@ -1712,6 +2013,11 @@ static MemTxResult smmu_readll(SMMUv3State *s, hwaddr offset,
> static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
> uint64_t *data, MemTxAttrs attrs)
> {
> + bool secure_impl = false;
> + if (offset >= SMMU_SECURE_BASE_OFFSET) {
> + secure_impl = smmu_hw_secure_implemented(s);
> + }
> +
> switch (offset) {
> case A_IDREGS ... A_IDREGS + 0x2f:
> *data = smmuv3_idreg(offset - A_IDREGS);
> @@ -1798,6 +2104,151 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
> case A_EVENTQ_CONS:
> *data = s->eventq.cons;
> return MEMTX_OK;
> + case A_S_IDR0 ... A_S_IDR4:
> + int idr_idx = (offset - A_S_IDR0) / 4;
> + g_assert(idr_idx >= 0 && idr_idx <= 4);
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + g_strdup_printf("S_IDR%d", idr_idx),
> + data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_idr[idr_idx];
> + return MEMTX_OK;
> + case A_S_CR0:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR0", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[0];
> + return MEMTX_OK;
> + case A_S_CR0ACK:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR0ACK", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr0ack;
> + return MEMTX_OK;
> + case A_S_CR1:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR1", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[1];
> + return MEMTX_OK;
> + case A_S_CR2:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CR2", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cr[2];
> + return MEMTX_OK;
> + case A_S_GBPA:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GBPA", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gbpa;
> + return MEMTX_OK;
> + case A_S_IRQ_CTRL:
> + case A_S_IRQ_CTRLACK:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_IRQ_CTRL", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_irq_ctrl;
> + return MEMTX_OK;
> + case A_S_GERROR:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror;
> + return MEMTX_OK;
> + case A_S_GERRORN:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERRORN", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerrorn;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0:
> + SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0");
> + *data = extract64(s->secure_gerror_irq_cfg0, 0, 32);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG0 + 4:
> + SMMU_CHECK_SECURE_READ("S_GERROR_IRQ_CFG0+4");
> + *data = extract64(s->secure_gerror_irq_cfg0, 32, 32);
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG1:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG1", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg1;
> + return MEMTX_OK;
> + case A_S_GERROR_IRQ_CFG2:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_GERROR_IRQ_CFG2", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_gerror_irq_cfg2;
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE");
> + *data = extract64(s->secure_strtab_base, 0, 32);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE + 4:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE+4");
> + *data = extract64(s->secure_strtab_base, 32, 32);
> + return MEMTX_OK;
> + case A_S_STRTAB_BASE_CFG:
> + SMMU_CHECK_ATTRS_SECURE_READ("S_STRTAB_BASE_CFG");
> + *data = s->secure_strtab_base_cfg;
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE:
> + SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
> + *data = extract64(s->secure_cmdq.base, 0, 32);
> + return MEMTX_OK;
> + case A_S_CMDQ_BASE + 4:
> + SMMU_CHECK_SECURE_READ("S_CMDQ_BASE");
> + *data = extract64(s->secure_cmdq.base, 32, 32);
> + return MEMTX_OK;
> + case A_S_CMDQ_PROD:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_PROD", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.prod;
> + return MEMTX_OK;
> + case A_S_CMDQ_CONS:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_CMDQ_CONS", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_cmdq.cons;
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE:
> + SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
> + *data = extract64(s->secure_eventq.base, 0, 32);
> + return MEMTX_OK;
> + case A_S_EVENTQ_BASE + 4:
> + SMMU_CHECK_SECURE_READ("S_EVENTQ_BASE");
> + *data = extract64(s->secure_eventq.base, 32, 32);
> + return MEMTX_OK;
> + case A_S_EVENTQ_PROD:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_PROD", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_eventq.prod;
> + return MEMTX_OK;
> + case A_S_EVENTQ_CONS:
> + if (!smmu_validate_secure_read(attrs, secure_impl, offset,
> + "S_EVENTQ_CONS", data)) {
> + return MEMTX_OK;
> + }
> + *data = s->secure_eventq.cons;
> + return MEMTX_OK;
> default:
> *data = 0;
> qemu_log_mask(LOG_UNIMP,
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization
2025-08-06 15:11 ` [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization Tao Tang
@ 2025-08-18 21:26 ` Mostafa Saleh
2025-08-20 16:01 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-18 21:26 UTC (permalink / raw)
To: Tao Tang; +Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi
On Wed, Aug 06, 2025 at 11:11:26PM +0800, Tao Tang wrote:
> This patch implements the S_INIT register, a secure-only register
> with no non-secure counterpart. It provides a simple mechanism for
> secure software to perform a global invalidation of all SMMU
> configuration and translation caches.
>
> This is typically the final step in a SMMU's probe sequence, marking
> the end of initialization for the SMMU's secure interface.
>
> With this and the previous change, a guest that is aware of the SMMUv3
> secure extensions can probe the device's capabilities and perform basic
> configuration of the secure interface, as is done by secure partition
> managers like Hafnium in its smmuv3_driver_init function.
>
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmuv3.c | 29 +++++++++++++++++++++++++++++
> hw/arm/trace-events | 1 +
> 2 files changed, 30 insertions(+)
>
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index 619180d204..0ea9d897af 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -920,6 +920,21 @@ static void smmuv3_flush_config(SMMUDevice *sdev)
> g_hash_table_remove(bc->configs, sdev);
> }
>
> +static void smmuv3_invalidate_all_caches(SMMUv3State *s)
> +{
> + trace_smmuv3_invalidate_all_caches();
> + SMMUState *bs = &s->smmu_state;
> +
> + /* Clear all cached configs including STE and CD*/
> + if (bs->configs) {
> + g_hash_table_remove_all(bs->configs);
> + }
> +
> + /* Invalidate all SMMU IOTLB entries */
> + smmu_inv_notifiers_all(&s->smmu_state);
> + smmu_iotlb_inv_all(bs);
> +}
> +
> /* Do translation with TLB lookup. */
> static SMMUTranslationStatus smmuv3_do_translate(SMMUv3State *s, hwaddr addr,
> SMMUTransCfg *cfg,
> @@ -1921,6 +1936,16 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
> SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
> s->secure_eventq_irq_cfg2 = data;
> return MEMTX_OK;
> + case A_S_INIT:
> + SMMU_CHECK_SECURE_WRITE("S_INIT");
> + if (data & R_S_INIT_INV_ALL_MASK) {
> + /* write S_INIT and poll*/
> + s->secure_init = data & R_S_INIT_INV_ALL_MASK;
> + smmuv3_invalidate_all_caches(s);
Do we need to check that the SMMU is enabled as the spec says?
> + }
> + /* initialization is completed and set to 0 to terminate the polling */
> + s->secure_init = 0;
All access to SMMU registers are serialised, so it’s safe to drop this and
just return zero on reads.
Thanks,
Mostafa
> + return MEMTX_OK;
> default:
> qemu_log_mask(LOG_UNIMP,
> "%s Unexpected 32-bit access to 0x%"PRIx64" (WI)\n",
> @@ -2249,6 +2274,10 @@ static MemTxResult smmu_readl(SMMUv3State *s, hwaddr offset,
> }
> *data = s->secure_eventq.cons;
> return MEMTX_OK;
> + case A_S_INIT:
> + SMMU_CHECK_SECURE_READ("S_INIT");
> + *data = s->secure_init;
> + return MEMTX_OK;
> default:
> *data = 0;
> qemu_log_mask(LOG_UNIMP,
> diff --git a/hw/arm/trace-events b/hw/arm/trace-events
> index f3386bd7ae..019129ea43 100644
> --- a/hw/arm/trace-events
> +++ b/hw/arm/trace-events
> @@ -64,6 +64,7 @@ smmuv3_cmdq_tlbi_s12_vmid(int vmid) "vmid=%d"
> smmuv3_notify_flag_add(const char *iommu) "ADD SMMUNotifier node for iommu mr=%s"
> smmuv3_notify_flag_del(const char *iommu) "DEL SMMUNotifier node for iommu mr=%s"
> smmuv3_inv_notifiers_iova(const char *name, int asid, int vmid, uint64_t iova, uint8_t tg, uint64_t num_pages, int stage) "iommu mr=%s asid=%d vmid=%d iova=0x%"PRIx64" tg=%d num_pages=0x%"PRIx64" stage=%d"
> +smmuv3_invalidate_all_caches(void) "Invalidate all SMMU caches and TLBs"
> smmu_reset_exit(void) ""
>
> # strongarm.c
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions
2025-08-06 15:11 ` [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions Tao Tang
@ 2025-08-18 21:28 ` Mostafa Saleh
2025-08-20 16:25 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-18 21:28 UTC (permalink / raw)
To: Tao Tang; +Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi
On Wed, Aug 06, 2025 at 11:11:29PM +0800, Tao Tang wrote:
> To support parallel processing of secure and non-secure streams, the
> SMMUv3 model needs to differentiate between the two contexts throughout
> its core logic. This commit is the foundational step to make the code
> security-state aware.
>
> An is_secure flag, which will be used in subsequent patches to represent
> the transaction's security state, is now plumbed through the main
> processing paths.
>
> This change is purely preparatory and introduces no functional changes
> for the existing non-secure path. All current call sites are updated
> to pass is_secure = false.
>
> This refactoring paves the way for upcoming patches that will introduce
> separate TLB entries for secure transactions and enable a fully
> parallel secure/non-secure SMMU model.
>
I think it’s easier to review if this patch was split (STE parsing,
page table handling and translation, TLB invalidation)
Also based on my comment on patch 2, stage-2 handling doesn’t seem correct to me.
Thanks,
Mostafa
> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> ---
> hw/arm/smmu-common.c | 11 ++-
> hw/arm/smmuv3-internal.h | 16 +++-
> hw/arm/smmuv3.c | 160 ++++++++++++++++++++++++-----------
> hw/arm/trace-events | 2 +-
> include/hw/arm/smmu-common.h | 1 +
> include/hw/arm/smmuv3.h | 4 +
> 6 files changed, 138 insertions(+), 56 deletions(-)
>
> diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
> index 0dcaf2f589..28d6d1bc7f 100644
> --- a/hw/arm/smmu-common.c
> +++ b/hw/arm/smmu-common.c
> @@ -332,13 +332,16 @@ void smmu_iotlb_inv_vmid_s1(SMMUState *s, int vmid)
> * @base_addr[@index]
> */
> static int get_pte(dma_addr_t baseaddr, uint32_t index, uint64_t *pte,
> - SMMUPTWEventInfo *info)
> + SMMUPTWEventInfo *info, bool is_secure)
> {
> int ret;
> dma_addr_t addr = baseaddr + index * sizeof(*pte);
> + MemTxAttrs attrs = is_secure ?
> + (MemTxAttrs) { .secure = 1 } :
> + (MemTxAttrs) { .unspecified = true };
>
> /* TODO: guarantee 64-bit single-copy atomicity */
> - ret = ldq_le_dma(&address_space_memory, addr, pte, MEMTXATTRS_UNSPECIFIED);
> + ret = ldq_le_dma(&address_space_memory, addr, pte, attrs);
>
> if (ret != MEMTX_OK) {
> info->type = SMMU_PTW_ERR_WALK_EABT;
> @@ -485,7 +488,7 @@ static int smmu_ptw_64_s1(SMMUState *bs, SMMUTransCfg *cfg,
> dma_addr_t pte_addr = baseaddr + offset * sizeof(pte);
> uint8_t ap;
>
> - if (get_pte(baseaddr, offset, &pte, info)) {
> + if (get_pte(baseaddr, offset, &pte, info, cfg->secure)) {
> goto error;
> }
> trace_smmu_ptw_level(stage, level, iova, subpage_size,
> @@ -621,7 +624,7 @@ static int smmu_ptw_64_s2(SMMUTransCfg *cfg,
> dma_addr_t pte_addr = baseaddr + offset * sizeof(pte);
> uint8_t s2ap;
>
> - if (get_pte(baseaddr, offset, &pte, info)) {
> + if (get_pte(baseaddr, offset, &pte, info, cfg->secure)) {
> goto error;
> }
> trace_smmu_ptw_level(stage, level, ipa, subpage_size,
> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> index 720d21652c..852186cea4 100644
> --- a/hw/arm/smmuv3-internal.h
> +++ b/hw/arm/smmuv3-internal.h
> @@ -243,9 +243,13 @@ REG64(S_EVENTQ_IRQ_CFG0, 0x80b0)
> REG32(S_EVENTQ_IRQ_CFG1, 0x80b8)
> REG32(S_EVENTQ_IRQ_CFG2, 0x80bc)
>
> -static inline int smmu_enabled(SMMUv3State *s)
> +static inline int smmu_enabled(SMMUv3State *s, bool is_secure)
> {
> - return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
> + if (is_secure) {
> + return FIELD_EX32(s->secure_cr[0], S_CR0, SMMUEN);
> + } else {
> + return FIELD_EX32(s->cr[0], CR0, SMMU_ENABLE);
> + }
> }
>
> /* Command Queue Entry */
> @@ -661,6 +665,10 @@ typedef struct CD {
> #define STE_S2S(x) extract32((x)->word[5], 25, 1)
> #define STE_S2R(x) extract32((x)->word[5], 26, 1)
>
> +#define STE_S_S2T0SZ(x) extract32((x)->word[9], 0 , 6)
> +#define STE_S_S2SL0(x) extract32((x)->word[9], 6 , 2)
> +#define STE_S_S2TG(x) extract32((x)->word[9], 14, 2)
> +
> #define STE_CTXPTR(x) \
> ((extract64((x)->word[1], 0, 16) << 32) | \
> ((x)->word[0] & 0xffffffc0))
> @@ -669,6 +677,10 @@ typedef struct CD {
> ((extract64((x)->word[7], 0, 16) << 32) | \
> ((x)->word[6] & 0xfffffff0))
>
> +#define STE_S_S2TTB(x) \
> + ((extract64((x)->word[13], 0, 16) << 32) | \
> + ((x)->word[12] & 0xfffffff0))
> +
> static inline int oas2bits(int oas_field)
> {
> switch (oas_field) {
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index 1f05cc983b..bcf06679e1 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -354,14 +354,14 @@ static void smmuv3_init_regs(SMMUv3State *s)
> }
>
> static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
> - SMMUEventInfo *event)
> + SMMUEventInfo *event, MemTxAttrs attrs)
> {
> int ret, i;
>
> trace_smmuv3_get_ste(addr);
> /* TODO: guarantee 64-bit single-copy atomicity */
> ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
> - MEMTXATTRS_UNSPECIFIED);
> + attrs);
> if (ret != MEMTX_OK) {
> qemu_log_mask(LOG_GUEST_ERROR,
> "Cannot fetch pte at address=0x%"PRIx64"\n", addr);
> @@ -390,6 +390,9 @@ static int smmu_get_cd(SMMUv3State *s, STE *ste, SMMUTransCfg *cfg,
> int ret, i;
> SMMUTranslationStatus status;
> SMMUTLBEntry *entry;
> + MemTxAttrs attrs = cfg->secure ?
> + (MemTxAttrs) { .secure = 1 } :
> + (MemTxAttrs) { .unspecified = true };
>
> trace_smmuv3_get_cd(addr);
>
> @@ -407,7 +410,7 @@ static int smmu_get_cd(SMMUv3State *s, STE *ste, SMMUTransCfg *cfg,
>
> /* TODO: guarantee 64-bit single-copy atomicity */
> ret = dma_memory_read(&address_space_memory, addr, buf, sizeof(*buf),
> - MEMTXATTRS_UNSPECIFIED);
> + attrs);
> if (ret != MEMTX_OK) {
> qemu_log_mask(LOG_GUEST_ERROR,
> "Cannot fetch pte at address=0x%"PRIx64"\n", addr);
> @@ -472,7 +475,8 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
> g_assert_not_reached();
> }
>
> - switch (STE_S2TG(ste)) {
> + uint32_t s2tg = cfg->secure ? STE_S_S2TG(ste) : STE_S2TG(ste);
> + switch (s2tg) {
> case 0x0: /* 4KB */
> cfg->s2cfg.granule_sz = 12;
> break;
> @@ -484,13 +488,13 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
> break;
> default:
> qemu_log_mask(LOG_GUEST_ERROR,
> - "SMMUv3 bad STE S2TG: %x\n", STE_S2TG(ste));
> + "SMMUv3 bad STE S2TG: %x\n", s2tg);
> goto bad_ste;
> }
>
> - cfg->s2cfg.vttb = STE_S2TTB(ste);
> + cfg->s2cfg.vttb = cfg->secure ? STE_S_S2TTB(ste) : STE_S2TTB(ste);
>
> - cfg->s2cfg.sl0 = STE_S2SL0(ste);
> + cfg->s2cfg.sl0 = cfg->secure ? STE_S_S2SL0(ste) : STE_S2SL0(ste);
> /* FEAT_TTST not supported. */
> if (cfg->s2cfg.sl0 == 0x3) {
> qemu_log_mask(LOG_UNIMP, "SMMUv3 S2SL0 = 0x3 has no meaning!\n");
> @@ -519,7 +523,7 @@ static int decode_ste_s2_cfg(SMMUv3State *s, SMMUTransCfg *cfg,
> goto bad_ste;
> }
>
> - cfg->s2cfg.tsz = STE_S2T0SZ(ste);
> + cfg->s2cfg.tsz = cfg->secure ? STE_S_S2T0SZ(ste) : STE_S2T0SZ(ste);
>
> if (!s2t0sz_valid(cfg)) {
> qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 bad STE S2T0SZ = %d\n",
> @@ -599,21 +603,52 @@ static int decode_ste(SMMUv3State *s, SMMUTransCfg *cfg,
> if (cfg->aborted || cfg->bypassed) {
> return 0;
> }
> + bool is_secure = cfg->secure;
>
> /*
> * If a stage is enabled in SW while not advertised, throw bad ste
> * according to user manual(IHI0070E) "5.2 Stream Table Entry".
> */
> - if (!STAGE1_SUPPORTED(s) && STE_CFG_S1_ENABLED(config)) {
> - qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 S1 used but not supported.\n");
> - goto bad_ste;
> - }
> - if (!STAGE2_SUPPORTED(s) && STE_CFG_S2_ENABLED(config)) {
> - qemu_log_mask(LOG_GUEST_ERROR, "SMMUv3 S2 used but not supported.\n");
> - goto bad_ste;
> + if (!is_secure) {
> + if (!STAGE1_SUPPORTED(s) && STE_CFG_S1_ENABLED(config)) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "SMMUv3 S1 used but not supported.\n");
> + goto bad_ste;
> + }
> + if (!STAGE2_SUPPORTED(s) && STE_CFG_S2_ENABLED(config)) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "SMMUv3 S2 used but not supported.\n");
> + goto bad_ste;
> + }
> + } else {
> + /*
> + * As described in user manual(IHI0070G.b) "3.10.2 Support for Secure
> + * state" , the SMMU supports stage 1 translation and might support
> + * stage 2 translation.
> + */
> + if (!SECURE_IMPLEMENTED(s) && STE_CFG_S1_ENABLED(config)) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "SMMUv3 S1 used but not supported in secure state.\n");
> + goto bad_ste;
> + }
> +
> + /*
> + * IHI0070G.b "6.3.53 SMMU_S_IDR1": SEL2 == 0 if SMMU_IDR0.S1P == 0 or
> + * if SMMU_IDR0.S2P == 0.
> + */
> + if (STE_CFG_S2_ENABLED(config)) {
> + if (!SECURE_S2_SUPPORTED(s) ||
> + (SECURE_S2_SUPPORTED(s) &&
> + (!STAGE1_SUPPORTED(s) || !STAGE2_SUPPORTED(s)))) {
> + qemu_log_mask(LOG_GUEST_ERROR,
> + "SMMUv3 S2 used but not supported in secure state.\n");
> + goto bad_ste;
> + }
> + }
> }
>
> - if (STAGE2_SUPPORTED(s)) {
> + if ((!is_secure && STAGE2_SUPPORTED(s)) ||
> + (is_secure && SECURE_S2_SUPPORTED(s))) {
> /* VMID is considered even if s2 is disabled. */
> cfg->s2cfg.vmid = STE_S2VMID(ste);
> } else {
> @@ -659,20 +694,29 @@ bad_ste:
> * @sid: stream ID
> * @ste: returned stream table entry
> * @event: handle to an event info
> + * @is_secure: true if the translation is for a secure domain
> *
> * Supports linear and 2-level stream table
> * Return 0 on success, -EINVAL otherwise
> */
> static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
> - SMMUEventInfo *event)
> + SMMUEventInfo *event, bool is_secure)
> {
> - dma_addr_t addr, strtab_base;
> + dma_addr_t addr;
> uint32_t log2size;
> int strtab_size_shift;
> int ret;
> + uint32_t features = is_secure ? s->secure_features : s->features;
> + dma_addr_t strtab_base = is_secure ? s->secure_strtab_base : s->strtab_base;
> + uint8_t sid_split = is_secure ? s->secure_sid_split : s->sid_split;
> + MemTxAttrs attrs = is_secure ?
> + (MemTxAttrs) { .secure = 1 } :
> + (MemTxAttrs) { .unspecified = true };
>
> - trace_smmuv3_find_ste(sid, s->features, s->sid_split);
> - log2size = FIELD_EX32(s->strtab_base_cfg, STRTAB_BASE_CFG, LOG2SIZE);
> + trace_smmuv3_find_ste(sid, features, sid_split, is_secure);
> + log2size = is_secure
> + ? FIELD_EX32(s->secure_strtab_base_cfg, S_STRTAB_BASE_CFG, LOG2SIZE)
> + : FIELD_EX32(s->strtab_base_cfg, STRTAB_BASE_CFG, LOG2SIZE);
> /*
> * Check SID range against both guest-configured and implementation limits
> */
> @@ -680,7 +724,7 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
> event->type = SMMU_EVT_C_BAD_STREAMID;
> return -EINVAL;
> }
> - if (s->features & SMMU_FEATURE_2LVL_STE) {
> + if (features & SMMU_FEATURE_2LVL_STE) {
> int l1_ste_offset, l2_ste_offset, max_l2_ste, span, i;
> dma_addr_t l1ptr, l2ptr;
> STEDesc l1std;
> @@ -689,15 +733,15 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
> * Align strtab base address to table size. For this purpose, assume it
> * is not bounded by SMMU_IDR1_SIDSIZE.
> */
> - strtab_size_shift = MAX(5, (int)log2size - s->sid_split - 1 + 3);
> - strtab_base = s->strtab_base & SMMU_BASE_ADDR_MASK &
> + strtab_size_shift = MAX(5, (int)log2size - sid_split - 1 + 3);
> + strtab_base = strtab_base & SMMU_BASE_ADDR_MASK &
> ~MAKE_64BIT_MASK(0, strtab_size_shift);
> - l1_ste_offset = sid >> s->sid_split;
> - l2_ste_offset = sid & ((1 << s->sid_split) - 1);
> + l1_ste_offset = sid >> sid_split;
> + l2_ste_offset = sid & ((1 << sid_split) - 1);
> l1ptr = (dma_addr_t)(strtab_base + l1_ste_offset * sizeof(l1std));
> /* TODO: guarantee 64-bit single-copy atomicity */
> ret = dma_memory_read(&address_space_memory, l1ptr, &l1std,
> - sizeof(l1std), MEMTXATTRS_UNSPECIFIED);
> + sizeof(l1std), attrs);
> if (ret != MEMTX_OK) {
> qemu_log_mask(LOG_GUEST_ERROR,
> "Could not read L1PTR at 0X%"PRIx64"\n", l1ptr);
> @@ -722,7 +766,7 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
> }
> max_l2_ste = (1 << span) - 1;
> l2ptr = l1std_l2ptr(&l1std);
> - trace_smmuv3_find_ste_2lvl(s->strtab_base, l1ptr, l1_ste_offset,
> + trace_smmuv3_find_ste_2lvl(strtab_base, l1ptr, l1_ste_offset,
> l2ptr, l2_ste_offset, max_l2_ste);
> if (l2_ste_offset > max_l2_ste) {
> qemu_log_mask(LOG_GUEST_ERROR,
> @@ -734,12 +778,12 @@ static int smmu_find_ste(SMMUv3State *s, uint32_t sid, STE *ste,
> addr = l2ptr + l2_ste_offset * sizeof(*ste);
> } else {
> strtab_size_shift = log2size + 5;
> - strtab_base = s->strtab_base & SMMU_BASE_ADDR_MASK &
> + strtab_base = strtab_base & SMMU_BASE_ADDR_MASK &
> ~MAKE_64BIT_MASK(0, strtab_size_shift);
> addr = strtab_base + sid * sizeof(*ste);
> }
>
> - if (smmu_get_ste(s, addr, ste, event)) {
> + if (smmu_get_ste(s, addr, ste, event, attrs)) {
> return -EINVAL;
> }
>
> @@ -868,7 +912,7 @@ static int smmuv3_decode_config(IOMMUMemoryRegion *mr, SMMUTransCfg *cfg,
> /* ASID defaults to -1 (if s1 is not supported). */
> cfg->asid = -1;
>
> - ret = smmu_find_ste(s, sid, &ste, event);
> + ret = smmu_find_ste(s, sid, &ste, event, cfg->secure);
> if (ret) {
> return ret;
> }
> @@ -897,12 +941,14 @@ static int smmuv3_decode_config(IOMMUMemoryRegion *mr, SMMUTransCfg *cfg,
> *
> * @sdev: SMMUDevice handle
> * @event: output event info
> + * @is_secure: true if the translation is for a secure domain
> *
> * The configuration cache contains data resulting from both STE and CD
> * decoding under the form of an SMMUTransCfg struct. The hash table is indexed
> * by the SMMUDevice handle.
> */
> -static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event)
> +static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event,
> + bool is_secure)
> {
> SMMUv3State *s = sdev->smmu;
> SMMUState *bc = &s->smmu_state;
> @@ -922,6 +968,7 @@ static SMMUTransCfg *smmuv3_get_config(SMMUDevice *sdev, SMMUEventInfo *event)
> 100 * sdev->cfg_cache_hits /
> (sdev->cfg_cache_hits + sdev->cfg_cache_misses));
> cfg = g_new0(SMMUTransCfg, 1);
> + cfg->secure = is_secure;
>
> if (!smmuv3_decode_config(&sdev->iommu, cfg, event)) {
> g_hash_table_insert(bc->configs, sdev, cfg);
> @@ -1103,19 +1150,25 @@ static IOMMUTLBEntry smmuv3_translate(IOMMUMemoryRegion *mr, hwaddr addr,
> .perm = IOMMU_NONE,
> };
> SMMUTLBEntry *cached_entry = NULL;
> + /* We don't support secure translation for now */
> + bool is_secure = false;
>
> qemu_mutex_lock(&s->mutex);
>
> - if (!smmu_enabled(s)) {
> - if (FIELD_EX32(s->gbpa, GBPA, ABORT)) {
> - status = SMMU_TRANS_ABORT;
> + if (!smmu_enabled(s, is_secure)) {
> + bool abort_flag;
> +
> + if (is_secure) {
> + abort_flag = FIELD_EX32(s->secure_gbpa, S_GBPA, ABORT);
> } else {
> - status = SMMU_TRANS_DISABLE;
> + abort_flag = FIELD_EX32(s->gbpa, GBPA, ABORT);
> }
> +
> + status = abort_flag ? SMMU_TRANS_ABORT : SMMU_TRANS_DISABLE;
> goto epilogue;
> }
>
> - cfg = smmuv3_get_config(sdev, &event);
> + cfg = smmuv3_get_config(sdev, &event, is_secure);
> if (!cfg) {
> status = SMMU_TRANS_ERROR;
> goto epilogue;
> @@ -1167,7 +1220,7 @@ epilogue:
> qemu_log_mask(LOG_GUEST_ERROR,
> "%s translation failed for iova=0x%"PRIx64" (%s)\n",
> mr->parent_obj.name, addr, smmu_event_string(event.type));
> - smmuv3_record_event(s, &event, false);
> + smmuv3_record_event(s, &event, cfg->secure);
> break;
> }
>
> @@ -1186,16 +1239,18 @@ epilogue:
> * @tg: translation granule (if communicated through range invalidation)
> * @num_pages: number of @granule sized pages (if tg != 0), otherwise 1
> * @stage: Which stage(1 or 2) is used
> + * @is_secure: true if the translation is for a secure domain
> */
> static void smmuv3_notify_iova(IOMMUMemoryRegion *mr,
> IOMMUNotifier *n,
> int asid, int vmid,
> dma_addr_t iova, uint8_t tg,
> - uint64_t num_pages, int stage)
> + uint64_t num_pages, int stage,
> + bool is_secure)
> {
> SMMUDevice *sdev = container_of(mr, SMMUDevice, iommu);
> SMMUEventInfo eventinfo = {.inval_ste_allowed = true};
> - SMMUTransCfg *cfg = smmuv3_get_config(sdev, &eventinfo);
> + SMMUTransCfg *cfg = smmuv3_get_config(sdev, &eventinfo, is_secure);
> IOMMUTLBEvent event;
> uint8_t granule;
>
> @@ -1251,7 +1306,8 @@ static void smmuv3_notify_iova(IOMMUMemoryRegion *mr,
> /* invalidate an asid/vmid/iova range tuple in all mr's */
> static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, int vmid,
> dma_addr_t iova, uint8_t tg,
> - uint64_t num_pages, int stage)
> + uint64_t num_pages, int stage,
> + bool is_secure)
> {
> SMMUDevice *sdev;
>
> @@ -1263,12 +1319,14 @@ static void smmuv3_inv_notifiers_iova(SMMUState *s, int asid, int vmid,
> iova, tg, num_pages, stage);
>
> IOMMU_NOTIFIER_FOREACH(n, mr) {
> - smmuv3_notify_iova(mr, n, asid, vmid, iova, tg, num_pages, stage);
> + smmuv3_notify_iova(mr, n, asid, vmid, iova, tg,
> + num_pages, stage, is_secure);
> }
> }
> }
>
> -static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> +static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage,
> + bool is_secure)
> {
> dma_addr_t end, addr = CMD_ADDR(cmd);
> uint8_t type = CMD_TYPE(cmd);
> @@ -1284,7 +1342,8 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> SMMUv3State *smmuv3 = ARM_SMMUV3(s);
>
> /* Only consider VMID if stage-2 is supported. */
> - if (STAGE2_SUPPORTED(smmuv3)) {
> + if (STAGE2_SUPPORTED(smmuv3) ||
> + (SECURE_IMPLEMENTED(smmuv3) && SECURE_S2_SUPPORTED(smmuv3))) {
> vmid = CMD_VMID(cmd);
> }
>
> @@ -1294,7 +1353,7 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
>
> if (!tg) {
> trace_smmuv3_range_inval(vmid, asid, addr, tg, 1, ttl, leaf, stage);
> - smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, 1, stage);
> + smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, 1, stage, is_secure);
> if (stage == SMMU_STAGE_1) {
> smmu_iotlb_inv_iova(s, asid, vmid, addr, tg, 1, ttl);
> } else {
> @@ -1317,7 +1376,8 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> num_pages = (mask + 1) >> granule;
> trace_smmuv3_range_inval(vmid, asid, addr, tg, num_pages,
> ttl, leaf, stage);
> - smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg, num_pages, stage);
> + smmuv3_inv_notifiers_iova(s, asid, vmid, addr, tg,
> + num_pages, stage, is_secure);
> if (stage == SMMU_STAGE_1) {
> smmu_iotlb_inv_iova(s, asid, vmid, addr, tg, num_pages, ttl);
> } else {
> @@ -1461,7 +1521,8 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
> * VMID is only matched when stage 2 is supported, otherwise set it
> * to -1 as the value used for stage-1 only VMIDs.
> */
> - if (STAGE2_SUPPORTED(s)) {
> + if (STAGE2_SUPPORTED(s) ||
> + (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
> vmid = CMD_VMID(&cmd);
> }
>
> @@ -1483,7 +1544,8 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
> * If stage-2 is supported, invalidate for this VMID only, otherwise
> * invalidate the whole thing.
> */
> - if (STAGE2_SUPPORTED(s)) {
> + if (STAGE2_SUPPORTED(s) ||
> + (SECURE_IMPLEMENTED(s) && SECURE_S2_SUPPORTED(s))) {
> vmid = CMD_VMID(&cmd);
> trace_smmuv3_cmdq_tlbi_nh(vmid);
> smmu_iotlb_inv_vmid_s1(bs, vmid);
> @@ -1502,7 +1564,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
> cmd_error = SMMU_CERROR_ILL;
> break;
> }
> - smmuv3_range_inval(bs, &cmd, SMMU_STAGE_1);
> + smmuv3_range_inval(bs, &cmd, SMMU_STAGE_1, false);
> break;
> case SMMU_CMD_TLBI_S12_VMALL:
> {
> @@ -1527,7 +1589,7 @@ static int smmuv3_cmdq_consume(SMMUv3State *s, bool is_secure)
> * As currently only either s1 or s2 are supported
> * we can reuse same function for s2.
> */
> - smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2);
> + smmuv3_range_inval(bs, &cmd, SMMU_STAGE_2, false);
> break;
> case SMMU_CMD_TLBI_EL3_ALL:
> case SMMU_CMD_TLBI_EL3_VA:
> diff --git a/hw/arm/trace-events b/hw/arm/trace-events
> index 7bb0bd0cc5..92c87f0b9e 100644
> --- a/hw/arm/trace-events
> +++ b/hw/arm/trace-events
> @@ -41,7 +41,7 @@ smmuv3_cmdq_consume_out(uint32_t prod, uint32_t cons, uint8_t prod_wrap, uint8_t
> smmuv3_cmdq_consume_error(const char *cmd_name, uint8_t cmd_error) "Error on %s command execution: %d"
> smmuv3_write_mmio(uint64_t addr, uint64_t val, unsigned size, uint32_t r) "addr: 0x%"PRIx64" val:0x%"PRIx64" size: 0x%x(%d)"
> smmuv3_record_event(const char *type, uint32_t sid, bool is_secure) "%s sid=0x%x is_secure=%d"
> -smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split) "sid=0x%x features:0x%x, sid_split:0x%x"
> +smmuv3_find_ste(uint16_t sid, uint32_t features, uint16_t sid_split, bool is_secure) "sid=0x%x features:0x%x, sid_split:0x%x is_secure=%d"
> smmuv3_find_ste_2lvl(uint64_t strtab_base, uint64_t l1ptr, int l1_ste_offset, uint64_t l2ptr, int l2_ste_offset, int max_l2_ste) "strtab_base:0x%"PRIx64" l1ptr:0x%"PRIx64" l1_off:0x%x, l2ptr:0x%"PRIx64" l2_off:0x%x max_l2_ste:%d"
> smmuv3_get_ste(uint64_t addr) "STE addr: 0x%"PRIx64
> smmuv3_translate_disable(const char *n, uint16_t sid, uint64_t addr, bool is_write) "%s sid=0x%x bypass (smmu disabled) iova:0x%"PRIx64" is_write=%d"
> diff --git a/include/hw/arm/smmu-common.h b/include/hw/arm/smmu-common.h
> index e5e2d09294..5d15a1212b 100644
> --- a/include/hw/arm/smmu-common.h
> +++ b/include/hw/arm/smmu-common.h
> @@ -116,6 +116,7 @@ typedef struct SMMUTransCfg {
> SMMUTransTableInfo tt[2];
> /* Used by stage-2 only. */
> struct SMMUS2Cfg s2cfg;
> + bool secure;
> } SMMUTransCfg;
>
> typedef struct SMMUDevice {
> diff --git a/include/hw/arm/smmuv3.h b/include/hw/arm/smmuv3.h
> index 72ad042514..43c7289a43 100644
> --- a/include/hw/arm/smmuv3.h
> +++ b/include/hw/arm/smmuv3.h
> @@ -110,4 +110,8 @@ OBJECT_DECLARE_TYPE(SMMUv3State, SMMUv3Class, ARM_SMMUV3)
> #define STAGE1_SUPPORTED(s) FIELD_EX32(s->idr[0], IDR0, S1P)
> #define STAGE2_SUPPORTED(s) FIELD_EX32(s->idr[0], IDR0, S2P)
>
> +#define SECURE_IMPLEMENTED(s) \
> + FIELD_DP32(s->secure_idr[1], S_IDR1, SECURE_IMPL, 1)
> +#define SECURE_S2_SUPPORTED(s) \
> + FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1)
> #endif
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
` (11 preceding siblings ...)
2025-08-06 21:28 ` Pierrick Bouvier
@ 2025-08-18 21:52 ` Mostafa Saleh
12 siblings, 0 replies; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-18 21:52 UTC (permalink / raw)
To: Tao Tang; +Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi
On Wed, Aug 06, 2025 at 11:11:23PM +0800, Tao Tang wrote:
> Hi all,
>
> This patch series introduces initial support for emulating the Arm SMMUv3
> Secure State.
>
> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
> emulation is a notable missing piece in QEMU. While the FVP model has
> some support, its limited PCIe capabilities make it challenging for
> complex use cases. The ability to properly manage device DMA from a
> secure context is a critical prerequisite for enabling device assignment
> (passthrough) for confidential computing solutions like Arm CCA and
> related research such as virtCCA [2]. This series aims to build that
> foundational support in QEMU.
>
> This work is being proposed as an RFC. It introduces a significant amount
> of new logic, including the core concept of modeling parallel secure and
> non-secure contexts within a single SMMUv3 device. I am seeking feedback
> on the overall approach, the core refactoring, and the implementation
> details before proceeding further.
>
> The series begins by implementing the components of the secure programming
> interface, then progressively refactors the core SMMU logic to handle
> secure and non-secure contexts in parallel.
>
> Secure Interface Implementation: The initial patches add the
> secure-side registers, implement their read/write logic, and enable
> the secure command and event queues. This includes the S_INIT
> mechanism and the new secure TLB invalidation commands.
>
> Core Logic Refactoring: The next set of patches makes the core SMMU
> functions security-state aware. This involves plumbing an is_secure
> context flag through the main code paths and adding logic to route
> SMMU-originated memory accesses to the correct (Secure or Non-secure)
> address space.
>
> Cache Isolation: With the core logic now aware of security states,
> the following patches refactor the configuration and translation
> lookup caches. The cache keys are modified to include the security
> context, ensuring that secure and non-secure entries for the same
> device or address are properly isolated and preventing aliasing.
>
> Framework Integration: The final patch connects the SMMU's internal
> security context to the generic QEMU IOMMU framework by using the
> iommu_index to represent the architectural SEC_SID.
>
> To validate this work, I performed the following tests:
>
> Non-Secure Regression: To ensure that existing functionality remains
> intact, I ran a nested virtualization test. A TCG guest was created on
> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned.
> Command line of TCG VM is below:
>
> qemu-system-aarch64 \
> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
> -cpu max -smp 1 -m 4080M \
> -accel tcg,thread=single,tb-size=512 \
> -kernel Image \
> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \
> -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
> -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
> -device virtio-blk-device,drive=hd0 \
> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
> -device virtio-net-device,netdev=eth0 \
> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
> -device nvme,drive=nvme0,serial=deadbeef \
> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic
>
> Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
> re-assigned to it via VFIO.
> Command line of KVM VM inside TCG VM is below:
>
> sudo qemu-system-aarch64 \
> -enable-kvm -m 1024 -cpu host -M virt \
> -machine virt,gic-version=3 \
> -cpu max -append "nokaslr" -smp 1 \
> -monitor stdio \
> -kernel 5.15.Image \
> -initrd rootfs.cpio.gz \
> -display vnc=:22,id=primary \
> -device vfio-pci,host=00:01.0
>
> The KVM guest was able to perform I/O on the device
> correctly, confirming that the non-secure path is not broken.
I gave the patches a quick test and they seem to have broken my
nested setup, I will look more into it and let you know what I find.
Thanks,
Mostafa
>
> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
> environment. Hafnium's smmuv3_driver_init function was used to test
> the secure register I/O and command queue functionality (excluding
> translation). As Hafnium assumes larger queue and StreamID sizes than
> are practical without TTST support, I temporarily patched Hafnium to
> use smaller values, allowing its driver to initialize the emulated
> secure SMMU successfully.
>
> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
> support QEMU, and no secure device assignment feature exists yet, I
> created a custom platform device to test the secure translation flow.
> To trigger the translation logic, I initiated MMIO writes to this
> device from within Hafnium. The device's MMIO callback handler then
> performed DMA accesses via its IOMMU region, exercising the secure
> translation path. While SMMUv3 is typically used for PCIe on
> physical SoCs, the architecture allows its use with platform devices
> via a stream-id binding in the device tree. The test harness
> required some non-standard modifications to decouple the SMMU from
> its tight integration with PCIe. The code for this test device is
> available for review at [3]. README.md with detailed instructions is
> also provided.
>
> I've attempted to follow all of the guidance in the "Submitting a Patch"
> guide, but as this is my first series of this scale, I apologize if I
> missed anything and welcome all feedback.
>
> Thanks,
> Tang
>
> [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
> [2] https://arxiv.org/abs/2306.11011
> [3] https://github.com/hnusdr/qemu
>
> Tao Tang (11):
> hw/arm/smmuv3: Introduce secure registers and commands
> hw/arm/smmuv3: Implement read/write logic for secure registers
> hw/arm/smmuv3: Implement S_INIT for secure initialization
> hw/arm/smmuv3: Enable command processing for the Secure state
> hw/arm/smmuv3: Support secure event queue and error handling
> hw/arm/smmuv3: Plumb security state through core functions
> hw/arm/smmuv3: Add separate address space for secure SMMU accesses
> hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
> hw/arm/smmuv3: Make the configuration cache security-state aware
> hw/arm/smmuv3: Differentiate secure TLB entries via keying
> hw/arm/smmuv3: Use iommu_index to represent SEC_SID
>
> hw/arm/smmu-common.c | 74 ++-
> hw/arm/smmuv3-internal.h | 128 +++++-
> hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++-----
> hw/arm/trace-events | 7 +-
> hw/arm/virt.c | 5 +
> include/hw/arm/smmu-common.h | 23 +-
> include/hw/arm/smmuv3.h | 27 ++
> 7 files changed, 968 insertions(+), 140 deletions(-)
>
> --
> 2.34.1
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-18 21:24 ` Mostafa Saleh
@ 2025-08-20 15:21 ` Tao Tang
2025-08-23 10:41 ` Mostafa Saleh
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-20 15:21 UTC (permalink / raw)
To: Mostafa Saleh
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
Pierrick Bouvier, Philippe Mathieu-Daudé, jean-philippe
On 2025/8/19 05:24, Mostafa Saleh wrote:
> On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
>> This patch builds upon the previous introduction of secure register
>> definitions by providing the functional implementation for their access.
>>
>> The availability of the secure programming interface is now correctly
>> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
>> secure functionality is enabled, the I/O handlers (smmuv3_read and
>> smmuv3_write) will correctly dispatch accesses to the secure
>> register space.
>>
>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>> ---
>> hw/arm/smmuv3-internal.h | 5 +
>> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 456 insertions(+)
>>
>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>> index 483aaa915e..1a8b1cb204 100644
>> --- a/hw/arm/smmuv3-internal.h
>> +++ b/hw/arm/smmuv3-internal.h
>> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>>
>> #define SMMU_CR0_RESERVED 0xFFFFFC20
>>
>> +/*
>> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
>> + */
>> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
>> +
>> REG32(CR0ACK, 0x24)
>> REG32(CR1, 0x28)
>> REG32(CR2, 0x2c)
>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>> index ab67972353..619180d204 100644
>> --- a/hw/arm/smmuv3.c
>> +++ b/hw/arm/smmuv3.c
>> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
>> s->gerrorn = 0;
>> s->statusr = 0;
>> s->gbpa = SMMU_GBPA_RESET_VAL;
>> +
>> + /* Initialize secure state */
>> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
>> + /* Secure EL2 and Secure stage 2 support */
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
> AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
> one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
> Which is not implemented in this series.
Hi Mostafa,
Thank you for the very detailed and helpful review. Your feedback is
spot on, and I'd like to address your points and ask for a quick
confirmation on them.
Regarding the SEL2 bit, you are absolutely right, my understanding was
incorrect. I've spent the last few days reviewing the manual to better
understand the selection between Secure and Non-secure Stage 2
translations. I would be very grateful if you could confirm if my new
understanding is correct:
- In Stage 2-only mode (Stage 1 bypassed), the choice between a Secure
or Non-secure IPA translation is determined solely by STE.NSCFG.
- In Stage 1-enabled mode, STE.NSCFG is ignored. The choice is
determined by the translation process, starting from CD.NSCFGx, with the
output NS attribute being the result of intermediate NSTable flags and
the final descriptor.NS bit (TTD.NSTable, TTD.NS).
Based on this, I plan to have an internal flag, perhaps named
target_ipa_is_ns in SMMUTransCfg.SMMUS2Cfg struct, to track the outcome
of this process. This flag will then determine whether S_S2TTB or S2TTB
is used for the Stage 2 translation.
>
>> + /* Secure state implemented */
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
>> + SECURE_IMPL, 1);
>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
>> + S_SIDSIZE, SMMU_IDR1_SIDSIZE);
>> +
>> + s->secure_gbpa = SMMU_GBPA_RESET_VAL;
>> }
>>
>> static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
>> @@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
>> }
>> }
>>
>> +/* Check if the SMMU hardware itself implements secure state features */
>> +static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
>> +{
>> + return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
>> +}
>> +
> I see that the secure SMMU support is unconditional. So, is this always true?
> Also, how that looks with migration?
For the v2 series, my plan is to make SECURE_IMPL a user-configurable
device property. I will add a "secure-enabled" property to
smmuv3_properties and ensure all necessary states are added to the
VMStateDescription to handle migration correctly. Does this approach
sound reasonable to you?
>
>> static int smmuv3_cmdq_consume(SMMUv3State *s)
>> {
>> SMMUState *bs = ARM_SMMU(s);
>> @@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
>> return 0;
>> }
>>
>> +/* Helper function for secure register write validation */
>> +static bool smmu_validate_secure_write(MemTxAttrs attrs, bool secure_impl,
>> + hwaddr offset, const char *reg_name)
>> +{
>> + if (!attrs.secure || !secure_impl) {
>> + const char *reason = !attrs.secure ?
>> + "Non-secure write attempt" :
>> + "SMMU didn't implement Security State";
>> + qemu_log_mask(LOG_GUEST_ERROR,
>> + "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
>> + __func__, reason, offset, reg_name);
>> + return false;
>> + }
>> + return true;
>> +}
>> +
>> +/* Helper function for secure register read validation */
>> +static bool smmu_validate_secure_read(MemTxAttrs attrs, bool secure_impl,
>> + hwaddr offset, const char *reg_name,
>> + uint64_t *data)
>> +{
>> + if (!attrs.secure || !secure_impl) {
>> + const char *reason = !attrs.secure ?
>> + "Non-secure read attempt" :
>> + "SMMU didn't implement Security State";
>> + qemu_log_mask(LOG_GUEST_ERROR,
>> + "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
>> + __func__, reason, offset, reg_name);
>> + *data = 0; /* RAZ */
>> + return false;
>> + }
>> + return true;
>> +}
>> +
>> +/* Macro for secure write validation - returns early if validation fails */
>> +#define SMMU_CHECK_SECURE_WRITE(reg_name) \
>> + do { \
>> + if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
>> + reg_name)) { \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>> +/* Macro for attrs.secure only validation */
>> +#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
>> + do { \
>> + if (!attrs.secure) { \
>> + qemu_log_mask(LOG_GUEST_ERROR, \
>> + "%s: Non-secure write attempt at offset " \
>> + "0x%" PRIx64 " (%s, WI)\n", \
>> + __func__, offset, reg_name); \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>> +/* Macro for secure read validation - returns RAZ if validation fails */
>> +#define SMMU_CHECK_SECURE_READ(reg_name) \
>> + do { \
>> + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
>> + reg_name, data)) { \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>> +/* Macro for attrs.secure only validation (read) */
>> +#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
>> + do { \
>> + if (!attrs.secure) { \
>> + qemu_log_mask(LOG_GUEST_ERROR, \
>> + "%s: Non-secure read attempt at offset " \
>> + "0x%" PRIx64 " (%s, RAZ)\n", \
>> + __func__, offset, reg_name); \
>> + *data = 0; \
>> + return MEMTX_OK; \
>> + } \
>> + } while (0)
>> +
>>
> Can’t we just have one check? If the access > SMMU_SECURE_BASE_OFFSET, just
> check the security state?
>
> And then based on banking, many of those switches will be common with
> non secure cases.
>
> Thanks,
> Mostafa
I have already refactored this part in my v2 series, exactly as you
proposed. This also addresses your earlier feedback on patch #1
regarding the overall structure:
> As Philippe mentioned, this would be better the secure state is separated
> in another instance of the struct, that seems it would reduce a lot of the
> duplication later around the logic of MMIO and queues... in the next
> patches.
The new code performs a single, necessary security state check at the
entry point of the MMIO handlers. The rest of the logic relies on the
banking mechanism, which makes the implementation generic for
Non-secure, Secure, and future states like Realm/Root. The new structure
looks like this:
/* Structure for one register bank */
typedef struct SMMUv3Bank {
uint32_t idr[6]; /* IDR0-IDR5, note: IDR5 only used for NS bank */
uint32_t cr[3]; /* CR0-CR2 */
uint32_t cr0ack;
uint32_t init; /* S_INIT register (secure only), reserved for
NS */
uint32_t gbpa;
......
SMMUQueue eventq, cmdq;
} SMMUv3Bank;
struct SMMUv3State {
SMMUState smmu_state;
/* Shared (non-banked) registers and state */
uint32_t features;
uint8_t sid_size;
uint8_t sid_split;
......
/* Banked registers for all access */
SMMUv3Bank bank[SMMU_SEC_IDX_NUM];
......
};
Thanks again for your valuable feedback. I've outlined my proposed plan
above and would be grateful for any thoughts on it to ensure I'm on the
right track for v2.
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization
2025-08-18 21:26 ` Mostafa Saleh
@ 2025-08-20 16:01 ` Tao Tang
0 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-08-20 16:01 UTC (permalink / raw)
To: Mostafa Saleh
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
jean-philippe, Philippe Mathieu-Daudé, Pierrick Bouvier
On 2025/8/19 05:26, Mostafa Saleh wrote:
> On Wed, Aug 06, 2025 at 11:11:26PM +0800, Tao Tang wrote:
>> This patch implements the S_INIT register, a secure-only register
>> with no non-secure counterpart. It provides a simple mechanism for
>> secure software to perform a global invalidation of all SMMU
>> configuration and translation caches.
>>
>> This is typically the final step in a SMMU's probe sequence, marking
>> the end of initialization for the SMMU's secure interface.
>>
>> With this and the previous change, a guest that is aware of the SMMUv3
>> secure extensions can probe the device's capabilities and perform basic
>> configuration of the secure interface, as is done by secure partition
>> managers like Hafnium in its smmuv3_driver_init function.
>>
>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>> ---
>> hw/arm/smmuv3.c | 29 +++++++++++++++++++++++++++++
>> hw/arm/trace-events | 1 +
>> 2 files changed, 30 insertions(+)
>>
>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>> index 619180d204..0ea9d897af 100644
>> --- a/hw/arm/smmuv3.c
>> +++ b/hw/arm/smmuv3.c
>> @@ -920,6 +920,21 @@ static void smmuv3_flush_config(SMMUDevice *sdev)
>> g_hash_table_remove(bc->configs, sdev);
>> }
>>
>> +static void smmuv3_invalidate_all_caches(SMMUv3State *s)
>> +{
>> + trace_smmuv3_invalidate_all_caches();
>> + SMMUState *bs = &s->smmu_state;
>> +
>> + /* Clear all cached configs including STE and CD*/
>> + if (bs->configs) {
>> + g_hash_table_remove_all(bs->configs);
>> + }
>> +
>> + /* Invalidate all SMMU IOTLB entries */
>> + smmu_inv_notifiers_all(&s->smmu_state);
>> + smmu_iotlb_inv_all(bs);
>> +}
>> +
>> /* Do translation with TLB lookup. */
>> static SMMUTranslationStatus smmuv3_do_translate(SMMUv3State *s, hwaddr addr,
>> SMMUTransCfg *cfg,
>> @@ -1921,6 +1936,16 @@ static MemTxResult smmu_writel(SMMUv3State *s, hwaddr offset,
>> SMMU_CHECK_ATTRS_SECURE("S_EVENTQ_IRQ_CFG2");
>> s->secure_eventq_irq_cfg2 = data;
>> return MEMTX_OK;
>> + case A_S_INIT:
>> + SMMU_CHECK_SECURE_WRITE("S_INIT");
>> + if (data & R_S_INIT_INV_ALL_MASK) {
>> + /* write S_INIT and poll*/
>> + s->secure_init = data & R_S_INIT_INV_ALL_MASK;
>> + smmuv3_invalidate_all_caches(s);
> Do we need to check that the SMMU is enabled as the spec says?
Hi Mostafa,
Thanks for the feedback on this patch.
You are right, I did miss that necessary check in v1. I will review the
specification and ensure that for the S_INIT register SMMUEN bit is
checked. Additionally, I will take this opportunity to audit whether
other registers also have dependencies on specific control bits before
they can be accessed.
>
>> + }
>> + /* initialization is completed and set to 0 to terminate the polling */
>> + s->secure_init = 0;
> All access to SMMU registers are serialised, so it’s safe to drop this and
> just return zero on reads.
>
This is a much cleaner way to model the Write-Only nature of this
register. In v2, I will remove the secure_init field from the state
structure and have the read handler for A_S_INIT always return zero.
Thanks again for the help.
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions
2025-08-18 21:28 ` Mostafa Saleh
@ 2025-08-20 16:25 ` Tao Tang
2025-08-23 10:43 ` Mostafa Saleh
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-08-20 16:25 UTC (permalink / raw)
To: Mostafa Saleh
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
jean-philippe, Philippe Mathieu-Daudé, Pierrick Bouvier
On 2025/8/19 05:28, Mostafa Saleh wrote:
> On Wed, Aug 06, 2025 at 11:11:29PM +0800, Tao Tang wrote:
>> To support parallel processing of secure and non-secure streams, the
>> SMMUv3 model needs to differentiate between the two contexts throughout
>> its core logic. This commit is the foundational step to make the code
>> security-state aware.
>>
>> An is_secure flag, which will be used in subsequent patches to represent
>> the transaction's security state, is now plumbed through the main
>> processing paths.
>>
>> This change is purely preparatory and introduces no functional changes
>> for the existing non-secure path. All current call sites are updated
>> to pass is_secure = false.
>>
>> This refactoring paves the way for upcoming patches that will introduce
>> separate TLB entries for secure transactions and enable a fully
>> parallel secure/non-secure SMMU model.
>>
> I think it’s easier to review if this patch was split (STE parsing,
> page table handling and translation, TLB invalidation)
> Also based on my comment on patch 2, stage-2 handling doesn’t seem correct to me.
>
> Thanks,
> Mostafa
>
Hi Mostafa,
Thank you your suggestion.
You've made a very good point. This patch is indeed too large and tries
to cover too many different areas. For the v2 series, I will break this
patch down into logical parts as you suggested (STE parsing, page table
handling, etc.).
I also acknowledge your concern about the stage-2 handling logic from
your comment on patch 2. I have sent a separate, detailed reply to your
feedback on patch #2 that outlines my new understanding.
And as you commented on patch #01:
>> Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
>> re-assigned to it via VFIO.
>> Command line of KVM VM inside TCG VM is below:
>>
>> sudo qemu-system-aarch64 \
>> -enable-kvm -m 1024 -cpu host -M virt \
>> -machine virt,gic-version=3 \
>> -cpu max -append "nokaslr" -smp 1 \
>> -monitor stdio \
>> -kernel 5.15.Image \
>> -initrd rootfs.cpio.gz \
>> -display vnc=:22,id=primary \
>> -device vfio-pci,host=00:01.0
>>
>> The KVM guest was able to perform I/O on the device
>> correctly, confirming that the non-secure path is not broken.
> I gave the patches a quick test and they seem to have broken my
> nested setup, I will look more into it and let you know what I find.
>
> Thanks,
> Mostafa
>
I'm sorry to hear that it has broken your environment. Please don't
hesitate to share any details, logs, or reproduction steps when you find
them. I am more than happy to help reproduce the issue on my end to get
it fixed as quickly as possible.
I would be delighted to hear back from you on any of the topics we've
discussed, as any further guidance you can offer would be invaluable.
Thanks,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-20 15:21 ` Tao Tang
@ 2025-08-23 10:41 ` Mostafa Saleh
2025-09-11 15:27 ` Tao Tang
0 siblings, 1 reply; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-23 10:41 UTC (permalink / raw)
To: Tao Tang
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
Pierrick Bouvier, Philippe Mathieu-Daudé, jean-philippe
On Wed, Aug 20, 2025 at 11:21:02PM +0800, Tao Tang wrote:
>
> On 2025/8/19 05:24, Mostafa Saleh wrote:
> > On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
> > > This patch builds upon the previous introduction of secure register
> > > definitions by providing the functional implementation for their access.
> > >
> > > The availability of the secure programming interface is now correctly
> > > gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
> > > secure functionality is enabled, the I/O handlers (smmuv3_read and
> > > smmuv3_write) will correctly dispatch accesses to the secure
> > > register space.
> > >
> > > Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> > > ---
> > > hw/arm/smmuv3-internal.h | 5 +
> > > hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
> > > 2 files changed, 456 insertions(+)
> > >
> > > diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> > > index 483aaa915e..1a8b1cb204 100644
> > > --- a/hw/arm/smmuv3-internal.h
> > > +++ b/hw/arm/smmuv3-internal.h
> > > @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
> > > #define SMMU_CR0_RESERVED 0xFFFFFC20
> > > +/*
> > > + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
> > > + */
> > > +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
> > > +
> > > REG32(CR0ACK, 0x24)
> > > REG32(CR1, 0x28)
> > > REG32(CR2, 0x2c)
> > > diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> > > index ab67972353..619180d204 100644
> > > --- a/hw/arm/smmuv3.c
> > > +++ b/hw/arm/smmuv3.c
> > > @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
> > > s->gerrorn = 0;
> > > s->statusr = 0;
> > > s->gbpa = SMMU_GBPA_RESET_VAL;
> > > +
> > > + /* Initialize secure state */
> > > + memset(s->secure_idr, 0, sizeof(s->secure_idr));
> > > + /* Secure EL2 and Secure stage 2 support */
> > > + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
> > AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
> > one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
> > Which is not implemented in this series.
>
>
> Hi Mostafa,
>
> Thank you for the very detailed and helpful review. Your feedback is spot
> on, and I'd like to address your points and ask for a quick confirmation on
> them.
>
> Regarding the SEL2 bit, you are absolutely right, my understanding was
> incorrect. I've spent the last few days reviewing the manual to better
> understand the selection between Secure and Non-secure Stage 2 translations.
> I would be very grateful if you could confirm if my new understanding is
> correct:
>
> - In Stage 2-only mode (Stage 1 bypassed), the choice between a Secure or
> Non-secure IPA translation is determined solely by STE.NSCFG.
>
Yes, but that's only with SMMU_IDR1.ATTR_PERMS_OVR which Qemu doesn't
advertise, so in our case it's always secure.
> - In Stage 1-enabled mode, STE.NSCFG is ignored. The choice is determined by
> the translation process, starting from CD.NSCFGx, with the output NS
> attribute being the result of intermediate NSTable flags and the final
> descriptor.NS bit (TTD.NSTable, TTD.NS).
>
You have to differentiate between the security state of the translation and
the security state of the translation table access.
For stage-1, the security state is determined by the NS bit in the last
level PTE, which in case of nested translation it will choose between S2TTB
or S_S2TTB.
Also, note that the stage-2 also have an NS which define the final attribute
of the transaction.
You have to also be careful around things such as NSCFG{0,1} as it might
change which stage-2 is used for the stage-1 TTB walk.
I see, in your patches, all the page-table access is done using the secure
state of the SID which is not correct.
> Based on this, I plan to have an internal flag, perhaps named
> target_ipa_is_ns in SMMUTransCfg.SMMUS2Cfg struct, to track the outcome of
> this process. This flag will then determine whether S_S2TTB or S2TTB is used
> for the Stage 2 translation.
>
I am worried that it's not that simple for a single secure nested translation
you can have multiple stage-2 walks where some might be secure and others not,
so I imagine this some how will be determined from each stage-1 walk and
some how returned (maybe in the TLB struct) which is then the stage-2
walk looks into.
I am not sure how complicated it is to manage 2 stage-2 with the current code
base, so my advice would be to split the problem; for now you can drop SEL2
from this series and rely on NS stage-2.
And later, it can be added separately, but that’s up to you and the maintainers
on how they want to do this, I will try to review as much as I can.
Also, according to the spec:
"STT is 1 in implementations where SMMU_S_IDR1.SEL2 == 1."
which requires extra work in the translation table code to support this
feature.
>
> >
> > > + /* Secure state implemented */
> > > + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> > > + SECURE_IMPL, 1);
> > > + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1,
> > > + S_SIDSIZE, SMMU_IDR1_SIDSIZE);
> > > +
> > > + s->secure_gbpa = SMMU_GBPA_RESET_VAL;
> > > }
> > > static int smmu_get_ste(SMMUv3State *s, dma_addr_t addr, STE *buf,
> > > @@ -1278,6 +1290,12 @@ static void smmuv3_range_inval(SMMUState *s, Cmd *cmd, SMMUStage stage)
> > > }
> > > }
> > > +/* Check if the SMMU hardware itself implements secure state features */
> > > +static inline bool smmu_hw_secure_implemented(SMMUv3State *s)
> > > +{
> > > + return FIELD_EX32(s->secure_idr[1], S_IDR1, SECURE_IMPL);
> > > +}
> > > +
> > I see that the secure SMMU support is unconditional. So, is this always true?
> > Also, how that looks with migration?
>
> For the v2 series, my plan is to make SECURE_IMPL a user-configurable device
> property. I will add a "secure-enabled" property to smmuv3_properties and
> ensure all necessary states are added to the VMStateDescription to handle
> migration correctly. Does this approach sound reasonable to you?
Yes, that makes sense.
>
>
> >
> > > static int smmuv3_cmdq_consume(SMMUv3State *s)
> > > {
> > > SMMUState *bs = ARM_SMMU(s);
> > > @@ -1508,9 +1526,91 @@ static int smmuv3_cmdq_consume(SMMUv3State *s)
> > > return 0;
> > > }
> > > +/* Helper function for secure register write validation */
> > > +static bool smmu_validate_secure_write(MemTxAttrs attrs, bool secure_impl,
> > > + hwaddr offset, const char *reg_name)
> > > +{
> > > + if (!attrs.secure || !secure_impl) {
> > > + const char *reason = !attrs.secure ?
> > > + "Non-secure write attempt" :
> > > + "SMMU didn't implement Security State";
> > > + qemu_log_mask(LOG_GUEST_ERROR,
> > > + "%s: %s at offset 0x%" PRIx64 " (%s, WI)\n",
> > > + __func__, reason, offset, reg_name);
> > > + return false;
> > > + }
> > > + return true;
> > > +}
> > > +
> > > +/* Helper function for secure register read validation */
> > > +static bool smmu_validate_secure_read(MemTxAttrs attrs, bool secure_impl,
> > > + hwaddr offset, const char *reg_name,
> > > + uint64_t *data)
> > > +{
> > > + if (!attrs.secure || !secure_impl) {
> > > + const char *reason = !attrs.secure ?
> > > + "Non-secure read attempt" :
> > > + "SMMU didn't implement Security State";
> > > + qemu_log_mask(LOG_GUEST_ERROR,
> > > + "%s: %s at offset 0x%" PRIx64 " (%s, RAZ)\n",
> > > + __func__, reason, offset, reg_name);
> > > + *data = 0; /* RAZ */
> > > + return false;
> > > + }
> > > + return true;
> > > +}
> > > +
> > > +/* Macro for secure write validation - returns early if validation fails */
> > > +#define SMMU_CHECK_SECURE_WRITE(reg_name) \
> > > + do { \
> > > + if (!smmu_validate_secure_write(attrs, secure_impl, offset, \
> > > + reg_name)) { \
> > > + return MEMTX_OK; \
> > > + } \
> > > + } while (0)
> > > +
> > > +/* Macro for attrs.secure only validation */
> > > +#define SMMU_CHECK_ATTRS_SECURE(reg_name) \
> > > + do { \
> > > + if (!attrs.secure) { \
> > > + qemu_log_mask(LOG_GUEST_ERROR, \
> > > + "%s: Non-secure write attempt at offset " \
> > > + "0x%" PRIx64 " (%s, WI)\n", \
> > > + __func__, offset, reg_name); \
> > > + return MEMTX_OK; \
> > > + } \
> > > + } while (0)
> > > +
> > > +/* Macro for secure read validation - returns RAZ if validation fails */
> > > +#define SMMU_CHECK_SECURE_READ(reg_name) \
> > > + do { \
> > > + if (!smmu_validate_secure_read(attrs, secure_impl, offset, \
> > > + reg_name, data)) { \
> > > + return MEMTX_OK; \
> > > + } \
> > > + } while (0)
> > > +
> > > +/* Macro for attrs.secure only validation (read) */
> > > +#define SMMU_CHECK_ATTRS_SECURE_READ(reg_name) \
> > > + do { \
> > > + if (!attrs.secure) { \
> > > + qemu_log_mask(LOG_GUEST_ERROR, \
> > > + "%s: Non-secure read attempt at offset " \
> > > + "0x%" PRIx64 " (%s, RAZ)\n", \
> > > + __func__, offset, reg_name); \
> > > + *data = 0; \
> > > + return MEMTX_OK; \
> > > + } \
> > > + } while (0)
> > > +
> > >
> > Can’t we just have one check? If the access > SMMU_SECURE_BASE_OFFSET, just
> > check the security state?
> >
> > And then based on banking, many of those switches will be common with
> > non secure cases.
> >
> > Thanks,
> > Mostafa
>
>
> I have already refactored this part in my v2 series, exactly as you
> proposed. This also addresses your earlier feedback on patch #1 regarding
> the overall structure:
>
> > As Philippe mentioned, this would be better the secure state is separated
> > in another instance of the struct, that seems it would reduce a lot of the
> > duplication later around the logic of MMIO and queues... in the next
> > patches.
>
> The new code performs a single, necessary security state check at the entry
> point of the MMIO handlers. The rest of the logic relies on the banking
> mechanism, which makes the implementation generic for Non-secure, Secure,
> and future states like Realm/Root. The new structure looks like this:
>
> /* Structure for one register bank */
> typedef struct SMMUv3Bank {
> uint32_t idr[6]; /* IDR0-IDR5, note: IDR5 only used for NS bank */
> uint32_t cr[3]; /* CR0-CR2 */
> uint32_t cr0ack;
> uint32_t init; /* S_INIT register (secure only), reserved for NS
> */
> uint32_t gbpa;
>
> ......
>
> SMMUQueue eventq, cmdq;
> } SMMUv3Bank;
>
> struct SMMUv3State {
> SMMUState smmu_state;
>
> /* Shared (non-banked) registers and state */
> uint32_t features;
> uint8_t sid_size;
> uint8_t sid_split;
>
> ......
>
> /* Banked registers for all access */
> SMMUv3Bank bank[SMMU_SEC_IDX_NUM];
> ......
> };
>
Yes, IMO,that’s the right approach. Although that might make the
migration code more complicated as we changed the state struct.
Thanks,
Mostafa
>
> Thanks again for your valuable feedback. I've outlined my proposed plan
> above and would be grateful for any thoughts on it to ensure I'm on the
> right track for v2.
>
> Best regards,
>
> Tao
>
>
>
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions
2025-08-20 16:25 ` Tao Tang
@ 2025-08-23 10:43 ` Mostafa Saleh
0 siblings, 0 replies; 47+ messages in thread
From: Mostafa Saleh @ 2025-08-23 10:43 UTC (permalink / raw)
To: Tao Tang
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
jean-philippe, Philippe Mathieu-Daudé, Pierrick Bouvier
On Thu, Aug 21, 2025 at 12:25:40AM +0800, Tao Tang wrote:
>
> On 2025/8/19 05:28, Mostafa Saleh wrote:
> > On Wed, Aug 06, 2025 at 11:11:29PM +0800, Tao Tang wrote:
> > > To support parallel processing of secure and non-secure streams, the
> > > SMMUv3 model needs to differentiate between the two contexts throughout
> > > its core logic. This commit is the foundational step to make the code
> > > security-state aware.
> > >
> > > An is_secure flag, which will be used in subsequent patches to represent
> > > the transaction's security state, is now plumbed through the main
> > > processing paths.
> > >
> > > This change is purely preparatory and introduces no functional changes
> > > for the existing non-secure path. All current call sites are updated
> > > to pass is_secure = false.
> > >
> > > This refactoring paves the way for upcoming patches that will introduce
> > > separate TLB entries for secure transactions and enable a fully
> > > parallel secure/non-secure SMMU model.
> > >
> > I think it’s easier to review if this patch was split (STE parsing,
> > page table handling and translation, TLB invalidation)
> > Also based on my comment on patch 2, stage-2 handling doesn’t seem correct to me.
> >
> > Thanks,
> > Mostafa
> >
> Hi Mostafa,
>
> Thank you your suggestion.
>
> You've made a very good point. This patch is indeed too large and tries to
> cover too many different areas. For the v2 series, I will break this patch
> down into logical parts as you suggested (STE parsing, page table handling,
> etc.).
>
> I also acknowledge your concern about the stage-2 handling logic from your
> comment on patch 2. I have sent a separate, detailed reply to your feedback
> on patch #2 that outlines my new understanding.
>
> And as you commented on patch #01:
>
> > > Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
> > > re-assigned to it via VFIO.
> > > Command line of KVM VM inside TCG VM is below:
> > >
> > > sudo qemu-system-aarch64 \
> > > -enable-kvm -m 1024 -cpu host -M virt \
> > > -machine virt,gic-version=3 \
> > > -cpu max -append "nokaslr" -smp 1 \
> > > -monitor stdio \
> > > -kernel 5.15.Image \
> > > -initrd rootfs.cpio.gz \
> > > -display vnc=:22,id=primary \
> > > -device vfio-pci,host=00:01.0
> > >
> > > The KVM guest was able to perform I/O on the device
> > > correctly, confirming that the non-secure path is not broken.
> > I gave the patches a quick test and they seem to have broken my
> > nested setup, I will look more into it and let you know what I find.
> >
> > Thanks,
> > Mostafa
> >
> I'm sorry to hear that it has broken your environment. Please don't hesitate
> to share any details, logs, or reproduction steps when you find them. I am
> more than happy to help reproduce the issue on my end to get it fixed as
> quickly as possible.
>
>
> I would be delighted to hear back from you on any of the topics we've
> discussed, as any further guidance you can offer would be invaluable.
>
So far, I couldn’t repro, I remember getting permission errors, I will
keep the patches in my stack, and will let you know if I hit that again.
Thanks,
Mostafa
> Thanks,
>
> Tao
>
>
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-08-23 10:41 ` Mostafa Saleh
@ 2025-09-11 15:27 ` Tao Tang
2025-09-15 9:14 ` Mostafa Saleh
0 siblings, 1 reply; 47+ messages in thread
From: Tao Tang @ 2025-09-11 15:27 UTC (permalink / raw)
To: Mostafa Saleh
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
Pierrick Bouvier, Philippe Mathieu-Daudé, jean-philippe
Hi Mostafa,
First, my apologies for the long delay in getting back to you. I was
away on paternity leave for the last few weeks.
Thank you for the detailed follow-up, your advice is very helpful for
simplifying the series.
On 2025/8/23 18:41, Mostafa Saleh wrote:
> On Wed, Aug 20, 2025 at 11:21:02PM +0800, Tao Tang wrote:
>> On 2025/8/19 05:24, Mostafa Saleh wrote:
>>> On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
>>>> This patch builds upon the previous introduction of secure register
>>>> definitions by providing the functional implementation for their access.
>>>>
>>>> The availability of the secure programming interface is now correctly
>>>> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
>>>> secure functionality is enabled, the I/O handlers (smmuv3_read and
>>>> smmuv3_write) will correctly dispatch accesses to the secure
>>>> register space.
>>>>
>>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>>> ---
>>>> hw/arm/smmuv3-internal.h | 5 +
>>>> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
>>>> 2 files changed, 456 insertions(+)
>>>>
>>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>>> index 483aaa915e..1a8b1cb204 100644
>>>> --- a/hw/arm/smmuv3-internal.h
>>>> +++ b/hw/arm/smmuv3-internal.h
>>>> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>>>> #define SMMU_CR0_RESERVED 0xFFFFFC20
>>>> +/*
>>>> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
>>>> + */
>>>> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
>>>> +
>>>> REG32(CR0ACK, 0x24)
>>>> REG32(CR1, 0x28)
>>>> REG32(CR2, 0x2c)
>>>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>>>> index ab67972353..619180d204 100644
>>>> --- a/hw/arm/smmuv3.c
>>>> +++ b/hw/arm/smmuv3.c
>>>> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
>>>> s->gerrorn = 0;
>>>> s->statusr = 0;
>>>> s->gbpa = SMMU_GBPA_RESET_VAL;
>>>> +
>>>> + /* Initialize secure state */
>>>> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
>>>> + /* Secure EL2 and Secure stage 2 support */
>>>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
>>> AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
>>> one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
>>> Which is not implemented in this series.
>>
>> Hi Mostafa,
>>
>> Thank you for the very detailed and helpful review. Your feedback is spot
>> on, and I'd like to address your points and ask for a quick confirmation on
>> them.
>>
>> Regarding the SEL2 bit, you are absolutely right, my understanding was
>> incorrect. I've spent the last few days reviewing the manual to better
>> understand the selection between Secure and Non-secure Stage 2 translations.
>> I would be very grateful if you could confirm if my new understanding is
>> correct:
>>
>> - In Stage 2-only mode (Stage 1 bypassed), the choice between a Secure or
>> Non-secure IPA translation is determined solely by STE.NSCFG.
>>
> Yes, but that's only with SMMU_IDR1.ATTR_PERMS_OVR which Qemu doesn't
> advertise, so in our case it's always secure.
>
>> - In Stage 1-enabled mode, STE.NSCFG is ignored. The choice is determined by
>> the translation process, starting from CD.NSCFGx, with the output NS
>> attribute being the result of intermediate NSTable flags and the final
>> descriptor.NS bit (TTD.NSTable, TTD.NS).
>>
> You have to differentiate between the security state of the translation and
> the security state of the translation table access.
>
> For stage-1, the security state is determined by the NS bit in the last
> level PTE, which in case of nested translation it will choose between S2TTB
> or S_S2TTB.
>
> Also, note that the stage-2 also have an NS which define the final attribute
> of the transaction.
>
> You have to also be careful around things such as NSCFG{0,1} as it might
> change which stage-2 is used for the stage-1 TTB walk.
>
> I see, in your patches, all the page-table access is done using the secure
> state of the SID which is not correct.
>
>
>> Based on this, I plan to have an internal flag, perhaps named
>> target_ipa_is_ns in SMMUTransCfg.SMMUS2Cfg struct, to track the outcome of
>> this process. This flag will then determine whether S_S2TTB or S2TTB is used
>> for the Stage 2 translation.
>>
> I am worried that it's not that simple for a single secure nested translation
> you can have multiple stage-2 walks where some might be secure and others not,
> so I imagine this some how will be determined from each stage-1 walk and
> some how returned (maybe in the TLB struct) which is then the stage-2
> walk looks into.
>
> I am not sure how complicated it is to manage 2 stage-2 with the current code
> base, so my advice would be to split the problem; for now you can drop SEL2
> from this series and rely on NS stage-2.
I would like to confirm my understanding of the implementation. Does
this mean that for the current RFC, we should set S_IDR1.SEL2=0, which
implies that all Stage-2 translations will begin with a Non-secure IPA?
And the final output PA space will then almost always be Non-secure PA,
with the sole exception being when S2SW, S2SA, S2NSW, and S2NSA are ALL
ZERO.
However, since these fields are RES0 when S_IDR1.SEL2=0, it seems we can
conclude that for this version, the output will definitively be a
Non-secure PA. I believe this is what you meant by your advice to "rely
on NS stage-2". I would be grateful if you could let me know whether
this interpretation is on the right track.
------------------------------<snip>------------------------------
>> The new code performs a single, necessary security state check at the entry
>> point of the MMIO handlers. The rest of the logic relies on the banking
>> mechanism, which makes the implementation generic for Non-secure, Secure,
>> and future states like Realm/Root. The new structure looks like this:
>>
>> /* Structure for one register bank */
>> typedef struct SMMUv3Bank {
>> uint32_t idr[6]; /* IDR0-IDR5, note: IDR5 only used for NS bank */
>> uint32_t cr[3]; /* CR0-CR2 */
>> uint32_t cr0ack;
>> uint32_t init; /* S_INIT register (secure only), reserved for NS
>> */
>> uint32_t gbpa;
>>
>> ......
>>
>> SMMUQueue eventq, cmdq;
>> } SMMUv3Bank;
>>
>> struct SMMUv3State {
>> SMMUState smmu_state;
>>
>> /* Shared (non-banked) registers and state */
>> uint32_t features;
>> uint8_t sid_size;
>> uint8_t sid_split;
>>
>> ......
>>
>> /* Banked registers for all access */
>> SMMUv3Bank bank[SMMU_SEC_IDX_NUM];
>> ......
>> };
>>
> Yes, IMO,that’s the right approach. Although that might make the
> migration code more complicated as we changed the state struct.
>
> Thanks,
> Mostafa
I have almost completed the refactoring based on this new structure, and
I will send out the v2 patch series in the next few days for review.
Thanks again for your invaluable guidance.
Best regards,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-09-11 15:27 ` Tao Tang
@ 2025-09-15 9:14 ` Mostafa Saleh
2025-09-15 9:34 ` Eric Auger
0 siblings, 1 reply; 47+ messages in thread
From: Mostafa Saleh @ 2025-09-15 9:14 UTC (permalink / raw)
To: Tao Tang
Cc: qemu-arm, qemu-devel, Eric Auger, Peter Maydell, Chen Baozi,
Pierrick Bouvier, Philippe Mathieu-Daudé, jean-philippe
Hi Tao,
On Thu, Sep 11, 2025 at 11:27:50PM +0800, Tao Tang wrote:
>
> Hi Mostafa,
>
> First, my apologies for the long delay in getting back to you. I was away on
> paternity leave for the last few weeks.
No worries!
>
> Thank you for the detailed follow-up, your advice is very helpful for
> simplifying the series.
>
>
> On 2025/8/23 18:41, Mostafa Saleh wrote:
> > On Wed, Aug 20, 2025 at 11:21:02PM +0800, Tao Tang wrote:
> > > On 2025/8/19 05:24, Mostafa Saleh wrote:
> > > > On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
> > > > > This patch builds upon the previous introduction of secure register
> > > > > definitions by providing the functional implementation for their access.
> > > > >
> > > > > The availability of the secure programming interface is now correctly
> > > > > gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
> > > > > secure functionality is enabled, the I/O handlers (smmuv3_read and
> > > > > smmuv3_write) will correctly dispatch accesses to the secure
> > > > > register space.
> > > > >
> > > > > Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
> > > > > ---
> > > > > hw/arm/smmuv3-internal.h | 5 +
> > > > > hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
> > > > > 2 files changed, 456 insertions(+)
> > > > >
> > > > > diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
> > > > > index 483aaa915e..1a8b1cb204 100644
> > > > > --- a/hw/arm/smmuv3-internal.h
> > > > > +++ b/hw/arm/smmuv3-internal.h
> > > > > @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
> > > > > #define SMMU_CR0_RESERVED 0xFFFFFC20
> > > > > +/*
> > > > > + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
> > > > > + */
> > > > > +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
> > > > > +
> > > > > REG32(CR0ACK, 0x24)
> > > > > REG32(CR1, 0x28)
> > > > > REG32(CR2, 0x2c)
> > > > > diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> > > > > index ab67972353..619180d204 100644
> > > > > --- a/hw/arm/smmuv3.c
> > > > > +++ b/hw/arm/smmuv3.c
> > > > > @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
> > > > > s->gerrorn = 0;
> > > > > s->statusr = 0;
> > > > > s->gbpa = SMMU_GBPA_RESET_VAL;
> > > > > +
> > > > > + /* Initialize secure state */
> > > > > + memset(s->secure_idr, 0, sizeof(s->secure_idr));
> > > > > + /* Secure EL2 and Secure stage 2 support */
> > > > > + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
> > > > AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
> > > > one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
> > > > Which is not implemented in this series.
> > >
> > > Hi Mostafa,
> > >
> > > Thank you for the very detailed and helpful review. Your feedback is spot
> > > on, and I'd like to address your points and ask for a quick confirmation on
> > > them.
> > >
> > > Regarding the SEL2 bit, you are absolutely right, my understanding was
> > > incorrect. I've spent the last few days reviewing the manual to better
> > > understand the selection between Secure and Non-secure Stage 2 translations.
> > > I would be very grateful if you could confirm if my new understanding is
> > > correct:
> > >
> > > - In Stage 2-only mode (Stage 1 bypassed), the choice between a Secure or
> > > Non-secure IPA translation is determined solely by STE.NSCFG.
> > >
> > Yes, but that's only with SMMU_IDR1.ATTR_PERMS_OVR which Qemu doesn't
> > advertise, so in our case it's always secure.
> >
> > > - In Stage 1-enabled mode, STE.NSCFG is ignored. The choice is determined by
> > > the translation process, starting from CD.NSCFGx, with the output NS
> > > attribute being the result of intermediate NSTable flags and the final
> > > descriptor.NS bit (TTD.NSTable, TTD.NS).
> > >
> > You have to differentiate between the security state of the translation and
> > the security state of the translation table access.
> >
> > For stage-1, the security state is determined by the NS bit in the last
> > level PTE, which in case of nested translation it will choose between S2TTB
> > or S_S2TTB.
> >
> > Also, note that the stage-2 also have an NS which define the final attribute
> > of the transaction.
> >
> > You have to also be careful around things such as NSCFG{0,1} as it might
> > change which stage-2 is used for the stage-1 TTB walk.
> >
> > I see, in your patches, all the page-table access is done using the secure
> > state of the SID which is not correct.
> >
> >
> > > Based on this, I plan to have an internal flag, perhaps named
> > > target_ipa_is_ns in SMMUTransCfg.SMMUS2Cfg struct, to track the outcome of
> > > this process. This flag will then determine whether S_S2TTB or S2TTB is used
> > > for the Stage 2 translation.
> > >
> > I am worried that it's not that simple for a single secure nested translation
> > you can have multiple stage-2 walks where some might be secure and others not,
> > so I imagine this some how will be determined from each stage-1 walk and
> > some how returned (maybe in the TLB struct) which is then the stage-2
> > walk looks into.
> >
> > I am not sure how complicated it is to manage 2 stage-2 with the current code
> > base, so my advice would be to split the problem; for now you can drop SEL2
> > from this series and rely on NS stage-2.
>
>
> I would like to confirm my understanding of the implementation. Does this
> mean that for the current RFC, we should set S_IDR1.SEL2=0, which implies
> that all Stage-2 translations will begin with a Non-secure IPA? And the
> final output PA space will then almost always be Non-secure PA, with the
> sole exception being when S2SW, S2SA, S2NSW, and S2NSA are ALL ZERO.
>
>
> However, since these fields are RES0 when S_IDR1.SEL2=0, it seems we can
> conclude that for this version, the output will definitively be a Non-secure
> PA. I believe this is what you meant by your advice to "rely on NS stage-2".
> I would be grateful if you could let me know whether this interpretation is
> on the right track.
Yes, that’s what I meant, I think that simplifies a lot, in this series we
can focus on the infrastructure for the secure SMMU (registers, TLBs..),
and then extra features such as secure stage-2 can be added later.
>
>
> ------------------------------<snip>------------------------------
>
> > > The new code performs a single, necessary security state check at the entry
> > > point of the MMIO handlers. The rest of the logic relies on the banking
> > > mechanism, which makes the implementation generic for Non-secure, Secure,
> > > and future states like Realm/Root. The new structure looks like this:
> > >
> > > /* Structure for one register bank */
> > > typedef struct SMMUv3Bank {
> > > uint32_t idr[6]; /* IDR0-IDR5, note: IDR5 only used for NS bank */
> > > uint32_t cr[3]; /* CR0-CR2 */
> > > uint32_t cr0ack;
> > > uint32_t init; /* S_INIT register (secure only), reserved for NS
> > > */
> > > uint32_t gbpa;
> > >
> > > ......
> > >
> > > SMMUQueue eventq, cmdq;
> > > } SMMUv3Bank;
> > >
> > > struct SMMUv3State {
> > > SMMUState smmu_state;
> > >
> > > /* Shared (non-banked) registers and state */
> > > uint32_t features;
> > > uint8_t sid_size;
> > > uint8_t sid_split;
> > >
> > > ......
> > >
> > > /* Banked registers for all access */
> > > SMMUv3Bank bank[SMMU_SEC_IDX_NUM];
> > > ......
> > > };
> > >
> > Yes, IMO,that’s the right approach. Although that might make the
> > migration code more complicated as we changed the state struct.
> >
> > Thanks,
> > Mostafa
> I have almost completed the refactoring based on this new structure, and I
> will send out the v2 patch series in the next few days for review.
Sounds good!
Thanks,
Mostafa
>
> Thanks again for your invaluable guidance.
>
> Best regards,
>
> Tao
>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers
2025-09-15 9:14 ` Mostafa Saleh
@ 2025-09-15 9:34 ` Eric Auger
0 siblings, 0 replies; 47+ messages in thread
From: Eric Auger @ 2025-09-15 9:34 UTC (permalink / raw)
To: Mostafa Saleh, Tao Tang
Cc: qemu-arm, qemu-devel, Peter Maydell, Chen Baozi, Pierrick Bouvier,
Philippe Mathieu-Daudé, jean-philippe
Hi Tao,
On 9/15/25 11:14 AM, Mostafa Saleh wrote:
> Hi Tao,
>
> On Thu, Sep 11, 2025 at 11:27:50PM +0800, Tao Tang wrote:
>> Hi Mostafa,
>>
>> First, my apologies for the long delay in getting back to you. I was away on
>> paternity leave for the last few weeks.
> No worries!
>
>> Thank you for the detailed follow-up, your advice is very helpful for
>> simplifying the series.
>>
>>
>> On 2025/8/23 18:41, Mostafa Saleh wrote:
>>> On Wed, Aug 20, 2025 at 11:21:02PM +0800, Tao Tang wrote:
>>>> On 2025/8/19 05:24, Mostafa Saleh wrote:
>>>>> On Wed, Aug 06, 2025 at 11:11:25PM +0800, Tao Tang wrote:
>>>>>> This patch builds upon the previous introduction of secure register
>>>>>> definitions by providing the functional implementation for their access.
>>>>>>
>>>>>> The availability of the secure programming interface is now correctly
>>>>>> gated by the S_IDR1.SECURE_IMPL bit. When this bit indicates that
>>>>>> secure functionality is enabled, the I/O handlers (smmuv3_read and
>>>>>> smmuv3_write) will correctly dispatch accesses to the secure
>>>>>> register space.
>>>>>>
>>>>>> Signed-off-by: Tao Tang <tangtao1634@phytium.com.cn>
>>>>>> ---
>>>>>> hw/arm/smmuv3-internal.h | 5 +
>>>>>> hw/arm/smmuv3.c | 451 +++++++++++++++++++++++++++++++++++++++
>>>>>> 2 files changed, 456 insertions(+)
>>>>>>
>>>>>> diff --git a/hw/arm/smmuv3-internal.h b/hw/arm/smmuv3-internal.h
>>>>>> index 483aaa915e..1a8b1cb204 100644
>>>>>> --- a/hw/arm/smmuv3-internal.h
>>>>>> +++ b/hw/arm/smmuv3-internal.h
>>>>>> @@ -122,6 +122,11 @@ REG32(CR0, 0x20)
>>>>>> #define SMMU_CR0_RESERVED 0xFFFFFC20
>>>>>> +/*
>>>>>> + * BIT1 and BIT4 are RES0 in SMMU_S_CRO
>>>>>> + */
>>>>>> +#define SMMU_S_CR0_RESERVED 0xFFFFFC12
>>>>>> +
>>>>>> REG32(CR0ACK, 0x24)
>>>>>> REG32(CR1, 0x28)
>>>>>> REG32(CR2, 0x2c)
>>>>>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>>>>>> index ab67972353..619180d204 100644
>>>>>> --- a/hw/arm/smmuv3.c
>>>>>> +++ b/hw/arm/smmuv3.c
>>>>>> @@ -317,6 +317,18 @@ static void smmuv3_init_regs(SMMUv3State *s)
>>>>>> s->gerrorn = 0;
>>>>>> s->statusr = 0;
>>>>>> s->gbpa = SMMU_GBPA_RESET_VAL;
>>>>>> +
>>>>>> + /* Initialize secure state */
>>>>>> + memset(s->secure_idr, 0, sizeof(s->secure_idr));
>>>>>> + /* Secure EL2 and Secure stage 2 support */
>>>>>> + s->secure_idr[1] = FIELD_DP32(s->secure_idr[1], S_IDR1, SEL2, 1);
>>>>> AFAIU, this is wrong, SEL2 means that the SMMU has dual stage-2,
>>>>> one for secure (S_S2TTB) and one for non-secure IPAs(S2TTB).
>>>>> Which is not implemented in this series.
>>>> Hi Mostafa,
>>>>
>>>> Thank you for the very detailed and helpful review. Your feedback is spot
>>>> on, and I'd like to address your points and ask for a quick confirmation on
>>>> them.
>>>>
>>>> Regarding the SEL2 bit, you are absolutely right, my understanding was
>>>> incorrect. I've spent the last few days reviewing the manual to better
>>>> understand the selection between Secure and Non-secure Stage 2 translations.
>>>> I would be very grateful if you could confirm if my new understanding is
>>>> correct:
>>>>
>>>> - In Stage 2-only mode (Stage 1 bypassed), the choice between a Secure or
>>>> Non-secure IPA translation is determined solely by STE.NSCFG.
>>>>
>>> Yes, but that's only with SMMU_IDR1.ATTR_PERMS_OVR which Qemu doesn't
>>> advertise, so in our case it's always secure.
>>>
>>>> - In Stage 1-enabled mode, STE.NSCFG is ignored. The choice is determined by
>>>> the translation process, starting from CD.NSCFGx, with the output NS
>>>> attribute being the result of intermediate NSTable flags and the final
>>>> descriptor.NS bit (TTD.NSTable, TTD.NS).
>>>>
>>> You have to differentiate between the security state of the translation and
>>> the security state of the translation table access.
>>>
>>> For stage-1, the security state is determined by the NS bit in the last
>>> level PTE, which in case of nested translation it will choose between S2TTB
>>> or S_S2TTB.
>>>
>>> Also, note that the stage-2 also have an NS which define the final attribute
>>> of the transaction.
>>>
>>> You have to also be careful around things such as NSCFG{0,1} as it might
>>> change which stage-2 is used for the stage-1 TTB walk.
>>>
>>> I see, in your patches, all the page-table access is done using the secure
>>> state of the SID which is not correct.
>>>
>>>
>>>> Based on this, I plan to have an internal flag, perhaps named
>>>> target_ipa_is_ns in SMMUTransCfg.SMMUS2Cfg struct, to track the outcome of
>>>> this process. This flag will then determine whether S_S2TTB or S2TTB is used
>>>> for the Stage 2 translation.
>>>>
>>> I am worried that it's not that simple for a single secure nested translation
>>> you can have multiple stage-2 walks where some might be secure and others not,
>>> so I imagine this some how will be determined from each stage-1 walk and
>>> some how returned (maybe in the TLB struct) which is then the stage-2
>>> walk looks into.
>>>
>>> I am not sure how complicated it is to manage 2 stage-2 with the current code
>>> base, so my advice would be to split the problem; for now you can drop SEL2
>>> from this series and rely on NS stage-2.
>>
>> I would like to confirm my understanding of the implementation. Does this
>> mean that for the current RFC, we should set S_IDR1.SEL2=0, which implies
>> that all Stage-2 translations will begin with a Non-secure IPA? And the
>> final output PA space will then almost always be Non-secure PA, with the
>> sole exception being when S2SW, S2SA, S2NSW, and S2NSA are ALL ZERO.
>>
>>
>> However, since these fields are RES0 when S_IDR1.SEL2=0, it seems we can
>> conclude that for this version, the output will definitively be a Non-secure
>> PA. I believe this is what you meant by your advice to "rely on NS stage-2".
>> I would be grateful if you could let me know whether this interpretation is
>> on the right track.
> Yes, that’s what I meant, I think that simplifies a lot, in this series we
> can focus on the infrastructure for the secure SMMU (registers, TLBs..),
> and then extra features such as secure stage-2 can be added later.
>
>>
>> ------------------------------<snip>------------------------------
>>
>>>> The new code performs a single, necessary security state check at the entry
>>>> point of the MMIO handlers. The rest of the logic relies on the banking
>>>> mechanism, which makes the implementation generic for Non-secure, Secure,
>>>> and future states like Realm/Root. The new structure looks like this:
>>>>
>>>> /* Structure for one register bank */
>>>> typedef struct SMMUv3Bank {
>>>> uint32_t idr[6]; /* IDR0-IDR5, note: IDR5 only used for NS bank */
>>>> uint32_t cr[3]; /* CR0-CR2 */
>>>> uint32_t cr0ack;
>>>> uint32_t init; /* S_INIT register (secure only), reserved for NS
>>>> */
>>>> uint32_t gbpa;
>>>>
>>>> ......
>>>>
>>>> SMMUQueue eventq, cmdq;
>>>> } SMMUv3Bank;
>>>>
>>>> struct SMMUv3State {
>>>> SMMUState smmu_state;
>>>>
>>>> /* Shared (non-banked) registers and state */
>>>> uint32_t features;
>>>> uint8_t sid_size;
>>>> uint8_t sid_split;
>>>>
>>>> ......
>>>>
>>>> /* Banked registers for all access */
>>>> SMMUv3Bank bank[SMMU_SEC_IDX_NUM];
>>>> ......
>>>> };
>>>>
>>> Yes, IMO,that’s the right approach. Although that might make the
>>> migration code more complicated as we changed the state struct.
>>>
>>> Thanks,
>>> Mostafa
>> I have almost completed the refactoring based on this new structure, and I
>> will send out the v2 patch series in the next few days for review.
> Sounds good!
Sorry for having failed to review the RFC on time. Thanks to other
reviewers I think you've got quite a lot of feedbacks already. I will
review v2.
Looking forward to receiving your respin.
Thanks
Eric
>
> Thanks,
> Mostafa
>
>> Thanks again for your invaluable guidance.
>>
>> Best regards,
>>
>> Tao
>>
^ permalink raw reply [flat|nested] 47+ messages in thread
* Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State
2025-08-15 5:49 ` Tao Tang
@ 2025-09-30 4:04 ` Tao Tang
0 siblings, 0 replies; 47+ messages in thread
From: Tao Tang @ 2025-09-30 4:04 UTC (permalink / raw)
To: Pierrick Bouvier, qemu-arm, qemu-devel
Cc: Eric Auger, Peter Maydell, Chen Baozi,
Philippe Mathieu-Daudé, Mostafa Saleh, Jean-Philippe Brucker
Hi Pierrick,
On 2025/8/15 13:49, Tao Tang wrote:
>
> On 2025/8/13 02:04, Pierrick Bouvier wrote:
>> On 8/10/25 9:11 AM, Tao Tang wrote:
>>>
>>> On 2025/8/7 05:28, Pierrick Bouvier wrote:
>>>> On 8/6/25 8:11 AM, Tao Tang wrote:
>>>>> Hi all,
>>>>>
>>>>> This patch series introduces initial support for emulating the Arm
>>>>> SMMUv3
>>>>> Secure State.
>>>>> ------------------------------<snip>------------------------------
>>>>>
>>>>>
>>>>> ------------------------------<snip>------------------------------
>>>>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
>>>>> environment. Hafnium's smmuv3_driver_init function was used to test
>>>>> the secure register I/O and command queue functionality (excluding
>>>>> translation). As Hafnium assumes larger queue and StreamID sizes than
>>>>> are practical without TTST support, I temporarily patched Hafnium to
>>>>> use smaller values, allowing its driver to initialize the emulated
>>>>> secure SMMU successfully.
>>>>>
>>>>
>>>> Would that be possible to share your changes, and build instructions
>>>> for this? While working on SMMU emulation, we finally left this on the
>>>> side due to lack of a software stack being able to use secure SMMU, as
>>>> we were not aware that Hafnium + op-tee could make use of it.
>>>>
>>> Hi Pierrick,
>>>
>>> Thanks for your interest! I'm very happy to share my work on this. I've
>>> documented the setup process, including our code modifications and the
>>> step-by-step build instructions in this link:
>>>
>>> https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG
>>>
>>
>> Thanks for taking the time to assemble all this in a comprehensible
>> post, I'll give it a try when I have some spare time.
>
> Hi Pierrick,
>
> You're welcome, and please feel free to let me know if you run into
> any issues.
>
>
>>
>>>
>>> The core point of these changes is to enable the SMMUv3 feature in
>>> Hafnium. This leads to numerous read/write operations on SMMUv3 secure
>>> registers and various queue manipulations within the smmuv3_driver_init
>>> function in Hafnium.
>>>
>>> However, it's important to note that this initialization process itself
>>> does not initiate any DMA memory access that would trigger the
>>> smmuv3_translate flow.
>>>
>>
>> I understand the difference. It can be tricky to generate specific
>> translation scenarios, which is where a custom test device can really
>> help.
>>
>>> Even so, we've devised a method to test the subsequent Secure
>>> Translation Path by leveraging the smmuv3-test platform device. This
>>> approach allows us to verify the entire SMMUv3 flow, from
>>> initialization
>>> to translation.
>>>
>>
>> Does it rely on a custom driver integration into an existing firmware
>> or the kernel?
>>
>>>
>>>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
>>>>> support QEMU, and no secure device assignment feature exists yet, I
>>>>> created a custom platform device to test the secure translation flow.
>>>>> To trigger the translation logic, I initiated MMIO writes to this
>>>>> device from within Hafnium. The device's MMIO callback handler then
>>>>> performed DMA accesses via its IOMMU region, exercising the secure
>>>>> translation path. While SMMUv3 is typically used for PCIe on
>>>>> physical SoCs, the architecture allows its use with platform devices
>>>>> via a stream-id binding in the device tree. The test harness
>>>>> required some non-standard modifications to decouple the SMMU from
>>>>> its tight integration with PCIe. The code for this test device is
>>>>> available for review at [3]. README.md with detailed instructions is
>>>>> also provided.
>>>>>
>>>>
>>>> I am not sure about the current policy in QEMU for test oriented
>>>> devices, but it would be really useful to have something similar
>>>> upstream (Note: it's out of the scope of this series).
>>>> One challenge working with SMMU emulation is that reproducing setups
>>>> and triggering specific code paths is hard to achieve, due to the
>>>> indirect use of SMMU feature (through DMA) and the complex software
>>>> stack usually involved.
>>>> Having something upstream available to work on SMMU emulation, at
>>>> least on device side, would be a great addition.
>>>>
>>>> Eric, Peter, is this something that would be acceptable to merge?
>>>>
>>>
>>> Looking ahead, my plan is to refactor the smmuv3-test platform device.
>>> The goal is to make it self-contained within QEMU, removing the current
>>> dependency on Hafnium to trigger its operations. I plan to submit this
>>> as a separate RFC patch series in the next few days.
>>>
>>
>> This is very welcome. Once this is in place, it would be great to add
>> a new test to make sure things don't regress, and from where we can
>> iterate.
>> By self-contained within QEMU, do you mean a QTest based test?
>>
>> Regards,
>> Pierrick
>
>
> Thanks for the follow-up and the great questions.
>
> To answer your question about the custom test driver: yes, the current
> implementation of the smmuv3-test device relies on integration with
> Hafnium to be triggered.
>
> The test flow is initiated when Hafnium performs an mmio_write32 to
> the smmuv3-test device's MMIO space. This triggers the device's
> read/write callback in QEMU. Inside this callback, I use
> address_space_write/read to first populate the necessary SMMU
> structures (STEs, CDs, PTEs) in guest secure memory, and then perform
> another address_space_write/read to an IOMMU-protected region of
> smmuv3-test. It is this final access that exercises the full secure
> translation path. So for now, it is indeed dependent on the Hafnium
> firmware for debugging.
>
> That brings me to your next point about making the test
> "self-contained". My goal is exactly to remove this dependency. I'm
> currently exploring ways to achieve this within QEMU—for instance, by
> using QMP commands or another monitor interface to replace the
> mmio_write32 action that Hafnium currently provides. This is what I
> meant by "self-contained": allowing us to test the entire SMMUv3
> translation flow relying only on the QEMU emulator itself, without any
> specific guest firmware or kernel.
>
> This leads perfectly to your question about QTest. I'm not very
> familiar with it, so your suggestion is very helpful. Is QTest the
> standard or required framework for implementing this kind of
> self-contained test device? I would appreciate any guidance or
> pointers you could provide on this topic.
Hi Pierrick,
I've just submitted a new SMMU test device that enables qtest-based
testing without needing to boot any kernel or firmware. You can find the
patch here:
https://lists.nongnu.org/archive/html/qemu-devel/2025-09/msg05366.html
I wanted to send a special thank you for your suggestion to use qtest.
It was a pivotal piece of advice that unblocked this entire effort.
To give you some context, my initial approach was much more complex.
Based on the method we discussed earlier
([https://github.com/hnusdr/qemu/tree/smmuv3-test-device]), I had added
a new memory map in virt.c and heavily modified the SMMU code to create
a non-standard platform device. This setup was coupled with the Hafnium
firmware for testing, which was clearly not a sustainable long-term
solution.
My next idea was to create a self-contained device using QMP to make
test easier. It was at that point you suggested I look into qtest, and
that's when I realized qtest already works on the exact QMP mechanism I
was considering. That was the missing piece for me, and it directly led
to this new test device patch.
If you or others have a moment, I would greatly appreciate any review or
feedback on whether this test device is a suitable approach for our
testing needs.
Thanks again for your invaluable guidance.
Best,
Tao
^ permalink raw reply [flat|nested] 47+ messages in thread
end of thread, other threads:[~2025-09-30 4:05 UTC | newest]
Thread overview: 47+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-06 15:11 [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Tao Tang
2025-08-06 15:11 ` [RFC 01/11] hw/arm/smmuv3: Introduce secure registers and commands Tao Tang
2025-08-11 10:22 ` Philippe Mathieu-Daudé
2025-08-11 10:43 ` Philippe Mathieu-Daudé
2025-08-18 21:21 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 02/11] hw/arm/smmuv3: Implement read/write logic for secure registers Tao Tang
2025-08-06 21:53 ` Pierrick Bouvier
2025-08-10 16:54 ` Tao Tang
2025-08-12 17:12 ` Pierrick Bouvier
2025-08-18 21:24 ` Mostafa Saleh
2025-08-20 15:21 ` Tao Tang
2025-08-23 10:41 ` Mostafa Saleh
2025-09-11 15:27 ` Tao Tang
2025-09-15 9:14 ` Mostafa Saleh
2025-09-15 9:34 ` Eric Auger
2025-08-06 15:11 ` [RFC 03/11] hw/arm/smmuv3: Implement S_INIT for secure initialization Tao Tang
2025-08-18 21:26 ` Mostafa Saleh
2025-08-20 16:01 ` Tao Tang
2025-08-06 15:11 ` [RFC 04/11] hw/arm/smmuv3: Enable command processing for the Secure state Tao Tang
2025-08-06 21:55 ` Pierrick Bouvier
2025-08-10 16:59 ` Tao Tang
2025-08-11 10:34 ` Philippe Mathieu-Daudé
2025-08-12 17:27 ` Pierrick Bouvier
2025-08-12 17:39 ` Philippe Mathieu-Daudé
2025-08-12 18:42 ` Peter Maydell
2025-08-15 6:02 ` Tao Tang
2025-08-15 14:53 ` Peter Maydell
2025-08-17 3:46 ` Tao Tang
2025-08-06 15:11 ` [RFC 05/11] hw/arm/smmuv3: Support secure event queue and error handling Tao Tang
2025-08-11 10:41 ` Philippe Mathieu-Daudé
2025-08-06 15:11 ` [RFC 06/11] hw/arm/smmuv3: Plumb security state through core functions Tao Tang
2025-08-18 21:28 ` Mostafa Saleh
2025-08-20 16:25 ` Tao Tang
2025-08-23 10:43 ` Mostafa Saleh
2025-08-06 15:11 ` [RFC 07/11] hw/arm/smmuv3: Add separate address space for secure SMMU accesses Tao Tang
2025-08-06 15:11 ` [RFC 08/11] hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations Tao Tang
2025-08-06 15:11 ` [RFC 09/11] hw/arm/smmuv3: Make the configuration cache security-state aware Tao Tang
2025-08-06 15:11 ` [RFC 10/11] hw/arm/smmuv3: Differentiate secure TLB entries via keying Tao Tang
2025-08-06 21:11 ` [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State Pierrick Bouvier
2025-08-06 21:28 ` Pierrick Bouvier
2025-08-10 16:11 ` Tao Tang
2025-08-11 10:26 ` Philippe Mathieu-Daudé
2025-08-12 17:50 ` Pierrick Bouvier
2025-08-12 18:04 ` Pierrick Bouvier
2025-08-15 5:49 ` Tao Tang
2025-09-30 4:04 ` Tao Tang
2025-08-18 21:52 ` Mostafa Saleh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).