* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Gavin Shan @ 2026-03-25 6:37 UTC (permalink / raw)
To: Steven Price, Mathieu Poirier
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Suzuki K Poulose, Zenghui Yu,
linux-arm-kernel, linux-kernel, Joey Gouly, Alexandru Elisei,
Christoffer Dall, Fuad Tabba, linux-coco, Ganapatrao Kulkarni,
Shanker Donthineni, Alper Gun, Aneesh Kumar K . V, Emi Kisanuki,
Vishal Annapurve
In-Reply-To: <37bc1222-6fc7-48f0-94d3-6eaac420aa55@arm.com>
Hi Steven,
On 3/21/26 2:45 AM, Steven Price wrote:
> On 19/03/2026 23:02, Mathieu Poirier wrote:
[...]
>>>
>>> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
>>> use the following branch:
>>>
>>> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>>
>> This RMM version is expecting a RMM EL3 interface version of at least 2.0. Do
>> you have a TF-A to use with it?
>
> You should be able to use the 'master' branch of the TF-A repository.
> For now you need to set RMM_V1_COMPAT=0 to enable 2.0 support.
>
In upstream TF-A repository [1], I don't see the config option 'RMM_V1_COMPAT'.
would it be something else?
[1] git@github.com:ARM-software/arm-trusted-firmware.git (branch: master)
I use the following command to build TF-A image. The RMM-EL3 compatible issue is
still seen.
TFA_PATH=$PWD
EDK2_IMAGE=${TFA_PATH}/../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
RMM_IMAGE=${TFA_PATH}/../tf-rmm/build-qemu/Debug/rmm.img
make CROSS_COMPILE=aarch64-none-elf- \
PLAT=qemu ENABLE_RME=1 RMM_V1_COMPAT=0 DEBUG=1 LOG_LEVEL=40 \
QEMU_USE_GIC_DRIVER=QEMU_GICV3 \
BL33=${EDK2_IMAGE} RMM=${RMM_IMAGE} \
-j 8 all fip
Booting messages
================
INFO: BL31: Initializing runtime services
INFO: RMM setup done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
ERROR: RMM init failed: -2
WARNING: BL31: RMM initialization failed
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
From: Dan Williams @ 2026-03-25 4:13 UTC (permalink / raw)
To: Jason Gunthorpe, Dan Williams
Cc: Greg KH, linux-coco, linux-pci, aik, aneesh.kumar, yilun.xu,
bhelgaas, alistair23, lukas, Christoph Hellwig, Marek Szyprowski,
Robin Murphy, Roman Kisel, Samuel Ortiz, Rafael J. Wysocki,
Danilo Krummrich
In-Reply-To: <20260324123649.GY7340@nvidia.com>
Jason Gunthorpe wrote:
[..]
> I feel like starting with trust=0 is much cleaner than using
> autoprobe. Especially since it would be nice that when you do
> ultimately set trust!=0 then you do want the kernel to do the normal
> autoprobe flow.
>
> Double so because I would like the iommu drivers to respond to trust 0
> by fully blocking the device 100% of the time without holes, so to
> make that work I would like to see the struct device report trust 0
> the moment the iommu framework attaches the iommu.
>
> How you decide the starting trust value for device during system boot
> is definately something we need to discuss properly..
>
> I liked your idea of using built in driver match, so if there is a
> simple command line paramater that says 'only built in is trusted'
> then we'd default all devices to untrusted and during device probe
> check if any built in driver is matching and auto-set trust to X based
> on the commandline parameter.
I do agree that forcing trust=0 at the beginning of time is attractive
and theoretically clean. I am concerned about subsystems that are not
prepared for driver attach failures. For example, I would not expect to
need to set trust for auxiliary bus devices if the host device is
trusted.
However, the work to set module autoprobe policy is on the same order as
adding a module scoped trust policy.
So something like "modprobe $module trust=X" automatically tries to set
the device trust level on attach to any drivers in that module. That
could allow a semantic of "attach iff device is able to go to level 4".
> With the idea that only devices required to get to the initrd are
> built in. Then the initrd userspace has the policy to bring more
> devices into trusted!=0 to get to the root file system, then the
> rootfs has more policy for further devices, and so on.
>
> Probably this would ultimately escalate into annotations in the
> modinfo about default policies for various drivers.
Yes, escalate over time for subsystems to say "devices I create are
trusted, the responsibility to manage trust lies with clients of my
APIs".
> A kernel default policy of trusting everything without a "trust ops"
> (see below) may also be quite reasonable, however boot ordering the
> trust ops might be really tricky...
Given the optionality in selecting a trust ops provider I think it gets
extra messy quickly. Let me see how far I can get with built-in auto
trust + module trust policy.
> > > > > The DMA API just wants a flag in the struct device that says if the
> > > > > device can access encrypted memory or only decrypted.
> > > >
> > > > You mean separate "trusted to access private" and "currently enabled to
> > > > access private" properties? I am trying to think of a situation where
> > > > "dev->trust >= 3" and a flag saying "disable bouncing for encrypted
> > > > memory" would ever disagree.
> > >
> > > I'm steering the trust level toward more of an acceptance criteria.
> > > If the trust level is you have access to private memory but the device
> > > can't actually do that then fail the trust level change.
> > >
> > > Same for the reverse, if the trust level says no private memory and the
> > > device is T=1 then fail the trust level change.
> >
> > Ok, so the uapi for PCI/TDISP would be:
> >
> > echo $tsm > $pdev/tsm/lock
> > <gather evidence, validate with relying party>
> > echo 3 > $pdev/trust
> >
> > ...where that @trust attribute is a generic device semantic, but in the
> > case of PCI device connected to a given TSM it invokes the TSM hypercall
> > to transition the device to the RUN state and the TSM local call to
> > unblock DMA to private memory.
>
> Maybe, but I was thinking the transition through run/locked would be
> done through TSM uAPIs too. trust setting in the kernel just confirms
> the device is in the right state.
>
> But I haven't thought of a reason why the final switch to RUN couldn't
> happen like this either.
Right, the potential to see in-between states concerns me because TSM
uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trust is still showing the device at some lower level of trust. So
I think trust modification needs to be synchronous with privileges
granted/revoked.
[..]
> > The complication vs benefit tradeoff is indeed not mathing, but wanted
> > to do justice to Aneesh's proposal and the suitability of the sysfs
> > uapi.
>
> I think if you want something like this then it is better to target
> the root - remove the ability for concurrent userspace to wrongly
> operate the TSM entirely. Ie use a cdev, make it so going to LOCKED
> isolates access to only this cdev fd and require only this cdev fd to
> go to RUN. Then these kinds of bugs don't exist.
The netlink evidence proposal can handle this, it just needs a
'validate' command. 'Validate' records a device evidence generation
number. Require a stable generation number between entering LOCK and the
trust=4 event transitioning the device to RUN.
Yes, not as safe as fd-private LOCK-to-RUN, but I like semantic of
"modprobe $module trust=4" to say "transition to RUN and attach to all
validated devices".
[..]
> Basically the 'trust' generic framework sits on top of some "trust
> ops" that will be provided by the security module that is affiliated
> with the struct device (ie thunderbolt, TSM TDISP, TSM Link IDE, etc,
> etc)
>
> Then it becomes a general synchronization point where on one side the
> "tust ops" can ack that the level is acceptable and consistent with
> the system when on the other side generic compoments like IOMMU,
> driver binding, etc can respond to it and change their behavior.
Something like this, yes.
> > For bare metal PCI device security the TSM 'connection' needs to be
> > established in order to enable device evidence collection.
> >
> > echo $tsm > $pdev/tsm/connect
> > <validate device evidence>
> > echo 2 > $pdev/trust
> >
> > Now, I question whether 5 trust levels instead of 4. This would be to
> > explicitly only trust devices where the TSM has established physical
> > link encryption, or the TSM has asserted that the link is protected by
> > other means. So the trust levels are:
>
> I probably wouldn't use an int for the uAPI,
Yes, but an int for now saves the bikeshed of the level names till a bit
later.
> ...but yes picking the initial levels is important. As above since
> this is a clearing point between two different worlds it needs to be
> defined in some way both sides can understand what it means for them.
>
> > 0 disconnected: bus does not attach drivers
> > 1 limited: core code deploys hostile device mitigations like disable
> > ATS, CC force shared memory bouncing.
> > 2 DMA: full DMA access, driver responsible for protecting against
> > adverarial devices.
> > 3 Link: mitigations against physical integrity and snooping attacks
> > deployed
> > 4 TCB: full device interface validation via a protocol like TDISP,
> > CC private memory access granted.
>
> This seems reasonable to me, the 3/4 distinction is not meaningful for
> the iommu&dev side, but it does provide a good check point for the
> "trust ops". If userspace ack's that it expects physical security and
> the kernel says it isn't physically secure (or becomes insecure later)
> then it should fail.
>
> > Where the native Rust library based SPDM driver only offers trust level
> > 2, bare metal TSMs can support trust level 3, and the TSM interfaces in
> > CC VMs can support trust level 4.
>
> I'm not sure that the SPDM driver even provides a "trust ops" right? I
> would guess that 0/1/2 are simply built in always available if trust ops are
> NULL and 3/4 require positive reply from the ops to accept it.
Right. Even though the SPDM driver allows device evidence to be
collected there is no event like LOCK-to-RUN that expects validated
evidence as a precondition.
> So #3 needs a "trust ops" linked to enabling link IDE.. If this is
> done in-kernel the link IDE module is providing the trust ops and just
> using SPDM as a library to establish the link IDE keys?
Yes, but note to date only Intel platforms allow for IDE establishment
without talking to the platform TSM.
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Gavin Shan @ 2026-03-25 4:07 UTC (permalink / raw)
To: Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Suzuki K Poulose, Zenghui Yu, linux-arm-kernel,
linux-kernel, Joey Gouly, Alexandru Elisei, Christoffer Dall,
Fuad Tabba, linux-coco, Ganapatrao Kulkarni, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-1-steven.price@arm.com>
Hi Steven,
On 3/19/26 1:53 AM, Steven Price wrote:
>
> This series is based on v7.0-rc1. It is also available as a git
> repository:
>
> https://gitlab.arm.com/linux-arm/linux-cca cca-host/v13
>
> Work in progress changes for kvmtool are available from the git
> repository below:
>
> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v11
>
Could you please share if we have a working qemu repository on top of this
(v13) series? The previous qemu repository [1] seems out of dated for long
time. I heard Jean won't be able to continue his efforts on QEMU part, who
is going to pick it up in this case.
[1] https://git.codelinaro.org/linaro/dcap/qemu.git (branch: cca/latest)
> Note that the kvmtool code has been tidied up (thanks to Suzuki) and
> this involves a minor change in flags. The "--restricted_mem" flag is no
> longer recognised (or necessary).
>
> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
> use the following branch:
>
> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>
I'm seeing error to initialize RMM with the suggested RMM branch (topics/rmm-v2.0-poc)
and the upstream TF-A [1]. It seems the problem is compatible issue in the
RMM-EL3 interface. RMM requires verion 2.0 while TF-A only supports 0.8. So
I guess I must be using a wrong TF-A repository. Could you please share which
TF-A repository you use for testing?
[1] git@github.com:ARM-software/arm-trusted-firmware.git (branch: master)
Booting logs
=============
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v2.14.0(debug):67edb4f8e
NOTICE: BL1: Built : 00:01:39, Mar 25 2026
INFO: BL1: RAM 0xe0ee000 - 0xe0f7000
INFO: BL1: Loading BL2
INFO: Loading image id=1 at address 0xe05b000
INFO: Image id=1 loaded: 0xe05b000 - 0xe0642bc
NOTICE: BL1: Booting BL2
INFO: Entry point address = 0xe05b000
INFO: SPSR = 0x3cd
NOTICE: BL2: v2.14.0(debug):67edb4f8e
NOTICE: BL2: Built : 00:01:39, Mar 25 2026
INFO: BL2: Doing platform setup
INFO: Reserved RMM memory [0x40100000, 0x418fffff] in Device tree
INFO: BL2: Loading image id 3
INFO: Loading image id=3 at address 0xe090000
INFO: Image id=3 loaded: 0xe090000 - 0xe0a292b
INFO: BL2: Loading image id 35
INFO: Loading image id=35 at address 0x40100000
INFO: Image id=35 loaded: 0x40100000 - 0x401a11e0
INFO: BL2: Loading image id 5
INFO: Loading image id=5 at address 0x60000000
INFO: Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE: BL2: Booting BL31
INFO: Entry point address = 0xe090000
INFO: SPSR = 0x3cd
INFO: GPT: Boot Configuration
INFO: PPS/T: 0x2/40
INFO: PGS/P: 0x0/12
INFO: L0GPTSZ/S: 0x0/30
INFO: PAS count: 6
INFO: L0 base: 0xedfe000
INFO: Enabling Granule Protection Checks
NOTICE: BL31: v2.14.0(debug):67edb4f8e
NOTICE: BL31: Built : 00:01:39, Mar 25 2026
INFO: GICv3 without legacy support detected.
INFO: ARM GICv3 driver initialized in EL3
INFO: Maximum SPI INTID supported: 287
INFO: BL31: Initializing runtime services
INFO: RMM setup done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
ERROR: RMM init failed: -2 <<<< Error raised by RMM here
WARNING: BL31: RMM initialization failed
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x60000000
INFO: SPSR = 0x3c9
UEFI firmware (version built at 19:33:51 on Mar 3 2026)
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v2 02/16] fs, x86/resctrl: Add architecture routines for kernel mode initialization
From: Reinette Chatre @ 2026-03-24 22:53 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <72315f83ddf35b0de53fb911f46623782d95f507.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Implement the resctrl kernel mode (kmode) arch initialization.
>
> - Add resctrl_arch_get_kmode_cfg() to fill the default kernel mode
> (INHERIT_CTRL_AND_MON). This can be extended later (e.g. for PLZA) to set
> additional modes.
I do not think this is something that the architecture should set, at least
at this time. Every mode has different requirements and this just lets the arch set
it without any support for what configurations it implies. For example, if
arch sets a different default mode than INHERIT_CTRL_AND_MON then PQR_PLZA_ASSOC
needs to be programmed as the CPUs come online and this does not seem to
accommodate this. This implementation appears to have significant assumptions on
what architecture will end up setting since it is only considering PLZA.
>
> - Add global resctrl_kcfg and resctrl_kmode_init() to initialize default
> values.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
> ---
> arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
> fs/resctrl/rdtgroup.c | 10 ++++++++++
> 2 files changed, 17 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 7667cf7c4e94..4c3ab2d93909 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -892,6 +892,13 @@ bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
> }
> }
>
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg)
> +{
> + kcfg->kmode = INHERIT_CTRL_AND_MON;
> + kcfg->kmode_cur = INHERIT_CTRL_AND_MON;
> + kcfg->k_rdtgrp = NULL;
> +}
I already commented on the arch vs filesystem settings.
When using an arch helper this forces all architectures to support this helper. Is a
helper required? Is it perhaps possible for arch to set a property instead? For example,
how enumeration is handled?
I think the assumption here is that INHERIT_CTRL_AND_MON is the default and expected to
be supported by all architectures. I do not see why arch should set this as default but
instead this should be from resctrl fs. At the same time it is expected that the
architecture supports this mode so there needs to be a failure if an architecture does
not support this mode?
I'm going to stop here. I think the comments so far may result in major changes already
making further detailed review of patches unnecessary.
Reinette
^ permalink raw reply
* Re: [PATCH v2 01/16] fs/resctrl: Add kernel mode (kmode) data structures and arch hook
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <c3381fbbce357b00895252572907645a92cd9718.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Add resctrl_kmode, resctrl_kmode_cfg, kernel mode bit defines, and
> resctrl_arch_get_kmode_cfg() for resctrl kernel mode (e.g. PLZA) support.
We should not have to start every series from scratch.
Documentation/process/maintainer-tip.rst. Always.
> ---
> include/linux/resctrl.h | 10 ++++++++++
> include/linux/resctrl_types.h | 30 ++++++++++++++++++++++++++++++
> 2 files changed, 40 insertions(+)
>
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 006e57fd7ca5..2c36d1ac392f 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -699,6 +699,16 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
> */
> bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
>
> +/**
> + * resctrl_arch_get_kmode_cfg() - Get resctrl kernel mode configuration
> + * @kcfg: Filled with current kernel mode config (kmode, kmode_cur, k_rdtgrp).
> + *
> + * Used by the arch (e.g. x86) to report which kernel mode is active and,
> + * when a global assign mode is in use, which rdtgroup is assigned to
> + * kernel work.
> + */
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg);
This interface does not look right. Would it not be resctrl fs that determines
which resource group is assigned? This cannot be set by arch. Why does arch decide
which mode is active? Is this not also resctrl fs? Should arch not just tell
resctrl fs what it supports?
> +
> extern unsigned int resctrl_rmid_realloc_threshold;
> extern unsigned int resctrl_rmid_realloc_limit;
>
> diff --git a/include/linux/resctrl_types.h b/include/linux/resctrl_types.h
> index a5f56faa18d2..6b78b08eab29 100644
> --- a/include/linux/resctrl_types.h
> +++ b/include/linux/resctrl_types.h
> @@ -65,7 +65,37 @@ enum resctrl_event_id {
> QOS_NUM_EVENTS,
> };
>
> +/**
> + * struct resctrl_kmode - Resctrl kernel mode descriptor
> + * @name: Human-readable name of the kernel mode.
> + * @val: Bitmask value for the kernel mode (e.g. INHERIT_CTRL_AND_MON).
> + */
> +struct resctrl_kmode {
> + char name[32];
> + u32 val;
> +};
There is no reason why this needs to be in a central header exposed to archs. Could
this not be a static within the only function that uses it? Something like
rdt_mode_str[]?
> +
> +/**
> + * struct resctrl_kmode_cfg - Resctrl kernel mode configuration
> + * @kmode: Requested kernel mode.
> + * @kmode_cur: Currently active kernel mode.
> + * @k_rdtgrp: Resource control structure in use, or NULL otherwise.
> + */
> +struct resctrl_kmode_cfg {
> + u32 kmode;
> + u32 kmode_cur;
> + struct rdtgroup *k_rdtgrp;
> +};
> +
> #define QOS_NUM_L3_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
> #define MBM_STATE_IDX(evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
>
> +/* Resctrl kernel mode bits (e.g. for PLZA). */
> +#define INHERIT_CTRL_AND_MON BIT(0) /* Kernel uses same CLOSID/RMID as user. */
> +/* One CLOSID for all kernel work; RMID inherited from user. */
> +#define GLOBAL_ASSIGN_CTRL_INHERIT_MON BIT(1)
> +/* One resource group (CLOSID+RMID) for all kernel work. */
> +#define GLOBAL_ASSIGN_CTRL_ASSIGN_MON BIT(2)
> +#define RESCTRL_KERNEL_MODES_NUM 3
I think it will make the code much easier to understand if the different modes are described by an
enum. For example,
enum resctrl_kernel_modes {
INHERIT_CTRL_AND_MON,
GLOBAL_ASSIGN_CTRL_INHERIT_MON,
GLOBAL_ASSIGN_CTRL_ASSIGN_MON,
RESCTRL_KMODE_LAST = GLOBAL_ASSIGN_CTRL_ASSIGN_MON
};
#define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
The supported kernel modes can still be managed as a bitmap with intuitive API using the
enum that will make the code easier to read. For example, __set_bit(INHERIT_CTRL_AND_MON, ...)
or BIT(INHERIT_CTRL_AND_MON). The naming is awkward at the moment though, we should improve here.
Reinette
^ permalink raw reply
* Re: [PATCH v2 00/16] fs,x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <cover.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
>
> This series adds support for Privilege-Level Zero Association (PLZA) to the
> resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID
> and/or RMID for execution in kernel mode (privilege level zero), so that
> kernel work is not subject to the same resource constrains as the current
> user-space task. This avoids kernel operations being aggressively throttled
> when a task's memory bandwidth is heavily limited.
>
> The feature documentation is not yet publicly available, but it is expected
> to be released in the next few weeks. In the meantime, a brief description
> of the features is provided below.
>
> Privilege Level Zero Association (PLZA)
>
> Privilege Level Zero Association (PLZA) allows the hardware to
> automatically associate execution in Privilege Level Zero (CPL=0) with a
> specific COS (Class of Service) and/or RMID (Resource Monitoring
> Identifier). The QoS feature set already has a mechanism to associate
> execution on each logical processor with an RMID or COS. PLZA allows the
> system to override this per-thread association for a thread that is
> executing with CPL=0.
> ------------------------------------------------------------------------
>
> The series introduces the feature in a way that supports the interface in
> a generic manner to accomodate MPAM or other vendor specific implimentation.
>
> Below is the detailed requirements provided by Reinette:
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
Our discussion considered how resctrl could support PLZA in a generic way while
also preparing to support MPAM's variants and how PLZA may evolve to have similar
capabilities when considering the capabilities of its registers.
This does not mean that your work needs to implement everything that was discussed.
Instead, this work is expected to just support what PLZA is capable of today but
do so in a way that the future enhancements could be added to.
This series is quite difficult to follow since it appears to implement a full
featured generic interface while PLZA cannot take advantage of it.
Could you please simplify this work to focus on just enabling PLZA and only
add interfaces needed to do so?
>
> Summary:
> 1. Kernel-mode/PLZA controls and status should be exposed under the resctrl
> info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.
>
> 2. Add two info files
>
> a. kernel_mode
> Purpose: Control how resource allocation and monitoring apply in kernel mode
> (e.g. inherit from task vs global assign).
>
> Read: List supported modes and show current one (e.g. with [brackets]).
> Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).
>
> b. kernel_mode_assignment
>
> Purpose: When a “global assign” kernel mode is active, specify which resctrl group
> (CLOSID/RMID) is used for kernel work.
>
> Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
> Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).
>
> The patches are based on top of commit (v7.0.0-rc3)
> 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
> ------------------------------------------------------------------------
>
> Examples: kernel_mode and kernel_mode_assignment
>
> All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means
> /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform
> must support the relevant modes (e.g. AMD with PLZA).
>
> 1) kernel_mode — show and set the current kernel mode
>
> Read supported modes and which one is active (current in brackets):
>
> $ cat info/kernel_mode
> [inherit_ctrl_and_mon]
> global_assign_ctrl_inherit_mon
> global_assign_ctrl_assign_mon
>
> Set the active mode (e.g. use one CLOSID+RMID for all kernel work):
>
> $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
> $ cat info/kernel_mode
> inherit_ctrl_and_mon
> global_assign_ctrl_inherit_mon
> [global_assign_ctrl_assign_mon]
>
> Mode meanings:
> - inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
> - global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
> - global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.
>
> 2) kernel_mode_assignment — show and set which group is used for kernel work
>
> Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the
To help with future usages please connect visibility of this file with the mode in
info/kernel_mode. This helps us to support future modes with other resctrl files, possible
within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon",
while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.
> currently assigned group (path format is "CTRL_MON/MON/"):
The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon"
then it should only contain a control group, alternatively, if the mode is
"global_assign_ctrl_assign_mon" then it contains control and mon group. This gives
resctrl future flexibility to change format for future modes.
We should also consider the scenario when it is a "monitoring only" system, which can
happen independent from what hardware actually supports, for example, if user boots
with "rdt=!l3cat,!l2cat,!mba,!smba". In this case I assume CLOS should just always be
zero and thus only "default control group" is accepted?
>
> $ cat info/kernel_mode_assignment
> //
>
> "//" means the default CTRL_MON group is assigned. Assign a specific
> group instead (e.g. a CTRL_MON group "ctrl1", or a MON group "mon1" under it):
>
> $ echo "ctrl1//" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1//
>
> $ echo "ctrl1/mon1/" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1/mon1/
>
> Clear the assignment (no dedicated group for kernel work):
>
> $ echo >> info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> Kmode is not configured
This does not look right. Would this not create a conflict between info/kernel_mode
and info/kernel_mode_assignment about what the current mode is? The way I see it
info/kernel_mode_assignment must always contain a valid group.
>
> Errors (e.g. invalid group name or unsupported mode) are reported in
> info/last_cmd_status.
>
> ---
>
> v2:
> This is similar to RFC with new proposal. Names of the some interfaces
> are not final. Lets fix that later as we move forward.
>
> Separated the two features: Global Bandwidth Enforcement (GLBE) and
> Privilege Level Zero Association (PLZA).
>
> This series only adds support for PLZA.
>
> Used the name of the feature as kmode instead of PLZA. That can be changed as well.
>
> Tony suggested using global variables to store the kernel mode
> CLOSID and RMID. However, the kernel mode CLOSID and RMID are
> coming from rdtgroup structure with the new interface. Accessing
> them requires holding the associated lock, which would make the
> context switch path unnecessarily expensive. So, dropped the idea.
> https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
> Let me know if there are other ways to optimize this.
I do not see why the context switch path needs to be touched at all with this
implementation. Since PLZA only supports global assignment does it not mean that resctrl
only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and
info/kernel_mode_assignment?
Consider some of the scenarios:
resctrl mount with default state:
# cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_inherit_mon
global_assign_ctrl_assign_mon
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
enable global_assign_ctrl_assign_mon mode:
# echo "global_assign_ctrl_assign_mon" > info/kernel_mode
Expectation here is that when user space sets this mode as above then resctrl would
in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
I do not see why it is necessary to maintain any per-CPU or per-task state or needing
to touch the context switch code. Since PLZA only supports global could it not
just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it?
Only caveat is that if a CPU is offline then this setting needs to be stashed
so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.
The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing
with RDT_RESOURCE_L3 and traverses the resource domain list and resource group
CPU mask seems unnecessary to me as well as error prone since the system may only
have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set
MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?
To continue the scenarios ...
After user's setting above related files read:
# cat info/kernel_mode
inherit_ctrl_and_mon
global_assign_ctrl_inherit_mon
[global_assign_ctrl_assign_mon]
# cat info/kernel_mode_assignment
//
Modify group used by global_assign_ctrl_assign_mon mode:
# echo 'ctrl1/mon1/' > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1>
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
Enable global_assign_ctrl_inherit_mon mode:
# echo "global_assign_ctrl_inherit_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode
inherit_ctrl_and_mon
[global_assign_ctrl_inherit_mon]
global_assign_ctrl_assign_mon
# cat info/kernel_mode_assignment <==== returns just a ctrl group
/
Modify group used by global_assign_ctrl_inherit_mon mode:
# echo ctrl1 > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode_assignment <==== returns just a ctrl group
ctrl/
Enable inherit_ctrl_and_mon mode:
# echo "inherit_ctrl_and_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=0
MSR_IA32_PQR_PLZA_ASSOC.plza_en=0
At this point info/kernel_mode_assignment is not visible anymore:
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
From what I understand above exposes and enables full capability of PLZA. All the other
per-task and per-cpu handling in this series is not something that PLZA can benefit from.
If this is not the case, what am I missing? Could this series be simplified to just support
PLZA today? When next hardware with more capability needs to be supported resctrl could be
enhanced to support it by using the more accurate information about what the hardware is
capable of.
We also do not really know what use cases users prefer. This may even be sufficient.
Reinette
^ permalink raw reply
* Re: [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Edgecombe, Rick P @ 2026-03-24 22:03 UTC (permalink / raw)
To: bp@alien8.de, marcandre.lureau@redhat.com, kas@kernel.org,
hpa@zytor.com, mingo@redhat.com, x86@kernel.org, tglx@kernel.org,
dave.hansen@linux.intel.com, Qiang, Chenyi
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev,
linux-kernel@vger.kernel.org
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-2-8f29f2c17278@redhat.com>
On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> accepted before use. The first access triggers a fatal "SEPT entry in
> PENDING state" EPT violation and KVM terminates the guest.
>
> Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> calls tdx_accept_memory() for the range being onlined.
>
> The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> memory from going online.
Does this depend on patch 1 somehow?
^ permalink raw reply
* Re: [PATCH 1/2] x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
From: Edgecombe, Rick P @ 2026-03-24 22:02 UTC (permalink / raw)
To: bp@alien8.de, marcandre.lureau@redhat.com, kas@kernel.org,
hpa@zytor.com, mingo@redhat.com, x86@kernel.org, tglx@kernel.org,
dave.hansen@linux.intel.com, Qiang, Chenyi
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev,
linux-kernel@vger.kernel.org
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-1-8f29f2c17278@redhat.com>
On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> try_accept_one() treats any non-zero return from __tdcall() as a
> failure. However, per the TDX Module Base Spec (Table SEPT Walk Cases),
> TDG.MEM.PAGE.ACCEPT returns a non-zero status code with bit 63 clear
> when the target page is already in MAPPED state (i.e., already
> accepted). This is a "success-with-warning" -- the page is usable and no
> action is needed.
>
> Check only bit 63 (TDX_ERROR) to distinguish real errors from
> success-with-warning returns, rather than treating all non-zero values
> as failures.
>
> Assisted-by: Claude:claude-opus-4-6
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Hmm. Accepting private memory is a security sensitive operation, so I think it
is probably bad to silently hide the detection of re-accepting.
For example, if the kernel accepts a page and sets some values in it, the VMM
could reset the data to zero by re-adding the page and letting the second accept
zero it. It allows the VMM to have some limited ability to mess with guest data.
If we detect a re-accept we should probably warn on it actually.
Not sure on if the specific case in this series is problematic, but this patch
changes the behavior generally.
^ permalink raw reply
* SVSM Development Call March 25, 2026
From: Jörg Rödel @ 2026-03-24 19:14 UTC (permalink / raw)
To: coconut-svsm, linux-coco
Hi,
Here is the call for agenda items for this weeks SVSM development call. Please
send any agenda items you have in mind as a reply to this email or raise them
in the meeting.
We will use the LF Zoom instance. Details of the meeting can be found in our
governance repository at:
https://github.com/coconut-svsm/governance
The link to the COCONUT-SVSM calendar is:
https://zoom-lfx.platform.linuxfoundation.org/meetings/coconut-svsm?view=week
The meeting will be recorded and the recording eventually published.
Regards,
Jörg
^ permalink raw reply
* Re: [PATCH v4 1/2] dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
From: Jason Gunthorpe @ 2026-03-24 19:11 UTC (permalink / raw)
To: Jiri Pirko
Cc: Robin Murphy, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, leon, sean.anderson,
ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco
In-Reply-To: <lufy3ppgiy7637iravgzabajj67chmqvg22qc3fd2yqe2p33ws@b54vibhehth2>
On Tue, Mar 17, 2026 at 09:26:21AM +0100, Jiri Pirko wrote:
> >...although, why *shouldn't* this be allowed with a vIOMMU? (Especially given
> >that a vIOMMU for untrusted devices can be emulated by the host VMM without
> >the CoCo hypervisor having to care at all - again, at least on Arm and other
> >architectures where IOMMUs are regular driver model devices)
>
> Well, when iommu path is able to consume the attr, this restriction
> should be lifted. This is basically a sanity check for the
> dma_map_phys() caller.
Right we eventually need a matching IOMMU_DECRYPTED.
It needs to mirror how the CPUs work - any place that would use
pgprot_decrypted to create a PTE should use IOMMU_PROT_DECRYPTED to
create an iommu mapping.
The current hack in AMD assumes IOMMU_DECRYPTED behavior for
IOMMU_MMIO, but that isn't general enough..
There is some maze to get there but for the moment I think it is fine
to just not support vIOMMU, it isn't like any vIOMMU drivers even
exist for CC VMs right now.
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Mostafa Saleh @ 2026-03-24 18:32 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260324175717.GE8437@ziepe.ca>
On Tue, Mar 24, 2026 at 5:57 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Mar 24, 2026 at 05:36:23PM +0000, Mostafa Saleh wrote:
> > But it's not about drivers in that case, it's about many places
> > (SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
> > ownership so in some cases they step on each other's toes, and I don't
> > think that will get simpler with yet another caller in this series
>
> I don't understand how this can be, ownership is clear. SWIOTLB owns
> the buffer, dma alloc coherent owns the buffer, user owns the
> buffer. There should be no other cases, and they don't step on each
> other unless the APIs are being used wrong.
>
Logically, that's the case, but the DMA-direct code currently loses
this information and assumes it can encrypt/decrypt any memory even
the SWIOTLB one.
That's what I am fixing in my series. When I respin, I can try to
introduce some more helpers around that to make it easier to integrate
new cases.
Thanks,
Mostafa
> > I am fine with the API design you mentioned, but I believe that it
> > needs clear documentation specifying who is responsible for
> > decryption. The code should provide wrappers checking for these cases
> > instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
> > everywhere in DMA-direct.
>
> Redoingt how dma-api works internally is some other project... It
> would be nice if swiotlb would sort of recursively DMA map using the
> new flag instead of open coding it but that is pretty minor.
>
> Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jason Gunthorpe @ 2026-03-24 17:57 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <CAFgf54qwA2D1Xa4rnruJ4Nfp5BsB=T_pB3hzz9HBjh22TL17uA@mail.gmail.com>
On Tue, Mar 24, 2026 at 05:36:23PM +0000, Mostafa Saleh wrote:
> But it's not about drivers in that case, it's about many places
> (SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
> ownership so in some cases they step on each other's toes, and I don't
> think that will get simpler with yet another caller in this series
I don't understand how this can be, ownership is clear. SWIOTLB owns
the buffer, dma alloc coherent owns the buffer, user owns the
buffer. There should be no other cases, and they don't step on each
other unless the APIs are being used wrong.
> I am fine with the API design you mentioned, but I believe that it
> needs clear documentation specifying who is responsible for
> decryption. The code should provide wrappers checking for these cases
> instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
> everywhere in DMA-direct.
Redoingt how dma-api works internally is some other project... It
would be nice if swiotlb would sort of recursively DMA map using the
new flag instead of open coding it but that is pretty minor.
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Mostafa Saleh @ 2026-03-24 17:36 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260324122416.GD8437@ziepe.ca>
On Tue, Mar 24, 2026 at 12:24 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Mar 24, 2026 at 12:14:36PM +0000, Mostafa Saleh wrote:
> > On Tue, Mar 24, 2026 at 12:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Tue, Mar 17, 2026 at 01:24:13PM +0000, Mostafa Saleh wrote:
> > >
> > > > On the other hand, for restricted-dma, the memory decryption is deep
> > > > in the DMA direct memory allocation and the DMA API callers (for ex
> > > > virtio drivers) are clueless about it and can’t pass any attrs.
> > > > My proposal was specific to restricted-dma and won’t work for your case.
> > >
> > > How is this any different from CC?
> > >
> > > If the device cannot dma to "encrypted" memory, whatever that means
> > > for you, then the DMA API:
> > > - Makes dma alloc coherent return "decrypted" memory, and the built
> > > in mapping of coherent memory knows about this
> > > - Makes dma_map_xxx use SWIOTLB to bounce to decrypted memory
> > >
> > > There is no need for something like virtio drivers to be aware of
> > > any of this.
> > >
> > > On the other hand if the driver deliberately allocates decrypted
> > > memory without using DMA API alloc coherent then it knows it did it
> > > and can pass the flag to map it.
> > >
> >
> > The problem is that the DMA API currently gets confused by this; it
> > can end up double decrypting the memory or using the wrong functions
> > as mentioned in [1]
>
> I fully belive there are bugs, but the API design is sound. If you use
> the coherent allocations from the DMA API then it knows decryption has
> happened when it generates a dma_addr_t and there should be no issue.
>
> Now, if drivers are using the DMA API wrong, like trying to double map
> coherent allocations then they are broken. I also would not be
> surprised to find cases like this.
>
But it's not about drivers in that case, it's about many places
(SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
ownership so in some cases they step on each other's toes, and I don't
think that will get simpler with yet another caller in this series
I am fine with the API design you mentioned, but I believe that it
needs clear documentation specifying who is responsible for
decryption. The code should provide wrappers checking for these cases
instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
everywhere in DMA-direct.
Thanks,
Mostafa
> Jason
^ permalink raw reply
* [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-0-8f29f2c17278@redhat.com>
In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
accepted before use. The first access triggers a fatal "SEPT entry in
PENDING state" EPT violation and KVM terminates the guest.
Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
calls tdx_accept_memory() for the range being onlined.
The notifier returns NOTIFY_BAD on acceptance failure, preventing the
memory from going online.
Assisted-by: Claude:claude-opus-4-6
Reported-by: Chenyi Qiang <chenyi.qiang@intel.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
index 7b2833705d475..89f90bc303258 100644
--- a/arch/x86/coco/tdx/tdx.c
+++ b/arch/x86/coco/tdx/tdx.c
@@ -8,6 +8,7 @@
#include <linux/export.h>
#include <linux/io.h>
#include <linux/kexec.h>
+#include <linux/memory.h>
#include <asm/coco.h>
#include <asm/tdx.h>
#include <asm/vmx.h>
@@ -1194,3 +1195,40 @@ void __init tdx_early_init(void)
tdx_announce();
}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static int tdx_guest_memory_notifier(struct notifier_block *nb,
+ unsigned long action, void *v)
+{
+ struct memory_notify *mn = v;
+ phys_addr_t start, end;
+
+ if (action != MEM_GOING_ONLINE)
+ return NOTIFY_OK;
+
+ start = PFN_PHYS(mn->start_pfn);
+ end = start + PFN_PHYS(mn->nr_pages);
+
+ if (!tdx_accept_memory(start, end)) {
+ pr_err("Failed to accept memory [0x%llx, 0x%llx)\n",
+ (unsigned long long)start,
+ (unsigned long long)end);
+ return NOTIFY_BAD;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block tdx_guest_memory_nb = {
+ .notifier_call = tdx_guest_memory_notifier,
+};
+
+static int __init tdx_guest_memory_init(void)
+{
+ if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
+ return 0;
+
+ return register_memory_notifier(&tdx_guest_memory_nb);
+}
+core_initcall(tdx_guest_memory_init);
+#endif
--
2.53.0
^ permalink raw reply related
* [PATCH 1/2] x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-0-8f29f2c17278@redhat.com>
try_accept_one() treats any non-zero return from __tdcall() as a
failure. However, per the TDX Module Base Spec (Table SEPT Walk Cases),
TDG.MEM.PAGE.ACCEPT returns a non-zero status code with bit 63 clear
when the target page is already in MAPPED state (i.e., already
accepted). This is a "success-with-warning" -- the page is usable and no
action is needed.
Check only bit 63 (TDX_ERROR) to distinguish real errors from
success-with-warning returns, rather than treating all non-zero values
as failures.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
arch/x86/coco/tdx/tdx-shared.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/coco/tdx/tdx-shared.c b/arch/x86/coco/tdx/tdx-shared.c
index 1655aa56a0a51..24983601a2ded 100644
--- a/arch/x86/coco/tdx/tdx-shared.c
+++ b/arch/x86/coco/tdx/tdx-shared.c
@@ -35,7 +35,7 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len,
}
args.rcx = start | page_size;
- if (__tdcall(TDG_MEM_PAGE_ACCEPT, &args))
+ if (__tdcall(TDG_MEM_PAGE_ACCEPT, &args) & TDX_ERROR)
return 0;
return accept_size;
--
2.53.0
^ permalink raw reply related
* [PATCH 0/2] x86/tdx: Fix memory hotplug in TDX guests
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In TDX guests, hotplugged memory (e.g., via virtio-mem) must be accepted
via TDG.MEM.PAGE.ACCEPT before use. The first access to an unaccepted
page triggers a fatal "SEPT entry in PENDING state" EPT violation and
KVM terminates the guest.
This was discovered while testing virtio-mem resize with TDX guests.
The associated QEMU virtio-mem + TDX patch series is under review at:
https://patchew.org/QEMU/20260226140001.3622334-1-marcandre.lureau@redhat.com/
The fix has two parts:
1. Handle TDG.MEM.PAGE.ACCEPT "success-with-warning" returns for pages
that are already in MAPPED state (e.g., after offline/re-online
cycles), instead of treating them as fatal errors.
2. Register a MEM_GOING_ONLINE memory hotplug notifier that calls
tdx_accept_memory() before pages are freed to the buddy allocator.
The TDCALL transparently triggers KVM-side page augmentation (AUG)
followed by acceptance, avoiding the fatal EPT violation path.
The solution was suggested by Claude Code (Anthropic) and has been
tested with virtio-mem hot-add on a TDX guest. I did my best to review
the produced code and comments. Apologies if the agent did hallucinate.
Let me know if I need to check or correct something.
Thanks,
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
Marc-André Lureau (2):
x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
x86/tdx: Accept hotplugged memory before online
arch/x86/coco/tdx/tdx-shared.c | 2 +-
arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
2 files changed, 39 insertions(+), 1 deletion(-)
---
base-commit: c369299895a591d96745d6492d4888259b004a9e
change-id: 20260324-tdx-hotplug-fixes-644d009dad63
Best regards,
--
Marc-André Lureau <marcandre.lureau@redhat.com>
^ permalink raw reply
* Re: [PATCH v4 2/2] dma-buf: heaps: system: add system_cc_decrypted heap for explicitly decrypted memory
From: Jiri Pirko @ 2026-03-24 12:58 UTC (permalink / raw)
To: T.J. Mercier
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, christian.koenig,
m.szyprowski, robin.murphy, jgg, leon, sean.anderson, ptesarik,
catalin.marinas, aneesh.kumar, suzuki.poulose, steven.price,
thomas.lendacky, john.allen, ashish.kalra, suravee.suthikulpanit,
linux-coco
In-Reply-To: <CABdmKX0Ux2HMTTuOnvZryBBRwCjH0zUe-EMvL9k1VxZtz+BWiA@mail.gmail.com>
Tue, Mar 24, 2026 at 12:25:46AM +0100, tjmercier@google.com wrote:
>On Mon, Mar 16, 2026 at 5:59 AM Jiri Pirko <jiri@resnulli.us> wrote:
[..]
>> --- a/drivers/dma-buf/heaps/system_heap.c
>> +++ b/drivers/dma-buf/heaps/system_heap.c
>> @@ -10,17 +10,25 @@
>> * Andrew F. Davis <afd@ti.com>
>> */
>>
>> +#include <linux/cc_platform.h>
>> #include <linux/dma-buf.h>
>> #include <linux/dma-mapping.h>
>> #include <linux/dma-heap.h>
>> #include <linux/err.h>
>> #include <linux/highmem.h>
>> +#include <linux/mem_encrypt.h>
>> #include <linux/mm.h>
>> +#include <linux/set_memory.h>
>> #include <linux/module.h>
>> +#include <linux/pgtable.h>
>> #include <linux/scatterlist.h>
>> #include <linux/slab.h>
>> #include <linux/vmalloc.h>
>>
>> +struct system_heap_priv {
>> + bool decrypted;
>> +};
>
>Hi Jiri,
>
>I wonder if it'd better to call this cc_decrypted (or I guess
>cc_shared based on Robin's comment in the previous patch) like the DMA
>attr? There's a separate effort for "restricted" heaps with TEE for
>(encrypted) video playback, which doesn't involve VMs or RDMA. I think
>the cc_ prefix might help avoid any confusion between the usecase here
>and restricted heaps.
Sure. I'll rename this.
Thanks!
[..]
^ permalink raw reply
* Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
From: Jason Gunthorpe @ 2026-03-24 12:38 UTC (permalink / raw)
To: Dan Williams
Cc: Xu Yilun, Aneesh Kumar K.V, linux-coco, linux-pci, gregkh, aik,
bhelgaas, alistair23, lukas, Arnd Bergmann
In-Reply-To: <69c2046287d35_7ee3100ed@dwillia2-mobl4.notmuch>
On Mon, Mar 23, 2026 at 08:26:26PM -0700, Dan Williams wrote:
> Xu Yilun wrote:
> [..]
> > And I've no idea why unlocked MSIX/PBA must be hidden? How about other
> > non-TEE ranges, must be hidden or mustn't? Is there a possibility we
> > enforce DSM to present all ranges, then the layout is clear to OS?
>
> Just to close this question, this was discussed at the last device
> security call. Indeed the expectation is that Linux will assume that all
> ranges besides MSIX/PBA must be present in the report, and that offset
> is always aligned. If / when an implementation violates that expectation
> they can help write the Linux quirk for that case or otherwise fix their
> implementation.
I don't think you can quirk it.
Implementations can always follow this requirement by setting the
offset to 0. If they cannot compute a proper aligned offset then this
is what they must do.
Jason
^ permalink raw reply
* Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
From: Jason Gunthorpe @ 2026-03-24 12:36 UTC (permalink / raw)
To: Dan Williams
Cc: Greg KH, linux-coco, linux-pci, aik, aneesh.kumar, yilun.xu,
bhelgaas, alistair23, lukas, Christoph Hellwig, Marek Szyprowski,
Robin Murphy, Roman Kisel, Samuel Ortiz, Rafael J. Wysocki,
Danilo Krummrich
In-Reply-To: <69c1f469f2814_51621100bc@dwillia2-mobl4.notmuch>
On Mon, Mar 23, 2026 at 07:18:17PM -0700, Dan Williams wrote:
> Jason Gunthorpe wrote:
> > On Fri, Mar 13, 2026 at 06:32:27PM -0700, Dan Williams wrote:
> >
> > > The problem is that for all the buses that do not currently have a
> > > "device authorization" concept only userspace can decide that a device
> > > should skip bind by default. For that, I propose module autoprobe policy
> > > [1]. Not yet convinced the kernel needs its own per-device "no bind"
> > > policy.
> >
> > I think it is just part of the broader definition of the level that
> > extends into the iommu and so on. It makes sense to have this kind of
> > no-binding security level, IMHO.
>
> Easy to include for completeness.
>
> In a CC VM userspace can set "$module autoprobe=0" to get a control
> point to set @trust to zero, but @trust otherwise defaults to 1. This
> allows for userspace policy to generically distrust devices, but without
> needing to build a new mechanism to specify which devices start life at
> trust == 0 (i.e. "device filter" proposal previously NAK'd).
I feel like starting with trust=0 is much cleaner than using
autoprobe. Especially since it would be nice that when you do
ultimately set trust!=0 then you do want the kernel to do the normal
autoprobe flow.
Double so because I would like the iommu drivers to respond to trust 0
by fully blocking the device 100% of the time without holes, so to
make that work I would like to see the struct device report trust 0
the moment the iommu framework attaches the iommu.
How you decide the starting trust value for device during system boot
is definately something we need to discuss properly..
I liked your idea of using built in driver match, so if there is a
simple command line paramater that says 'only built in is trusted'
then we'd default all devices to untrusted and during device probe
check if any built in driver is matching and auto-set trust to X based
on the commandline parameter.
With the idea that only devices required to get to the initrd are
built in. Then the initrd userspace has the policy to bring more
devices into trusted!=0 to get to the root file system, then the
rootfs has more policy for further devices, and so on.
Probably this would ultimately escalate into annotations in the
modinfo about default policies for various drivers.
A kernel default policy of trusting everything without a "trust ops"
(see below) may also be quite reasonable, however boot ordering the
trust ops might be really tricky...
> > > > The DMA API just wants a flag in the struct device that says if the
> > > > device can access encrypted memory or only decrypted.
> > >
> > > You mean separate "trusted to access private" and "currently enabled to
> > > access private" properties? I am trying to think of a situation where
> > > "dev->trust >= 3" and a flag saying "disable bouncing for encrypted
> > > memory" would ever disagree.
> >
> > I'm steering the trust level toward more of an acceptance criteria.
> > If the trust level is you have access to private memory but the device
> > can't actually do that then fail the trust level change.
> >
> > Same for the reverse, if the trust level says no private memory and the
> > device is T=1 then fail the trust level change.
>
> Ok, so the uapi for PCI/TDISP would be:
>
> echo $tsm > $pdev/tsm/lock
> <gather evidence, validate with relying party>
> echo 3 > $pdev/trust
>
> ...where that @trust attribute is a generic device semantic, but in the
> case of PCI device connected to a given TSM it invokes the TSM hypercall
> to transition the device to the RUN state and the TSM local call to
> unblock DMA to private memory.
Maybe, but I was thinking the transition through run/locked would be
done through TSM uAPIs too. trust setting in the kernel just confirms
the device is in the right state.
But I haven't thought of a reason why the final switch to RUN couldn't
happen like this either.
> So, userspace can generically understand what privileges come with which
> trust levels, but the mechanism to get those privileges remains bus
> specific.
Yes
> > If we fall out of RUN then the level auto-resets back to 0 and
> > userspace has to go around and fix it again. (ignoring driver RAS)
>
> Yes, at the moment that the bus detects that an event like SPDM session
> loss, IDE link loss, TDISP ERROR state entry has occurred it can
> downgrade trust and notify. That notification fits well with netlink
> because all of those events are downstream of evidence validation.
Which is also why it would be nice to be consistent and rely on
trust=0 to isolate the device in all cases, not a mixture of
autoprobe.
> The complication vs benefit tradeoff is indeed not mathing, but wanted
> to do justice to Aneesh's proposal and the suitability of the sysfs
> uapi.
I think if you want something like this then it is better to target
the root - remove the ability for concurrent userspace to wrongly
operate the TSM entirely. Ie use a cdev, make it so going to LOCKED
isolates access to only this cdev fd and require only this cdev fd to
go to RUN. Then these kinds of bugs don't exist.
> > It brings it into the whole 'measure the device and then decide what
> > to do with it' framework. The trust level is still the generic ack
> > that the device is allowed the participate in the system with whatever
> > level of security.
>
> Some thought experiments to confirm alignment...
>
> 'Generic ack' is a synchronous mechanism for the bus to evaluate. So if
> @trust appears for any device, and by consequence alongside @authorized
> for a thunderbolt device, it should be the case that these operations
> are equivalent:
>
> # echo 1 > $dev/trust
> # echo 1 > $dev/authorized
>
> ...and the result is cross-reflected for comptability:
>
> # echo 1 > $dev/trust
> # cat $dev/authorized
> 1
>
> Consequently this:
>
> # echo 2 > $dev/trust
>
> ...would be equivalent to authorizing the device and unblocking ATS (if
> such a thing existed).
Yes, I don't know anything about thunderbolt, but this seems
reasonable. You could also do as you suggested for TDISP that trust!=0
auto-authorizes.
Basically the 'trust' generic framework sits on top of some "trust
ops" that will be provided by the security module that is affiliated
with the struct device (ie thunderbolt, TSM TDISP, TSM Link IDE, etc,
etc)
Then it becomes a general synchronization point where on one side the
"tust ops" can ack that the level is acceptable and consistent with
the system when on the other side generic compoments like IOMMU,
driver binding, etc can respond to it and change their behavior.
> For bare metal PCI device security the TSM 'connection' needs to be
> established in order to enable device evidence collection.
>
> echo $tsm > $pdev/tsm/connect
> <validate device evidence>
> echo 2 > $pdev/trust
>
> Now, I question whether 5 trust levels instead of 4. This would be to
> explicitly only trust devices where the TSM has established physical
> link encryption, or the TSM has asserted that the link is protected by
> other means. So the trust levels are:
I probably wouldn't use an int for the uAPI, but yes picking the
initial levels is important. As above since this is a clearing point
between two different worlds it needs to be defined in some way both
sides can understand what it means for them.
> 0 disconnected: bus does not attach drivers
> 1 limited: core code deploys hostile device mitigations like disable
> ATS, CC force shared memory bouncing.
> 2 DMA: full DMA access, driver responsible for protecting against
> adverarial devices.
> 3 Link: mitigations against physical integrity and snooping attacks
> deployed
> 4 TCB: full device interface validation via a protocol like TDISP,
> CC private memory access granted.
This seems reasonable to me, the 3/4 distinction is not meaningful for
the iommu&dev side, but it does provide a good check point for the
"trust ops". If userspace ack's that it expects physical security and
the kernel says it isn't physically secure (or becomes insecure later)
then it should fail.
> Where the native Rust library based SPDM driver only offers trust level
> 2, bare metal TSMs can support trust level 3, and the TSM interfaces in
> CC VMs can support trust level 4.
I'm not sure that the SPDM driver even provides a "trust ops" right? I
would guess that 0/1/2 are simply built in always available if trust ops are
NULL and 3/4 require positive reply from the ops to accept it.
So #3 needs a "trust ops" linked to enabling link IDE.. If this is
done in-kernel the link IDE module is providing the trust ops and just
using SPDM as a library to establish the link IDE keys?
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jason Gunthorpe @ 2026-03-24 12:24 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <CAFgf54qoFhvVR2yfROGaTJmLcjuLACswVFv=trqgJwGb_DB3zA@mail.gmail.com>
On Tue, Mar 24, 2026 at 12:14:36PM +0000, Mostafa Saleh wrote:
> On Tue, Mar 24, 2026 at 12:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Tue, Mar 17, 2026 at 01:24:13PM +0000, Mostafa Saleh wrote:
> >
> > > On the other hand, for restricted-dma, the memory decryption is deep
> > > in the DMA direct memory allocation and the DMA API callers (for ex
> > > virtio drivers) are clueless about it and can’t pass any attrs.
> > > My proposal was specific to restricted-dma and won’t work for your case.
> >
> > How is this any different from CC?
> >
> > If the device cannot dma to "encrypted" memory, whatever that means
> > for you, then the DMA API:
> > - Makes dma alloc coherent return "decrypted" memory, and the built
> > in mapping of coherent memory knows about this
> > - Makes dma_map_xxx use SWIOTLB to bounce to decrypted memory
> >
> > There is no need for something like virtio drivers to be aware of
> > any of this.
> >
> > On the other hand if the driver deliberately allocates decrypted
> > memory without using DMA API alloc coherent then it knows it did it
> > and can pass the flag to map it.
> >
>
> The problem is that the DMA API currently gets confused by this; it
> can end up double decrypting the memory or using the wrong functions
> as mentioned in [1]
I fully belive there are bugs, but the API design is sound. If you use
the coherent allocations from the DMA API then it knows decryption has
happened when it generates a dma_addr_t and there should be no issue.
Now, if drivers are using the DMA API wrong, like trying to double map
coherent allocations then they are broken. I also would not be
surprised to find cases like this.
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Mostafa Saleh @ 2026-03-24 12:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260324120057.GC8437@ziepe.ca>
On Tue, Mar 24, 2026 at 12:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Mar 17, 2026 at 01:24:13PM +0000, Mostafa Saleh wrote:
>
> > On the other hand, for restricted-dma, the memory decryption is deep
> > in the DMA direct memory allocation and the DMA API callers (for ex
> > virtio drivers) are clueless about it and can’t pass any attrs.
> > My proposal was specific to restricted-dma and won’t work for your case.
>
> How is this any different from CC?
>
> If the device cannot dma to "encrypted" memory, whatever that means
> for you, then the DMA API:
> - Makes dma alloc coherent return "decrypted" memory, and the built
> in mapping of coherent memory knows about this
> - Makes dma_map_xxx use SWIOTLB to bounce to decrypted memory
>
> There is no need for something like virtio drivers to be aware of
> any of this.
>
> On the other hand if the driver deliberately allocates decrypted
> memory without using DMA API alloc coherent then it knows it did it
> and can pass the flag to map it.
>
The problem is that the DMA API currently gets confused by this; it
can end up double decrypting the memory or using the wrong functions
as mentioned in [1]
In addition to the complexity it adds to the already complicated DMA
code. I don't have a strong opinion on how to solve this, but I
believe we need clear boundaries (and wrappers) for cases where memory
encryption is expected as it is starting to spill into the kernel.
[1] https://lore.kernel.org/all/20260305170335.963568-1-smostafa@google.com/
Thanks,
Mostafa
> > I am wondering if the kernel should have a more solid, unified method
> > for identifying already-decrypted memory instead. Perhaps we need a
> > way for the DMA API to natively recognize the encryption state of a
> > physical page (working alongside force_dma_unencrypted(dev)), rather
> > than relying on caller-provided attributes?
>
> Definately not, we do not want the DMA API inspecting things like
> this.
>
> Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jason Gunthorpe @ 2026-03-24 12:00 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <ablV_f_l7wD2m63E@google.com>
On Tue, Mar 17, 2026 at 01:24:13PM +0000, Mostafa Saleh wrote:
> On the other hand, for restricted-dma, the memory decryption is deep
> in the DMA direct memory allocation and the DMA API callers (for ex
> virtio drivers) are clueless about it and can’t pass any attrs.
> My proposal was specific to restricted-dma and won’t work for your case.
How is this any different from CC?
If the device cannot dma to "encrypted" memory, whatever that means
for you, then the DMA API:
- Makes dma alloc coherent return "decrypted" memory, and the built
in mapping of coherent memory knows about this
- Makes dma_map_xxx use SWIOTLB to bounce to decrypted memory
There is no need for something like virtio drivers to be aware of
any of this.
On the other hand if the driver deliberately allocates decrypted
memory without using DMA API alloc coherent then it knows it did it
and can pass the flag to map it.
> I am wondering if the kernel should have a more solid, unified method
> for identifying already-decrypted memory instead. Perhaps we need a
> way for the DMA API to natively recognize the encryption state of a
> physical page (working alongside force_dma_unencrypted(dev)), rather
> than relying on caller-provided attributes?
Definately not, we do not want the DMA API inspecting things like
this.
Jason
^ permalink raw reply
* Re: [PATCH v2 3/5] x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
From: Chao Gao @ 2026-03-24 10:18 UTC (permalink / raw)
To: Vishal Verma
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini, linux-kernel, linux-coco, kvm
In-Reply-To: <20260323-fuller_tdx_kexec_support-v2-3-87a36409e051@intel.com>
On Mon, Mar 23, 2026 at 02:59:06PM -0600, Vishal Verma wrote:
>Some early TDX-capable platforms have an erratum where a partial write
>to TDX private memory can cause a machine check on a subsequent read.
>On these platforms, kexec and kdump have been disabled in these cases,
>because the old kernel cannot safely hand off TDX state to the new
>kernel. Later TDX modules support the TDH.SYS.DISABLE SEAMCALL, which
>provides a way to cleanly disable TDX and allow kexec to proceed.
>
>The new SEAMCALL has an enumeration bit, but that is ignored. It is
>expected that users will be using the latest TDX module, and the failure
>mode for running the missing SEAMCALL on an older module is not fatal.
>
>This can be a long running operation, and the time needed largely
>depends on the amount of memory that has been allocated to TDs. If all
>TDs have been destroyed prior to the sys_disable call, then it is fast,
>with only needing to override the TDX module memory.
>
>After the SEAMCALL completes, the TDX module is disabled and all memory
>resources allocated to TDX are freed and reset. The next kernel can then
>re-initialize the TDX module from scratch via the normal TDX bring-up
>sequence.
>
>The SEAMCALL can return two different error codes that expect a retry.
> - TDX_INTERRUPTED_RESUMABLE can be returned in the case of a host
> interrupt. However, it will not return until it makes some forward
> progress, so we can expect to complete even in the case of interrupt
> storms.
> - TDX_SYS_BUSY will be returned on contention with other TDH.SYS.*
> SEAMCALLs, however a side effect of TDH.SYS.DISABLE is that it will
> block other SEAMCALLs once it gets going. So this contention will be
> short lived.
>
>So loop infinitely on either of these error codes, until success or other
>error.
>
>Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
^ permalink raw reply
* Re: [PATCH v2 2/5] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Chao Gao @ 2026-03-24 10:03 UTC (permalink / raw)
To: Vishal Verma
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini, linux-kernel, linux-coco, kvm,
Kai Huang
In-Reply-To: <20260323-fuller_tdx_kexec_support-v2-2-87a36409e051@intel.com>
On Mon, Mar 23, 2026 at 02:59:05PM -0600, Vishal Verma wrote:
>From: Rick Edgecombe <rick.p.edgecombe@intel.com>
>
>KVM tries to take care of some required cache flushing earlier in the
>kexec path in order to be kind to some long standing races that can occur
>later in the operation. Until recently, VMXOFF was handled within KVM.
>Since VMX being enabled is required to make a SEAMCALL, it had the best
>per-cpu scoped operation to plug the flushing into. So it is kicked off
>from there.
>
>This early kexec cache flushing in KVM happens via a syscore shutdown
>callback. Now that VMX enablement control has moved to arch/x86, which has
>grown its own syscore shutdown callback, it no longer make sense for it to
>live in KVM. It fits better with the TDX enablement managing code.
>
>In addition, future changes will add a SEAMCALL that happens immediately
>before VMXOFF, which means the cache flush in KVM will be too late to
>flush the cache before the last SEAMCALL. So move it to the newly added TDX
>arch/x86 syscore shutdown handler.
>
>Since tdx_cpu_flush_cache_for_kexec() is no longer needed by KVM, make it
>static and remove the export. Since it is also not part of an operation
>spread across disparate components, remove the redundant comments and
>verbose naming.
>
>In the existing KVM based code, CPU offline also funnels through
>tdx_cpu_flush_cache_for_kexec(). So the centralization to the arch/x86
>syscore shutdown callback elides this CPU offline time behavior. However,
>WBINVD is already generally done at CPU offline as matter of course. So
>don't bother adding TDX specific logic for this, and rely on the normal
>WBINVD to handle it.
>
>Acked-by: Kai Huang <kai.huang@intel.com>
>Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
^ permalink raw reply
* Re: [PATCH v2 1/5] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>
From: Chao Gao @ 2026-03-24 9:49 UTC (permalink / raw)
To: Vishal Verma
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini, linux-kernel, linux-coco, kvm
In-Reply-To: <20260323-fuller_tdx_kexec_support-v2-1-87a36409e051@intel.com>
On Mon, Mar 23, 2026 at 02:59:04PM -0600, Vishal Verma wrote:
>From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
>
>Today there are two separate locations where TDX error codes are defined:
> arch/x86/include/asm/tdx.h
> arch/x86/kvm/vmx/tdx_errno.h
>
>They have some overlap that is already defined similarly. Reduce the
>duplication and prepare to introduce some helpers for these error codes in
>the central place by unifying them. Join them at:
> asm/shared/tdx_errno.h
>...and update the headers that contained the duplicated definitions to
>include the new unified header.
>
>"asm/shared" is used for sharing TDX code between the early compressed
>code and the normal kernel code. While the compressed code for the guest
>doesn't use these error code header definitions today, it does make the
>types of calls that return the values they define. So place the defines in
>"shared" location so that it can, but leave such cleanups for future
>changes.
>
>Also, adjust BITUL() -> _BITULL() to address 32 bit build errors after the
>move.
>
>Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
>[enhance log]
>Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox