* Re: [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Edgecombe, Rick P @ 2026-03-25 17:21 UTC (permalink / raw)
To: marcandre.lureau@redhat.com
Cc: pbonzini@redhat.com, bp@alien8.de, kas@kernel.org, Qiang, Chenyi,
hpa@zytor.com, mingo@redhat.com, linux-kernel@vger.kernel.org,
dave.hansen@linux.intel.com, tglx@kernel.org, kvm@vger.kernel.org,
linux-coco@lists.linux.dev, x86@kernel.org
In-Reply-To: <CAMxuvaytU-pM+rviUbNGWMhvbAYutwvaSXW_O=sn+QfOzF35Xw@mail.gmail.com>
On Wed, 2026-03-25 at 14:29 +0400, Marc-André Lureau wrote:
> > Does this depend on patch 1 somehow?
>
> Yes, if I plug, unplug and plug again I get this without PATCH 1:
> [root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
> size: 0x80000000
> [ 5707.395583] virtio_mem virtio5: requested size: 0x0
>
> [root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
> size: 0x2e00000
> [ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
> [ 5714.676296] tdx: Failed to accept memory [0x108000000,
> 0x110000000)
> [ 5714.683980] tdx: Failed to accept memory [0x110000000,
> 0x118000000)
> [ 5714.686997] tdx: Failed to accept memory [0x140000000,
> 0x148000000)
> [ 5714.689989] tdx: Failed to accept memory [0x128000000,
> 0x130000000)
> [ 5714.694981] tdx: Failed to accept memory [0x148000000,
> 0x150000000)
> [ 5714.704064] tdx: Failed to accept memory [0x138000000,
> 0x140000000)
> [ 5714.710144] tdx: Failed to accept memory [0x118000000,
> 0x120000000)
> [ 5714.722532] tdx: Failed to accept memory [0x130000000,
> 0x138000000)
>
> My understanding is that QEMU should eventually unplug the memory and
> PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
> to happen. Is this strictly required? According to the specification,
> it may not be.
Ah, I see now! So the problem is not that the kernel is accidentally
re-accepting the memory. It's that host userspace is not actually
removing the memory during unplug. Hmm. Why not fix userspace then? If
the memory is unplugged it should not be usable anymore by the guest.
If it is still accessible then it seems kind of like a bug, no?
And! This totally justifies the warning. If the error is ignored, the
guest would think the memory is zeroed, but it could have old data in
it. It's exactly the kind of tricks a VMM could play to attack the
guest.
Another option could be to perform a TDG.MEM.PAGE.RELEASE TDCALL from
the guest when it unplugs the memory, to put it in an unaccepted state.
This would be more robust to buggy VMM behavior. But working around
buggy VM behavior would need a high bar.
^ permalink raw reply
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Edgecombe, Rick P @ 2026-03-25 16:57 UTC (permalink / raw)
To: Hansen, Dave, Zhao, Yan Y
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Huang, Kai,
Li, Xiaoyao, dave.hansen@linux.intel.com, kas@kernel.org,
seanjc@google.com, mingo@redhat.com, pbonzini@redhat.com,
binbin.wu@linux.intel.com, ackerleytng@google.com,
linux-kernel@vger.kernel.org, Yamahata, Isaku, sagis@google.com,
Annapurve, Vishal, bp@alien8.de, tglx@kernel.org,
yilun.xu@linux.intel.com, x86@kernel.org
In-Reply-To: <acOmoOP2fFUyOByC@yzhao56-desk.sh.intel.com>
On Wed, 2026-03-25 at 17:10 +0800, Yan Zhao wrote:
> > I don't really understand what this is saying.
> >
> > Is the concern that KVM might want to set up page tables for memory
> > that differ from how it was allocated? I'm a bit worried that this
> > assumes something about folios that doesn't always hold.
> >
> > I think the hugetlbfs gigantic support uses folios in at least a
> > few spots today.
> Below is the background of this problem. I'll try to include a short
> summary in the next version's patch logs.
While this patchset is kind of pre-work for TDX huge pages, the reason
to separate it out and push it earlier is because it has some value on
it's own. So I'd think to focus mostly on the impact of the change
today.
How about this justification:
1. Because KVM handles guest memory as PFNs, and the SEAMCALLs under
discussion are only used there, PFN is more natural.
2. The struct page was partly making sure we didn't pass a wrong arg
(typical type safety) and partly ensuring that KVM doesn't pass non-
convertible memory, however the SEAMCALLs themselves can check this for
the kernel. So the case is already covered by warnings.
In conclusion, the PFN is more natural and the original purpose of
struct page is already covered.
Sean said somewhere IIRC that he would have NAKed the struct page thing
if he had seen it, for even the base support. And the two points above
don't actually require discussion of even huge pages. So does it
actually add any value to dive into the issues you list below?
>
> In TDX huge page v3, I added logic that assumes PFNs are contained in
> a single folio in both TDX's map/unmap paths [1][2]:
> if (start_idx + npages > folio_nr_pages(folio))
> return TDX_OPERAND_INVALID;
> This not only assumes the PFNs have corresponding struct page, but
> also assumes they must be contained in a single folio, since with
> only base_page + npages, it's not easy to get the ith page's pointer
> without first ensuring the pages are contained in a single folio.
>
> This should work since current KVM/guest_memfd only allocates memory
> with struct page and maps them into S-EPT at a level lower than or
> equal to the backend folio size. That is, a single S-EPT mapping
> cannot span multiple backend folios.
>
> However, Ackerley's 1G hugetlb-based gmem splits the backend folio
> [3] ahead of splitting/unmapping them from S-EPT [4], due to
> implementation limitations mentioned at [5]. It makes the warning in
> [1] hit upon invoking TDX's unmap callback.
>
> Moreover, Google's future gmem may manage PFNs independently in the
> future,
I think we can adapt to such changes when they eventually come up. It's
not just about not merging code to be used by out-of-tree code. It also
shrinks what we have to consider at each stage. So we can eventually
get there.
> so TDX's private memory may have no corresponding struct page, and
> KVM would map them via VM_PFNMAP, similar to mapping pass-through
> MMIOs or other PFNs without struct page or with non-refcounted struct
> page in normal VMs. Given that KVM has
> suffered a lot from handling VM_PFNMAP memory for non-refcounted
> struct page [6] in normal VMs, and TDX mapping/unmapping callbacks
> have no semantic reason to dictate where and how KVM/guest_memfd
> should allocate and map memory, Sean suggested dropping the
> unnecessary assumption that memory to be mapped/unmapped to/from S-
> EPT must be contained in a single folio (though he didn't object
> reasonable sanity checks on if the PFNs are TDX convertible).
>
>
> [1]
> https://lore.kernel.org/kvm/20260106101929.24937-1-yan.y.zhao@intel.com
> [2]
> https://lore.kernel.org/kvm/20260106101826.24870-1-yan.y.zhao@intel.com
> [3]
> https://github.com/googleprodkernel/linux-cc/blob/wip-gmem-conversions-hugetlb-restructuring-12-08-25/virt/kvm/guest_memfd.c#L909
> [4]
> https://github.com/googleprodkernel/linux-cc/blob/wip-gmem-conversions-hugetlb-restructuring-12-08-25/virt/kvm/guest_memfd.c#L918
> [5] https://lore.kernel.org/kvm/diqzqzrzdfvh.fsf@google.com/
> [6]
> https://lore.kernel.org/all/20241010182427.1434605-1-seanjc@google.com
^ permalink raw reply
* Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
From: Jason Gunthorpe @ 2026-03-25 11:56 UTC (permalink / raw)
To: Dan Williams
Cc: Greg KH, linux-coco, linux-pci, aik, aneesh.kumar, yilun.xu,
bhelgaas, alistair23, lukas, Christoph Hellwig, Marek Szyprowski,
Robin Murphy, Roman Kisel, Samuel Ortiz, Rafael J. Wysocki,
Danilo Krummrich
In-Reply-To: <69c360d2107ca_7ee310052@dwillia2-mobl4.notmuch>
On Tue, Mar 24, 2026 at 09:13:06PM -0700, Dan Williams wrote:
> Jason Gunthorpe wrote:
> [..]
> > I feel like starting with trust=0 is much cleaner than using
> > autoprobe. Especially since it would be nice that when you do
> > ultimately set trust!=0 then you do want the kernel to do the normal
> > autoprobe flow.
> >
> > Double so because I would like the iommu drivers to respond to trust 0
> > by fully blocking the device 100% of the time without holes, so to
> > make that work I would like to see the struct device report trust 0
> > the moment the iommu framework attaches the iommu.
> >
> > How you decide the starting trust value for device during system boot
> > is definately something we need to discuss properly..
> >
> > I liked your idea of using built in driver match, so if there is a
> > simple command line paramater that says 'only built in is trusted'
> > then we'd default all devices to untrusted and during device probe
> > check if any built in driver is matching and auto-set trust to X based
> > on the commandline parameter.
>
> I do agree that forcing trust=0 at the beginning of time is attractive
> and theoretically clean. I am concerned about subsystems that are not
> prepared for driver attach failures. For example, I would not expect to
> need to set trust for auxiliary bus devices if the host device is
> trusted.
Yeah, IDK here either. Maybe it is some per-bus opt int.
I think the most important thing is we get a clean story for devices
to be isolated by the iommu until user space ack's them.
> Right, the potential to see in-between states concerns me because TSM
> uAPIs would have fully enabled the device to wreak havoc, meanwhile
> dev->trust is still showing the device at some lower level of trust. So
> I think trust modification needs to be synchronous with privileges
> granted/revoked.
If an iommu is present then the device will still be blocked even
though it is in RUN, I'm not sure this synchronicity is so important.
Jason
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Suzuki K Poulose @ 2026-03-25 11:32 UTC (permalink / raw)
To: Gavin Shan, Steven Price, Mathieu Poirier
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Zenghui Yu, linux-arm-kernel,
linux-kernel, Joey Gouly, Alexandru Elisei, Christoffer Dall,
Fuad Tabba, linux-coco, Ganapatrao Kulkarni, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <9247d9ea-64b4-4e8e-81f8-3c8e00750acf@arm.com>
On 25/03/2026 10:16, Suzuki K Poulose wrote:
> Hi Gavin
>
> Steven is on holidays, so I am jumping in here.
>
> On 25/03/2026 06:37, Gavin Shan wrote:
>> Hi Steven,
>>
>> On 3/21/26 2:45 AM, Steven Price wrote:
>>> On 19/03/2026 23:02, Mathieu Poirier wrote:
>>
>> [...]
>>
>>>>>
>>>>> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
>>>>> use the following branch:
>>>>>
>>>>> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>>>>
>>>> This RMM version is expecting a RMM EL3 interface version of at
>>>> least 2.0. Do
>>>> you have a TF-A to use with it?
>>>
>>> You should be able to use the 'master' branch of the TF-A repository.
>>> For now you need to set RMM_V1_COMPAT=0 to enable 2.0 support.
>>>
>>
>> In upstream TF-A repository [1], I don't see the config option
>> 'RMM_V1_COMPAT'.
>> would it be something else?
>>
>> [1] git@github.com:ARM-software/arm-trusted-firmware.git (branch:
>> master)
>>
>
> suzuki@ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
> Makefile: RMM_V1_COMPAT \
> Makefile: RMM_V1_COMPAT \
> docs/getting_started/build-options.rst:- ``RMM_V1_COMPAT``: Boolean
> flag to enable support for RMM v1.x compatibility
> include/services/rmmd_svc.h:#if RMM_V1_COMPAT
> include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
> make_helpers/defaults.mk:RMM_V1_COMPAT := 1
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> suzuki@ewhatever:trusted-firmware-a$ git log --oneline -1
> 8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge
> changes from topic "qti_lemans_evk" into integration
> suzuki@ewhatever:trusted-firmware-a$ git remote get-url origin
> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
>
>
>
>> I use the following command to build TF-A image. The RMM-EL3
>> compatible issue is
>> still seen.
>>
>> TFA_PATH=$PWD
>> EDK2_IMAGE=${TFA_PATH}/../edk2/Build/ArmVirtQemuKernel-AARCH64/
>> RELEASE_GCC5/FV/QEMU_EFI.fd
>> RMM_IMAGE=${TFA_PATH}/../tf-rmm/build-qemu/Debug/rmm.img
>> make CROSS_COMPILE=aarch64-none-elf- \
>> PLAT=qemu ENABLE_RME=1 RMM_V1_COMPAT=0 DEBUG=1 LOG_LEVEL=40 \
>> QEMU_USE_GIC_DRIVER=QEMU_GICV3 \
>> BL33=${EDK2_IMAGE} RMM=${RMM_IMAGE} \
>> -j 8 all fip
>>
>
>
>
>
>> Booting messages
>> ================
>> INFO: BL31: Initializing runtime services
>> INFO: RMM setup done.
>> INFO: BL31: Initializing RMM
>> INFO: RMM init start.
>> ERROR: RMM init failed: -2
>> WARNING: BL31: RMM initialization failed
>
> This is definitely the TF-A RMM incompatibility.
>
> Btw, the shrinkwrap overlay configs in tf-RMM repository should work.
> But unfortunately the Linux/kvmtool repositories are pointing to
> internal repositories. The following patch should fix it and get
> it all working. I am working with the tf-rmm team to fix this.
This is now fixed in the branch :
https://git.trustedfirmware.org/plugins/gitiles/TF-RMM/tf-rmm/+/refs/heads/topics/rmm-v2.0-poc/tools/shrinkwrap/configs/cca.yaml
Cheers
Suzuki
^ permalink raw reply
* Re: [PATCH kernel 4/9] dma/swiotlb: Stop forcing SWIOTLB for TDISP devices
From: Alexey Kardashevskiy @ 2026-03-25 10:42 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: dan.j.williams, Robin Murphy, x86, linux-kernel, kvm, linux-pci,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
H. Peter Anvin, Sean Christopherson, Paolo Bonzini,
Andy Lutomirski, Peter Zijlstra, Bjorn Helgaas, Marek Szyprowski,
Andrew Morton, Catalin Marinas, Michael Ellerman, Mike Rapoport,
Tom Lendacky, Ard Biesheuvel, Neeraj Upadhyay, Ashish Kalra,
Stefano Garzarella, Melody Wang, Seongman Lee, Joerg Roedel,
Nikunj A Dadhania, Michael Roth, Suravee Suthikulpanit,
Andi Kleen, Kuppuswamy Sathyanarayanan, Tony Luck,
David Woodhouse, Greg Kroah-Hartman, Denis Efremov, Geliang Tang,
Piotr Gregor, Michael S. Tsirkin, Alex Williamson, Arnd Bergmann,
Jesse Barnes, Jacob Pan, Yinghai Lu, Kevin Brodsky,
Jonathan Cameron, Aneesh Kumar K.V (Arm), Xu Yilun, Herbert Xu,
Kim Phillips, Konrad Rzeszutek Wilk, Stefano Stabellini,
Claire Chang, linux-coco, iommu
In-Reply-To: <20260304124316.GL972761@nvidia.com>
On 04/03/2026 23:43, Jason Gunthorpe wrote:
> On Wed, Mar 04, 2026 at 05:45:31PM +1100, Alexey Kardashevskiy wrote:
>
>>> I suspect AMD needs to use their vTOM feature to allow shared memory
>>> to remain available to TDISP RUN with a high/low address split.
>>
>> I could probably do something about it bit I wonder what is the real
>> live use case which requires leaking SME mask, have a live example
>> which I could try recreating?
>
> We need shared memory allocated through a DMABUF heap:
>
> https://lore.kernel.org/all/20260223095136.225277-1-jiri@resnulli.us/
>
> To work with all PCI devices in the system, TDISP or not.
>
> Without this the ability for a TDISP device to ingest (encrypted) data
> requires all kinds of memcpy..
>
> So the DMA API should see the DMA_ATTR_CC_DECRYPTED and setup the
> correct dma_dddr_t either by choosing the shared alias for the TDISP
> device's vTOM, or setting the C bit in a vIOMMU S1.
I can make this work by using some high enough bit as "Sbit" in vTOM to mark shared pages but I'll have to decouple dma_addr_encrypted/dma_addr_canonical (not many uses but still scary) from sme_me_mask and I wonder if anyone has recently attempted this so I won't have to reinvent?
(a reminder: below vTOM address == private, above vTOM == shared)
Thanks,
--
Alexey
^ permalink raw reply
* Re: [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Marc-André Lureau @ 2026-03-25 10:29 UTC (permalink / raw)
To: Edgecombe, Rick P
Cc: bp@alien8.de, kas@kernel.org, hpa@zytor.com, mingo@redhat.com,
x86@kernel.org, tglx@kernel.org, dave.hansen@linux.intel.com,
Qiang, Chenyi, kvm@vger.kernel.org, linux-coco@lists.linux.dev,
linux-kernel@vger.kernel.org, Bonzini, Paolo
In-Reply-To: <56190adc345148396ba6b3e52672e662145f7dc7.camel@intel.com>
Hi
On Wed, Mar 25, 2026 at 2:04 AM Edgecombe, Rick P
<rick.p.edgecombe@intel.com> wrote:
>
> On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> > In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> > accepted before use. The first access triggers a fatal "SEPT entry in
> > PENDING state" EPT violation and KVM terminates the guest.
> >
> > Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> > calls tdx_accept_memory() for the range being onlined.
> >
> > The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> > memory from going online.
>
> Does this depend on patch 1 somehow?
Yes, if I plug, unplug and plug again I get this without PATCH 1:
[root@rhel10-server ~]# [ 5707.392231] virtio_mem virtio5: plugged
size: 0x80000000
[ 5707.395583] virtio_mem virtio5: requested size: 0x0
[root@rhel10-server ~]# [ 5714.648501] virtio_mem virtio5: plugged
size: 0x2e00000
[ 5714.651808] virtio_mem virtio5: requested size: 0x80000000
[ 5714.676296] tdx: Failed to accept memory [0x108000000, 0x110000000)
[ 5714.683980] tdx: Failed to accept memory [0x110000000, 0x118000000)
[ 5714.686997] tdx: Failed to accept memory [0x140000000, 0x148000000)
[ 5714.689989] tdx: Failed to accept memory [0x128000000, 0x130000000)
[ 5714.694981] tdx: Failed to accept memory [0x148000000, 0x150000000)
[ 5714.704064] tdx: Failed to accept memory [0x138000000, 0x140000000)
[ 5714.710144] tdx: Failed to accept memory [0x118000000, 0x120000000)
[ 5714.722532] tdx: Failed to accept memory [0x130000000, 0x138000000)
My understanding is that QEMU should eventually unplug the memory and
PUNCH_HOLE then KVM should TDH.MEM.PAGE.REMOVE, but that doesn't seem
to happen. Is this strictly required? According to the specification,
it may not be.
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Suzuki K Poulose @ 2026-03-25 10:19 UTC (permalink / raw)
To: Gavin Shan, Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Zenghui Yu, linux-arm-kernel, linux-kernel,
Joey Gouly, Alexandru Elisei, Christoffer Dall, Fuad Tabba,
linux-coco, Ganapatrao Kulkarni, Shanker Donthineni, Alper Gun,
Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <779079e1-7193-46e9-b18c-9ab5e69bb077@redhat.com>
Hi Gavin
On 25/03/2026 04:07, Gavin Shan wrote:
> Hi Steven,
>
> On 3/19/26 1:53 AM, Steven Price wrote:
>>
>> This series is based on v7.0-rc1. It is also available as a git
>> repository:
>>
>> https://gitlab.arm.com/linux-arm/linux-cca cca-host/v13
>>
>> Work in progress changes for kvmtool are available from the git
>> repository below:
>>
>> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v11
>>
>
> Could you please share if we have a working qemu repository on top of this
> (v13) series? The previous qemu repository [1] seems out of dated for long
> time. I heard Jean won't be able to continue his efforts on QEMU part, who
> is going to pick it up in this case.
>
> [1] https://git.codelinaro.org/linaro/dcap/qemu.git (branch: cca/latest)
Unfortunately not at the moment. We have moved on to a simpler UABI,
which drops most of the previous configuration steps.
>
>> Note that the kvmtool code has been tidied up (thanks to Suzuki) and
>> this involves a minor change in flags. The "--restricted_mem" flag is no
>> longer recognised (or necessary).
>>
>> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
>> use the following branch:
>>
>> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>>
>
> I'm seeing error to initialize RMM with the suggested RMM branch
> (topics/rmm-v2.0-poc)
> and the upstream TF-A [1]. It seems the problem is compatible issue in the
> RMM-EL3 interface. RMM requires verion 2.0 while TF-A only supports 0.8. So
> I guess I must be using a wrong TF-A repository. Could you please share
> which
> TF-A repository you use for testing?
See the other thread with Mathieu.
Kind regards
Suzuki
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Suzuki K Poulose @ 2026-03-25 10:16 UTC (permalink / raw)
To: Gavin Shan, Steven Price, Mathieu Poirier
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Zenghui Yu, linux-arm-kernel,
linux-kernel, Joey Gouly, Alexandru Elisei, Christoffer Dall,
Fuad Tabba, linux-coco, Ganapatrao Kulkarni, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <0b7121e7-3c74-4303-a200-01d2b7c535ce@redhat.com>
Hi Gavin
Steven is on holidays, so I am jumping in here.
On 25/03/2026 06:37, Gavin Shan wrote:
> Hi Steven,
>
> On 3/21/26 2:45 AM, Steven Price wrote:
>> On 19/03/2026 23:02, Mathieu Poirier wrote:
>
> [...]
>
>>>>
>>>> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
>>>> use the following branch:
>>>>
>>>> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>>>
>>> This RMM version is expecting a RMM EL3 interface version of at least
>>> 2.0. Do
>>> you have a TF-A to use with it?
>>
>> You should be able to use the 'master' branch of the TF-A repository.
>> For now you need to set RMM_V1_COMPAT=0 to enable 2.0 support.
>>
>
> In upstream TF-A repository [1], I don't see the config option
> 'RMM_V1_COMPAT'.
> would it be something else?
>
> [1] git@github.com:ARM-software/arm-trusted-firmware.git (branch:
> master)
>
suzuki@ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
Makefile: RMM_V1_COMPAT \
Makefile: RMM_V1_COMPAT \
docs/getting_started/build-options.rst:- ``RMM_V1_COMPAT``: Boolean
flag to enable support for RMM v1.x compatibility
include/services/rmmd_svc.h:#if RMM_V1_COMPAT
include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
make_helpers/defaults.mk:RMM_V1_COMPAT := 1
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
suzuki@ewhatever:trusted-firmware-a$ git log --oneline -1
8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge
changes from topic "qti_lemans_evk" into integration
suzuki@ewhatever:trusted-firmware-a$ git remote get-url origin
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
> I use the following command to build TF-A image. The RMM-EL3 compatible
> issue is
> still seen.
>
> TFA_PATH=$PWD
> EDK2_IMAGE=${TFA_PATH}/../edk2/Build/ArmVirtQemuKernel-AARCH64/
> RELEASE_GCC5/FV/QEMU_EFI.fd
> RMM_IMAGE=${TFA_PATH}/../tf-rmm/build-qemu/Debug/rmm.img
> make CROSS_COMPILE=aarch64-none-elf- \
> PLAT=qemu ENABLE_RME=1 RMM_V1_COMPAT=0 DEBUG=1 LOG_LEVEL=40 \
> QEMU_USE_GIC_DRIVER=QEMU_GICV3 \
> BL33=${EDK2_IMAGE} RMM=${RMM_IMAGE} \
> -j 8 all fip
>
> Booting messages
> ================
> INFO: BL31: Initializing runtime services
> INFO: RMM setup done.
> INFO: BL31: Initializing RMM
> INFO: RMM init start.
> ERROR: RMM init failed: -2
> WARNING: BL31: RMM initialization failed
This is definitely the TF-A RMM incompatibility.
Btw, the shrinkwrap overlay configs in tf-RMM repository should work.
But unfortunately the Linux/kvmtool repositories are pointing to
internal repositories. The following patch should fix it and get
it all working. I am working with the tf-rmm team to fix this.
--8>--
diff --git a/tools/shrinkwrap/configs/cca.yaml
b/tools/shrinkwrap/configs/cca.yaml
index 1c0455ba..0d70a582 100644
--- a/tools/shrinkwrap/configs/cca.yaml
+++ b/tools/shrinkwrap/configs/cca.yaml
@@ -25,8 +25,8 @@ build:
linux:
repo:
- remote: https://gitlab.geo.arm.com/software/linux-arm/fkvm.git
- revision: stepri01/cca/v13-wip+sro
+ remote: https://gitlab.arm.com/linux-arm/linux-cca
+ revision: cca-host/v13
prebuild:
- ./scripts/config --file ${param:builddir}/.config --enable
CONFIG_PCI_TSM
- ./scripts/config --file ${param:builddir}/.config --enable
CONFIG_PCI_DOE
@@ -50,5 +50,5 @@ build:
remote: https://git.kernel.org/pub/scm/utils/dtc/dtc.git
revision: v1.7.2
kvmtool:
- remote: https://gitlab.geo.arm.com/software/linux-arm/fkvmtool.git
- revision: stepri01/cca/v13-wip
+ remote: https://gitlab.arm.com/linux-arm/kvmtool-cca
+ revision: cca/v11
Kind regards
Suzuki
>
> Thanks,
> Gavin
>
>
>
^ permalink raw reply related
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Yan Zhao @ 2026-03-25 9:10 UTC (permalink / raw)
To: Dave Hansen
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, kas, x86,
linux-kernel, kvm, linux-coco, kai.huang, rick.p.edgecombe,
yilun.xu, vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <a14531ab-f069-41f9-8c5c-9fe6f28a9454@intel.com>
On Thu, Mar 19, 2026 at 11:05:09AM -0700, Dave Hansen wrote:
> On 3/18/26 17:57, Yan Zhao wrote:
> > Remove the completely unnecessary assumption that memory mapped into a TDX
> > guest is backed by refcounted struct page memory. From KVM's point of view,
> > TDH_MEM_PAGE_ADD and TDH_MEM_PAGE_AUG are glorified writes to PTEs, so they
> > have no business placing requirements on how KVM and guest_memfd manage
> > memory.
>
> I think this goes a bit too far.
>
> It's one thing to say that it's more convenient for KVM to stick with
> pfns because it's what KVM uses now. Or, that the goals of using 'struct
> page' can be accomplished other ways. It's quite another to say what
> other bits of the codebase have "business" doing.
I explained the background in the cover letter, thinking we could add the link
to the final patches when they are merged.
I can expand the patch logs by providing background explanation as well.
> Sean, can we tone this down a _bit_ to help guide folks in the future?
Sorry for being lazy and not expanding the patch logs from Sean's original
patch tagged "DO NOT MERGE".
> > Rip out the misguided struct page assumptions/constraints and instead have
>
> Could we maybe tone down the editorializing a bit, please? Folks can
> have honest disagreements about this stuff while not being "misguided".
You are right. I need to make it clear.
> > the two SEAMCALL wrapper APIs take PFN directly. This ensures that for
> > future huge page support in S-EPT, the kernel doesn't pick up even worse
> > assumptions like "a hugepage must be contained in a single folio".
>
> I don't really understand what this is saying.
>
> Is the concern that KVM might want to set up page tables for memory that
> differ from how it was allocated? I'm a bit worried that this assumes
> something about folios that doesn't always hold.
>
> I think the hugetlbfs gigantic support uses folios in at least a few
> spots today.
Below is the background of this problem. I'll try to include a short summary in
the next version's patch logs.
In TDX huge page v3, I added logic that assumes PFNs are contained in a single
folio in both TDX's map/unmap paths [1][2]:
if (start_idx + npages > folio_nr_pages(folio))
return TDX_OPERAND_INVALID;
This not only assumes the PFNs have corresponding struct page, but also assumes
they must be contained in a single folio, since with only base_page + npages,
it's not easy to get the ith page's pointer without first ensuring the pages are
contained in a single folio.
This should work since current KVM/guest_memfd only allocates memory with
struct page and maps them into S-EPT at a level lower than or equal to the
backend folio size. That is, a single S-EPT mapping cannot span multiple backend
folios.
However, Ackerley's 1G hugetlb-based gmem splits the backend folio [3] ahead of
splitting/unmapping them from S-EPT [4], due to implementation limitations
mentioned at [5]. It makes the warning in [1] hit upon invoking TDX's unmap
callback.
Moreover, Google's future gmem may manage PFNs independently in the future, so
TDX's private memory may have no corresponding struct page, and KVM would map
them via VM_PFNMAP, similar to mapping pass-through MMIOs or other PFNs without
struct page or with non-refcounted struct page in normal VMs. Given that KVM has
suffered a lot from handling VM_PFNMAP memory for non-refcounted struct page [6]
in normal VMs, and TDX mapping/unmapping callbacks have no semantic reason to
dictate where and how KVM/guest_memfd should allocate and map memory, Sean
suggested dropping the unnecessary assumption that memory to be mapped/unmapped
to/from S-EPT must be contained in a single folio (though he didn't object
reasonable sanity checks on if the PFNs are TDX convertible).
[1] https://lore.kernel.org/kvm/20260106101929.24937-1-yan.y.zhao@intel.com
[2] https://lore.kernel.org/kvm/20260106101826.24870-1-yan.y.zhao@intel.com
[3] https://github.com/googleprodkernel/linux-cc/blob/wip-gmem-conversions-hugetlb-restructuring-12-08-25/virt/kvm/guest_memfd.c#L909
[4] https://github.com/googleprodkernel/linux-cc/blob/wip-gmem-conversions-hugetlb-restructuring-12-08-25/virt/kvm/guest_memfd.c#L918
[5] https://lore.kernel.org/kvm/diqzqzrzdfvh.fsf@google.com/
[6] https://lore.kernel.org/all/20241010182427.1434605-1-seanjc@google.com
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Gavin Shan @ 2026-03-25 6:37 UTC (permalink / raw)
To: Steven Price, Mathieu Poirier
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Suzuki K Poulose, Zenghui Yu,
linux-arm-kernel, linux-kernel, Joey Gouly, Alexandru Elisei,
Christoffer Dall, Fuad Tabba, linux-coco, Ganapatrao Kulkarni,
Shanker Donthineni, Alper Gun, Aneesh Kumar K . V, Emi Kisanuki,
Vishal Annapurve
In-Reply-To: <37bc1222-6fc7-48f0-94d3-6eaac420aa55@arm.com>
Hi Steven,
On 3/21/26 2:45 AM, Steven Price wrote:
> On 19/03/2026 23:02, Mathieu Poirier wrote:
[...]
>>>
>>> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
>>> use the following branch:
>>>
>>> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>>
>> This RMM version is expecting a RMM EL3 interface version of at least 2.0. Do
>> you have a TF-A to use with it?
>
> You should be able to use the 'master' branch of the TF-A repository.
> For now you need to set RMM_V1_COMPAT=0 to enable 2.0 support.
>
In upstream TF-A repository [1], I don't see the config option 'RMM_V1_COMPAT'.
would it be something else?
[1] git@github.com:ARM-software/arm-trusted-firmware.git (branch: master)
I use the following command to build TF-A image. The RMM-EL3 compatible issue is
still seen.
TFA_PATH=$PWD
EDK2_IMAGE=${TFA_PATH}/../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
RMM_IMAGE=${TFA_PATH}/../tf-rmm/build-qemu/Debug/rmm.img
make CROSS_COMPILE=aarch64-none-elf- \
PLAT=qemu ENABLE_RME=1 RMM_V1_COMPAT=0 DEBUG=1 LOG_LEVEL=40 \
QEMU_USE_GIC_DRIVER=QEMU_GICV3 \
BL33=${EDK2_IMAGE} RMM=${RMM_IMAGE} \
-j 8 all fip
Booting messages
================
INFO: BL31: Initializing runtime services
INFO: RMM setup done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
ERROR: RMM init failed: -2
WARNING: BL31: RMM initialization failed
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
From: Dan Williams @ 2026-03-25 4:13 UTC (permalink / raw)
To: Jason Gunthorpe, Dan Williams
Cc: Greg KH, linux-coco, linux-pci, aik, aneesh.kumar, yilun.xu,
bhelgaas, alistair23, lukas, Christoph Hellwig, Marek Szyprowski,
Robin Murphy, Roman Kisel, Samuel Ortiz, Rafael J. Wysocki,
Danilo Krummrich
In-Reply-To: <20260324123649.GY7340@nvidia.com>
Jason Gunthorpe wrote:
[..]
> I feel like starting with trust=0 is much cleaner than using
> autoprobe. Especially since it would be nice that when you do
> ultimately set trust!=0 then you do want the kernel to do the normal
> autoprobe flow.
>
> Double so because I would like the iommu drivers to respond to trust 0
> by fully blocking the device 100% of the time without holes, so to
> make that work I would like to see the struct device report trust 0
> the moment the iommu framework attaches the iommu.
>
> How you decide the starting trust value for device during system boot
> is definately something we need to discuss properly..
>
> I liked your idea of using built in driver match, so if there is a
> simple command line paramater that says 'only built in is trusted'
> then we'd default all devices to untrusted and during device probe
> check if any built in driver is matching and auto-set trust to X based
> on the commandline parameter.
I do agree that forcing trust=0 at the beginning of time is attractive
and theoretically clean. I am concerned about subsystems that are not
prepared for driver attach failures. For example, I would not expect to
need to set trust for auxiliary bus devices if the host device is
trusted.
However, the work to set module autoprobe policy is on the same order as
adding a module scoped trust policy.
So something like "modprobe $module trust=X" automatically tries to set
the device trust level on attach to any drivers in that module. That
could allow a semantic of "attach iff device is able to go to level 4".
> With the idea that only devices required to get to the initrd are
> built in. Then the initrd userspace has the policy to bring more
> devices into trusted!=0 to get to the root file system, then the
> rootfs has more policy for further devices, and so on.
>
> Probably this would ultimately escalate into annotations in the
> modinfo about default policies for various drivers.
Yes, escalate over time for subsystems to say "devices I create are
trusted, the responsibility to manage trust lies with clients of my
APIs".
> A kernel default policy of trusting everything without a "trust ops"
> (see below) may also be quite reasonable, however boot ordering the
> trust ops might be really tricky...
Given the optionality in selecting a trust ops provider I think it gets
extra messy quickly. Let me see how far I can get with built-in auto
trust + module trust policy.
> > > > > The DMA API just wants a flag in the struct device that says if the
> > > > > device can access encrypted memory or only decrypted.
> > > >
> > > > You mean separate "trusted to access private" and "currently enabled to
> > > > access private" properties? I am trying to think of a situation where
> > > > "dev->trust >= 3" and a flag saying "disable bouncing for encrypted
> > > > memory" would ever disagree.
> > >
> > > I'm steering the trust level toward more of an acceptance criteria.
> > > If the trust level is you have access to private memory but the device
> > > can't actually do that then fail the trust level change.
> > >
> > > Same for the reverse, if the trust level says no private memory and the
> > > device is T=1 then fail the trust level change.
> >
> > Ok, so the uapi for PCI/TDISP would be:
> >
> > echo $tsm > $pdev/tsm/lock
> > <gather evidence, validate with relying party>
> > echo 3 > $pdev/trust
> >
> > ...where that @trust attribute is a generic device semantic, but in the
> > case of PCI device connected to a given TSM it invokes the TSM hypercall
> > to transition the device to the RUN state and the TSM local call to
> > unblock DMA to private memory.
>
> Maybe, but I was thinking the transition through run/locked would be
> done through TSM uAPIs too. trust setting in the kernel just confirms
> the device is in the right state.
>
> But I haven't thought of a reason why the final switch to RUN couldn't
> happen like this either.
Right, the potential to see in-between states concerns me because TSM
uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trust is still showing the device at some lower level of trust. So
I think trust modification needs to be synchronous with privileges
granted/revoked.
[..]
> > The complication vs benefit tradeoff is indeed not mathing, but wanted
> > to do justice to Aneesh's proposal and the suitability of the sysfs
> > uapi.
>
> I think if you want something like this then it is better to target
> the root - remove the ability for concurrent userspace to wrongly
> operate the TSM entirely. Ie use a cdev, make it so going to LOCKED
> isolates access to only this cdev fd and require only this cdev fd to
> go to RUN. Then these kinds of bugs don't exist.
The netlink evidence proposal can handle this, it just needs a
'validate' command. 'Validate' records a device evidence generation
number. Require a stable generation number between entering LOCK and the
trust=4 event transitioning the device to RUN.
Yes, not as safe as fd-private LOCK-to-RUN, but I like semantic of
"modprobe $module trust=4" to say "transition to RUN and attach to all
validated devices".
[..]
> Basically the 'trust' generic framework sits on top of some "trust
> ops" that will be provided by the security module that is affiliated
> with the struct device (ie thunderbolt, TSM TDISP, TSM Link IDE, etc,
> etc)
>
> Then it becomes a general synchronization point where on one side the
> "tust ops" can ack that the level is acceptable and consistent with
> the system when on the other side generic compoments like IOMMU,
> driver binding, etc can respond to it and change their behavior.
Something like this, yes.
> > For bare metal PCI device security the TSM 'connection' needs to be
> > established in order to enable device evidence collection.
> >
> > echo $tsm > $pdev/tsm/connect
> > <validate device evidence>
> > echo 2 > $pdev/trust
> >
> > Now, I question whether 5 trust levels instead of 4. This would be to
> > explicitly only trust devices where the TSM has established physical
> > link encryption, or the TSM has asserted that the link is protected by
> > other means. So the trust levels are:
>
> I probably wouldn't use an int for the uAPI,
Yes, but an int for now saves the bikeshed of the level names till a bit
later.
> ...but yes picking the initial levels is important. As above since
> this is a clearing point between two different worlds it needs to be
> defined in some way both sides can understand what it means for them.
>
> > 0 disconnected: bus does not attach drivers
> > 1 limited: core code deploys hostile device mitigations like disable
> > ATS, CC force shared memory bouncing.
> > 2 DMA: full DMA access, driver responsible for protecting against
> > adverarial devices.
> > 3 Link: mitigations against physical integrity and snooping attacks
> > deployed
> > 4 TCB: full device interface validation via a protocol like TDISP,
> > CC private memory access granted.
>
> This seems reasonable to me, the 3/4 distinction is not meaningful for
> the iommu&dev side, but it does provide a good check point for the
> "trust ops". If userspace ack's that it expects physical security and
> the kernel says it isn't physically secure (or becomes insecure later)
> then it should fail.
>
> > Where the native Rust library based SPDM driver only offers trust level
> > 2, bare metal TSMs can support trust level 3, and the TSM interfaces in
> > CC VMs can support trust level 4.
>
> I'm not sure that the SPDM driver even provides a "trust ops" right? I
> would guess that 0/1/2 are simply built in always available if trust ops are
> NULL and 3/4 require positive reply from the ops to accept it.
Right. Even though the SPDM driver allows device evidence to be
collected there is no event like LOCK-to-RUN that expects validated
evidence as a precondition.
> So #3 needs a "trust ops" linked to enabling link IDE.. If this is
> done in-kernel the link IDE module is providing the trust ops and just
> using SPDM as a library to establish the link IDE keys?
Yes, but note to date only Intel platforms allow for IDE establishment
without talking to the platform TSM.
^ permalink raw reply
* Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Gavin Shan @ 2026-03-25 4:07 UTC (permalink / raw)
To: Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Suzuki K Poulose, Zenghui Yu, linux-arm-kernel,
linux-kernel, Joey Gouly, Alexandru Elisei, Christoffer Dall,
Fuad Tabba, linux-coco, Ganapatrao Kulkarni, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-1-steven.price@arm.com>
Hi Steven,
On 3/19/26 1:53 AM, Steven Price wrote:
>
> This series is based on v7.0-rc1. It is also available as a git
> repository:
>
> https://gitlab.arm.com/linux-arm/linux-cca cca-host/v13
>
> Work in progress changes for kvmtool are available from the git
> repository below:
>
> https://gitlab.arm.com/linux-arm/kvmtool-cca cca/v11
>
Could you please share if we have a working qemu repository on top of this
(v13) series? The previous qemu repository [1] seems out of dated for long
time. I heard Jean won't be able to continue his efforts on QEMU part, who
is going to pick it up in this case.
[1] https://git.codelinaro.org/linaro/dcap/qemu.git (branch: cca/latest)
> Note that the kvmtool code has been tidied up (thanks to Suzuki) and
> this involves a minor change in flags. The "--restricted_mem" flag is no
> longer recognised (or necessary).
>
> The TF-RMM has not yet merged the RMMv2.0 support, so you will need to
> use the following branch:
>
> https://git.trustedfirmware.org/TF-RMM/tf-rmm.git topics/rmm-v2.0-poc
>
I'm seeing error to initialize RMM with the suggested RMM branch (topics/rmm-v2.0-poc)
and the upstream TF-A [1]. It seems the problem is compatible issue in the
RMM-EL3 interface. RMM requires verion 2.0 while TF-A only supports 0.8. So
I guess I must be using a wrong TF-A repository. Could you please share which
TF-A repository you use for testing?
[1] git@github.com:ARM-software/arm-trusted-firmware.git (branch: master)
Booting logs
=============
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v2.14.0(debug):67edb4f8e
NOTICE: BL1: Built : 00:01:39, Mar 25 2026
INFO: BL1: RAM 0xe0ee000 - 0xe0f7000
INFO: BL1: Loading BL2
INFO: Loading image id=1 at address 0xe05b000
INFO: Image id=1 loaded: 0xe05b000 - 0xe0642bc
NOTICE: BL1: Booting BL2
INFO: Entry point address = 0xe05b000
INFO: SPSR = 0x3cd
NOTICE: BL2: v2.14.0(debug):67edb4f8e
NOTICE: BL2: Built : 00:01:39, Mar 25 2026
INFO: BL2: Doing platform setup
INFO: Reserved RMM memory [0x40100000, 0x418fffff] in Device tree
INFO: BL2: Loading image id 3
INFO: Loading image id=3 at address 0xe090000
INFO: Image id=3 loaded: 0xe090000 - 0xe0a292b
INFO: BL2: Loading image id 35
INFO: Loading image id=35 at address 0x40100000
INFO: Image id=35 loaded: 0x40100000 - 0x401a11e0
INFO: BL2: Loading image id 5
INFO: Loading image id=5 at address 0x60000000
INFO: Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE: BL2: Booting BL31
INFO: Entry point address = 0xe090000
INFO: SPSR = 0x3cd
INFO: GPT: Boot Configuration
INFO: PPS/T: 0x2/40
INFO: PGS/P: 0x0/12
INFO: L0GPTSZ/S: 0x0/30
INFO: PAS count: 6
INFO: L0 base: 0xedfe000
INFO: Enabling Granule Protection Checks
NOTICE: BL31: v2.14.0(debug):67edb4f8e
NOTICE: BL31: Built : 00:01:39, Mar 25 2026
INFO: GICv3 without legacy support detected.
INFO: ARM GICv3 driver initialized in EL3
INFO: Maximum SPI INTID supported: 287
INFO: BL31: Initializing runtime services
INFO: RMM setup done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
ERROR: RMM init failed: -2 <<<< Error raised by RMM here
WARNING: BL31: RMM initialization failed
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x60000000
INFO: SPSR = 0x3c9
UEFI firmware (version built at 19:33:51 on Mar 3 2026)
Thanks,
Gavin
^ permalink raw reply
* Re: [PATCH v2 02/16] fs, x86/resctrl: Add architecture routines for kernel mode initialization
From: Reinette Chatre @ 2026-03-24 22:53 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <72315f83ddf35b0de53fb911f46623782d95f507.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Implement the resctrl kernel mode (kmode) arch initialization.
>
> - Add resctrl_arch_get_kmode_cfg() to fill the default kernel mode
> (INHERIT_CTRL_AND_MON). This can be extended later (e.g. for PLZA) to set
> additional modes.
I do not think this is something that the architecture should set, at least
at this time. Every mode has different requirements and this just lets the arch set
it without any support for what configurations it implies. For example, if
arch sets a different default mode than INHERIT_CTRL_AND_MON then PQR_PLZA_ASSOC
needs to be programmed as the CPUs come online and this does not seem to
accommodate this. This implementation appears to have significant assumptions on
what architecture will end up setting since it is only considering PLZA.
>
> - Add global resctrl_kcfg and resctrl_kmode_init() to initialize default
> values.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> v2: New patch to handle PLZA interfaces with /sys/fs/resctrl/info/ directory.
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
> ---
> arch/x86/kernel/cpu/resctrl/core.c | 7 +++++++
> fs/resctrl/rdtgroup.c | 10 ++++++++++
> 2 files changed, 17 insertions(+)
>
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 7667cf7c4e94..4c3ab2d93909 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -892,6 +892,13 @@ bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
> }
> }
>
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg)
> +{
> + kcfg->kmode = INHERIT_CTRL_AND_MON;
> + kcfg->kmode_cur = INHERIT_CTRL_AND_MON;
> + kcfg->k_rdtgrp = NULL;
> +}
I already commented on the arch vs filesystem settings.
When using an arch helper this forces all architectures to support this helper. Is a
helper required? Is it perhaps possible for arch to set a property instead? For example,
how enumeration is handled?
I think the assumption here is that INHERIT_CTRL_AND_MON is the default and expected to
be supported by all architectures. I do not see why arch should set this as default but
instead this should be from resctrl fs. At the same time it is expected that the
architecture supports this mode so there needs to be a failure if an architecture does
not support this mode?
I'm going to stop here. I think the comments so far may result in major changes already
making further detailed review of patches unnecessary.
Reinette
^ permalink raw reply
* Re: [PATCH v2 01/16] fs/resctrl: Add kernel mode (kmode) data structures and arch hook
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <c3381fbbce357b00895252572907645a92cd9718.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
> Add resctrl_kmode, resctrl_kmode_cfg, kernel mode bit defines, and
> resctrl_arch_get_kmode_cfg() for resctrl kernel mode (e.g. PLZA) support.
We should not have to start every series from scratch.
Documentation/process/maintainer-tip.rst. Always.
> ---
> include/linux/resctrl.h | 10 ++++++++++
> include/linux/resctrl_types.h | 30 ++++++++++++++++++++++++++++++
> 2 files changed, 40 insertions(+)
>
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index 006e57fd7ca5..2c36d1ac392f 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -699,6 +699,16 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable);
> */
> bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r);
>
> +/**
> + * resctrl_arch_get_kmode_cfg() - Get resctrl kernel mode configuration
> + * @kcfg: Filled with current kernel mode config (kmode, kmode_cur, k_rdtgrp).
> + *
> + * Used by the arch (e.g. x86) to report which kernel mode is active and,
> + * when a global assign mode is in use, which rdtgroup is assigned to
> + * kernel work.
> + */
> +void resctrl_arch_get_kmode_cfg(struct resctrl_kmode_cfg *kcfg);
This interface does not look right. Would it not be resctrl fs that determines
which resource group is assigned? This cannot be set by arch. Why does arch decide
which mode is active? Is this not also resctrl fs? Should arch not just tell
resctrl fs what it supports?
> +
> extern unsigned int resctrl_rmid_realloc_threshold;
> extern unsigned int resctrl_rmid_realloc_limit;
>
> diff --git a/include/linux/resctrl_types.h b/include/linux/resctrl_types.h
> index a5f56faa18d2..6b78b08eab29 100644
> --- a/include/linux/resctrl_types.h
> +++ b/include/linux/resctrl_types.h
> @@ -65,7 +65,37 @@ enum resctrl_event_id {
> QOS_NUM_EVENTS,
> };
>
> +/**
> + * struct resctrl_kmode - Resctrl kernel mode descriptor
> + * @name: Human-readable name of the kernel mode.
> + * @val: Bitmask value for the kernel mode (e.g. INHERIT_CTRL_AND_MON).
> + */
> +struct resctrl_kmode {
> + char name[32];
> + u32 val;
> +};
There is no reason why this needs to be in a central header exposed to archs. Could
this not be a static within the only function that uses it? Something like
rdt_mode_str[]?
> +
> +/**
> + * struct resctrl_kmode_cfg - Resctrl kernel mode configuration
> + * @kmode: Requested kernel mode.
> + * @kmode_cur: Currently active kernel mode.
> + * @k_rdtgrp: Resource control structure in use, or NULL otherwise.
> + */
> +struct resctrl_kmode_cfg {
> + u32 kmode;
> + u32 kmode_cur;
> + struct rdtgroup *k_rdtgrp;
> +};
> +
> #define QOS_NUM_L3_MBM_EVENTS (QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
> #define MBM_STATE_IDX(evt) ((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
>
> +/* Resctrl kernel mode bits (e.g. for PLZA). */
> +#define INHERIT_CTRL_AND_MON BIT(0) /* Kernel uses same CLOSID/RMID as user. */
> +/* One CLOSID for all kernel work; RMID inherited from user. */
> +#define GLOBAL_ASSIGN_CTRL_INHERIT_MON BIT(1)
> +/* One resource group (CLOSID+RMID) for all kernel work. */
> +#define GLOBAL_ASSIGN_CTRL_ASSIGN_MON BIT(2)
> +#define RESCTRL_KERNEL_MODES_NUM 3
I think it will make the code much easier to understand if the different modes are described by an
enum. For example,
enum resctrl_kernel_modes {
INHERIT_CTRL_AND_MON,
GLOBAL_ASSIGN_CTRL_INHERIT_MON,
GLOBAL_ASSIGN_CTRL_ASSIGN_MON,
RESCTRL_KMODE_LAST = GLOBAL_ASSIGN_CTRL_ASSIGN_MON
};
#define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
The supported kernel modes can still be managed as a bitmap with intuitive API using the
enum that will make the code easier to read. For example, __set_bit(INHERIT_CTRL_AND_MON, ...)
or BIT(INHERIT_CTRL_AND_MON). The naming is awkward at the moment though, we should improve here.
Reinette
^ permalink raw reply
* Re: [PATCH v2 00/16] fs,x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem
From: Reinette Chatre @ 2026-03-24 22:51 UTC (permalink / raw)
To: Babu Moger, corbet, tony.luck, Dave.Martin, james.morse, tglx,
mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <cover.1773347820.git.babu.moger@amd.com>
Hi Babu,
On 3/12/26 1:36 PM, Babu Moger wrote:
>
> This series adds support for Privilege-Level Zero Association (PLZA) to the
> resctrl subsystem. PLZA is an AMD feature that allows specifying a CLOSID
> and/or RMID for execution in kernel mode (privilege level zero), so that
> kernel work is not subject to the same resource constrains as the current
> user-space task. This avoids kernel operations being aggressively throttled
> when a task's memory bandwidth is heavily limited.
>
> The feature documentation is not yet publicly available, but it is expected
> to be released in the next few weeks. In the meantime, a brief description
> of the features is provided below.
>
> Privilege Level Zero Association (PLZA)
>
> Privilege Level Zero Association (PLZA) allows the hardware to
> automatically associate execution in Privilege Level Zero (CPL=0) with a
> specific COS (Class of Service) and/or RMID (Resource Monitoring
> Identifier). The QoS feature set already has a mechanism to associate
> execution on each logical processor with an RMID or COS. PLZA allows the
> system to override this per-thread association for a thread that is
> executing with CPL=0.
> ------------------------------------------------------------------------
>
> The series introduces the feature in a way that supports the interface in
> a generic manner to accomodate MPAM or other vendor specific implimentation.
>
> Below is the detailed requirements provided by Reinette:
> https://lore.kernel.org/lkml/2ab556af-095b-422b-9396-f845c6fd0342@intel.com/
Our discussion considered how resctrl could support PLZA in a generic way while
also preparing to support MPAM's variants and how PLZA may evolve to have similar
capabilities when considering the capabilities of its registers.
This does not mean that your work needs to implement everything that was discussed.
Instead, this work is expected to just support what PLZA is capable of today but
do so in a way that the future enhancements could be added to.
This series is quite difficult to follow since it appears to implement a full
featured generic interface while PLZA cannot take advantage of it.
Could you please simplify this work to focus on just enabling PLZA and only
add interfaces needed to do so?
>
> Summary:
> 1. Kernel-mode/PLZA controls and status should be exposed under the resctrl
> info directory:/sys/fs/resctrl/info/, not as a separate or arch-specific path.
>
> 2. Add two info files
>
> a. kernel_mode
> Purpose: Control how resource allocation and monitoring apply in kernel mode
> (e.g. inherit from task vs global assign).
>
> Read: List supported modes and show current one (e.g. with [brackets]).
> Write: Set current mode by name (e.g. inherit_ctrl_and_mon, global_assign_ctrl_assign_mon).
>
> b. kernel_mode_assignment
>
> Purpose: When a “global assign” kernel mode is active, specify which resctrl group
> (CLOSID/RMID) is used for kernel work.
>
> Read: Show the assigned group in a path-like form (e.g. //, ctrl1//, ctrl1/mon1/).
> Write: Assign or clear the group used for kernel mode (and optionally clear with an empty write).
>
> The patches are based on top of commit (v7.0.0-rc3)
> 839e91ce3f41b (tip/master) Merge branch into tip/master: 'x86/tdx'
> ------------------------------------------------------------------------
>
> Examples: kernel_mode and kernel_mode_assignment
>
> All paths below are under /sys/fs/resctrl/ (e.g. info/kernel_mode means
> /sys/fs/resctrl/info/kernel_mode). Resctrl must be mounted and the platform
> must support the relevant modes (e.g. AMD with PLZA).
>
> 1) kernel_mode — show and set the current kernel mode
>
> Read supported modes and which one is active (current in brackets):
>
> $ cat info/kernel_mode
> [inherit_ctrl_and_mon]
> global_assign_ctrl_inherit_mon
> global_assign_ctrl_assign_mon
>
> Set the active mode (e.g. use one CLOSID+RMID for all kernel work):
>
> $ echo "global_assign_ctrl_assign_mon" > info/kernel_mode
> $ cat info/kernel_mode
> inherit_ctrl_and_mon
> global_assign_ctrl_inherit_mon
> [global_assign_ctrl_assign_mon]
>
> Mode meanings:
> - inherit_ctrl_and_mon: kernel uses same CLOSID/RMID as the current task (default).
> - global_assign_ctrl_inherit_mon: one CLOSID for all kernel work; RMID inherited from user.
> - global_assign_ctrl_assign_mon: one resource group (CLOSID+RMID) for all kernel work.
>
> 2) kernel_mode_assignment — show and set which group is used for kernel work
>
> Only relevant when kernel_mode is not "inherit_ctrl_and_mon". Read the
To help with future usages please connect visibility of this file with the mode in
info/kernel_mode. This helps us to support future modes with other resctrl files, possible
within each resource group.
Specifically, kernel_mode_assignment is not visible to user space if mode is "inherit_ctrl_and_mon",
while it is visible when mode is global_assign_ctrl_inherit_mon or global_assign_ctrl_assign_mon.
> currently assigned group (path format is "CTRL_MON/MON/"):
The format depends on the mode, right? If the mode is "global_assign_ctrl_inherit_mon"
then it should only contain a control group, alternatively, if the mode is
"global_assign_ctrl_assign_mon" then it contains control and mon group. This gives
resctrl future flexibility to change format for future modes.
We should also consider the scenario when it is a "monitoring only" system, which can
happen independent from what hardware actually supports, for example, if user boots
with "rdt=!l3cat,!l2cat,!mba,!smba". In this case I assume CLOS should just always be
zero and thus only "default control group" is accepted?
>
> $ cat info/kernel_mode_assignment
> //
>
> "//" means the default CTRL_MON group is assigned. Assign a specific
> group instead (e.g. a CTRL_MON group "ctrl1", or a MON group "mon1" under it):
>
> $ echo "ctrl1//" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1//
>
> $ echo "ctrl1/mon1/" > info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> ctrl1/mon1/
>
> Clear the assignment (no dedicated group for kernel work):
>
> $ echo >> info/kernel_mode_assignment
> $ cat info/kernel_mode_assignment
> Kmode is not configured
This does not look right. Would this not create a conflict between info/kernel_mode
and info/kernel_mode_assignment about what the current mode is? The way I see it
info/kernel_mode_assignment must always contain a valid group.
>
> Errors (e.g. invalid group name or unsupported mode) are reported in
> info/last_cmd_status.
>
> ---
>
> v2:
> This is similar to RFC with new proposal. Names of the some interfaces
> are not final. Lets fix that later as we move forward.
>
> Separated the two features: Global Bandwidth Enforcement (GLBE) and
> Privilege Level Zero Association (PLZA).
>
> This series only adds support for PLZA.
>
> Used the name of the feature as kmode instead of PLZA. That can be changed as well.
>
> Tony suggested using global variables to store the kernel mode
> CLOSID and RMID. However, the kernel mode CLOSID and RMID are
> coming from rdtgroup structure with the new interface. Accessing
> them requires holding the associated lock, which would make the
> context switch path unnecessarily expensive. So, dropped the idea.
> https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
> Let me know if there are other ways to optimize this.
I do not see why the context switch path needs to be touched at all with this
implementation. Since PLZA only supports global assignment does it not mean that resctrl
only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and
info/kernel_mode_assignment?
Consider some of the scenarios:
resctrl mount with default state:
# cat info/kernel_mode
[inherit_ctrl_and_mon]
global_assign_ctrl_inherit_mon
global_assign_ctrl_assign_mon
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
enable global_assign_ctrl_assign_mon mode:
# echo "global_assign_ctrl_assign_mon" > info/kernel_mode
Expectation here is that when user space sets this mode as above then resctrl would
in turn program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
I do not see why it is necessary to maintain any per-CPU or per-task state or needing
to touch the context switch code. Since PLZA only supports global could it not
just set MSR_IA32_PQR_PLZA_ASSOC on all online CPUs and be done with it?
Only caveat is that if a CPU is offline then this setting needs to be stashed
so that MSR_IA32_PQR_PLZA_ASSOC can be set when new CPU comes online.
The way that rdtgroup_config_kmode() introduced in patch #11 assumes it is dealing
with RDT_RESOURCE_L3 and traverses the resource domain list and resource group
CPU mask seems unnecessary to me as well as error prone since the system may only
have, for example, RDT_RESOURCE_MBA enabled or even just monitoring. Why not just set
MSR_IA32_PQR_PLZA_ASSOC on all CPUs and be done?
To continue the scenarios ...
After user's setting above related files read:
# cat info/kernel_mode
inherit_ctrl_and_mon
global_assign_ctrl_inherit_mon
[global_assign_ctrl_assign_mon]
# cat info/kernel_mode_assignment
//
Modify group used by global_assign_ctrl_assign_mon mode:
# echo 'ctrl1/mon1/' > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=<rmid of mon1>
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=1
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
Enable global_assign_ctrl_inherit_mon mode:
# echo "global_assign_ctrl_inherit_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode
inherit_ctrl_and_mon
[global_assign_ctrl_inherit_mon]
global_assign_ctrl_assign_mon
# cat info/kernel_mode_assignment <==== returns just a ctrl group
/
Modify group used by global_assign_ctrl_inherit_mon mode:
# echo ctrl1 > info/kernel_mode_assignment
Expectation here is that when user space sets this then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=<closid of ctrl1>
MSR_IA32_PQR_PLZA_ASSOC.closid_en=1
MSR_IA32_PQR_PLZA_ASSOC.plza_en=1
# cat info/kernel_mode_assignment <==== returns just a ctrl group
ctrl/
Enable inherit_ctrl_and_mon mode:
# echo "inherit_ctrl_and_mon" > info/kernel_mode
Expectation here is that when user space sets this mode then resctrl would
program MSR_IA32_PQR_PLZA_ASSOC on all CPUs to be:
MSR_IA32_PQR_PLZA_ASSOC.rmid=0
MSR_IA32_PQR_PLZA_ASSOC.rmid_en=0
MSR_IA32_PQR_PLZA_ASSOC.closid=0
MSR_IA32_PQR_PLZA_ASSOC.closid_en=0
MSR_IA32_PQR_PLZA_ASSOC.plza_en=0
At this point info/kernel_mode_assignment is not visible anymore:
# ls info/kernel_mode_assignment
ls: cannot access 'info/kernel_mode_assignment': No such file or directory
From what I understand above exposes and enables full capability of PLZA. All the other
per-task and per-cpu handling in this series is not something that PLZA can benefit from.
If this is not the case, what am I missing? Could this series be simplified to just support
PLZA today? When next hardware with more capability needs to be supported resctrl could be
enhanced to support it by using the more accurate information about what the hardware is
capable of.
We also do not really know what use cases users prefer. This may even be sufficient.
Reinette
^ permalink raw reply
* Re: [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Edgecombe, Rick P @ 2026-03-24 22:03 UTC (permalink / raw)
To: bp@alien8.de, marcandre.lureau@redhat.com, kas@kernel.org,
hpa@zytor.com, mingo@redhat.com, x86@kernel.org, tglx@kernel.org,
dave.hansen@linux.intel.com, Qiang, Chenyi
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev,
linux-kernel@vger.kernel.org
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-2-8f29f2c17278@redhat.com>
On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
> accepted before use. The first access triggers a fatal "SEPT entry in
> PENDING state" EPT violation and KVM terminates the guest.
>
> Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
> calls tdx_accept_memory() for the range being onlined.
>
> The notifier returns NOTIFY_BAD on acceptance failure, preventing the
> memory from going online.
Does this depend on patch 1 somehow?
^ permalink raw reply
* Re: [PATCH 1/2] x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
From: Edgecombe, Rick P @ 2026-03-24 22:02 UTC (permalink / raw)
To: bp@alien8.de, marcandre.lureau@redhat.com, kas@kernel.org,
hpa@zytor.com, mingo@redhat.com, x86@kernel.org, tglx@kernel.org,
dave.hansen@linux.intel.com, Qiang, Chenyi
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev,
linux-kernel@vger.kernel.org
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-1-8f29f2c17278@redhat.com>
On Tue, 2026-03-24 at 19:21 +0400, Marc-André Lureau wrote:
> try_accept_one() treats any non-zero return from __tdcall() as a
> failure. However, per the TDX Module Base Spec (Table SEPT Walk Cases),
> TDG.MEM.PAGE.ACCEPT returns a non-zero status code with bit 63 clear
> when the target page is already in MAPPED state (i.e., already
> accepted). This is a "success-with-warning" -- the page is usable and no
> action is needed.
>
> Check only bit 63 (TDX_ERROR) to distinguish real errors from
> success-with-warning returns, rather than treating all non-zero values
> as failures.
>
> Assisted-by: Claude:claude-opus-4-6
> Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Hmm. Accepting private memory is a security sensitive operation, so I think it
is probably bad to silently hide the detection of re-accepting.
For example, if the kernel accepts a page and sets some values in it, the VMM
could reset the data to zero by re-adding the page and letting the second accept
zero it. It allows the VMM to have some limited ability to mess with guest data.
If we detect a re-accept we should probably warn on it actually.
Not sure on if the specific case in this series is problematic, but this patch
changes the behavior generally.
^ permalink raw reply
* SVSM Development Call March 25, 2026
From: Jörg Rödel @ 2026-03-24 19:14 UTC (permalink / raw)
To: coconut-svsm, linux-coco
Hi,
Here is the call for agenda items for this weeks SVSM development call. Please
send any agenda items you have in mind as a reply to this email or raise them
in the meeting.
We will use the LF Zoom instance. Details of the meeting can be found in our
governance repository at:
https://github.com/coconut-svsm/governance
The link to the COCONUT-SVSM calendar is:
https://zoom-lfx.platform.linuxfoundation.org/meetings/coconut-svsm?view=week
The meeting will be recorded and the recording eventually published.
Regards,
Jörg
^ permalink raw reply
* Re: [PATCH v4 1/2] dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
From: Jason Gunthorpe @ 2026-03-24 19:11 UTC (permalink / raw)
To: Jiri Pirko
Cc: Robin Murphy, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, leon, sean.anderson,
ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco
In-Reply-To: <lufy3ppgiy7637iravgzabajj67chmqvg22qc3fd2yqe2p33ws@b54vibhehth2>
On Tue, Mar 17, 2026 at 09:26:21AM +0100, Jiri Pirko wrote:
> >...although, why *shouldn't* this be allowed with a vIOMMU? (Especially given
> >that a vIOMMU for untrusted devices can be emulated by the host VMM without
> >the CoCo hypervisor having to care at all - again, at least on Arm and other
> >architectures where IOMMUs are regular driver model devices)
>
> Well, when iommu path is able to consume the attr, this restriction
> should be lifted. This is basically a sanity check for the
> dma_map_phys() caller.
Right we eventually need a matching IOMMU_DECRYPTED.
It needs to mirror how the CPUs work - any place that would use
pgprot_decrypted to create a PTE should use IOMMU_PROT_DECRYPTED to
create an iommu mapping.
The current hack in AMD assumes IOMMU_DECRYPTED behavior for
IOMMU_MMIO, but that isn't general enough..
There is some maze to get there but for the moment I think it is fine
to just not support vIOMMU, it isn't like any vIOMMU drivers even
exist for CC VMs right now.
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Mostafa Saleh @ 2026-03-24 18:32 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260324175717.GE8437@ziepe.ca>
On Tue, Mar 24, 2026 at 5:57 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Mar 24, 2026 at 05:36:23PM +0000, Mostafa Saleh wrote:
> > But it's not about drivers in that case, it's about many places
> > (SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
> > ownership so in some cases they step on each other's toes, and I don't
> > think that will get simpler with yet another caller in this series
>
> I don't understand how this can be, ownership is clear. SWIOTLB owns
> the buffer, dma alloc coherent owns the buffer, user owns the
> buffer. There should be no other cases, and they don't step on each
> other unless the APIs are being used wrong.
>
Logically, that's the case, but the DMA-direct code currently loses
this information and assumes it can encrypt/decrypt any memory even
the SWIOTLB one.
That's what I am fixing in my series. When I respin, I can try to
introduce some more helpers around that to make it easier to integrate
new cases.
Thanks,
Mostafa
> > I am fine with the API design you mentioned, but I believe that it
> > needs clear documentation specifying who is responsible for
> > decryption. The code should provide wrappers checking for these cases
> > instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
> > everywhere in DMA-direct.
>
> Redoingt how dma-api works internally is some other project... It
> would be nice if swiotlb would sort of recursively DMA map using the
> new flag instead of open coding it but that is pretty minor.
>
> Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jason Gunthorpe @ 2026-03-24 17:57 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <CAFgf54qwA2D1Xa4rnruJ4Nfp5BsB=T_pB3hzz9HBjh22TL17uA@mail.gmail.com>
On Tue, Mar 24, 2026 at 05:36:23PM +0000, Mostafa Saleh wrote:
> But it's not about drivers in that case, it's about many places
> (SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
> ownership so in some cases they step on each other's toes, and I don't
> think that will get simpler with yet another caller in this series
I don't understand how this can be, ownership is clear. SWIOTLB owns
the buffer, dma alloc coherent owns the buffer, user owns the
buffer. There should be no other cases, and they don't step on each
other unless the APIs are being used wrong.
> I am fine with the API design you mentioned, but I believe that it
> needs clear documentation specifying who is responsible for
> decryption. The code should provide wrappers checking for these cases
> instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
> everywhere in DMA-direct.
Redoingt how dma-api works internally is some other project... It
would be nice if swiotlb would sort of recursively DMA map using the
new flag instead of open coding it but that is pretty minor.
Jason
^ permalink raw reply
* Re: [PATCH net-next v3 0/2] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Mostafa Saleh @ 2026-03-24 17:36 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260324122416.GD8437@ziepe.ca>
On Tue, Mar 24, 2026 at 12:24 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Mar 24, 2026 at 12:14:36PM +0000, Mostafa Saleh wrote:
> > On Tue, Mar 24, 2026 at 12:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Tue, Mar 17, 2026 at 01:24:13PM +0000, Mostafa Saleh wrote:
> > >
> > > > On the other hand, for restricted-dma, the memory decryption is deep
> > > > in the DMA direct memory allocation and the DMA API callers (for ex
> > > > virtio drivers) are clueless about it and can’t pass any attrs.
> > > > My proposal was specific to restricted-dma and won’t work for your case.
> > >
> > > How is this any different from CC?
> > >
> > > If the device cannot dma to "encrypted" memory, whatever that means
> > > for you, then the DMA API:
> > > - Makes dma alloc coherent return "decrypted" memory, and the built
> > > in mapping of coherent memory knows about this
> > > - Makes dma_map_xxx use SWIOTLB to bounce to decrypted memory
> > >
> > > There is no need for something like virtio drivers to be aware of
> > > any of this.
> > >
> > > On the other hand if the driver deliberately allocates decrypted
> > > memory without using DMA API alloc coherent then it knows it did it
> > > and can pass the flag to map it.
> > >
> >
> > The problem is that the DMA API currently gets confused by this; it
> > can end up double decrypting the memory or using the wrong functions
> > as mentioned in [1]
>
> I fully belive there are bugs, but the API design is sound. If you use
> the coherent allocations from the DMA API then it knows decryption has
> happened when it generates a dma_addr_t and there should be no issue.
>
> Now, if drivers are using the DMA API wrong, like trying to double map
> coherent allocations then they are broken. I also would not be
> surprised to find cases like this.
>
But it's not about drivers in that case, it's about many places
(SWIOTLB and DMA-direct) calling set_memory_decrypted() without clear
ownership so in some cases they step on each other's toes, and I don't
think that will get simpler with yet another caller in this series
I am fine with the API design you mentioned, but I believe that it
needs clear documentation specifying who is responsible for
decryption. The code should provide wrappers checking for these cases
instead of having is_swiotlb_for_alloc() and force_dma_unencrypted()
everywhere in DMA-direct.
Thanks,
Mostafa
> Jason
^ permalink raw reply
* [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-0-8f29f2c17278@redhat.com>
In TDX guests, hotplugged memory (e.g., via virtio-mem) is never
accepted before use. The first access triggers a fatal "SEPT entry in
PENDING state" EPT violation and KVM terminates the guest.
Fix this by registering a MEM_GOING_ONLINE memory hotplug notifier that
calls tdx_accept_memory() for the range being onlined.
The notifier returns NOTIFY_BAD on acceptance failure, preventing the
memory from going online.
Assisted-by: Claude:claude-opus-4-6
Reported-by: Chenyi Qiang <chenyi.qiang@intel.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c
index 7b2833705d475..89f90bc303258 100644
--- a/arch/x86/coco/tdx/tdx.c
+++ b/arch/x86/coco/tdx/tdx.c
@@ -8,6 +8,7 @@
#include <linux/export.h>
#include <linux/io.h>
#include <linux/kexec.h>
+#include <linux/memory.h>
#include <asm/coco.h>
#include <asm/tdx.h>
#include <asm/vmx.h>
@@ -1194,3 +1195,40 @@ void __init tdx_early_init(void)
tdx_announce();
}
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static int tdx_guest_memory_notifier(struct notifier_block *nb,
+ unsigned long action, void *v)
+{
+ struct memory_notify *mn = v;
+ phys_addr_t start, end;
+
+ if (action != MEM_GOING_ONLINE)
+ return NOTIFY_OK;
+
+ start = PFN_PHYS(mn->start_pfn);
+ end = start + PFN_PHYS(mn->nr_pages);
+
+ if (!tdx_accept_memory(start, end)) {
+ pr_err("Failed to accept memory [0x%llx, 0x%llx)\n",
+ (unsigned long long)start,
+ (unsigned long long)end);
+ return NOTIFY_BAD;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block tdx_guest_memory_nb = {
+ .notifier_call = tdx_guest_memory_notifier,
+};
+
+static int __init tdx_guest_memory_init(void)
+{
+ if (!cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
+ return 0;
+
+ return register_memory_notifier(&tdx_guest_memory_nb);
+}
+core_initcall(tdx_guest_memory_init);
+#endif
--
2.53.0
^ permalink raw reply related
* [PATCH 1/2] x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In-Reply-To: <20260324-tdx-hotplug-fixes-v1-0-8f29f2c17278@redhat.com>
try_accept_one() treats any non-zero return from __tdcall() as a
failure. However, per the TDX Module Base Spec (Table SEPT Walk Cases),
TDG.MEM.PAGE.ACCEPT returns a non-zero status code with bit 63 clear
when the target page is already in MAPPED state (i.e., already
accepted). This is a "success-with-warning" -- the page is usable and no
action is needed.
Check only bit 63 (TDX_ERROR) to distinguish real errors from
success-with-warning returns, rather than treating all non-zero values
as failures.
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
arch/x86/coco/tdx/tdx-shared.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/coco/tdx/tdx-shared.c b/arch/x86/coco/tdx/tdx-shared.c
index 1655aa56a0a51..24983601a2ded 100644
--- a/arch/x86/coco/tdx/tdx-shared.c
+++ b/arch/x86/coco/tdx/tdx-shared.c
@@ -35,7 +35,7 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len,
}
args.rcx = start | page_size;
- if (__tdcall(TDG_MEM_PAGE_ACCEPT, &args))
+ if (__tdcall(TDG_MEM_PAGE_ACCEPT, &args) & TDX_ERROR)
return 0;
return accept_size;
--
2.53.0
^ permalink raw reply related
* [PATCH 0/2] x86/tdx: Fix memory hotplug in TDX guests
From: Marc-André Lureau @ 2026-03-24 15:21 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe, Chenyi Qiang
Cc: linux-kernel, linux-coco, kvm, Marc-André Lureau
In TDX guests, hotplugged memory (e.g., via virtio-mem) must be accepted
via TDG.MEM.PAGE.ACCEPT before use. The first access to an unaccepted
page triggers a fatal "SEPT entry in PENDING state" EPT violation and
KVM terminates the guest.
This was discovered while testing virtio-mem resize with TDX guests.
The associated QEMU virtio-mem + TDX patch series is under review at:
https://patchew.org/QEMU/20260226140001.3622334-1-marcandre.lureau@redhat.com/
The fix has two parts:
1. Handle TDG.MEM.PAGE.ACCEPT "success-with-warning" returns for pages
that are already in MAPPED state (e.g., after offline/re-online
cycles), instead of treating them as fatal errors.
2. Register a MEM_GOING_ONLINE memory hotplug notifier that calls
tdx_accept_memory() before pages are freed to the buddy allocator.
The TDCALL transparently triggers KVM-side page augmentation (AUG)
followed by acceptance, avoiding the fatal EPT violation path.
The solution was suggested by Claude Code (Anthropic) and has been
tested with virtio-mem hot-add on a TDX guest. I did my best to review
the produced code and comments. Apologies if the agent did hallucinate.
Let me know if I need to check or correct something.
Thanks,
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
Marc-André Lureau (2):
x86/tdx: Handle TDG.MEM.PAGE.ACCEPT success-with-warning returns
x86/tdx: Accept hotplugged memory before online
arch/x86/coco/tdx/tdx-shared.c | 2 +-
arch/x86/coco/tdx/tdx.c | 38 ++++++++++++++++++++++++++++++++++++++
2 files changed, 39 insertions(+), 1 deletion(-)
---
base-commit: c369299895a591d96745d6492d4888259b004a9e
change-id: 20260324-tdx-hotplug-fixes-644d009dad63
Best regards,
--
Marc-André Lureau <marcandre.lureau@redhat.com>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox