Re: RFC: Kernel CXL cache support (and IOMMU implications)

[Jonathan Cameron <Jonathan.Cameron@huawei.com>]

On Fri, 13 Dec 2024 14:15:18 +0000
Alejandro Lucero Palau <alucerop@amd.com> wrote:

> On 11/20/24 22:33, Zhi Wang wrote:
> > On Tue, 19 Nov 2024 16:52:15 +0000
> > Alejandro Lucero Palau <alucerop@amd.com> wrote:
> >
> > Thanks so much for the doc. I just quickly went through the doc and here
> > are my comments.  
> 
> 
> Hi Zhi,
> 
> 
> Thanks for your comments. I did not reply earlier waiting for more 
> feedback from, mainly, the IOMMU kernel guys. Maybe CXL support is 
> something most of them neither have contemplated nor aware of (maybe) 
> requiring special handling. I really think IOMMU/DMA API will need some 
> change, but this document is for discussing it and maybe proving me wrong.
> 
> 
> Let's hope replying to your comments keep things moving somehow ...

Sorry it took me so long to get to this! 

I replied as I read through it, so thoughts may not be totally coherent.
Key points:

1. If you are doing CXL.cache on a device then either your host should be
   doing checking that the device is not accessing something it shouldn't or
   you should have done the work to ensure it is part of your trusted
   compute base.
2. Using restrictions on memory in the IOMMU page tables to avoid
   thrashing of coherency tracking resources / cache in the host is a non
   starter.  That puts a bound on the number used, but at the cost of
   breaking many use cases.  Page table coverage != cachelines in device.
   a) Endpoint should be part of TCB, trusted not to use more than it is
      told.
   b) Host should not do rubbish QoS so the burden should be mainly on the
      badly behaving device 
3. I'm not sure why (for what is discussed here) there is any problem with
   VM usecases. The same model used for ATS etc for PCIe VFs should apply
   just fine here.

Anyhow, that's enough muddying the waters for today.

Jonathan


> 
> >> November, 2024
> >>
> >> Tittle: CXL Cache support by the kernel
> >> Author: Alejandro Lucero (alucerop@amd.com)
> >>
> >> Version 0.1
> >>
> >> Introduction
> >> ========
> >>
> >> After the LPC where I presented the current status of the Type2 CXL.mem
> >> support
> >> patchset, and some ideas about supporting CXL.cache, it is time to dig
> >> deeper in
> >> this second goal, and discussing the security/reliability aspect as well.
> >>
> >> It is also important to try to describe how this is going to work and
> >> what the
> >> kernel needs to know and enforce. Reading the CXL specs when having in
> >> mind some
> >> specific use case can easily lead to assuming certains aspects with a
> >> different
> >> perspective from other readers/use cases. To start with, it is necessary to
> >> differentiate two "CXL cache" functionalities when a Type2 device is in
> >> place:
> >>
> >> 1) A Type2 device caching Host memory.
> >>
> >> 2) The Host caching HDM memory, that is the memory inside the Type2 CXL
> >> device.
> >>
> >> The first option is also what a Type1 device can do, and the kernel support
> >> needs to manage all those Type1/2 per CXL Root Complex knowing the resources
> >> limitation, that is the snooping cache size.
> >>
> >> A snoop cache allows the host to track which memory is being used/cached by
> >> those devices, enforcing the cache coherency. The specs are not clear
> >> about some
> >> important aspects regarding how the host can enforce the proper use of
> >> this by
> >> devices or even if the snoop cache needs to do so. At pages 786 and 787
> >> of CXL
> >> specs 3.1, how the system software should deal with CXL cache devices is
> >> given,
> >> but this is inside a Hot-plug section. I think we can assume the Host
> >> firmware/BIOS will follow same approach for enabling CXL cache, and the
> >> kernel
> >> needs to look at those devices with CXL cache enabled by the BIOS for
> >> properly
> >> handling the available space in the snoop cache.
> >>
> >> It is also worth to mention the CXL.cache protocol can be used in the two
> >> "CXL cache" functionalities listed above. However, the last CXL spec implies
> >> CXL.cache only used for the first case. Some comments about what the
> >> specs say
> >>
> >> regarding number of devices with a cache for host memory:
> >>
> >>           - up to 16 Type1 and/or Type2 devices allowed per VH.
> >>
> >> can be easily confused with the limitations of just one CXL Type2 device
> >> using
> >> CXL.cache for enforcing coherency of its HDM. 

It has been a while since I read the relevant sections, but CXL 3.0 introduced
a cache ID that I thought was precisely to allow for multiple CXL.Cache agents
per VCS (not the HDM-DB stuff that for this purpose is replacing bias
based coherency)  There may well not be any hosts that support that yet though
and it doesnt' really matter for rest of this discussion.

> >> This limitation is
> >> overcome with
> >> forcing Type2 device using HDM-DB, which relies on CXL.mem instead of
> >> CXL.cache
> >> for HDM cache coherency.
> >>
> >> While the Host is assumed to be able to access HDM in a Type2 device, and
> >> keeping data in the host cpu caches, it is the Type2 device
> >> responsibility to
> >> properly manage cache coherency of its HDM. There is nothing the kernel can
> >> control here.
> >>
> >> Therefore the interesting part and what this documents tries to cover is the
> >> Host memory being cached by Type2 or Type1 devices. While the main goal is
> >> discussing how the kernel needs to handle this, and to describe how it
> >> should
> >> work when CXL devices are used by the system/Host, some comments are made to
> >> cover the virtualization case where those CXL devices can potenetially
> >> be used
> >> (device passthrough) by guests/VMs. I try to expose the current security
> >> problems where IOMMU is used for restricting what a guest controlled
> >> CXL.cache
> >> device can read/write in Host memory what I think needs to be clarified by
> >> hardware vendors.
> >>
> >>
> >> Understanding the memory accesses from CXL devices
> >> ==================================
> >>
> >> For the sake of presenting the case about kernel CXL.cache support, I'll
> >> try to
> >> explain how it works (I should say "how I think it works") and the main
> >> points
> >> to discuss regarding how to implement this support. So, do not take the next
> >> explanation as the definitive answer or guide, and if you think there
> >> are errors
> >> or maybe too much generalization at some points, please help fixing or
> >> adding
> >> further details. Also, consider some parts as just me thinking out loud,
> >> what
> >> maybe help other people (or confuse them!).
> >>
> >> The CXL.cache protocol allows devices to be part of the coherency ring

Probably avoid 'ring' in terminology. Just "coherency of the system" is fine
I think.

> >> of the
> >> system.
> >>
> >> Let's start with a Type2 device reading from a specific host memory
> >> address. The
> >> final situation is 64bytes (cache line) from host memory copied to the
> >> device
> >> cache, supposedly for being used by the device/accelerator. If the data
> >> changes,
> >> because some host cpu modifies it, the device will be signalled by the
> >> coherency
> >> ring, so the device will know. The important point here is the device can be
> >> told because the Host knows the device has a copy or the only copy of that
> >> data/memory. And that is thanks to the snoop cache implemented by the

Probably refer to "coherency tracking" rather than say a snoop cache which is
just one way of doing it and don't specify where it is.  Could be in any number
of places depending on system design.

> >> CXL Root
> >> Complex.
> >>
> >> A device caching host memory can be used as well for writes to host memory
> >> through the cache coherency ring. A device can not just read host memory and
> >> keep it, but it can modified it. The implications of writes versus reads
> >> are not
> >> important for the goal of this document. It requires the device to
> >> support more
> >> protocol exchange cases, but regarding the snoop cache, it is irrelevant.
> >>
> >> There arise obvious questions about how this snoop cache is going to work.
> >>
> >> First, with the simple case of just one device caching Host memory. From the
> >> specs, the device CXL.cache should not be enabled by the Host if the device
> >> cache is bigger than the snoop cache. However, what does preclude a
> >> device to do
> >> more memory accesses than what the snoop cache can cover? This can be partly
> >> explained with some allocation control for CXL.cache what is discussed
> >> in the
> >> next section. But a "rogue" device could try things like this, what for
> >> the case
> >> of a single device using the snoop cache and without any other concern about
> >> security, is probably fine:

If you are letting a device into your host coherency and you haven't done
a bunch of stuff to ensure it is not rogue you are on your own.  That stuff is the
domain of technologies such as attestation and more basic stuff like supply chain
management.

Having said that the host can easily identify such a problem and refuse to do
anything that would cause it to loose track + issue appropriate RAS event
(maybe including isolating the device).

> >>
> >>           - With a Type2, the snoop cache will tell the device to release
> >> another
> >>             line, meaning any modified line to be sent back to the Host.
> >>           - Any performance problem will only have an impact on the
> >> device itself.

Agreed the whole sizing thing is a performance question not so much a
correctness - though  a device might not make forwards progress
if it can't get enough data to do what it wants to do.

> >>
> >> Then the case of multiple CXL devices caching Host memory in the same
> >> CXL Root
> >> Complex and therefore same CXL Snoop Cache:
> >>
> >> * How can the snoop cache track reads from different devices without one
> >> device
> >>     monopolizing the full space?
> >>
> >>           - enforcing snoop cache slices by software?
> >>           - allowing specific/limited host ranges by the kernel?

To me, that's a hardware problem.  Hardware that doesn't do the
QoS handling for this is broken. Sure we can quirk that if needed
but I'd do it in the first instance by declaring the hardware so
broken we only support one device doing CXL.cache for each set of
tracking resources.  Seems you say that later :)

> >>  
> > I would like to compare it with the approaches that solves the similar
> > problem of the CPU cache since they might have similar essence.
> >
> > CPU cache suffered from the similar problems that noisy and
> > restless neighborhood keep poking the cache that might cause performance
> > drop. Nowadays, it is solved by the HW mechanism, cache allocation. For
> > Intel, it is called cache allocation technology(CAT) which is a subset of
> > Resource Director Technology(RDT). They can be also used in the
> > virtualization world.

Fine in theory in practice not used all that widely, but agreed similar
solutions could be applied here. They are a pain to tune though so I'd
expect to see better non configurable solutions for QoS first and the
ability to tweak only in a few generations time (could be wrong though!)

> >
> > Before SW gets the support from the HW, many research papers were talking
> > about solving it via page color. E.g. allocate the VM memory with page
> > color awareness for different VMs. But I don't think those ideas eventually
> > land in the mainline.
> >
> > Back to this prob, I think probably SW is going to rely on a HW mechanism
> > to solve this problem nicely and decently, the same as CPU side.  
> 
> 
> I agree with the need of relying on HW, what the following sentence (in 
> the original doc) tried to summarize.
> 
> But we need how this is done by HW for avoiding some undesirable 
> situations if we just blindly configure CXL.cache to those devices 
> advertising it and apparently without no problems regarding the snoop 
> cache size.

I'm not convinced we need to do anything in software.  This is no worse than
head of line blocking on PCIe (bandwidth to host is often
less than that if all devices below some switches want to DMA to 
host memory at the same time).  In theory we can tweak demand by
messing around with device specific stuff, or tweaking link controls
but in practice does anyone do this in a general purpose system?
Don't think so. We rely on sane QoS handling via credit allocations
etc and the switch doing something sensible.

Sure, a particular implementation might not do this, but that to
me is a quirk that we need to handle on a case by case basis.

> 
> 
> >> AFAIK, there is not any kind of hardware control for avoiding this
> >> contention.
> >> Note that with the proper checking by the BIOS and by the kernel (for
> >> hotplug or
> >> those not enabled devices yet during boot time), the size of total
> >> device caches
> >> allowed per CXL Root Complex should not be bigger than the snoop cache
> >> size, and
> >> therefore theoretically no contention at all ... if the devices do the right
> >> thing. From software the only thing we can do is to ensure the CXL.cache
> >> accesses from a device are within a range with same size than the enabled
> >> CXL.cache.
> >>  
> > What would be the consequence if we violate this rule?  
> 
> 
> Contention or just one device getting less snoop cache coverage implying 
> requests from the snoop cache for flushing cached data before trying to 
> access more data in the host.
> 
> With a full snoop cache, a new access to uncached addresses will trigger 
> some action by the snoop cache. I'm assuming it will be the device where 
> that new access comes from the one receiving orders for first flushing 
> cached data for making space, but I may be wrong.

That might be a design choice, but I don't see that being true in general.
Imagine a very simple QoS policy that aims to balance devices use of tracking
entrees.  In that case if only one agent is sending requests for a bit it
will rightly get all the entrees.  New agent wakes up and sends requests, the
eviction will be an entry for the original agent.  Lots of other policies
will lead to that.

> 
> But a rogue device, and the first using CXL.cache, could get most of the 
> snoop cache if no other control existing. Has the snoop cache have this 
> control per device about amount of tracked cachelines allowed? If so, 
> who is configuring it properly? I would expect the kernel CXL core being 
> the one after doing other checks for validation. If the CXL specs do not 
> specify how, we can expect different implementations, and the kernel 
> will need to implement a generic frontend layer with per vendor backends.

It's a host implementation thing. Out of scope for CXL. I'd expect we might
see RDT / MPAM whatever your favourite arch uses for resource controls for
this but even those are often soft limits so if only one agent
doing anything it still gets the whole set of tracking entrees.

As before though, I'm unconcerned about rogue devices as if such a device
is plugged in it can do a lot more damage than this! (your later thing
on lying about translated addresses).

> 
> 
> >> Therefore, some memory allocation API is required for dealing with the
> >> amount of
> >> memory the snoop cache can track, and the host memory a device can
> >> access to.
> >> The device needs the physical address to work with, and it is in this
> >> required
> >> translation from virtual to physical addresses where we can enforce the
> >> restriction. Of course, such an API does already exist, although not
> >> with the
> >> checking we need: the kernel DMA API.
> >>
> >>
> >> (Secure) memory allocation  and CXL.cache
> >> ===========================
> >>
> >> DMAs allow devices to perform read/write operations to system memory
> >> without any
> >> cpu intervention after the (meta)data about how to perform the DMA is
> >> given to
> >> the device. CXL.cache is more than DMA because the system memory caches are
> >> implicitly involved but for the sake of handling this by the operating
> >> system,
> >> not too much different. The important point here is there is no restriction
> >> about the DMAble memory to be used by a device, but due to the snoop cache
> >> limitations, this needs to change for CXL: code aware of the snoop cache
> >> state
> >> and what a device requires needs to be consulted for properly handling the
> >> available space.

I'm not sure how this is connected to DMA mappings etc.

What we map in the page tables is an upper bound on what might be cached by
the device, but it is not the only thing applying that upper bound.
The device is making guarantees not to cache more than a certain amount of memory
at a time - likely a tiny subset of what is mapped. Imagine a DB accelerator.
Those will typically having DMA mappings for many TB of data, but the query engines
will only be using a tiny amount of it at any time. They will evict the cachelines
when they are done with reading particular data (usually as part of a pointer chase).
Hence they will use only a few cachelines, but the page tables and indeed
address translation cache will cover much more.


> >>  
> > As what I replied above, I think we probably need a HW mechanism to solve
> > this problem nicely and decently. (Thinking sharing cache is
> > also a pre-condition of side-channel attack, even here is a snoop state
> > cahce.) With the HW mechanism, allocating the space of snoop state
> > cache might imply a glue layer of snoop cache management for different
> > CXL HB vendors to plug into the CXL core.  
> 
> 
> Just what I did mention above :-)
> 
> 
> Glad to have someone else seeing the problem.

I have no problem with an RDT type scheme but that is about performance
not correctness and is a fine turning on top of telling the devices we
have restricted tracking capacity.

> 
> 
> >
> > So when the CXL driver is initialized, the space of the snoop state cache
> > is allocated. With that is solved, for restricting the device to access the
> > memory (creating/mapping an IOVA for the DMA memory), SW can still leverage
> > the current Linux IOMMU/DMA APIs.
> >  
> 
> This is my main concern. Note the DMA/IOMMU is likely needed for normal 
> device operations, and that will be through CXL.io. Same mapping should 
> then not be shared for CXL.cache, or it can, but with additional per 
> mapping flags and obviously API changes. The implications here are 
> obviously more important if IOMMU is enabled, at least if we take what 
> the specs  say about some ATS/IOMMU mapping only to be allowed by 
> CXL.io. Without IOMMU, it turns into the problem of rogue devices 
> monopolizing the snoop cache.

If the snoop cache implementation is that bad, go annoy the hardware
folk. + Rogue devices are not a thing in production systems, well they
are but constraining them is not a problem for this layer of the stack
(if FPGA or similar then checks for rogue accesses need to belong in the shell
around the bit you allow customers to program - if not cloud vendors
etc will not plug in your devices)


> 
> 
> >> Should we use the kernel DMA API for CXL.cache allocations? This API
> >> deals with
> >> memory coherency what is not needed for the CXL.cache case. However, it is
> >> connected with the IOMMU functionality what is required for CXL.cache if
> >> it is
> >> enabled.
> >>
> >> I think the solution should be to implement a CXL.cache allocation API
> >> inside
> >> the CXL core dealing with the snoop cache available space, and to
> >> connect with
> >> IOMMU kernel code when it is enabled.

Just to check, does anyone care about these devices with iommu disabled?
In my opinion that should just be blocked on day one.

> >>
> >> A security aspect behind DMAs is a device has (usually) no restrictions for
> >> memory access. This is true in a system with no IOMMU hardware, and
> >> CXL.cache
> >> is not different in this case. With IOMMU is a different game though.
> >>
> >> First of all, IOMMU will be in place for CXL.io, what implies legacy TLP
> >> PCIe
> >> packets. A CXL.cache operation can not be handled by the IOMMU hardware
> >> and the
> >> spec states ATS to be used beforehand, that is, the CXL device asking
> >> the IOMMU
> >> hardware about the physical address to work with, and keeping that
> >> translation
> >> internally. The CXL spec specifies ATS service extensions for CXL, and
> >> some ATS
> >> requests can tell the device some addresses only to be used through
> >> CXL.io. This
> >> implies some sort of knowledge about CXL is required by the IOMMU/ATS
> >> hardware
> >> which depends on how the per device tables are programmed by the Host.

I think that is not (only?) about security, it is about correctness as some hosts may
not support CXL.cache accesses to some regions of the host address map, e.g. peer
PCI device BARs for which we are doing UIO or similar.

> >> However,
> >> AFAIK, this is not supported yet by any Linux kernel IOMMU vendor
> >> support. Note
> >> the usual IOMMU device/domain tables will/can be used for normal DMA
> >> transfers,
> >> so IOMMU configuration, both in the Host and by the HW, needs to know
> >> which parts
> >> of the domain are for DMAs and which are for CXL.cache.

Potentially yes.  Though there are bunch of ways to do that which don't
necessarily expose them to the OS, they may be characterstics of the underlying
HPA memory map.

> >>
> >> Assuming this support will be implemented at some point in the future, the
> >> questions are, when?, and, how safe is it?
> >>
> >> Can a device issue CXL.cache operations using arbitrary physical
> >> addresses?

Of course, same as a PCIe device doing DMA with translated addresses.
You rely on attestation and in some cases sanity checks in the host
(they are still subject to the Confidential compute type checks on physical
 address space permissions for example).

This is what all the fun of device security is all about.  You must know
and trust devices before you let them do anything at all to host memory.

> >> It
> >> seems there are some cases where the hardware can take control of PCIe TLP
> >> packets with the ATS bit on. For example, if there is a PCIe bridge in
> >> the path,
> >> and with that bridge using a specific redirection table based on
> >> configured ATS
> >> per device ranges, any TLP with the ATS bit on will be redirected based
> >> on such
> >> a table, and implying no redirection if no table entry. However, that
> >> does not
> >> seem to be in place for PCIe Root Complex implementations. For example, AMD
> >> IOMMU documentation states ATS TLP packets are not handled at all, implying
> >> trusting the device, and if more security is required, the IOMMU
> >> hardware can  
> > Are you referring to the ATS translated request here? I think ATS itself
> > doesn't consider the security in its mind.  
> 
> 
> It seems so, but I guess we agree that is not an option for VMs ... 

Why not?  There is a still quite a bit of infrastructure needed for all the
iommufd work to land, but it's getting close.  That absolutely allows
for ATS - up to the host to check it trusts the device before assigning
any part of it to the VM.  On at least some architectures the translation
is the full 2 stage one to host physical address.

> Without IOMMU you can not have DMAs from passthrough devices, and if 
> CXL.cache dodges the IOMMU checks (and none other security mechanism in 
> place), CXL.cache should not be allowed in virtualization.

No difference to ATS for conventional PCIe. Sure you need to do your security
checks. That's standard stuff. See what Lukas has been working on for last few
years around CMA + the bits of relevant confidential compute that are actually
useful for non CoCo usecases.

> 
> 
> >> check those TLP ATS packets as well, spoiling the ATS advantage. Note  
> > Yes, AMD IOMMU has the secure ATS support, but as you said, it is pretty
> > straight-forward, basically just check every translated request when
> > enabled.

Interesting approach. I'd not noticed that before.

There are other solutions that keep enough tracking data in host to verify
translated requests are fine but if you've checked the device, none of that
should be needed.

> >  
> >> this is
> >> PCIe, so CXL.io will likely keep the functionality, but CXL.cache operations
> >> follow another path with apparently no further control to enforce the right
> >> addresses within the allowed memory ranges per device are used.
> >>
> >> Because this apparently lack of security for IOMMU and CXL.cache, this
> >> implies a
> >> CXL device should not be used by VMs or any other user space controlled
> >> driver
> >> with CXL.cache being enabled. This seems a really serious limitation, so
> >> maybe
> >> I'm missing something here.
> >>  
> > I think at least for CXL path, IOMMU should have the similar mechanism like
> > secure ATS, and let the user to choose if they want it to be enabled or
> > not.
> >
> > In reality, many CSP design the HW by themselves and trust their HW won't
> > do messy things, they may want to enable it only on the 3rd party HW.

Exactly.  They do a lot of paperwork and security audits.  These devices have
to be in the trusted computing base to operate.

> >
> > For confidential computing world, secure ATS is mandatory, and performance
> > drop is the price of security.  

That's a choice, but it's not the only one for confidential compute
and from my recollection of other CoCo solutions they don't all do
this.

> 
> 
> We can let the user to choose ... but in the virtualization world the 
> provider does not want the user to choose ... and if CXL.cache is like 
> DMAs without IOMMU, I would say this is a really good reason.

Just require appropriate attestation. That's a userspace policy (see
Lukas' series) + the one Dan has for confidential compute.

> 
> 
> >> Regarding virtualization, assuming the security problems do not exist or
> >> will be
> >> solved, while CXL.mem can be supported with an ahead mapping by the
> >> Host, with
> >> CXL.cache this needs to be handled when the related driver asks for specific
> >> memory to access, and then to configure the IOMMU/ATS tables by the
> >> Host. This
> >> implies the emulation needs a backend, what an ahead mapping, as currently
> >> proposed for CXL.mem can avoid.
> >>
> >> Finally, if my concerns about the security of CXL.cache with IOMMU are
> >> unfounded, at least this document should describe how is this solved and the
> >> security enforced by the hardware, and if the kernel requires to handle it
> >> specifically (what I really think is the case, at least with IOMMU changes
> >> managed by the CXL core).

It's an implementation specific question. If you are selling a device that
uses cxl.cache, expect to do a lot of paperwork and security audits + probably
reveal a lot of implementation details to you major customers.

We should expose policy control to host userspace and provide it all the relevant
information.  I'm not yet convinced this has anything to do with DMA mapping or
the IOMMU core code.  There might be some advantages in locking down accesses
via CXL.io vs CXL.cache for some IOMMU designs but my understanding is that is
not a universal thing at all.  If userspace says accept the device after doing
all the certs etc are checked then it is part of the trusted compute base and
there will not need to be anything in the IOMMU page tables that is unique
to this.

> >>
> >>
> >> Summary
> >> ======
> >>
> >>
> >> Next the proposed tasks to perform for supporting CXL.cache:
> >>
> >>           - CXL core handling per device CXL.cache enabling based on CXL Root
> >>             Complex snoop cache state.

Agreed.

> >>
> >>           - CXL core implementing a CXL.cache host memory allocation
> >> restricting
> >>             the physical memory a a device can access to through CXL.cache.

No on this one. It's a broken solution to a potential hardware solution.

> >>
> >>           - IOMMU being CXL aware and dealing with CXL.cache vs CXL.io
> >> requests.

Not required in general. May be required for some host IOMMU architectures.
So we may need some hooks.

> >>
> >>           - Clarify CXL.cache and security with IOMMU.


Standard device security flows.  Policy for what counts as 'secure' may
be tighter, but it's no different to flows for PCIe devices in general.
Bring them into your TCB.

Jonathan

> >>
> >>
> >>  
> 
>