RE: [PATCH RFC] iommu: Enable per-device SSID space for SVA

RE: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Friday, April 24, 2026 9:40 PM
> 
> On Fri, Apr 24, 2026 at 08:53:39AM +0000, Joonwon Kang wrote:
> > For SVA, the IOMMU core always allocates PASID from the global PASID
> > space. The use of this global PASID space comes from the limitation of
> > the ENQCMD instruction in Intel CPUs that it fetches its PASID operand
> > from IA32_PASID, which is per-task.
> 
> That's right, and all the iommu drivers should have no issue with
> per-device pasid or they are not following the API contract..  I
> believe that has been taking care of already.
> 
> So, I don't think this is an iommu driver capability.
> 
> Instead, you have to decide if the PASID is per device or not based on
> if the system will use ENQCMD or any similar instruction.  I
> understand ARM has introduced a similar instruction.
> 
> So you may be better off with some kind of 'arch has enqcmd like
> instruction' to control this instead of involving the iommu driver.
> 

if both arch and device support enqcmd-like insn...

I recalled this was discussed years ago. For devices like this, just
let driver manage its own pasid space then have a new interface
e.g. iommu_sva_bind_device_pasid(dev, mm, pasid) to use the
specified pasid.

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Joonwon Kang]

Hi Jason, thank you for your review and sorry for the late reply.

> On Fri, Apr 24, 2026 at 08:53:39AM +0000, Joonwon Kang wrote:
> > For SVA, the IOMMU core always allocates PASID from the global PASID
> > space. The use of this global PASID space comes from the limitation of
> > the ENQCMD instruction in Intel CPUs that it fetches its PASID operand
> > from IA32_PASID, which is per-task.
> 
> That's right, and all the iommu drivers should have no issue with
> per-device pasid or they are not following the API contract..  I
> believe that has been taking care of already.
> 

Thanks for this info that every IOMMU should support per-device PASID
space already, i.e. each device behind the IOMMU can have its own PASID
space.

Let me clarify my understanding first to prevent future confusion.

The reason of using the global PASID space in the first place, i.e.
`iommu_global_pasid_ida`, is to support the case where a userspace driver
wants to communicate with multiple devices using the ENQCMD instruction
without kernel's intervention. Since the ENQCMD instruction fetches PASID
from the per-process IA32_PASID, the userspace driver could not use a
different PASID for each device. If a syscall had been provided to change
the process' current PASID, however, we might have been able to get rid of
the use of the global PASID space, although it may cause other issues and
require research on feasibility and effectiveness.

Please let me know if there is any other reason of the global PASID space
use that the team considered back then.

> So, I don't think this is an iommu driver capability.
> 
> Instead, you have to decide if the PASID is per device or not based on
> if the system will use ENQCMD or any similar instruction.  I
> understand ARM has introduced a similar instruction.
> 

By "similar instruction" on ARM, I guess you mean ST64BV0, which fetches
the bottom 32 bits data from ACCDATA_EL1. Please let me know if you meant
others as it will matter. If ST64BV0 is supported on ARM, however, it
would mean that ST64B and ST64BV are also supported already according to
the ID_AA64ISAR1_EL1's LS64 field. The latter 2 instructions are just to
atomically store whatever user wants to a memory location without
referring to ACCDATA_EL1 and all the 3 instructions can be run at EL0. So,
the userspace driver would have enough capability to designate arbitrary
PASID as it wants via the latter 2 instructions when communicating with
multiple devices.

> So you may be better off with some kind of 'arch has enqcmd like
> instruction' to control this instead of involving the iommu driver.
> 

If the above makes sense, I guess we could lift the use of the global
PASID space on ARM unconditionally. What do you think?

> > - The device is not a PCIe device.
> > - The device is to use SVA.
> > - The supported SSID/PASID space is very small for the device; only 1 to
> >   3 SSIDs are supported.
> 
> Yuk
> 
> > With this setup, when other modules have allocated all the PASIDs that
> > our device is expected to use from the global PASID space via APIs like
> > iommu_alloc_global_pasid() or iommu_sva_bind_device(), SVA binding to
> > our device fails due to the lack of available PASIDs.
> 
> So you have multiple SVA using devices as well? Or multiple instances
> of the same device?

We have multiple processes and a single device, those processes want to
do SVA with the same device, and only one process will do SVA with the
device at a time. Though, the problem occurs even when irrelevant
processes allocate the PASIDs from the global PASID space for their own
irrelevant purposes.

Thanks,
Joonwon Kang

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Jason Gunthorpe]

On Thu, May 07, 2026 at 08:15:21AM +0000, Tian, Kevin wrote:
> if both arch and device support enqcmd-like insn...
> 
> I recalled this was discussed years ago. For devices like this, just
> let driver manage its own pasid space then have a new interface
> e.g. iommu_sva_bind_device_pasid(dev, mm, pasid) to use the
> specified pasid.

Yeah, that makes sense. If the driver knows it doesn't use an ENQCMD
like programming model at all then it can use this API and it should
also avoid programming the MSRs/etc.

Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Jason Gunthorpe]

On Thu, May 07, 2026 at 09:58:51AM +0000, Joonwon Kang wrote:

> By "similar instruction" on ARM, I guess you mean ST64BV0, which fetches
> the bottom 32 bits data from ACCDATA_EL1. Please let me know if you meant
> others as it will matter. If ST64BV0 is supported on ARM, however, it
> would mean that ST64B and ST64BV are also supported already according to
> the ID_AA64ISAR1_EL1's LS64 field. The latter 2 instructions are just to
> atomically store whatever user wants to a memory location without
> referring to ACCDATA_EL1 and all the 3 instructions can be run at EL0. So,
> the userspace driver would have enough capability to designate arbitrary
> PASID as it wants via the latter 2 instructions when communicating with
> multiple devices.

IDK exactly what ARM did. IIRC on Intel ENQCMD forms a special
non-posted write TLP and the device can tell the TLP came from ENQCMD
and so it trusts the encoded PASID. ARM has to have done the same
thing - allowing anyone to forge the PASID by using a different
instruction misses the point of the Intel design.

Honestly, I'm not sure why they even implemented it. SMMUv3 can't do
the translation scheme required to use ENQCMD from a VM anyhow, so it
is pretty useless.

> We have multiple processes and a single device, those processes want to
> do SVA with the same device, and only one process will do SVA with the
> device at a time. Though, the problem occurs even when irrelevant
> processes allocate the PASIDs from the global PASID space for their own
> irrelevant purposes.

The only way to allocate a PASID from the global PASID space is to
establish another SVA, so you have multiple devices doing SVA?

Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Robin Murphy]

On 2026-05-09 6:10 pm, Jason Gunthorpe wrote:
> On Thu, May 07, 2026 at 09:58:51AM +0000, Joonwon Kang wrote:
> 
>> By "similar instruction" on ARM, I guess you mean ST64BV0, which fetches
>> the bottom 32 bits data from ACCDATA_EL1. Please let me know if you meant
>> others as it will matter. If ST64BV0 is supported on ARM, however, it
>> would mean that ST64B and ST64BV are also supported already according to
>> the ID_AA64ISAR1_EL1's LS64 field. The latter 2 instructions are just to
>> atomically store whatever user wants to a memory location without
>> referring to ACCDATA_EL1 and all the 3 instructions can be run at EL0. So,
>> the userspace driver would have enough capability to designate arbitrary
>> PASID as it wants via the latter 2 instructions when communicating with
>> multiple devices.
> 
> IDK exactly what ARM did. IIRC on Intel ENQCMD forms a special
> non-posted write TLP and the device can tell the TLP came from ENQCMD
> and so it trusts the encoded PASID. ARM has to have done the same
> thing - allowing anyone to forge the PASID by using a different
> instruction misses the point of the Intel design.

Yes, ACCDATA_EL1 is a privileged register neither writeable nor readable 
by userspace[1], so it should be functionally equivalent from an SVA 
point of view.

> Honestly, I'm not sure why they even implemented it. SMMUv3 can't do
> the translation scheme required to use ENQCMD from a VM anyhow, so it
> is pretty useless.

Not sure what you mean there - indeed you can't do the SIOV thing of 
assigning individual ADIs to _different_ VMs, but there's still no 
reason you couldn't give the whole accelerator device to one VM, and run 
the "full" kernel driver in that VM to hand out ADIs to processes, same 
as for non-virtualised ST64BV0/ENQCMD usage. It's entirely usable, just 
not so "scalable".

Thanks,
Robin.

[1] 
https://developer.arm.com/documentation/ddi0601/2026-03/AArch64-Registers/ACCDATA-EL1--Accelerator-Data

> 
>> We have multiple processes and a single device, those processes want to
>> do SVA with the same device, and only one process will do SVA with the
>> device at a time. Though, the problem occurs even when irrelevant
>> processes allocate the PASIDs from the global PASID space for their own
>> irrelevant purposes.
> 
> The only way to allocate a PASID from the global PASID space is to
> establish another SVA, so you have multiple devices doing SVA?
> 
> Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Jason Gunthorpe]

On Mon, May 11, 2026 at 01:39:06PM +0100, Robin Murphy wrote:
> On 2026-05-09 6:10 pm, Jason Gunthorpe wrote:
> > On Thu, May 07, 2026 at 09:58:51AM +0000, Joonwon Kang wrote:
> > 
> > > By "similar instruction" on ARM, I guess you mean ST64BV0, which fetches
> > > the bottom 32 bits data from ACCDATA_EL1. Please let me know if you meant
> > > others as it will matter. If ST64BV0 is supported on ARM, however, it
> > > would mean that ST64B and ST64BV are also supported already according to
> > > the ID_AA64ISAR1_EL1's LS64 field. The latter 2 instructions are just to
> > > atomically store whatever user wants to a memory location without
> > > referring to ACCDATA_EL1 and all the 3 instructions can be run at EL0. So,
> > > the userspace driver would have enough capability to designate arbitrary
> > > PASID as it wants via the latter 2 instructions when communicating with
> > > multiple devices.
> > 
> > IDK exactly what ARM did. IIRC on Intel ENQCMD forms a special
> > non-posted write TLP and the device can tell the TLP came from ENQCMD
> > and so it trusts the encoded PASID. ARM has to have done the same
> > thing - allowing anyone to forge the PASID by using a different
> > instruction misses the point of the Intel design.
> 
> Yes, ACCDATA_EL1 is a privileged register neither writeable nor readable by
> userspace[1], so it should be functionally equivalent from an SVA point of
> view.

There is a bit more going on though, I think that is what Joonwon is
mentioning by asking about ST64B and ST64BV. I *think* the answer is:

- ST64B uses a posted write
- ST64BV can be restricted so EL0 cannot execute it, it uses a
  non-posted write (AI tells me via EnASR)
- ST64BV0 can be used by EL0, always uses a non-posted write, and always
  uses ACCDATA_EL1

Which is similar to Intel. The device only processes the PASID from a
non-posted write, and the CPU prevents userspace from forming
non-posted writes except through ST64BV0

> > Honestly, I'm not sure why they even implemented it. SMMUv3 can't do
> > the translation scheme required to use ENQCMD from a VM anyhow, so it
> > is pretty useless.
> 
> Not sure what you mean there - indeed you can't do the SIOV thing of
> assigning individual ADIs to _different_ VMs, but there's still no reason
> you couldn't give the whole accelerator device to one VM, and run the "full"
> kernel driver in that VM to hand out ADIs to processes, same as for
> non-virtualised ST64BV0/ENQCMD usage. It's entirely usable, just not so
> "scalable".

Well yes, technically, but I'm not sure this is attractive in
practice.

The value of ENQCMD on Intel was it can eliminate any HW side
per-context state for simple HW like DMA engines, including for
virtualization.

You pay for that value with some performance loss, but it can be
attractive because of the universal scalability.

However complex devices don't seem to want to use it, once you have to
have per-context state for any other reason the performance downsides
of ENQCMD make it unappealing.

So, IDK, maybe some embedded on-chip device will find a way to make
good use of it, but also I'm not aware of any adoption on x86..

Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Joonwon Kang]

Hi Jason and Robin, thanks a lot for sharing your insights! Could you help
to answer the further questions below? or just let me know if it is better
to use other channels for them like ARM support.

> On Mon, May 11, 2026 at 01:39:06PM +0100, Robin Murphy wrote:
> > On 2026-05-09 6:10 pm, Jason Gunthorpe wrote:
> > > On Thu, May 07, 2026 at 09:58:51AM +0000, Joonwon Kang wrote:
> > > 
> > > > By "similar instruction" on ARM, I guess you mean ST64BV0, which fetches
> > > > the bottom 32 bits data from ACCDATA_EL1. Please let me know if you meant
> > > > others as it will matter. If ST64BV0 is supported on ARM, however, it
> > > > would mean that ST64B and ST64BV are also supported already according to
> > > > the ID_AA64ISAR1_EL1's LS64 field. The latter 2 instructions are just to
> > > > atomically store whatever user wants to a memory location without
> > > > referring to ACCDATA_EL1 and all the 3 instructions can be run at EL0. So,
> > > > the userspace driver would have enough capability to designate arbitrary
> > > > PASID as it wants via the latter 2 instructions when communicating with
> > > > multiple devices.
> > > 
> > > IDK exactly what ARM did. IIRC on Intel ENQCMD forms a special
> > > non-posted write TLP and the device can tell the TLP came from ENQCMD
> > > and so it trusts the encoded PASID. ARM has to have done the same
> > > thing - allowing anyone to forge the PASID by using a different
> > > instruction misses the point of the Intel design.
> > 
> > Yes, ACCDATA_EL1 is a privileged register neither writeable nor readable by
> > userspace[1], so it should be functionally equivalent from an SVA point of
> > view.
> 
> There is a bit more going on though, I think that is what Joonwon is
> mentioning by asking about ST64B and ST64BV. I *think* the answer is:
> 
> - ST64B uses a posted write
> - ST64BV can be restricted so EL0 cannot execute it, it uses a
>   non-posted write (AI tells me via EnASR)
> - ST64BV0 can be used by EL0, always uses a non-posted write, and always
>   uses ACCDATA_EL1
> 
> Which is similar to Intel.

Ah, I missed that ST64BV is currently being trapped to EL1 while ST64B is
not [1]. However, I am not sure if the trap is to disallow EL0 to use it.
Can it be instead to pass the response value of the non-posted write to
EL0 while using the EL0-given PASID as-is? If so, I believe EL0 still can
specify arbitrary PASID as it wants via ST64BV.

Since I guess ST64B* instructions are to serve generic purposes not only
for communication with accelerators with SIOV but also with any memory
location or device without SIOV, I am not sure if it is always okay to
make those instructions work the way Jason mentioned.

> The device only processes the PASID from a non-posted write,
> 

Regarding ST64B, are the ARM devices behind ARM SMMU v3 supposed to work
this way too? If not, EL0 can specify arbitrary PASID via ST64B with the
kernel today [1].

[1] https://github.com/torvalds/linux/blob/50897c955902c93ae71c38698abb910525ebdc89/arch/arm64/kernel/cpufeature.c#L3166-L3181

Thanks,
Joonwon Kang

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Joonwon Kang]

> > We have multiple processes and a single device, those processes want to
> > do SVA with the same device, and only one process will do SVA with the
> > device at a time. Though, the problem occurs even when irrelevant
> > processes allocate the PASIDs from the global PASID space for their own
> > irrelevant purposes.
> 
> The only way to allocate a PASID from the global PASID space is to
> establish another SVA, so you have multiple devices doing SVA?

Another way is via iommu_alloc_global_pasid() [1] and it does not require
SVA. No, we currently do not have multiple devices doing SVA. But, the
problem is that it is out of our control when any device is to allocate a
PASID or do SVA for its own purpose.

[1] https://lore.kernel.org/all/20230802212427.1497170-3-jacob.jun.pan@linux.intel.com/

Thanks,
Joonwon Kang

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Jason Gunthorpe]

On Tue, May 12, 2026 at 09:57:14AM +0000, Joonwon Kang wrote:
> > There is a bit more going on though, I think that is what Joonwon is
> > mentioning by asking about ST64B and ST64BV. I *think* the answer is:
> > 
> > - ST64B uses a posted write
> > - ST64BV can be restricted so EL0 cannot execute it, it uses a
> >   non-posted write (AI tells me via EnASR)
> > - ST64BV0 can be used by EL0, always uses a non-posted write, and always
> >   uses ACCDATA_EL1
> > 
> > Which is similar to Intel.
> 
> Ah, I missed that ST64BV is currently being trapped to EL1 while ST64B is
> not [1]. However, I am not sure if the trap is to disallow EL0 to use it.
> Can it be instead to pass the response value of the non-posted write to
> EL0 while using the EL0-given PASID as-is? If so, I believe EL0 still can
> specify arbitrary PASID as it wants via ST64BV.

I think if an OS implements things this way it is would security
broken as far as ENQCMD compatible HW goes.

> Since I guess ST64B* instructions are to serve generic purposes not only
> for communication with accelerators with SIOV but also with any memory
> location or device without SIOV, I am not sure if it is always okay to
> make those instructions work the way Jason mentioned.

The end point has to use the posted vs non-posted write distinction
for security.

> > The device only processes the PASID from a non-posted write,
> 
> Regarding ST64B, are the ARM devices behind ARM SMMU v3 supposed to work
> this way too? If not, EL0 can specify arbitrary PASID via ST64B with the
> kernel today [1].

If you want ENQCMD compatible semantics then yes you have to do all of
these things, it is part of the security design.

Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Robin Murphy]

On 12/05/2026 1:40 pm, Jason Gunthorpe wrote:
> On Tue, May 12, 2026 at 09:57:14AM +0000, Joonwon Kang wrote:
>>> There is a bit more going on though, I think that is what Joonwon is
>>> mentioning by asking about ST64B and ST64BV. I *think* the answer is:
>>>
>>> - ST64B uses a posted write
>>> - ST64BV can be restricted so EL0 cannot execute it, it uses a
>>>    non-posted write (AI tells me via EnASR)
>>> - ST64BV0 can be used by EL0, always uses a non-posted write, and always
>>>    uses ACCDATA_EL1
>>>
>>> Which is similar to Intel.
>>
>> Ah, I missed that ST64BV is currently being trapped to EL1 while ST64B is
>> not [1]. However, I am not sure if the trap is to disallow EL0 to use it.
>> Can it be instead to pass the response value of the non-posted write to
>> EL0 while using the EL0-given PASID as-is? If so, I believe EL0 still can
>> specify arbitrary PASID as it wants via ST64BV.
> 
> I think if an OS implements things this way it is would security
> broken as far as ENQCMD compatible HW goes.

Yes, I think it's rather the point that the EnALS/EnASR traps to EL1 
allow EL1 to sanitise the data that ST64B/ST64BV are sending, and do 
exactly things like substituting a valid PASID. ST64BV0 offers a way of 
doing so _without_ needing the overhead of trapping, but conversely that 
needs the EnAS0 opt-in all the way down to indicate both EL1's awareness 
of programming ACCDATA_EL1 appropriately and EL2/3's awareness of 
context-switching it.

I've not looked closely at what exactly the arm64 arch code is doing 
today and how well it actually fits the expected ENQCMD usage model, but 
I can well believe it might need a bit of tweaking.

Thanks,
Robin.

>> Since I guess ST64B* instructions are to serve generic purposes not only
>> for communication with accelerators with SIOV but also with any memory
>> location or device without SIOV, I am not sure if it is always okay to
>> make those instructions work the way Jason mentioned.
> 
> The end point has to use the posted vs non-posted write distinction
> for security.
> 
>>> The device only processes the PASID from a non-posted write,
>>
>> Regarding ST64B, are the ARM devices behind ARM SMMU v3 supposed to work
>> this way too? If not, EL0 can specify arbitrary PASID via ST64B with the
>> kernel today [1].
> 
> If you want ENQCMD compatible semantics then yes you have to do all of
> these things, it is part of the security design.
> 
> Jason

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Joonwon Kang]

> On 12/05/2026 1:40 pm, Jason Gunthorpe wrote:
> > On Tue, May 12, 2026 at 09:57:14AM +0000, Joonwon Kang wrote:
> >>> There is a bit more going on though, I think that is what Joonwon is
> >>> mentioning by asking about ST64B and ST64BV. I *think* the answer is:
> >>>
> >>> - ST64B uses a posted write
> >>> - ST64BV can be restricted so EL0 cannot execute it, it uses a
> >>>    non-posted write (AI tells me via EnASR)
> >>> - ST64BV0 can be used by EL0, always uses a non-posted write, and always
> >>>    uses ACCDATA_EL1
> >>>
> >>> Which is similar to Intel.
> >>
> >> Ah, I missed that ST64BV is currently being trapped to EL1 while ST64B is
> >> not [1]. However, I am not sure if the trap is to disallow EL0 to use it.
> >> Can it be instead to pass the response value of the non-posted write to
> >> EL0 while using the EL0-given PASID as-is? If so, I believe EL0 still can
> >> specify arbitrary PASID as it wants via ST64BV.
> > 
> > I think if an OS implements things this way it is would security
> > broken as far as ENQCMD compatible HW goes.
> 
> Yes, I think it's rather the point that the EnALS/EnASR traps to EL1 
> allow EL1 to sanitise the data that ST64B/ST64BV are sending, and do 
> exactly things like substituting a valid PASID. ST64BV0 offers a way of 
> doing so _without_ needing the overhead of trapping, but conversely that 
> needs the EnAS0 opt-in all the way down to indicate both EL1's awareness 
> of programming ACCDATA_EL1 appropriately and EL2/3's awareness of 
> context-switching it.
> 
> I've not looked closely at what exactly the arm64 arch code is doing 
> today and how well it actually fits the expected ENQCMD usage model, but 
> I can well believe it might need a bit of tweaking.
> 
> Thanks,
> Robin.
> 
> >> Since I guess ST64B* instructions are to serve generic purposes not only
> >> for communication with accelerators with SIOV but also with any memory
> >> location or device without SIOV, I am not sure if it is always okay to
> >> make those instructions work the way Jason mentioned.
> > 
> > The end point has to use the posted vs non-posted write distinction
> > for security.
> > 
> >>> The device only processes the PASID from a non-posted write,
> >>
> >> Regarding ST64B, are the ARM devices behind ARM SMMU v3 supposed to work
> >> this way too? If not, EL0 can specify arbitrary PASID via ST64B with the
> >> kernel today [1].
> > 
> > If you want ENQCMD compatible semantics then yes you have to do all of
> > these things, it is part of the security design.
> > 
> > Jason

Appreciate all your clarifications here. So, my understanding is that if
our system does not support ST64BV and ST64BV0 or if our device does not
distinguish between the posted write and the non-posted write regarding
PASID, then we can lift the use of the global PASID space. Can I say this?

If yes, I will create a new patch which checks for that case. It might as
well add a new device tree property to represent if the device supports
the posted write vs. non-posted write distinction.

Thanks,
Joonwon Kang

Re: [PATCH RFC] iommu: Enable per-device SSID space for SVA

[Jason Gunthorpe]

On Tue, May 12, 2026 at 02:51:38PM +0000, Joonwon Kang wrote:

> Appreciate all your clarifications here. So, my understanding is that if
> our system does not support ST64BV and ST64BV0 or if our device does not
> distinguish between the posted write and the non-posted write regarding
> PASID, then we can lift the use of the global PASID space. Can I say this?

You should do what Robin said - just have your driver use a per-device
PASID that it allocates and never use the global pasid allocator.

To do this lightly re-organize the SVA code so the driver can supply
its own PASID, and in this mode we wouldn't activate the ENQCMD
features in the mm.

Jason