Device specific pass through in host systems - discuss user interface

Device specific pass through in host systems - discuss user interface

[Prakhya, Sai Praneeth]

[-- Attachment #1.1: Type: text/plain, Size: 1852 bytes --]

Hi All,

I am working on an IOMMU driver feature that allows a user to specify if the DMA from a device should be translated by IOMMU or not. Presently, we support only all devices or none mode i.e. if user specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through kernel command line, all the devices would be in pass through mode and we don't have per device granularity, but, we were requested by a customer to selectively put devices in pass through mode and not all.

Since, this feature could be generic across architectures, we thought it would be better if the user interface is discussed in the community first. We are envisioning this to be used both during boot time and runtime and hence having a kernel command line argument along with a sysfs entry are needed. So, please pour in your suggestions on how the user interface should look like to make it architecture agnostic.


1.      Have a kernel command line argument that takes a list of BDF's as an input and puts them in pass through mode

a.      Accepting BDF as an input has a downside - BDF is dynamic and could change if BIOS/OS enumerates a new device in next reboot

b.      Accepting <vendor_id:device_id> pair as an input has a downside - What to do when there are multiple such devices and user would like to put only some of them in PT mode

2.      Have a sysfs file which takes 1 or 0 as an input to enable/disable pass through mode. Some places that seem to be reasonable are

a.      /sys/class/iommu/dmar0/devices/

b.      /sys/kernel/iommu_groups/<id>/devices

I am looking for a consensus on *how the kernel command line argument should look like and path for sysfs entry*. Also, please note that if a device is put in pass through mode it won't be available for the guest and that's ok.

Regards,
Sai

PS: Idea credits: Ashok Raj

[-- Attachment #1.2: Type: text/html, Size: 7404 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Robin Murphy]

On 07/06/2019 03:24, Prakhya, Sai Praneeth wrote:
> Hi All,
> 
> I am working on an IOMMU driver feature that allows a user to specify if the DMA from a device should be translated by IOMMU or not. Presently, we support only all devices or none mode i.e. if user specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through kernel command line, all the devices would be in pass through mode and we don't have per device granularity, but, we were requested by a customer to selectively put devices in pass through mode and not all.

It's interesting to see this from a fresh angle which isn't clouded by 
other SoC GPU details - thanks for the proposal! A couple more thoughts 
jump out immediately...

> Since, this feature could be generic across architectures, we thought it would be better if the user interface is discussed in the community first. We are envisioning this to be used both during boot time and runtime and hence having a kernel command line argument along with a sysfs entry are needed. So, please pour in your suggestions on how the user interface should look like to make it architecture agnostic.
> 
> 
> 1.      Have a kernel command line argument that takes a list of BDF's as an input and puts them in pass through mode
> 
> a.      Accepting BDF as an input has a downside - BDF is dynamic and could change if BIOS/OS enumerates a new device in next reboot
> 
> b.      Accepting <vendor_id:device_id> pair as an input has a downside - What to do when there are multiple such devices and user would like to put only some of them in PT mode
> 

c. Not all devices are PCI in the first place.

> 2.      Have a sysfs file which takes 1 or 0 as an input to enable/disable pass through mode. Some places that seem to be reasonable are
> 
> a.      /sys/class/iommu/dmar0/devices/
> 
> b.      /sys/kernel/iommu_groups/<id>/devices

Note that this this works out a bit tricky to actually use, since 
changing the meaning of DMA addresses under the device's feet would be 
disastrous. It can only safely take effect by unbinding and rebinding 
the driver and/or resetting the device (as a side note, this starts to 
overlap with the IOMMU-drivers-as-modules concept, and whatever solution 
to this we end up with may be helpful in making that work too). In most 
cases that's probably viable, but not every driver supports unbinding, 
and either way what if the device in question is the one hosting the 
root filesystem... :/

Robin.

> 
> I am looking for a consensus on *how the kernel command line argument should look like and path for sysfs entry*. Also, please note that if a device is put in pass through mode it won't be available for the guest and that's ok.
> 
> Regards,
> Sai
> 
> PS: Idea credits: Ashok Raj
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Sai Praneeth Prakhya]

> It's interesting to see this from a fresh angle which isn't clouded by 
> other SoC GPU details - thanks for the proposal! A couple more thoughts 
> jump out immediately...
> 

Thanks a lot! for jumping in and providing your valuable insights :)
Sorry! for taking time to reply on this. Did some quick testing on
unbinding/binding stuff.

> > Since, this feature could be generic across architectures, we thought it
> > would be better if the user interface is discussed in the community first.
> > We are envisioning this to be used both during boot time and runtime and
> > hence having a kernel command line argument along with a sysfs entry are
> > needed. So, please pour in your suggestions on how the user interface
> > should look like to make it architecture agnostic.
> > 
> > 
> > 1.      Have a kernel command line argument that takes a list of BDF's as
> > an input and puts them in pass through mode
> > 
> > a.      Accepting BDF as an input has a downside - BDF is dynamic and
> > could change if BIOS/OS enumerates a new device in next reboot
> > 
> > b.      Accepting <vendor_id:device_id> pair as an input has a downside -
> > What to do when there are multiple such devices and user would like to put
> > only some of them in PT mode
> > 
> 
> c. Not all devices are PCI in the first place.

Agreed, maybe we could limit this feature only to PCIe devices :(

> 
> > 2.      Have a sysfs file which takes 1 or 0 as an input to enable/disable
> > pass through mode. Some places that seem to be reasonable are
> > 
> > a.      /sys/class/iommu/dmar0/devices/
> > 
> > b.      /sys/kernel/iommu_groups/<id>/devices
> 
> Note that this this works out a bit tricky to actually use, since 
> changing the meaning of DMA addresses under the device's feet would be 
> disastrous.

Yes, that makes sense.

> It can only safely take effect by unbinding and rebinding 
> the driver and/or resetting the device (as a side note, this starts to 
> overlap with the IOMMU-drivers-as-modules concept, and whatever solution 
> to this we end up with may be helpful in making that work too). In most 
> cases that's probably viable, but not every driver supports unbinding, 
> and either way what if the device in question is the one hosting the 
> root filesystem... :/

We would like to propose something like this

1. User will first unbind the device driver by
echo "BDF" > /sys/bus/pci/drivers/<device_driver>/unbind

2. Set the Pass Through bit (this is to say the intent of the user that upon
the next rebind, please make this device as pass through)
echo 1 > /sys/class/iommu/dmar0/devices/<BDF>/pt

3. Re-bind the driver again
echo "BDF" > /sys/bus/pci/drivers/<device_driver>/bind

So, if the driver doesn't support unbind then the user cannot put the device
in pass through mode, in which case he has to use kernel command line
argument.

I did unbind on my device that has root file system and after that I was
unable to run any command (like cat or echo), which made sense. So, it's upto
the user to decide which devices can go through unbind and hence can be put in
pass through mode. Since this feature requires sudo privileges, I think, we
can assume admin does the right thing.

Please let me know what you think about this and also please do suggest if you
have any other better ways to deal with this.

Regards,
Sai

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[hch]

Just curious, what exactly is the use case?  Explaining how someone
would wan to use this should drive the way we design an interface for it.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Sai Praneeth Prakhya]

On Sat, 2019-06-08 at 09:27 +0200, hch@lst.de wrote:
> Just curious, what exactly is the use case?  Explaining how someone
> would wan to use this should drive the way we design an interface for it.

Makes sense.

Some example use cases:
1. Assume an SR-IOV device and if the admin decides to use some VF's natively
(i.e. they are available only to host) and he wants them to be high performing
and also believes that his OS is secure enough (so decides to by pass IOMMU).

Presently, we don't support this use case because "iommu=pt" kernel command
line argument is an all or none feature i.e. either all BDF's are translated
through IOMMU or none. So, we would like to propose a per-BDF on/off feature.

We would also want it to be run-time (i.e. sysfs based interface) rather than
just boot-time interface (kernel command line argument).

Also, want to be clear that it's not just SR-IOV devices that we are looking
at but could be any unrelated PCIe devices i.e. a SATA device, USB, NIC or
GFx. Wherein the admin wants to selectively put NIC and GFx through IOMMU and
does not want transactions by SATA and USB to be translated by IOMMU.

Regards,
Sai

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Lu, Baolu]

Hi Sai,


On 6/7/19 10:24 AM, Prakhya, Sai Praneeth wrote:
> Hi All,
> 
> I am working on an IOMMU driver feature that allows a user to specify if 
> the DMA from a device should be translated by IOMMU or not. Presently, 
> we support only all devices or none mode i.e. if user specifies 
> “iommu=pt” [X86] or “iommu.passthrough” [ARM64] through kernel command 
> line, all the devices would be in pass through mode and we don’t have 
> per device granularity, but, we were requested by a customer to 
> selectively put devices in pass through mode and not all.

Most iommu vendor drivers have switched from per-device to per-group
domain (a.k.a. default domain). So per-group pass-through mode makes
more sense?

By the way, can we extend this to "per-group default domain type",
instead of only "per-group pass-through mode"? Currently we have system
level default domain type, if we have finer granularity of default
domain type, both iommu drivers and end users will benefit from it.

> 
> Since, this feature could be generic across architectures, we thought it 
> would be better if the user interface is discussed in the community 
> first. We are envisioning this to be used both during boot time and 
> runtime and hence having a kernel command line argument along with a 
> sysfs entry are needed. So, please pour in your suggestions on how the 
> user interface should look like to make it architecture agnostic.
> 
> 1.Have a kernel command line argument that takes a list of BDF’s as an 
> input and puts them in pass through mode
> 
> a.Accepting BDF as an input has a downside – BDF is dynamic and could 
> change if BIOS/OS enumerates a new device in next reboot
> 
> b.Accepting <vendor_id:device_id> pair as an input has a downside – What 
> to do when there are multiple such devices and user would like to put 
> only some of them in PT mode
> 
> 2.Have a sysfs file which takes 1 or 0 as an input to enable/disable 
> pass through mode. Some places that seem to be reasonable are
> 
> a./sys/class/iommu/dmar0/devices/
> 
> b./sys/kernel/iommu_groups/<id>/devices
> 
> I am looking for a consensus on **how the kernel command line argument 
> should look like and path for sysfs entry**. Also, please note that if a 
> device is put in pass through mode it won’t be available for the guest 
> and that’s ok.

Just out of curiosity, what's the limitation for a device using pass-
through DMA domain to be assignable.

Best regards,
Baolu

> 
> Regards,
> 
> Sai
> 
> PS: Idea credits: Ashok Raj
> 
> 
> _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: Device specific pass through in host systems - discuss user interface

[Prakhya, Sai Praneeth]

> > I am working on an IOMMU driver feature that allows a user to specify
> > if the DMA from a device should be translated by IOMMU or not.
> > Presently, we support only all devices or none mode i.e. if user
> > specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through
> > kernel command line, all the devices would be in pass through mode and
> > we don't have per device granularity, but, we were requested by a
> > customer to selectively put devices in pass through mode and not all.
> 
> Most iommu vendor drivers have switched from per-device to per-group domain
> (a.k.a. default domain). So per-group pass-through mode makes more sense?
> 
> By the way, can we extend this to "per-group default domain type", instead of
> only "per-group pass-through mode"? Currently we have system level default
> domain type, if we have finer granularity of default domain type, both iommu
> drivers and end users will benefit from it.

Sure! Makes sense.. per-group default domain type sounds good.

> > I am looking for a consensus on **how the kernel command line argument
> > should look like and path for sysfs entry**. Also, please note that if
> > a device is put in pass through mode it won't be available for the
> > guest and that's ok.
> 
> Just out of curiosity, what's the limitation for a device using pass- through DMA
> domain to be assignable.

Sorry! I don't know about assignable devices. Probably, Ashok or Jacob could answer this question

Regards,
Sai
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Raj, Ashok]

Hi Sai

On Sun, Jun 09, 2019 at 10:41:10PM -0700, Sai Praneeth Prakhya wrote:
> > > I am working on an IOMMU driver feature that allows a user to specify
> > > if the DMA from a device should be translated by IOMMU or not.
> > > Presently, we support only all devices or none mode i.e. if user
> > > specifies "iommu=pt" [X86] or "iommu.passthrough" [ARM64] through
> > > kernel command line, all the devices would be in pass through mode and
> > > we don't have per device granularity, but, we were requested by a
> > > customer to selectively put devices in pass through mode and not all.
> > 
> > Most iommu vendor drivers have switched from per-device to per-group domain
> > (a.k.a. default domain). So per-group pass-through mode makes more sense?
> > 
> > By the way, can we extend this to "per-group default domain type", instead of
> > only "per-group pass-through mode"? Currently we have system level default
> > domain type, if we have finer granularity of default domain type, both iommu
> > drivers and end users will benefit from it.
> 
> Sure! Makes sense.. per-group default domain type sounds good.
> 
> > > I am looking for a consensus on **how the kernel command line argument
> > > should look like and path for sysfs entry**. Also, please note that if
> > > a device is put in pass through mode it won't be available for the
> > > guest and that's ok.
> > 
> > Just out of curiosity, what's the limitation for a device using pass- through DMA
> > domain to be assignable.
> 
> Sorry! I don't know about assignable devices. Probably, Ashok or Jacob could answer this question

We don't switch the domain for assigned devices. Only the "type" of the default domain is 
changed from dma-protected to passthrough type.

When assigning devices to user-space, there is no change in this proposal.

> 
> Regards,
> Sai
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: Device specific pass through in host systems - discuss user interface

[Prakhya, Sai Praneeth]

Hi All,

+ Sohil and Rob Clark (as there are dropped from CC'list)

> > > Most iommu vendor drivers have switched from per-device to per-group
> > > domain (a.k.a. default domain). So per-group pass-through mode makes
> more sense?
> > >
> > > By the way, can we extend this to "per-group default domain type",
> > > instead of only "per-group pass-through mode"? Currently we have
> > > system level default domain type, if we have finer granularity of
> > > default domain type, both iommu drivers and end users will benefit from it.
> >
> > Sure! Makes sense.. per-group default domain type sounds good.

I am planning to implement an RFC (supporting only runtime case for now) which works as below

1. User unbinds the driver by writing to sysfs
2. User puts a group in pass through mode by writing "1" to
/sys/kernel/iommu_groups/<group_id>/pt
3. User re-binds the driver by writing to sysfs

As suggested by Lu, Baolu will look into implementing this by using "per-group default domain type"

If anyone has suggestions/comments/concerns, please reply.

Regards,
Sai
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[Raj, Ashok]

On Mon, Jun 10, 2019 at 09:38:11PM -0700, Sai Praneeth Prakhya wrote:
> Hi All,
> 
> + Sohil and Rob Clark (as there are dropped from CC'list)
> 
> > > > Most iommu vendor drivers have switched from per-device to per-group
> > > > domain (a.k.a. default domain). So per-group pass-through mode makes
> > more sense?
> > > >
> > > > By the way, can we extend this to "per-group default domain type",
> > > > instead of only "per-group pass-through mode"? Currently we have
> > > > system level default domain type, if we have finer granularity of
> > > > default domain type, both iommu drivers and end users will benefit from it.
> > >
> > > Sure! Makes sense.. per-group default domain type sounds good.
> 
> I am planning to implement an RFC (supporting only runtime case for now) which works as below
> 
> 1. User unbinds the driver by writing to sysfs
> 2. User puts a group in pass through mode by writing "1" to
> /sys/kernel/iommu_groups/<group_id>/pt

might be better to read current value of default domain for that group.. 
/sys/kernel/iommu_groups/<group_id>/default_domain

reading the above value shows current setting.
provide a differnet file next_def_domain, and you can echo "pt" or "dma_domain"
to switch to pass-through, or normal dma isolation mode.

For devices that automatically set to pass through today like graphics, or isoch audio
you can show "pt" as default_domain.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: Device specific pass through in host systems - discuss user interface

[Prakhya, Sai Praneeth]

> > > > Sure! Makes sense.. per-group default domain type sounds good.
> >
> > I am planning to implement an RFC (supporting only runtime case for
> > now) which works as below
> >
> > 1. User unbinds the driver by writing to sysfs 2. User puts a group in
> > pass through mode by writing "1" to
> > /sys/kernel/iommu_groups/<group_id>/pt
> 
> might be better to read current value of default domain for that group..
> /sys/kernel/iommu_groups/<group_id>/default_domain

Presently, we already have a file that gives out "type" of default_domain and the file is
/sys/kernel/iommu_groups/<group_id>/type

> reading the above value shows current setting.
> provide a differnet file next_def_domain, and you can echo "pt" or
> "dma_domain"
> to switch to pass-through, or normal dma isolation mode.

We have couple of options here:

1. Since we already have "type" file, which is "read-only", we could make it R/W.

The present value shows the existing type of default domain.
If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he attempts to write the new value.
Kernel performs checks to make sure that the driver in unbinded and it's safe to change the default domain type.
After successfully changing the default_domain type internally, kernel reflects the new value in the file.
Ay errors in the process will be reported in dmesg.

2. As you have suggested, we could have a *new* file named "next_def_domain_type", which takes string as an input.

Please let me know if there is any preference among these approaches and why :)

Regards,
Sai
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: Device specific pass through in host systems - discuss user interface

[jroedel]

On Tue, Jun 11, 2019 at 05:27:15PM +0000, Prakhya, Sai Praneeth wrote:
> 1. Since we already have "type" file, which is "read-only", we could make it R/W.
> 
> The present value shows the existing type of default domain.
> If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he attempts to write the new value.
> Kernel performs checks to make sure that the driver in unbinded and it's safe to change the default domain type.
> After successfully changing the default_domain type internally, kernel reflects the new value in the file.
> Ay errors in the process will be reported in dmesg.

I prefer this way. Writing to the file should fail with -EBUSY when it
is not safe to change the default domain-type. Writing should only
succeed when no device in the group is assigned to a device driver.

Regards,

	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: Device specific pass through in host systems - discuss user interface

[Prakhya, Sai Praneeth]

> > The present value shows the existing type of default domain.
> > If user wants to change it (Eg: from DMA to IDENTITY or vice versa), he
> attempts to write the new value.
> > Kernel performs checks to make sure that the driver in unbinded and it's safe
> to change the default domain type.
> > After successfully changing the default_domain type internally, kernel reflects
> the new value in the file.
> > Ay errors in the process will be reported in dmesg.
> 
> I prefer this way. Writing to the file should fail with -EBUSY when it is not safe to
> change the default domain-type. Writing should only succeed when no device in
> the group is assigned to a device driver.

Thanks a lot! Joerg for the reply.
I have just sent out an RFC of this patch set to the IOMMU mailing list.
Could you please take a look at it and let me know your feedback?

Regards,
Sai
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu