* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-10 10:37 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <CALCETrVPQc04Ah7FXcRMb4RhpUJ1WPCoC_4dbacB8a+u5XpmwA@mail.gmail.com>
On Mon, 2015-11-09 at 21:35 -0800, Andy Lutomirski wrote:
>
> We could do it the other way around: on powerpc, if a PCI device is in
> that range and doesn't have the "bypass" property at all, then it's
> assumed to bypass the IOMMU. This means that everything that
> currently works continues working. If someone builds a physical
> virtio device or uses another system in PCIe target mode speaking
> virtio, then it won't work until they upgrade their firmware to set
> bypass=0. Meanwhile everyone using hypothetical new QEMU also gets
> bypass=0 and no ambiguity.
>
> vfio will presumably notice the bypass and correctly refuse to map any
> current virtio devices.
>
> Would that work?
That would be extremely strange from a platform perspective. Any device
in that vendor/device range would bypass the iommu unless some new
property "actually-works-like-a-real-pci-device" happens to exist in
the device-tree, which we would then need to define somewhere and
handle accross at least 3 different platforms who get their device-tree
from widly different places.
Also if tomorrow I create a PCI device that implements virtio-net and
put it in a machine running IBM proprietary firmware (or Apple's or
Sun's), it won't have that property...
This is not hypothetical. People are using virtio to do point-to-point
communication between machines via PCIe today.
Cheers,
Ben.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Joerg Roedel @ 2015-11-10 10:27 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: linux-s390, KVM, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Andy Lutomirski,
Christian Borntraeger, Christoph Hellwig, Andy Lutomirski,
sparclinux, Paolo Bonzini, Linux Virtualization, David Woodhouse,
David S. Miller, Martin Schwidefsky
In-Reply-To: <1447121076.31884.61.camel@kernel.crashing.org>
On Tue, Nov 10, 2015 at 01:04:36PM +1100, Benjamin Herrenschmidt wrote:
> The "in absence of the new DT binding" doesn't make that much sense.
>
> Those platforms use device-trees defined since the dawn of ages by
> actual open firmware implementations, they either have no iommu
> representation in there (Macs, the platform code hooks it all up) or
> have various properties related to the iommu but no concept of "bypass"
> in there.
>
> We can *add* a new property under some circumstances that indicates a
> bypass on a per-device basis, however that doesn't completely solve it:
>
> - As I said above, what does the absence of that property mean ? An
> old qemu that does bypass on all virtio or a new qemu trying to tell
> you that the virtio device actually does use the iommu (or some other
> environment that isn't qemu) ?
You have the same problem when real PCIe devices appear that speak
virtio. I think the only real (still not very nice) solution is to add a
quirk to powerpc platform code that sets noop dma-ops for the existing
virtio vendor/device-ids and add a DT property to opt-out of that quirk.
New vendor/device-ids (as for real devices) would just not be covered by
the quirk and existing emulated devices continue to work.
The absence of the property just means that the quirk is in place and
the system assumes no translation for virtio devices.
Joerg
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-10 10:26 UTC (permalink / raw)
To: Knut Omang, Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447148714.3005.133.camel@oracle.com>
On Tue, 2015-11-10 at 10:45 +0100, Knut Omang wrote:
> Can something be done by means of PCIe capabilities?
> ATS (Address Translation Support) seems like a natural choice?
Euh no... ATS is something else completely....
Cheers,
Ben.
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Knut Omang @ 2015-11-10 9:45 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447121076.31884.61.camel@kernel.crashing.org>
On Tue, 2015-11-10 at 13:04 +1100, Benjamin Herrenschmidt wrote:
> On Mon, 2015-11-09 at 16:46 -0800, Andy Lutomirski wrote:
> > The problem here is that in some of the problematic cases the
> > virtio
> > driver may not even be loaded. If someone runs an L1 guest with an
> > IOMMU-bypassing virtio device and assigns it to L2 using vfio, then
> > *boom* L1 crashes. (Same if, say, DPDK gets used, I think.)
> >
> > >
> > > The only way out of this while keeping the "platform" stuff would
> > > be to
> > > also bump some kind of version in the virtio config (or PCI
> > > header). I
> > > have no other way to differenciate between "this is an old qemu
> > > that
> > > doesn't do the 'bypass property' yet" from "this is a virtio
> > > device
> > > that doesn't bypass".
> > >
> > > Any better idea ?
> >
> > I'd suggest that, in the absence of the new DT binding, we assume
> > that
> > any PCI device with the virtio vendor ID is passthrough on powerpc.
> > I
> > can do this in the virtio driver, but if it's in the platform code
> > then vfio gets it right too (i.e. fails to load).
>
> The problem is there isn't *a* virtio vendor ID. It's the RedHat
> vendor
> ID which will be used by more than just virtio, so we need to
> specifically list the devices.
>
> Additionally, that still means that once we have a virtio device that
> actually uses the iommu, powerpc will not work since the "workaround"
> above will kick in.
>
> The "in absence of the new DT binding" doesn't make that much sense.
>
> Those platforms use device-trees defined since the dawn of ages by
> actual open firmware implementations, they either have no iommu
> representation in there (Macs, the platform code hooks it all up) or
> have various properties related to the iommu but no concept of
> "bypass"
> in there.
>
> We can *add* a new property under some circumstances that indicates a
> bypass on a per-device basis, however that doesn't completely solve
> it:
>
> - As I said above, what does the absence of that property mean ? An
> old qemu that does bypass on all virtio or a new qemu trying to tell
> you that the virtio device actually does use the iommu (or some other
> environment that isn't qemu) ?
>
> - On things like macs, the device-tree is generated by openbios, it
> would have to have some added logic to try to figure that out, which
> means it needs to know *via different means* that some or all virtio
> devices bypass the iommu.
>
> I thus go back to my original statement, it's a LOT easier to handle
> if
> the device itself is self describing, indicating whether it is set to
> bypass a host iommu or not. For L1->L2, well, that wouldn't be the
> first time qemu/VFIO plays tricks with the passed through device
> configuration space...
>
> Note that the above can be solved via some kind of compromise: The
> device self describes the ability to honor the iommu, along with the
> property (or ACPI table entry) that indicates whether or not it does.
>
> IE. We could use the revision or ProgIf field of the config space for
> example. Or something in virtio config. If it's an "old" device, we
> know it always bypass. If it's a new device, we know it only bypasses
> if the corresponding property is in. I still would have to sort out
> the
> openbios case for mac among others but it's at least a workable
> direction.
>
> BTW. Don't you have a similar problem on x86 that today qemu claims
> that everything honors the iommu in ACPI ?
>
> Unless somebody can come up with a better idea...
Can something be done by means of PCIe capabilities?
ATS (Address Translation Support) seems like a natural choice?
Knut
> Cheers,
> Ben.
>
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Jan Kiszka @ 2015-11-10 7:28 UTC (permalink / raw)
To: Andy Lutomirski, Benjamin Herrenschmidt
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <CALCETrW5_bKCX5gKYaH5y4rvD9jrjY5O5d=oX8hHtAM9EE2Bew@mail.gmail.com>
On 2015-11-10 03:18, Andy Lutomirski wrote:
> On Mon, Nov 9, 2015 at 6:04 PM, Benjamin Herrenschmidt
>> I thus go back to my original statement, it's a LOT easier to handle if
>> the device itself is self describing, indicating whether it is set to
>> bypass a host iommu or not. For L1->L2, well, that wouldn't be the
>> first time qemu/VFIO plays tricks with the passed through device
>> configuration space...
>
> Which leaves the special case of Xen, where even preexisting devices
> don't bypass the IOMMU. Can we keep this specific to powerpc and
> sparc? On x86, this problem is basically nonexistent, since the IOMMU
> is properly self-describing.
>
> IOW, I think that on x86 we should assume that all virtio devices
> honor the IOMMU.
From the guest driver POV, that is OK because either there is no IOMMU
to program (the current situation with qemu), there can be one that
doesn't need it (the current situation with qemu and iommu=on) or there
is (Xen) or will be (future qemu) one that requires it.
>
>>
>> Note that the above can be solved via some kind of compromise: The
>> device self describes the ability to honor the iommu, along with the
>> property (or ACPI table entry) that indicates whether or not it does.
>>
>> IE. We could use the revision or ProgIf field of the config space for
>> example. Or something in virtio config. If it's an "old" device, we
>> know it always bypass. If it's a new device, we know it only bypasses
>> if the corresponding property is in. I still would have to sort out the
>> openbios case for mac among others but it's at least a workable
>> direction.
>>
>> BTW. Don't you have a similar problem on x86 that today qemu claims
>> that everything honors the iommu in ACPI ?
>
> Only on a single experimental configuration, and that can apparently
> just be fixed going forward without any real problems being caused.
BTW, I once tried to describe the current situation on QEMU x86 with
IOMMU enabled via ACPI. While you can easily add IOMMU device exceptions
to the static tables, the fun starts when considering device hotplug for
virtio. Unless I missed some trick, ACPI doesn't seem like being
designed for that level of flexibility.
You would have to reserve a complete PCI bus, declare that one as not
being IOMMU-governed, and then only add new virtio devices to that bus.
Possible, but a lot of restrictions that existing management software
would have to be aware of as well.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Andy Lutomirski @ 2015-11-10 5:35 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447133316.31884.67.camel@kernel.crashing.org>
On Mon, Nov 9, 2015 at 9:28 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Mon, 2015-11-09 at 18:18 -0800, Andy Lutomirski wrote:
>>
>> /* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF.
>> */
>> static const struct pci_device_id virtio_pci_id_table[] = {
>> { PCI_DEVICE(0x1af4, PCI_ANY_ID) },
>> { 0 }
>> };
>>
>> Can we match on that range?
>
> We can, but the problem remains, how do we differenciate an existing
> device that does bypass vs. a newer one that needs the IOMMU and thus
> doesn't have the new "bypass" property in the device-tree.
>
We could do it the other way around: on powerpc, if a PCI device is in
that range and doesn't have the "bypass" property at all, then it's
assumed to bypass the IOMMU. This means that everything that
currently works continues working. If someone builds a physical
virtio device or uses another system in PCIe target mode speaking
virtio, then it won't work until they upgrade their firmware to set
bypass=0. Meanwhile everyone using hypothetical new QEMU also gets
bypass=0 and no ambiguity.
vfio will presumably notice the bypass and correctly refuse to map any
current virtio devices.
Would that work?
--Andy
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Andy Lutomirski @ 2015-11-10 5:33 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447133190.31884.65.camel@kernel.crashing.org>
On Mon, Nov 9, 2015 at 9:26 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Mon, 2015-11-09 at 18:18 -0800, Andy Lutomirski wrote:
>>
>> Which leaves the special case of Xen, where even preexisting devices
>> don't bypass the IOMMU. Can we keep this specific to powerpc and
>> sparc? On x86, this problem is basically nonexistent, since the IOMMU
>> is properly self-describing.
>>
>> IOW, I think that on x86 we should assume that all virtio devices
>> honor the IOMMU.
>
> You don't like performances ? :-)
This should have basically no effect. Every non-experimental x86
virtio setup in existence either doesn't work at all (Xen) or has DMA
ops that are no-ops.
--Andy
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-10 5:28 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <CALCETrW5_bKCX5gKYaH5y4rvD9jrjY5O5d=oX8hHtAM9EE2Bew@mail.gmail.com>
On Mon, 2015-11-09 at 18:18 -0800, Andy Lutomirski wrote:
>
> /* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF.
> */
> static const struct pci_device_id virtio_pci_id_table[] = {
> { PCI_DEVICE(0x1af4, PCI_ANY_ID) },
> { 0 }
> };
>
> Can we match on that range?
We can, but the problem remains, how do we differenciate an existing
device that does bypass vs. a newer one that needs the IOMMU and thus
doesn't have the new "bypass" property in the device-tree.
Cheers,
Ben.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-10 5:26 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <CALCETrW5_bKCX5gKYaH5y4rvD9jrjY5O5d=oX8hHtAM9EE2Bew@mail.gmail.com>
On Mon, 2015-11-09 at 18:18 -0800, Andy Lutomirski wrote:
>
> Which leaves the special case of Xen, where even preexisting devices
> don't bypass the IOMMU. Can we keep this specific to powerpc and
> sparc? On x86, this problem is basically nonexistent, since the IOMMU
> is properly self-describing.
>
> IOW, I think that on x86 we should assume that all virtio devices
> honor the IOMMU.
You don't like performances ? :-)
Cheers,
Ben.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Andy Lutomirski @ 2015-11-10 2:18 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447121076.31884.61.camel@kernel.crashing.org>
On Mon, Nov 9, 2015 at 6:04 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Mon, 2015-11-09 at 16:46 -0800, Andy Lutomirski wrote:
>> The problem here is that in some of the problematic cases the virtio
>> driver may not even be loaded. If someone runs an L1 guest with an
>> IOMMU-bypassing virtio device and assigns it to L2 using vfio, then
>> *boom* L1 crashes. (Same if, say, DPDK gets used, I think.)
>>
>> >
>> > The only way out of this while keeping the "platform" stuff would be to
>> > also bump some kind of version in the virtio config (or PCI header). I
>> > have no other way to differenciate between "this is an old qemu that
>> > doesn't do the 'bypass property' yet" from "this is a virtio device
>> > that doesn't bypass".
>> >
>> > Any better idea ?
>>
>> I'd suggest that, in the absence of the new DT binding, we assume that
>> any PCI device with the virtio vendor ID is passthrough on powerpc. I
>> can do this in the virtio driver, but if it's in the platform code
>> then vfio gets it right too (i.e. fails to load).
>
> The problem is there isn't *a* virtio vendor ID. It's the RedHat vendor
> ID which will be used by more than just virtio, so we need to
> specifically list the devices.
Really?
/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
static const struct pci_device_id virtio_pci_id_table[] = {
{ PCI_DEVICE(0x1af4, PCI_ANY_ID) },
{ 0 }
};
Can we match on that range?
>
> Additionally, that still means that once we have a virtio device that
> actually uses the iommu, powerpc will not work since the "workaround"
> above will kick in.
I don't know how to solve that problem, though, especially since the
vendor of such a device (especially if it's real hardware) might not
set any new bit.
>
> The "in absence of the new DT binding" doesn't make that much sense.
>
> Those platforms use device-trees defined since the dawn of ages by
> actual open firmware implementations, they either have no iommu
> representation in there (Macs, the platform code hooks it all up) or
> have various properties related to the iommu but no concept of "bypass"
> in there.
>
> We can *add* a new property under some circumstances that indicates a
> bypass on a per-device basis, however that doesn't completely solve it:
>
> - As I said above, what does the absence of that property mean ? An
> old qemu that does bypass on all virtio or a new qemu trying to tell
> you that the virtio device actually does use the iommu (or some other
> environment that isn't qemu) ?
>
> - On things like macs, the device-tree is generated by openbios, it
> would have to have some added logic to try to figure that out, which
> means it needs to know *via different means* that some or all virtio
> devices bypass the iommu.
>
> I thus go back to my original statement, it's a LOT easier to handle if
> the device itself is self describing, indicating whether it is set to
> bypass a host iommu or not. For L1->L2, well, that wouldn't be the
> first time qemu/VFIO plays tricks with the passed through device
> configuration space...
Which leaves the special case of Xen, where even preexisting devices
don't bypass the IOMMU. Can we keep this specific to powerpc and
sparc? On x86, this problem is basically nonexistent, since the IOMMU
is properly self-describing.
IOW, I think that on x86 we should assume that all virtio devices
honor the IOMMU.
>
> Note that the above can be solved via some kind of compromise: The
> device self describes the ability to honor the iommu, along with the
> property (or ACPI table entry) that indicates whether or not it does.
>
> IE. We could use the revision or ProgIf field of the config space for
> example. Or something in virtio config. If it's an "old" device, we
> know it always bypass. If it's a new device, we know it only bypasses
> if the corresponding property is in. I still would have to sort out the
> openbios case for mac among others but it's at least a workable
> direction.
>
> BTW. Don't you have a similar problem on x86 that today qemu claims
> that everything honors the iommu in ACPI ?
Only on a single experimental configuration, and that can apparently
just be fixed going forward without any real problems being caused.
--Andy
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-10 2:04 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <CALCETrX7Gkw3WrBHff=TpCFHj444E8hHcR6sAqOghQFBo5wp_A@mail.gmail.com>
On Mon, 2015-11-09 at 16:46 -0800, Andy Lutomirski wrote:
> The problem here is that in some of the problematic cases the virtio
> driver may not even be loaded. If someone runs an L1 guest with an
> IOMMU-bypassing virtio device and assigns it to L2 using vfio, then
> *boom* L1 crashes. (Same if, say, DPDK gets used, I think.)
>
> >
> > The only way out of this while keeping the "platform" stuff would be to
> > also bump some kind of version in the virtio config (or PCI header). I
> > have no other way to differenciate between "this is an old qemu that
> > doesn't do the 'bypass property' yet" from "this is a virtio device
> > that doesn't bypass".
> >
> > Any better idea ?
>
> I'd suggest that, in the absence of the new DT binding, we assume that
> any PCI device with the virtio vendor ID is passthrough on powerpc. I
> can do this in the virtio driver, but if it's in the platform code
> then vfio gets it right too (i.e. fails to load).
The problem is there isn't *a* virtio vendor ID. It's the RedHat vendor
ID which will be used by more than just virtio, so we need to
specifically list the devices.
Additionally, that still means that once we have a virtio device that
actually uses the iommu, powerpc will not work since the "workaround"
above will kick in.
The "in absence of the new DT binding" doesn't make that much sense.
Those platforms use device-trees defined since the dawn of ages by
actual open firmware implementations, they either have no iommu
representation in there (Macs, the platform code hooks it all up) or
have various properties related to the iommu but no concept of "bypass"
in there.
We can *add* a new property under some circumstances that indicates a
bypass on a per-device basis, however that doesn't completely solve it:
- As I said above, what does the absence of that property mean ? An
old qemu that does bypass on all virtio or a new qemu trying to tell
you that the virtio device actually does use the iommu (or some other
environment that isn't qemu) ?
- On things like macs, the device-tree is generated by openbios, it
would have to have some added logic to try to figure that out, which
means it needs to know *via different means* that some or all virtio
devices bypass the iommu.
I thus go back to my original statement, it's a LOT easier to handle if
the device itself is self describing, indicating whether it is set to
bypass a host iommu or not. For L1->L2, well, that wouldn't be the
first time qemu/VFIO plays tricks with the passed through device
configuration space...
Note that the above can be solved via some kind of compromise: The
device self describes the ability to honor the iommu, along with the
property (or ACPI table entry) that indicates whether or not it does.
IE. We could use the revision or ProgIf field of the config space for
example. Or something in virtio config. If it's an "old" device, we
know it always bypass. If it's a new device, we know it only bypasses
if the corresponding property is in. I still would have to sort out the
openbios case for mac among others but it's at least a workable
direction.
BTW. Don't you have a similar problem on x86 that today qemu claims
that everything honors the iommu in ACPI ?
Unless somebody can come up with a better idea...
Cheers,
Ben.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Andy Lutomirski @ 2015-11-10 0:46 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: Joerg Roedel, KVM, linux-s390, Michael S. Tsirkin, Sebastian Ott,
linux-kernel@vger.kernel.org, Christoph Hellwig,
Christian Borntraeger, Andy Lutomirski, sparclinux, Paolo Bonzini,
Linux Virtualization, David Woodhouse, David S. Miller,
Martin Schwidefsky
In-Reply-To: <1447109937.31884.42.camel@kernel.crashing.org>
On Mon, Nov 9, 2015 at 2:58 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> So ...
>
> I've finally tried to sort that out for powerpc and I can't find a way
> to make that work that isn't a complete pile of stinking shit.
>
> I'm very tempted to go back to my original idea: virtio itself should
> indicate it's "bypassing ability" via the virtio config space or some
> other bit (like the ProgIf of the PCI header etc...).
>
> I don't see how I can make it work otherwise.
>
> The problem with the statement "it's a platform matter" is that:
>
> - It's not entirely correct. It's actually a feature of a specific
> virtio implementation (qemu's) that it bypasses all the platform IOMMU
> facilities.
>
> - The platforms (on powerpc there's at least 3 or 4 that have qemu
> emulation and support some form of PCI) don't have an existing way to
> convey the information that a device bypasses the IOMMU (if any).
>
> - Even if we add such a mechanism (new property in the device-tree),
> we end up with a big turd: Because we need to be compatible with older
> qemus, we essentially need a quirk that will make all virtio devices
> assume that property is present. That will of course break whenever we
> try to use another implementation of virtio on powerpc which doesn't
> bypass the iommu.
>
> We don't have a way to differenciate a virtio device that does the
> bypass from one that doesn't and the backward compatibility requirement
> forces us to treat all existing virtio devices as doing bypass.
The problem here is that in some of the problematic cases the virtio
driver may not even be loaded. If someone runs an L1 guest with an
IOMMU-bypassing virtio device and assigns it to L2 using vfio, then
*boom* L1 crashes. (Same if, say, DPDK gets used, I think.)
>
> The only way out of this while keeping the "platform" stuff would be to
> also bump some kind of version in the virtio config (or PCI header). I
> have no other way to differenciate between "this is an old qemu that
> doesn't do the 'bypass property' yet" from "this is a virtio device
> that doesn't bypass".
>
> Any better idea ?
I'd suggest that, in the absence of the new DT binding, we assume that
any PCI device with the virtio vendor ID is passthrough on powerpc. I
can do this in the virtio driver, but if it's in the platform code
then vfio gets it right too (i.e. fails to load).
--Andy
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Benjamin Herrenschmidt @ 2015-11-09 22:58 UTC (permalink / raw)
To: Andy Lutomirski, dwmw2
Cc: linux-s390, Joerg Roedel, KVM, Michael S. Tsirkin, Sebastian Ott,
linux-kernel, Christian Borntraeger, sparclinux, Paolo Bonzini,
virtualization, Christoph Hellwig, David S. Miller,
Martin Schwidefsky
In-Reply-To: <20151109133624-mutt-send-email-mst@redhat.com>
So ...
I've finally tried to sort that out for powerpc and I can't find a way
to make that work that isn't a complete pile of stinking shit.
I'm very tempted to go back to my original idea: virtio itself should
indicate it's "bypassing ability" via the virtio config space or some
other bit (like the ProgIf of the PCI header etc...).
I don't see how I can make it work otherwise.
The problem with the statement "it's a platform matter" is that:
- It's not entirely correct. It's actually a feature of a specific
virtio implementation (qemu's) that it bypasses all the platform IOMMU
facilities.
- The platforms (on powerpc there's at least 3 or 4 that have qemu
emulation and support some form of PCI) don't have an existing way to
convey the information that a device bypasses the IOMMU (if any).
- Even if we add such a mechanism (new property in the device-tree),
we end up with a big turd: Because we need to be compatible with older
qemus, we essentially need a quirk that will make all virtio devices
assume that property is present. That will of course break whenever we
try to use another implementation of virtio on powerpc which doesn't
bypass the iommu.
We don't have a way to differenciate a virtio device that does the
bypass from one that doesn't and the backward compatibility requirement
forces us to treat all existing virtio devices as doing bypass.
The only way out of this while keeping the "platform" stuff would be to
also bump some kind of version in the virtio config (or PCI header). I
have no other way to differenciate between "this is an old qemu that
doesn't do the 'bypass property' yet" from "this is a virtio device
that doesn't bypass".
Any better idea ?
Cheers,
Ben.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Paolo Bonzini @ 2015-11-09 12:27 UTC (permalink / raw)
To: Michael S. Tsirkin, Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, benh, Sebastian Ott,
virtualization, linux-kernel, Christian Borntraeger, sparclinux,
dwmw2, Christoph Hellwig, David S. Miller, Martin Schwidefsky
In-Reply-To: <20151109133624-mutt-send-email-mst@redhat.com>
On 09/11/2015 13:15, Michael S. Tsirkin wrote:
> Well that's not exactly true. I think we would like to make
> it possible to put virtio devices behind an IOMMU on x86,
> but if this means existing guests break, then many people won't be able
> to use this option: having to find out which kernel version your guest
> is running is a significant burden.
>
> So on the host side, we need to detect guests that
> don't program the IOMMU and make QEMU ignore it.
> I think we need to figure out a way to do this
> before we commit to the guest change.
What is the usecase for putting virtio devices behind an IOMMU, apart from:
1) "because you can"
2) using VFIO within the guest
? Case 1 can be ignored, and in case 2 the guest will do the right thing.
> Additionally, IOMMU overhead is very high when running within the VM.
> So for uses such as VFIO, we'd like a way to make something like
> iommu-pt the default.
That's not something that the kernel cares about. It's just a
configuration issue.
Paolo
^ permalink raw reply
* Re: [PATCH v4 0/6] virtio core DMA API conversion
From: Michael S. Tsirkin @ 2015-11-09 12:15 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, benh, Sebastian Ott,
virtualization, linux-kernel, Christian Borntraeger, sparclinux,
Paolo Bonzini, dwmw2, Christoph Hellwig, David S. Miller,
Martin Schwidefsky
In-Reply-To: <cover.1446162273.git.luto@kernel.org>
On Thu, Oct 29, 2015 at 06:09:45PM -0700, Andy Lutomirski wrote:
> This switches virtio to use the DMA API unconditionally. I'm sure
> it breaks things, but it seems to work on x86 using virtio-pci, with
> and without Xen, and using both the modern 1.0 variant and the
> legacy variant.
>
> This appears to work on native and Xen x86_64 using both modern and
> legacy virtio-pci. It also appears to work on arm and arm64.
>
> It definitely won't work as-is on s390x, and I haven't been able to
> test Christian's patches because I can't get virtio-ccw to work in
> QEMU at all. I don't know what I'm doing wrong.
>
> It doesn't work on ppc64. Ben, consider yourself pinged to send me
> a patch :)
>
> It doesn't work on sparc64. I didn't realize at Kernel Summit that
> sparc64 has the same problem as ppc64.
>
> DaveM, for background, we're trying to fix virtio to use the DMA
> API. That will require that every platform that uses virtio
> supplies valid DMA operations on devices that use virtio_ring.
> Unfortunately, QEMU historically ignores the IOMMU on virtio
> devices.
>
> On x86, this isn't really a problem. x86 has a nice way for the
> platform to describe which devices are behind an IOMMU, and QEMU
> will be adjusted accordingly. The only thing that will break is a
> recently-added experimental mode.
Well that's not exactly true. I think we would like to make
it possible to put virtio devices behind an IOMMU on x86,
but if this means existing guests break, then many people won't be able
to use this option: having to find out which kernel version your guest
is running is a significant burden.
So on the host side, we need to detect guests that
don't program the IOMMU and make QEMU ignore it.
I think we need to figure out a way to do this
before we commit to the guest change.
Additionally, IOMMU overhead is very high when running within the VM.
So for uses such as VFIO, we'd like a way to make something like
iommu-pt the default.
> Ben's plan for powerpc is to add a quirk for existing virtio-pci
> devices and to eventually update the devicetree stuff to allow QEMU
> to tell the guest which devices use the IOMMU.
>
> AFAICT sparc has a similar problem to powerpc. DaveM, can you come
> up with a straightforward way to get sparc's DMA API to work
> correctly for virtio-pci devices?
>
> NB: Sadly, the platforms I've successfully tested on don't include any
> big-endian platforms, so there could still be lurking endian problems.
>
> Changes from v3:
> - More big-endian fixes.
> - Added better virtio-ring APIs that handle allocation and use them in
> virtio-mmio and virtio-pci.
> - Switch to Michael's virtio-net patch.
>
> Changes from v2:
> - Fix vring_mapping_error incorrect argument
>
> Changes from v1:
> - Fix an endian conversion error causing a BUG to hit.
> - Fix a DMA ordering issue (swiotlb=force works now).
> - Minor cleanups.
>
> Andy Lutomirski (5):
> virtio_ring: Support DMA APIs
> virtio_pci: Use the DMA API
> virtio: Add improved queue allocation API
> virtio_mmio: Use the DMA API
> virtio_pci: Use the DMA API
>
> Michael S. Tsirkin (1):
> virtio-net: Stop doing DMA from the stack
>
> drivers/net/virtio_net.c | 34 ++--
> drivers/virtio/Kconfig | 2 +-
> drivers/virtio/virtio_mmio.c | 67 ++-----
> drivers/virtio/virtio_pci_common.h | 6 -
> drivers/virtio/virtio_pci_legacy.c | 42 ++---
> drivers/virtio/virtio_pci_modern.c | 61 ++-----
> drivers/virtio/virtio_ring.c | 348 ++++++++++++++++++++++++++++++-------
> include/linux/virtio.h | 23 ++-
> include/linux/virtio_ring.h | 35 ++++
> tools/virtio/linux/dma-mapping.h | 17 ++
> 10 files changed, 426 insertions(+), 209 deletions(-)
> create mode 100644 tools/virtio/linux/dma-mapping.h
>
> --
> 2.4.3
^ permalink raw reply
* Re: [PATCH v3 0/3] virtio DMA API core stuff
From: David Woodhouse @ 2015-11-08 12:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-s390, Joerg Roedel, KVM, Benjamin Herrenschmidt,
Sebastian Ott, linux-kernel@vger.kernel.org, Andy Lutomirski,
Christian Borntraeger, Andy Lutomirski, Paolo Bonzini,
Linux Virtualization, Christoph Hellwig, Martin Schwidefsky
In-Reply-To: <20151108120627-mutt-send-email-mst@redhat.com>
[-- Attachment #1.1: Type: text/plain, Size: 2813 bytes --]
On Sun, 2015-11-08 at 12:37 +0200, Michael S. Tsirkin wrote:
> On Thu, Oct 29, 2015 at 05:18:56PM +0100, David Woodhouse wrote:
> > On Thu, 2015-10-29 at 11:01 +0200, Michael S. Tsirkin wrote:
> > >
> > > But you trust your hypervisor (you have no choice anyway),
> > > and you don't want the overhead of tweaking IOMMU
> > > on data path for virtio. Thus iommu=on is out too.
> >
> > That's not at all special for virtio or guest VMs. Even with real
> > hardware, we might want performance from *some* devices, and security
> > from others. See the DMA_ATTR_IOMMU_BYPASS which is currently being
> > discussed.
>
> Right. So let's wait for that discussion to play out?
That discussion is purely about a requested optimisation. This one is
about correctness.
> > But of course the easy answer in *your* case it just to ask the
> > hypervisor not to put the virtio devices behind an IOMMU at all. Which
> > we were planning to remain the default behaviour.
>
> One can't do this for x86 ATM, can one?
The converse is true, in fact — currently, there's no way to tell
qemu-system-x86 that you *do* want it to put the virtio devices behind
the emulated IOMMU, as it has no support for that.
Which is a bit sad really, since the DMAR table that qemu advertises to
the guest does *tell* the guest that the virtio devices are behind the
emulated IOMMU.
In the short term, we'll be fixing the DMAR table, and still not
actually making it possible to put the virtio devices behind the
emulated IOMMU.
In the fullness of time, however, we *will* be fixing the qemu IOMMU
code so that it can translate for virtio devices — and for assigned
physical devices, which I believe are also broken at the moment when
qemu emulates an IOMMU.
> > In all cases, the DMA API shall do the right thing.
>
> I have no problem with that. For example, can we teach
> the DMA API on intel x86 to use PT for virtio by default?
> That would allow merging Andy's patches with
> full compatibility with old guests and hosts.
A quirk so that we *notice* the bug in the existing qemu DMAR table,
and disbelieve what it says about the virtio devices?
Alternatively, we could just recognise that the emulated IOMMU support
in qemu is an experimental feature and doesn't work right, yet. Are
people really using it in anger?
If we do want to do a quirk, then we should make it get it right for
assigned devices too.
To start with, do you want to try to express the criteria for "the DMAR
table lies and <this> device is actually untranslated" in a form of
prose which could reasonably be translated into code?
--
David Woodhouse Open Source Technology Centre
David.Woodhouse@intel.com Intel Corporation
[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5691 bytes --]
[-- Attachment #2: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v3 0/3] virtio DMA API core stuff
From: Joerg Roedel @ 2015-11-08 11:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-s390, KVM, Benjamin Herrenschmidt, Sebastian Ott,
linux-kernel@vger.kernel.org, Andy Lutomirski,
Christian Borntraeger, Andy Lutomirski, Paolo Bonzini,
Linux Virtualization, David Woodhouse, Christoph Hellwig,
Martin Schwidefsky
In-Reply-To: <20151108120627-mutt-send-email-mst@redhat.com>
On Sun, Nov 08, 2015 at 12:37:47PM +0200, Michael S. Tsirkin wrote:
> I have no problem with that. For example, can we teach
> the DMA API on intel x86 to use PT for virtio by default?
> That would allow merging Andy's patches with
> full compatibility with old guests and hosts.
Well, the only incompatibility comes from an experimental qemu feature,
more explicitly from a bug in that features implementation. So why
should we work around that in the kernel? I think it is not too hard to
fix qemu to generate a correct DMAR table which excludes the virtio
devices from iommu translation.
Joerg
^ permalink raw reply
* Re: [PATCH v3 0/3] virtio DMA API core stuff
From: Michael S. Tsirkin @ 2015-11-08 10:37 UTC (permalink / raw)
To: David Woodhouse
Cc: linux-s390, Joerg Roedel, KVM, Benjamin Herrenschmidt,
Sebastian Ott, linux-kernel@vger.kernel.org, Andy Lutomirski,
Christian Borntraeger, Andy Lutomirski, Paolo Bonzini,
Linux Virtualization, Christoph Hellwig, Martin Schwidefsky
In-Reply-To: <1446135536.3405.279.camel@infradead.org>
On Thu, Oct 29, 2015 at 05:18:56PM +0100, David Woodhouse wrote:
> On Thu, 2015-10-29 at 11:01 +0200, Michael S. Tsirkin wrote:
> >
> > Example: you have a mix of assigned devices and virtio devices. You
> > don't trust your assigned device vendor not to corrupt your memory so
> > you want to limit the damage your assigned device can do to your
> > guest,
> > so you use an IOMMU for that. Thus existing iommu=pt within guest is
> > out.
> >
> > But you trust your hypervisor (you have no choice anyway),
> > and you don't want the overhead of tweaking IOMMU
> > on data path for virtio. Thus iommu=on is out too.
>
> That's not at all special for virtio or guest VMs. Even with real
> hardware, we might want performance from *some* devices, and security
> from others. See the DMA_ATTR_IOMMU_BYPASS which is currently being
> discussed.
Right. So let's wait for that discussion to play out?
> But of course the easy answer in *your* case it just to ask the
> hypervisor not to put the virtio devices behind an IOMMU at all. Which
> we were planning to remain the default behaviour.
One can't do this for x86 ATM, can one?
> In all cases, the DMA API shall do the right thing.
I have no problem with that. For example, can we teach
the DMA API on intel x86 to use PT for virtio by default?
That would allow merging Andy's patches with
full compatibility with old guests and hosts.
> --
> dwmw2
>
>
--
MST
^ permalink raw reply
* Re: [GIT PULL v4 3/3] s390/dma: Allow per device dma ops
From: Cornelia Huck @ 2015-11-06 8:35 UTC (permalink / raw)
To: Christian Borntraeger
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, virtualization, Andy Lutomirski,
Martin Schwidefsky, Paolo Bonzini, dwmw2, Christoph Hellwig
In-Reply-To: <1446754136-125772-4-git-send-email-borntraeger@de.ibm.com>
On Thu, 5 Nov 2015 21:08:56 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> As virtio-ccw will have dma ops, we can no longer default to the
> zPCI ones. Make use of dev_archdata to keep the dma_ops per device.
> The pci devices now use that to override the default, and the
> default is changed to use the noop ops for everything that does not
> specify a device specific one.
> To compile without PCI support we will enable HAS_DMA all the time,
> via the default config in lib/Kconfig.
>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Reviewed-by: Joerg Roedel <jroedel@suse.de>
> Acked-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
> ---
> arch/s390/Kconfig | 7 ++-----
> arch/s390/include/asm/device.h | 6 +++++-
> arch/s390/include/asm/dma-mapping.h | 6 ++++--
> arch/s390/pci/pci.c | 1 +
> arch/s390/pci/pci_dma.c | 4 ++--
> 5 files changed, 14 insertions(+), 10 deletions(-)
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
^ permalink raw reply
* Re: [GIT PULL v4 0/3] dma and virtio prep patches
From: Andy Lutomirski @ 2015-11-05 20:31 UTC (permalink / raw)
To: Christian Borntraeger
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin,
Benjamin Herrenschmidt, Sebastian Ott, Linux Virtualization,
Martin Schwidefsky, Andy Lutomirski, Paolo Bonzini,
David Woodhouse, Christoph Hellwig
In-Reply-To: <1446754136-125772-1-git-send-email-borntraeger@de.ibm.com>
On Thu, Nov 5, 2015 at 12:08 PM, Christian Borntraeger
<borntraeger@de.ibm.com> wrote:
> Andy,
>
> to make it obvious which version is the latest, here is a branch
>
> The following changes since commit 6a13feb9c82803e2b815eca72fa7a9f5561d7861:
>
> Linux 4.3 (2015-11-01 16:05:25 -0800)
>
> are available in the git repository at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/borntraeger/linux.git dma
>
> for you to fetch changes up to fc7f9754db6ce0c12281da4055281f731d36bdee:
>
> s390/dma: Allow per device dma ops (2015-11-05 21:02:40 +0100)
Pulled, thanks.
--Andy
^ permalink raw reply
* [GIT PULL v4 3/3] s390/dma: Allow per device dma ops
From: Christian Borntraeger @ 2015-11-05 20:08 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, Christian Borntraeger, virtualization,
Martin Schwidefsky, Paolo Bonzini, dwmw2, Christoph Hellwig
In-Reply-To: <1446754136-125772-1-git-send-email-borntraeger@de.ibm.com>
As virtio-ccw will have dma ops, we can no longer default to the
zPCI ones. Make use of dev_archdata to keep the dma_ops per device.
The pci devices now use that to override the default, and the
default is changed to use the noop ops for everything that does not
specify a device specific one.
To compile without PCI support we will enable HAS_DMA all the time,
via the default config in lib/Kconfig.
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
---
arch/s390/Kconfig | 7 ++-----
arch/s390/include/asm/device.h | 6 +++++-
arch/s390/include/asm/dma-mapping.h | 6 ++++--
arch/s390/pci/pci.c | 1 +
arch/s390/pci/pci_dma.c | 4 ++--
5 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 1d57000..e2a885b 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -124,6 +124,8 @@ config S390
select HAVE_CMPXCHG_DOUBLE
select HAVE_CMPXCHG_LOCAL
select HAVE_DEBUG_KMEMLEAK
+ select HAVE_DMA_ATTRS
+ select HAVE_DMA_API_DEBUG
select HAVE_DYNAMIC_FTRACE
select HAVE_DYNAMIC_FTRACE_WITH_REGS
select HAVE_FTRACE_MCOUNT_RECORD
@@ -580,7 +582,6 @@ config QDIO
menuconfig PCI
bool "PCI support"
- select HAVE_DMA_ATTRS
select PCI_MSI
help
Enable PCI support.
@@ -620,10 +621,6 @@ config HAS_IOMEM
config IOMMU_HELPER
def_bool PCI
-config HAS_DMA
- def_bool PCI
- select HAVE_DMA_API_DEBUG
-
config NEED_SG_DMA_LENGTH
def_bool PCI
diff --git a/arch/s390/include/asm/device.h b/arch/s390/include/asm/device.h
index d8f9872..4a9f35e 100644
--- a/arch/s390/include/asm/device.h
+++ b/arch/s390/include/asm/device.h
@@ -3,5 +3,9 @@
*
* This file is released under the GPLv2
*/
-#include <asm-generic/device.h>
+struct dev_archdata {
+ struct dma_map_ops *dma_ops;
+};
+struct pdev_archdata {
+};
diff --git a/arch/s390/include/asm/dma-mapping.h b/arch/s390/include/asm/dma-mapping.h
index b3fd54d..cb05f5c 100644
--- a/arch/s390/include/asm/dma-mapping.h
+++ b/arch/s390/include/asm/dma-mapping.h
@@ -11,11 +11,13 @@
#define DMA_ERROR_CODE (~(dma_addr_t) 0x0)
-extern struct dma_map_ops s390_dma_ops;
+extern struct dma_map_ops s390_pci_dma_ops;
static inline struct dma_map_ops *get_dma_ops(struct device *dev)
{
- return &s390_dma_ops;
+ if (dev && dev->archdata.dma_ops)
+ return dev->archdata.dma_ops;
+ return &dma_noop_ops;
}
static inline void dma_cache_sync(struct device *dev, void *vaddr, size_t size,
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 7ef12a3..fa41605 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -649,6 +649,7 @@ int pcibios_add_device(struct pci_dev *pdev)
zdev->pdev = pdev;
pdev->dev.groups = zpci_attr_groups;
+ pdev->dev.archdata.dma_ops = &s390_pci_dma_ops;
zpci_map_resources(pdev);
for (i = 0; i < PCI_BAR_COUNT; i++) {
diff --git a/arch/s390/pci/pci_dma.c b/arch/s390/pci/pci_dma.c
index 37505b8..ea39c3f 100644
--- a/arch/s390/pci/pci_dma.c
+++ b/arch/s390/pci/pci_dma.c
@@ -495,7 +495,7 @@ static int __init dma_debug_do_init(void)
}
fs_initcall(dma_debug_do_init);
-struct dma_map_ops s390_dma_ops = {
+struct dma_map_ops s390_pci_dma_ops = {
.alloc = s390_dma_alloc,
.free = s390_dma_free,
.map_sg = s390_dma_map_sg,
@@ -506,7 +506,7 @@ struct dma_map_ops s390_dma_ops = {
.is_phys = 0,
/* dma_supported is unconditionally true without a callback */
};
-EXPORT_SYMBOL_GPL(s390_dma_ops);
+EXPORT_SYMBOL_GPL(s390_pci_dma_ops);
static int __init s390_iommu_setup(char *str)
{
--
2.4.3
^ permalink raw reply related
* [GIT PULL v4 2/3] alpha/dma: use common noop dma ops
From: Christian Borntraeger @ 2015-11-05 20:08 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, Christian Borntraeger, virtualization,
Martin Schwidefsky, Paolo Bonzini, dwmw2, Christoph Hellwig
In-Reply-To: <1446754136-125772-1-git-send-email-borntraeger@de.ibm.com>
Some of the alpha pci noop dma ops are identical to the common ones.
Use them.
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Joerg Roedel <jroedel@suse.de>
---
arch/alpha/kernel/pci-noop.c | 46 ++++----------------------------------------
1 file changed, 4 insertions(+), 42 deletions(-)
diff --git a/arch/alpha/kernel/pci-noop.c b/arch/alpha/kernel/pci-noop.c
index 2b1f4a1..8e735b5e 100644
--- a/arch/alpha/kernel/pci-noop.c
+++ b/arch/alpha/kernel/pci-noop.c
@@ -123,44 +123,6 @@ static void *alpha_noop_alloc_coherent(struct device *dev, size_t size,
return ret;
}
-static void alpha_noop_free_coherent(struct device *dev, size_t size,
- void *cpu_addr, dma_addr_t dma_addr,
- struct dma_attrs *attrs)
-{
- free_pages((unsigned long)cpu_addr, get_order(size));
-}
-
-static dma_addr_t alpha_noop_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
-{
- return page_to_pa(page) + offset;
-}
-
-static int alpha_noop_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
- enum dma_data_direction dir, struct dma_attrs *attrs)
-{
- int i;
- struct scatterlist *sg;
-
- for_each_sg(sgl, sg, nents, i) {
- void *va;
-
- BUG_ON(!sg_page(sg));
- va = sg_virt(sg);
- sg_dma_address(sg) = (dma_addr_t)virt_to_phys(va);
- sg_dma_len(sg) = sg->length;
- }
-
- return nents;
-}
-
-static int alpha_noop_mapping_error(struct device *dev, dma_addr_t dma_addr)
-{
- return 0;
-}
-
static int alpha_noop_supported(struct device *dev, u64 mask)
{
return mask < 0x00ffffffUL ? 0 : 1;
@@ -168,10 +130,10 @@ static int alpha_noop_supported(struct device *dev, u64 mask)
struct dma_map_ops alpha_noop_ops = {
.alloc = alpha_noop_alloc_coherent,
- .free = alpha_noop_free_coherent,
- .map_page = alpha_noop_map_page,
- .map_sg = alpha_noop_map_sg,
- .mapping_error = alpha_noop_mapping_error,
+ .free = dma_noop_free_coherent,
+ .map_page = dma_noop_map_page,
+ .map_sg = dma_noop_map_sg,
+ .mapping_error = dma_noop_mapping_error,
.dma_supported = alpha_noop_supported,
};
--
2.4.3
^ permalink raw reply related
* [GIT PULL v4 1/3] dma: Provide simple noop dma ops
From: Christian Borntraeger @ 2015-11-05 20:08 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, Christian Borntraeger, virtualization,
Martin Schwidefsky, Paolo Bonzini, dwmw2, Christoph Hellwig
In-Reply-To: <1446754136-125772-1-git-send-email-borntraeger@de.ibm.com>
We are going to require dma_ops for several common drivers, even for
systems that do have an identity mapping. Lets provide some minimal
no-op dma_ops that can be used for that purpose.
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Joerg Roedel <jroedel@suse.de>
---
include/linux/dma-mapping.h | 2 ++
lib/Makefile | 1 +
lib/dma-noop.c | 75 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 78 insertions(+)
create mode 100644 lib/dma-noop.c
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index ac07ff0..7912f54 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -66,6 +66,8 @@ struct dma_map_ops {
int is_phys;
};
+extern struct dma_map_ops dma_noop_ops;
+
#define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1))
#define DMA_MASK_NONE 0x0ULL
diff --git a/lib/Makefile b/lib/Makefile
index 13a7c6a..92d6135 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -18,6 +18,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
obj-$(CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS) += usercopy.o
lib-$(CONFIG_MMU) += ioremap.o
lib-$(CONFIG_SMP) += cpumask.o
+lib-$(CONFIG_HAS_DMA) += dma-noop.o
lib-y += kobject.o klist.o
obj-y += lockref.o
diff --git a/lib/dma-noop.c b/lib/dma-noop.c
new file mode 100644
index 0000000..7214564
--- /dev/null
+++ b/lib/dma-noop.c
@@ -0,0 +1,75 @@
+/*
+ * lib/dma-noop.c
+ *
+ * Simple DMA noop-ops that map 1:1 with memory
+ */
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/dma-mapping.h>
+#include <linux/scatterlist.h>
+
+static void *dma_noop_alloc(struct device *dev, size_t size,
+ dma_addr_t *dma_handle, gfp_t gfp,
+ struct dma_attrs *attrs)
+{
+ void *ret;
+
+ ret = (void *)__get_free_pages(gfp, get_order(size));
+ if (ret)
+ *dma_handle = virt_to_phys(ret);
+ return ret;
+}
+
+static void dma_noop_free(struct device *dev, size_t size,
+ void *cpu_addr, dma_addr_t dma_addr,
+ struct dma_attrs *attrs)
+{
+ free_pages((unsigned long)cpu_addr, get_order(size));
+}
+
+static dma_addr_t dma_noop_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ return page_to_phys(page) + offset;
+}
+
+static int dma_noop_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ int i;
+ struct scatterlist *sg;
+
+ for_each_sg(sgl, sg, nents, i) {
+ void *va;
+
+ BUG_ON(!sg_page(sg));
+ va = sg_virt(sg);
+ sg_dma_address(sg) = (dma_addr_t)virt_to_phys(va);
+ sg_dma_len(sg) = sg->length;
+ }
+
+ return nents;
+}
+
+static int dma_noop_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+ return 0;
+}
+
+static int dma_noop_supported(struct device *dev, u64 mask)
+{
+ return 1;
+}
+
+struct dma_map_ops dma_noop_ops = {
+ .alloc = dma_noop_alloc,
+ .free = dma_noop_free,
+ .map_page = dma_noop_map_page,
+ .map_sg = dma_noop_map_sg,
+ .mapping_error = dma_noop_mapping_error,
+ .dma_supported = dma_noop_supported,
+};
+
+EXPORT_SYMBOL(dma_noop_ops);
--
2.4.3
^ permalink raw reply related
* [GIT PULL v4 0/3] dma and virtio prep patches
From: Christian Borntraeger @ 2015-11-05 20:08 UTC (permalink / raw)
To: Andy Lutomirski
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, Christian Borntraeger, virtualization,
Martin Schwidefsky, Paolo Bonzini, dwmw2, Christoph Hellwig
Andy,
to make it obvious which version is the latest, here is a branch
The following changes since commit 6a13feb9c82803e2b815eca72fa7a9f5561d7861:
Linux 4.3 (2015-11-01 16:05:25 -0800)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/borntraeger/linux.git dma
for you to fetch changes up to fc7f9754db6ce0c12281da4055281f731d36bdee:
s390/dma: Allow per device dma ops (2015-11-05 21:02:40 +0100)
----------------------------------------------------------------
Christian Borntraeger (3):
dma: Provide simple noop dma ops
alpha/dma: use common noop dma ops
s390/dma: Allow per device dma ops
arch/alpha/kernel/pci-noop.c | 46 ++---------------------
arch/s390/Kconfig | 7 +---
arch/s390/include/asm/device.h | 6 ++-
arch/s390/include/asm/dma-mapping.h | 6 ++-
arch/s390/pci/pci.c | 1 +
arch/s390/pci/pci_dma.c | 4 +-
include/linux/dma-mapping.h | 2 +
lib/Makefile | 1 +
lib/dma-noop.c | 75 +++++++++++++++++++++++++++++++++++++
9 files changed, 96 insertions(+), 52 deletions(-)
create mode 100644 lib/dma-noop.c
^ permalink raw reply
* Re: [PATCH 3/3] s390/dma: Allow per device dma ops
From: Andy Lutomirski @ 2015-11-05 19:22 UTC (permalink / raw)
To: Christian Borntraeger, Cornelia Huck
Cc: linux-s390, Joerg Roedel, KVM, Michael Tsirkin, benh,
Sebastian Ott, virtualization, Martin Schwidefsky, Paolo Bonzini,
dwmw2, Christoph Hellwig
In-Reply-To: <563B2275.3010406@de.ibm.com>
On 11/05/2015 01:33 AM, Christian Borntraeger wrote:
> Am 03.11.2015 um 13:26 schrieb Cornelia Huck:
>> On Tue, 3 Nov 2015 12:54:39 +0100
>> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
>>
>>> As virtio-ccw now has dma ops, we can no longer default to the PCI ones.
>>> Make use of dev_archdata to keep the dma_ops per device. The pci devices
>>> now use that to override the default, and the default is changed to use
>>> the noop ops for everything that is not PCI. To compile without PCI
>>> support we also have to enable the DMA api with virtio.
>>
>> Not only with virtio, but generally, right?
>
> Yes, will update the patch description.
>>
>>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>> Reviewed-by: Joerg Roedel <jroedel@suse.de>
>>> Acked-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
>>> ---
>>> arch/s390/Kconfig | 3 ++-
>>> arch/s390/include/asm/device.h | 6 +++++-
>>> arch/s390/include/asm/dma-mapping.h | 6 ++++--
>>> arch/s390/pci/pci.c | 1 +
>>> arch/s390/pci/pci_dma.c | 4 ++--
>>> 5 files changed, 14 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
>>> index 1d57000..04f0e02 100644
>>> --- a/arch/s390/Kconfig
>>> +++ b/arch/s390/Kconfig
>>> @@ -113,6 +113,7 @@ config S390
>>> select GENERIC_FIND_FIRST_BIT
>>> select GENERIC_SMP_IDLE_THREAD
>>> select GENERIC_TIME_VSYSCALL
>>> + select HAS_DMA
>>> select HAVE_ALIGNED_STRUCT_PAGE if SLUB
>>> select HAVE_ARCH_AUDITSYSCALL
>>> select HAVE_ARCH_EARLY_PFN_TO_NID
>>> @@ -124,6 +125,7 @@ config S390
>>> select HAVE_CMPXCHG_DOUBLE
>>> select HAVE_CMPXCHG_LOCAL
>>> select HAVE_DEBUG_KMEMLEAK
>>> + select HAVE_DMA_ATTRS
>>> select HAVE_DYNAMIC_FTRACE
>>> select HAVE_DYNAMIC_FTRACE_WITH_REGS
>>> select HAVE_FTRACE_MCOUNT_RECORD
>>> @@ -580,7 +582,6 @@ config QDIO
>>>
>>> menuconfig PCI
>>> bool "PCI support"
>>> - select HAVE_DMA_ATTRS
>>> select PCI_MSI
>>> help
>>> Enable PCI support.
>>
>> Hm. Further down in this file, there's
>>
>> config HAS_DMA
>> def_bool PCI
>> select HAVE_DMA_API_DEBUG
>>
>> Should we maybe select HAVE_DMA_API_DEBUG above, drop the HAS_DMA
>> config option and rely on not defining NO_DMA instead?
>
> Hmm, yes. That would simplify things a lot. Right now we include
> lib/Kconfig (which defines HAS_DMA) and define it ourselfes in
> arch/s390/Kconfig. WHoever comes first wins. Adding a select statement
> would make this even more complicated.
>
> Andy, I will simply send you a respin of this patch.
>
I'm slightly concerned that I'm going to screw this all up and apply the
wrong version. Could you resend the whole series with git format-patch
-vN for some appropriate N (or similar)?
Thanks,
Andy
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox