[PATCH v2 0/2] sparc: IOMMU pools for better scalability

[PATCH v2 0/2] sparc: IOMMU pools for better scalability

[Sowmini Varadhan]

V2 version ofthis patch-set incorporates the feedback from
  http://marc.info/?l=linux-sparc&m\x142527471211108&w=2
The sparc specific fields have been moved  off into iommu_64.h, 
into struct iommu_sparc, leaving a minimalist struct iommu_table
in the common functions..

In addition to those changes, there are some additional enhancements to
improve the generality of this library that were based on inspection of other 
iommu use-case such as LDC, which have a much smaller map size 
(8k, vs 256k for pci_sun4v.c), and some other variations. These 
change are:

  (a) When the number of map entries is small, it may not be desirable
      to have too many pools: make npools an input parameter to the library
  (b) the cookie-to-index mapping function could be more complex,
      depending on the device. So added a ->cookie_to_index() indirection
      in the iommu_tbl_ops
  (c) Depending on the device, the span-boundary check may not be needed. 
      Allow for optimizations of this check.


Sowmini (2):
  Break up monolithic iommu table/lock into finer graularity pools and
    lock
  Make sparc64 use scalable lib/iommu-common.c functions

 arch/sparc/include/asm/iommu_64.h |   19 ++++
 arch/sparc/kernel/pci_impl.h      |    7 +-
 arch/sparc/kernel/pci_sun4v.c     |  217 +++++++++++++++++--------------------
 include/linux/iommu-common.h      |   57 ++++++++++
 lib/Makefile                      |    2 +-
 lib/iommu-common.c                |  217 +++++++++++++++++++++++++++++++++++++
 6 files changed, 402 insertions(+), 117 deletions(-)
 create mode 100644 include/linux/iommu-common.h
 create mode 100644 lib/iommu-common.c

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[David Miller]

From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Date: Tue, 3 Mar 2015 09:22:02 -0500

> 
> V2 version ofthis patch-set incorporates the feedback from
>   http://marc.info/?l=linux-sparc&m\x142527471211108&w=2
> The sparc specific fields have been moved  off into iommu_64.h, 
> into struct iommu_sparc, leaving a minimalist struct iommu_table
> in the common functions..

Still have some sparc specific stuff in the generic header file.

For example, IOMMU_NUM_CTXS.

I don't see anything using cookie_to_index.

Furthermore, who so many TABS in the dma_4v_iommu_ops initializer?

Next, canonically "linux/" headers are included before "asm/" ones.

Please don't put "ifdef notyet" kind of stuff into your changes,
that is not appropriate.  In any event, I want you to translate all
of the PCI DMA implementations, because it is the only way you can
show me that your abstractions are sufficient.

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[Sowmini Varadhan]

On (03/03/15 17:15), David Miller wrote:
> 
> For example, IOMMU_NUM_CTXS.
> 
> I don't see anything using cookie_to_index.
> 
> Furthermore, who so many TABS in the dma_4v_iommu_ops initializer?
> 
> Next, canonically "linux/" headers are included before "asm/" ones.

sure I can fix all of the above for v3. The cookie_to_index
comes from ldc_iommu conversion, which I had not added to the set
because it does nothing whatsoever for ldc perf, where we are only
dealing with an 8k map size, though I see the value of less code
duplication.

I'll add ldc to the v3 patch-set.

However..

> Please don't put "ifdef notyet" kind of stuff into your changes,
> that is not appropriate.  In any event, I want you to translate all
> of the PCI DMA implementations, because it is the only way you can
> show me that your abstractions are sufficient.

That means I have to fix all kinds of dinosaurs in the neighborhood,
things like pci_fire, pci_schizo, pci_psycho, sun4u. Some of this
hardware was EOL'ed a long time ago. And the last time I remember
using a sun4u was back in 2008 (and even then, it was really not
very useful for any sort of non-trivial SMP/perf development).

But whatever floats your boat. 

Just want to note that this will no longer be a simple 2-part patch set.

--Sowmini

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[David Miller]

From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Date: Wed, 4 Mar 2015 14:10:17 -0500

> On (03/03/15 17:15), David Miller wrote:
>> Please don't put "ifdef notyet" kind of stuff into your changes,
>> that is not appropriate.  In any event, I want you to translate all
>> of the PCI DMA implementations, because it is the only way you can
>> show me that your abstractions are sufficient.
> 
> That means I have to fix all kinds of dinosaurs in the neighborhood,
> things like pci_fire, pci_schizo, pci_psycho, sun4u. Some of this
> hardware was EOL'ed a long time ago. And the last time I remember
> using a sun4u was back in 2008 (and even then, it was really not
> very useful for any sort of non-trivial SMP/perf development).
> 
> But whatever floats your boat. 

It's a conversion that has to be done anyways, and otherwise you'll
have no idea whether your abstractions are sufficient or not.

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[David Ahern]

On 3/4/15 12:15 PM, David Miller wrote:
> From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
> Date: Wed, 4 Mar 2015 14:10:17 -0500
>
>> On (03/03/15 17:15), David Miller wrote:
>>> Please don't put "ifdef notyet" kind of stuff into your changes,
>>> that is not appropriate.  In any event, I want you to translate all
>>> of the PCI DMA implementations, because it is the only way you can
>>> show me that your abstractions are sufficient.
>>
>> That means I have to fix all kinds of dinosaurs in the neighborhood,
>> things like pci_fire, pci_schizo, pci_psycho, sun4u. Some of this
>> hardware was EOL'ed a long time ago. And the last time I remember
>> using a sun4u was back in 2008 (and even then, it was really not
>> very useful for any sort of non-trivial SMP/perf development).
>>
>> But whatever floats your boat.
>
> It's a conversion that has to be done anyways, and otherwise you'll
> have no idea whether your abstractions are sufficient or not.

Doesn't that run the risk of breaking older platforms, which cannot 
realistically be tested? Why not refactor the code such that old 
platforms are not touched and focus on abstractions for modern h/w?

David

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[David Miller]

From: David Ahern <dsahern@gmail.com>
Date: Wed, 04 Mar 2015 12:32:32 -0700

> On 3/4/15 12:15 PM, David Miller wrote:
>> From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
>> Date: Wed, 4 Mar 2015 14:10:17 -0500
>>
>>> On (03/03/15 17:15), David Miller wrote:
>>>> Please don't put "ifdef notyet" kind of stuff into your changes,
>>>> that is not appropriate.  In any event, I want you to translate all
>>>> of the PCI DMA implementations, because it is the only way you can
>>>> show me that your abstractions are sufficient.
>>>
>>> That means I have to fix all kinds of dinosaurs in the neighborhood,
>>> things like pci_fire, pci_schizo, pci_psycho, sun4u. Some of this
>>> hardware was EOL'ed a long time ago. And the last time I remember
>>> using a sun4u was back in 2008 (and even then, it was really not
>>> very useful for any sort of non-trivial SMP/perf development).
>>>
>>> But whatever floats your boat.
>>
>> It's a conversion that has to be done anyways, and otherwise you'll
>> have no idea whether your abstractions are sufficient or not.
> 
> Doesn't that run the risk of breaking older platforms, which cannot
> realistically be tested? Why not refactor the code such that old
> platforms are not touched and focus on abstractions for modern h/w?

Are you kidding me?  I have a sun4u sitting right under my desk, and
several more (all IOMMU variants) at my Seattle house, I'll test it
for you.

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Wed, 04 Mar 2015 15:35:38 -0500 (EST)

> Are you kidding me?  I have a sun4u sitting right under my desk, and
> several more (all IOMMU variants) at my Seattle house, I'll test it
> for you.

Furthermore, people like Meelis Roos are here testing sun4u based
sparc64 systems all the time.

People do actively use these systems, report bugs, and will help
us work on any regressions.

So please don't use this as an excuse to not implement things
properly.  Just because Sun EOL'd the machine doesn't mean it's
not actively in use and supported by Linus.

Re: [PATCH v2 0/2] sparc: IOMMU pools for better scalability

[Sowmini Varadhan]

On (03/04/15 15:37), David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Wed, 04 Mar 2015 15:35:38 -0500 (EST)
> 
> > Are you kidding me?  I have a sun4u sitting right under my desk, and
> > several more (all IOMMU variants) at my Seattle house, I'll test it
> > for you.

Thanks for the offer, and I may take you up on it.
But at the moment I'm trying to see what we have in-house, so let me
pursue those chains first, and then ping this list if we need
additional coverage.

> Furthermore, people like Meelis Roos are here testing sun4u based
> sparc64 systems all the time.
> 
> People do actively use these systems, report bugs, and will help
> us work on any regressions.
> 
> So please don't use this as an excuse to not implement things

Understood.

--Sowmini