Modularizing IOMMUs (devel/iommu-0.4)

Modularizing IOMMUs (devel/iommu-0.4)

[Konrad Rzeszutek Wilk]

Hey Peter, Tomo-san, Jan and Jeremy, and LKML:

I was wondering if I could run this idea by you guys. One of the things that
Peter proposed was in some ways to "modularize" the IOMMUs (and potentially
later on use that framework to "modularize" other built-in components). For details:
http://lkml.org/lkml/2010/8/2/282 and the follow-on threads.

If you want to look at code right away, I've some beginnings at:

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git devel/iommu-0.4
(look at patches past swiotlb-xen: Remove the unnecessary EXPORT_SYMBOL_GPL macros)

The 10,000 ft view is that during bootup, we want to jettison those IOMMUs
that are built-in, but actually aren't used b/c the machine does not have it.
Once it is identified which one we don't want, we mark those sections as available
for the kernel memory subsystem and we can shed some kilobytes.

So the first step is to identify the functions for the IOMMUs. And also
group them tightly together. The path I pursued was to stick all of the
executable sections in clearly identifiable sections that can be
easily identified.

The first idea I had that was to define in vmlinux.ld.S this little macro:

.PAGE_ALIGN(PAGE_SIZE) = .;
.iommu_text: {
   PROVIDE_HIDDEN(__iommu_swiotlb_start) := .;
   swiotlb*.o(*.text)
   PROVIDE_HIDDEN(__iommu_gart_start) := .;
   pci-gart_64.o(*.text)
   .. 
} = 0x9090

So essentially renaming .text section of specific object files to be 
instead .iommu_text section.

Well, that did not work as during linking GCC makes the object files in /tmp, so
instead of pci-gart_64.o we get /tmp/f892343.o. So question #1: Do you know of any
mechanism to enforce the naming of the object files? Or perhaps another way
to identify in the linker script file the <object-file><.text section> ?

Another way, less elegant was to manually enforce each function to be stuck
in the .iommu_text section. For that I made a macro: __iommu that would
force the function to be stuck in section specific for the IOMMU. So all
functions in pci-gart_64.c would be funneled in .iommu.gart.text. For calgary:
.iommu.calgary.text., and so on. This is accomplished by having at the beginning
of the the file the name of the IOMMU section, as so:

#define IOMMU_MODULE "gart"

And all of the functions would get stuck in .iommu.gart.text. This actually
worked out (hadn't tried to boot yet) and based on the compilation these are
the sizes:

1 .iommu.gart.text 00000be4  ..sip.  0000000001436fe2  00636fe2  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .iommu.calgary.text 00000c94  ..snip..  0000000001437bc6  00637bc6  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  3 .iommu.amdvi.text 00002d74  ..snip..  000000000143885a  0063885a  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  4 .iommu.dmar.text 00004ff7  ..snip..  000000000143b5ce  0063b5ce  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 .iommu.xen-swiotlb.text 000005f3  ..snip.  00000000014405c5  006405c5  2**0

For those that can't read nativly hexadecimal, here are the sizes:
  Xen-SWIOTLB: 1523
  GART: 3044 bytes
  Calgary: 3220 bytes
  AMD V-I: 11636
  Intel DMAR: 20471.

Roughly it comes out to 40kB of executable. I haven't run the math, but
estimating this it looks as if on AMD machine we can dump about eight pages
(VT-D, Calgary, Xen-SWIOTBL), while on Intel with VT-D we can dump six pages
(GART, Calgary, AMD VI, Xen-SWIOTLB), and on Intel with non-VT, around eleven
pages (all IOMMUs, just stitch with the bare-metal SWIOTLB). Each .iommu.<x>.text
would be PAGE_ALIGN-ed of course.

The next stage (not yet written) would have to:
 - scan the EXPORT_SYMBOL definitions and see if any of them map within
   the sections that are to be discarded. If so, either neuter them
   or don't discard the IOMMU.
 - Mark the discarded sections to be freed. And utilize the framework for
   doing so.
 - Give the same treatment as the _stext -> _etext sections have. So
   setting of the various flags, etc.

But the Question #2 I have is: is it worth it? We can save on average around
eight pages (6+7+11/3), which is not exactly earth-shattering. This could be
become more generic so in the long run it could jettison more code and not
just IOMMUs, but lets start small first.

And then Question  #3): Is there a better way?

Re: Modularizing IOMMUs (devel/iommu-0.4)

[H. Peter Anvin]

On 10/20/2010 01:20 PM, Konrad Rzeszutek Wilk wrote:
> Hey Peter, Tomo-san, Jan and Jeremy, and LKML:
> 
> I was wondering if I could run this idea by you guys. One of the things that
> Peter proposed was in some ways to "modularize" the IOMMUs (and potentially
> later on use that framework to "modularize" other built-in components). For details:
> http://lkml.org/lkml/2010/8/2/282 and the follow-on threads.
> 
> If you want to look at code right away, I've some beginnings at:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git devel/iommu-0.4
> (look at patches past swiotlb-xen: Remove the unnecessary EXPORT_SYMBOL_GPL macros)
> 
> The 10,000 ft view is that during bootup, we want to jettison those IOMMUs
> that are built-in, but actually aren't used b/c the machine does not have it.
> Once it is identified which one we don't want, we mark those sections as available
> for the kernel memory subsystem and we can shed some kilobytes.
> 
> So the first step is to identify the functions for the IOMMUs. And also
> group them tightly together. The path I pursued was to stick all of the
> executable sections in clearly identifiable sections that can be
> easily identified.
> 

I think a much better way to do this is something I discussed with Linus
at least 12 years ago, which is allow modules to be built into the
kernel, but still be unloadable using the actual module mechanism.  This
means they are part of the full kernel image and therefore immediately
available, but appear to the module system as an already-installed
module.  Then we can unload modules to free up space.

This would be a handy thing to have in general, and would simplify this
kind of stuff tremendously.

	-hpa

Re: Modularizing IOMMUs (devel/iommu-0.4)

[Jan Beulich]

>>> On 20.10.10 at 22:20, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> Another way, less elegant was to manually enforce each function to be stuck
> in the .iommu_text section. For that I made a macro: __iommu that would
> force the function to be stuck in section specific for the IOMMU. So all
> functions in pci-gart_64.c would be funneled in .iommu.gart.text. For 
> calgary:
> .iommu.calgary.text., and so on. This is accomplished by having at the 
> beginning
> of the the file the name of the IOMMU section, as so:
> 
> #define IOMMU_MODULE "gart"
> 
> And all of the functions would get stuck in .iommu.gart.text.

For this particular approach - did you consider using objcopy's
--rename-section option?

> And then Question  #3): Is there a better way?

Generally I like hpa's suggestion of using pre-loaded but unloadable
modules much better, not the least because in the past I had seen
potential uses for such a mechanism in other places of the kernel.

Jan

Re: Modularizing IOMMUs (devel/iommu-0.4)

[Konrad Rzeszutek Wilk]

On Fri, Oct 22, 2010 at 08:52:41AM +0100, Jan Beulich wrote:
> >>> On 20.10.10 at 22:20, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > Another way, less elegant was to manually enforce each function to be stuck
> > in the .iommu_text section. For that I made a macro: __iommu that would
> > force the function to be stuck in section specific for the IOMMU. So all
> > functions in pci-gart_64.c would be funneled in .iommu.gart.text. For 
> > calgary:
> > .iommu.calgary.text., and so on. This is accomplished by having at the 
> > beginning
> > of the the file the name of the IOMMU section, as so:
> > 
> > #define IOMMU_MODULE "gart"
> > 
> > And all of the functions would get stuck in .iommu.gart.text.
> 
> For this particular approach - did you consider using objcopy's
> --rename-section option?

<goes off to look at that>
> 
> > And then Question  #3): Is there a better way?
> 
> Generally I like hpa's suggestion of using pre-loaded but unloadable
> modules much better, not the least because in the past I had seen
> potential uses for such a mechanism in other places of the kernel.

<nods> It does sound like the right option. 12 years ago it was suggested,
so.. how come nobody worked on it in the past? Were there some epic battles?

Re: Modularizing IOMMUs (devel/iommu-0.4)

[H. Peter Anvin]

On 10/22/2010 06:54 AM, Konrad Rzeszutek Wilk wrote:
> <nods>  It does sound like the right option. 12 years ago it was suggested,
> so.. how come nobody worked on it in the past? Were there some epic battles?

No, quite the opposite... deafening silence.  However, 12 years ago the 
module infrastructure looked very different than it does today, with 
large chunks implemented in user space.  These days the module 
infrastructure is almost entirely in the kernel, which should make it a 
lot easier to implement.

I thus suspect that it was just too hard when it was first proposed, and 
then got forgotten about.

	-hpa