Re: [RFC PATCH 0/3] allow fallback to swiotlb on hw iommu init failures

[Chris Wright <chrisw@sous-sol.org>]

* Andi Kleen (andi@firstfloor.org) wrote:
> Chris Wright <chrisw@sous-sol.org> writes:
> 
> > This short series gives us the ability to allocate the swiotlb and then
> > conditionally free it if we discover it isn't needed.  This allows us to
> > put swiotlb to use when the hw iommu fails to initialize properly.
> >
> > This needs some changes to the bootmem allocator to give the ability to
> > free reserved bootmem directly to the page allocator after bootmem is
> > torn down.
> 
> You forgot to state what motivated you to that change?

I thought I did ;-)  Here's another more verbose attempt:

The HW IOMMU, for example Intel VT-d, may fail initialization (typically
due to BIOS bugs).  In that case the existing fallback is nommu, which is
clearly insufficient for many boxen which need some bounce buffering if
there is no HW IOMMU.  The problem is that at the point of this failure
the decision to allocate and initialize the swiotlb has long since past.

There are 4 ways to handle this:

1) Give up and panic the box.  This is not a user friendly option since
the box will boot and function fine (minus any isolation gained from the
HW IOMMU) if there is either not much phys mem or an swiotlb.

2) Do the discovery that causes the initialization failure earlier so
that HW IOMMU detection fails.  Compilcated by the HW IOMMU's use of the
run time env that includes a real allocator and PCI enumeration, etc.

3) Allow the swiotlb to be allocated later in pci_iommu_init() instead
of early in pci_iommu_alloc(), IOW don't use bootmem for the swiotlb.
This is possible, although it will hit 2 limitations.  The first is any
possible fragmentation that limits the availability of a 64M region
by the time this runs.  The second is that x86 has MAX_ORDER of 11,
so at most we can allocate a 4M region from the page allocator which is
inusufficient for swiotlb.

4) Allow the swiotlb to be allocated in pci_iommu_alloc(), but not
initialized until pci_iommu_init().  This allows using bootmem allocator
to reserve a nice large contiguous chunk of memory, but requires some
way to give that memory back in the case that a HW IOMMU is properly
both detected and initialized (else it'd be a 64M memory leak for
effectively all HW IOMMU enabled boxen).

This series implements the fourth option.

thanks,
-chris