Re: [PATCH 2.6.34] ehci-hcd: add option to enable 64-bit DMA support

[David Brownell <david-b@pacbell.net>]

On Thursday 18 February 2010, Robert Hancock wrote:
> >
> > But we disabled it on purpose, because of problems, do we want those
> > problems again?
> 
> AFAICS, it was disabled because of problems with kernel code, not with
> hardware (and it appears the issue was with the code that detected the
> expanded DMA mask in the USB device driver code, not the HCD driver).
> CCing David Brownell who may know more.

That's a good summary of the high points.  Testing was potentially an
issue, but it never quite got that far.  So I have no idea if there are
systems where EHCI advertises 64-bit DMA support but that support is
broken (e.g. "Yet Another Hardware Mechanism MS-Windows Ignores", so that
only Linux would ever trip over the relevant BIOS breakage).

I won't attempt to go into details, but I recall a few basic issues:

 * Not all clients or implementors of the "dma mask" mechanism agreed
   on what it was supposed to achieve.  Few, for example, really used
   it as a mask ... and it rarely affects allocation of buffers that
   will later get used for DMA.

 * Confusing semantics for the various types of DMA restriction which
   hardware may impose, and upper layers in driver stacks would thus
   need (in some cases) to cope with.

 * How to pass such restrictions up the driver stack ... as for example
   that NETIF_* flag.  ISTR there was some block layer issue too, but
   at this remove I can't remember any details at all.  (If networking
   and the block layer can use 64-bit DMA, I can't imagine many other
   subsystems would deliver wins as big.)  For example, how would one
   pass up the knowledge that a driver for a particular USB peripheral
   across a few hubs) can do DMA to/from address 0x1234567890abcdef, but
   the same driver can't do that for an otherwise identical peripheral
   connected through a different HCD?

 * There were probably a few PCI-specific issues too.  I don't think
   at that time there were many users of 64-bit DMA which weren't
   specific to PCI.  Wanting to use the generic DMA calls for such
   stuff wasn't really done back then.  But ... the USB stack uses
   the generic calls, and drivers sitting on top of usbcore (and its
   tightly coupled HCDs) will never issue PCI-specific calls, since
   they need to work on systems that don't even have PCI.

I basically think that if the controller can do 64-bit DMA, it should
be enabling it by default ... assuming the software stack can handle
that.  (What would be the benefit of adding needless restrictions,
and making systems needlessly apply bounce buffering.)  So while I'd
like to see the 64-bit DMA working, it should IMO be done without any
options to cause trouble/confusion.

But at that time it wasn't straightforward to manage 64-bit DMA except
in the very lowest level PCI drivers.  That is, EHCI could do it ...
but driver layers on top of it had no good way to do their part.  For
example, when they manage DMA mappings themselves.)

- Dave