[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

James Bottomley wrote:

> On Thu, 2003-05-15 at 08:17, Colin Ngam wrote:
> > Anyway, to make a long story short, we like to "extend" these
> > interfaces to return DMA addresses of 64 bits if and when the
> > driver has the full capabality to support DAC - via it's
> > DMA mask capability setting.
> >
> > This extention will not break current usages of these interface,
> > and as driver changes, this extension will enable platforms to
> > support all of it's Physical Memory via the above 2 Interfaces
> > without exhausting limited system mapping resources.
>
> The intent of the dma_alloc_.. was to follow the PCI spec (i.e. lower 32
> bits).  However, it does take the GFP_ flags, and thus could be adjusted
> to use the region specifier portion of those.

Hi James,

Do you mean to use the flag parameter to advice that we can return a
64Bit DMA Address?  What is a region specifier?

Thanks.

colin


>
>
> We're currently debating this very issue, since it came up in the
> context of the 2.6 must fix list.
>
> James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[James Bottomley]

On Thu, 2003-05-15 at 08:17, Colin Ngam wrote:
> Anyway, to make a long story short, we like to "extend" these
> interfaces to return DMA addresses of 64 bits if and when the
> driver has the full capabality to support DAC - via it's
> DMA mask capability setting.
> 
> This extention will not break current usages of these interface,
> and as driver changes, this extension will enable platforms to
> support all of it's Physical Memory via the above 2 Interfaces
> without exhausting limited system mapping resources.

The intent of the dma_alloc_.. was to follow the PCI spec (i.e. lower 32
bits).  However, it does take the GFP_ flags, and thus could be adjusted
to use the region specifier portion of those.

We're currently debating this very issue, since it came up in the
context of the 2.6 must fix list.

James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

David Mosberger wrote:

> >>>>> On Thu, 15 May 2003 08:17:58 -0500, Colin Ngam <cngam@sgi.com> said:
>
>   Colin> Anyway, to make a long story short, we like to "extend" these
>   Colin> interfaces to return DMA addresses of 64 bits if and when the
>   Colin> driver has the full capabality to support DAC - via it's DMA
>   Colin> mask capability setting.
>
>   Colin> This extention will not break current usages of these
>   Colin> interface, and as driver changes, this extension will enable
>   Colin> platforms to support all of it's Physical Memory via the
>   Colin> above 2 Interfaces without exhausting limited system mapping
>   Colin> resources.
>
> James and I have been discussing this very point for the last week or
> so.  My preferred solution is to add an explicit dma-mask argument to
> coherent/non-coherent allocators, but James prefers to do this via a
> new GFP flag (GFP_DMA32 or some such).

Hi David,

I actually like the flag as a parameter because it allows platform specific
extensions to pass in platform specific capabilities that can be ignored
by platform that does not support these capabilities.

I do not know whether using the same set of flags i.e.GFP_XXX would overload
it's meanings to other MM routines but I really like the a flag parameter, that
will provide capability for Platform specific hints.

Thanks.

Colin

>
>
>         --david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David Mosberger]

>>>>> On Thu, 15 May 2003 08:17:58 -0500, Colin Ngam <cngam@sgi.com> said:

  Colin> Anyway, to make a long story short, we like to "extend" these
  Colin> interfaces to return DMA addresses of 64 bits if and when the
  Colin> driver has the full capabality to support DAC - via it's DMA
  Colin> mask capability setting.

  Colin> This extention will not break current usages of these
  Colin> interface, and as driver changes, this extension will enable
  Colin> platforms to support all of it's Physical Memory via the
  Colin> above 2 Interfaces without exhausting limited system mapping
  Colin> resources.

James and I have been discussing this very point for the last week or
so.  My preferred solution is to add an explicit dma-mask argument to
coherent/non-coherent allocators, but James prefers to do this via a
new GFP flag (GFP_DMA32 or some such).

	--david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[James Bottomley]

On Thu, 2003-05-15 at 10:06, David Mosberger wrote:
> James and I have been discussing this very point for the last week or
> so.  My preferred solution is to add an explicit dma-mask argument to
> coherent/non-coherent allocators, but James prefers to do this via a
> new GFP flag (GFP_DMA32 or some such).

This is still being hashed out.  It looks like adding a the _mask
variants may still be a viable runner for 2.6.  If it turns out not to
be, the GFP_DMA32 will almost definitely be there.

James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David Mosberger]

>>>>> On Thu, 15 May 2003 10:05:28 -0500, Colin Ngam <cngam@sgi.com> said:

  Colin> I do not know whether using the same set of flags i.e.GFP_XXX
  Colin> would overload it's meanings to other MM routines but I
  Colin> really like the a flag parameter, that will provide
  Colin> capability for Platform specific hints.

Overloading the GFP flags with platform-specific DMA stuff is certainly
not an option.

If you want to argue for yet-another-flags argument, go ahead and do
so.  Be prepared for having to make a VERY strong argument...

	--david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David Mosberger]

>>>>> On 15 May 2003 10:12:48 -0500, James Bottomley <James.Bottomley@HansenPartnership.com> said:

  James> On Thu, 2003-05-15 at 10:06, David Mosberger wrote:
  >> James and I have been discussing this very point for the last
  >> week or so.  My preferred solution is to add an explicit dma-mask
  >> argument to coherent/non-coherent allocators, but James prefers
  >> to do this via a new GFP flag (GFP_DMA32 or some such).

  James> This is still being hashed out.  It looks like adding a the
  James> _mask variants may still be a viable runner for 2.6.

That would be good.

	--david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

David Mosberger wrote:

> >>>>> On 15 May 2003 10:12:48 -0500, James Bottomley <James.Bottomley@HansenPartnership.com> said:
>
>   James> On Thu, 2003-05-15 at 10:06, David Mosberger wrote:
>   >> James and I have been discussing this very point for the last
>   >> week or so.  My preferred solution is to add an explicit dma-mask
>   >> argument to coherent/non-coherent allocators, but James prefers
>   >> to do this via a new GFP flag (GFP_DMA32 or some such).
>
>   James> This is still being hashed out.  It looks like adding a the
>   James> _mask variants may still be a viable runner for 2.6.

David,

Can you provide a little detail on how the _mask vriant is used and
passed into the call?

Thanks.

colin

>
>
> That would be good.
>
>         --david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

David Mosberger wrote:

> >>>>> On Thu, 15 May 2003 10:05:28 -0500, Colin Ngam <cngam@sgi.com> said:
>
>   Colin> I do not know whether using the same set of flags i.e.GFP_XXX
>   Colin> would overload it's meanings to other MM routines but I
>   Colin> really like the a flag parameter, that will provide
>   Colin> capability for Platform specific hints.
>
> Overloading the GFP flags with platform-specific DMA stuff is certainly
> not an option.
>
> If you want to argue for yet-another-flags argument, go ahead and do
> so.  Be prepared for having to make a VERY strong argument...

David,

I was argueing for using the flag arguement and define it's own set
of values versus using the GFP_XXX defines.  And not an additional
flag arguement.

Thanks.

colin

>
>
>         --david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 08:06:04AM -0700, David Mosberger wrote:
> James and I have been discussing this very point for the last week or
> so.  My preferred solution is to add an explicit dma-mask argument to
> coherent/non-coherent allocators, but James prefers to do this via a
> new GFP flag (GFP_DMA32 or some such).

For Streaming DMA, I'd rather stick with the current scheme.
The main reason is 64-bit addressing support will be hard coded
in the each driver as appropriate.  ie it's not a parameter than
needs to be passed in with each call to dma_map_single().

I'm assuming there is some performance impact to adding another argument
to the existing dma_* calls. (more instructions and another register
consumed). That might be a bad assumption for ia64.

Since Consistent DMA allocations aren't in the performance path normally,
implementation details here don't matter as much (to me).


One of the original goals for my talk at OLS was to propose extensions
to 2.4.x to provide "DMA Hints". This would help chipset specific DMA
support pick "optimal" settings for DMA operations.  I was thinking of
redefining "direction" parameter to be "hints".  It would continue to
support Read or Write "hint" but then use upper bits as flags for other
things (don't want to completely spoil my talk).  But I can't justify
the change based on the performance differences I've measured to date
and probably will not offer a proposal.  I haven't gotten to more complex
testing (only single card tests so far) and HPUX certainly saw performance
benefits in a similar scheme on PARISC (using PCI, not PCI-X).

hth,
grant

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David Mosberger]

>>>>> On Thu, 15 May 2003 12:06:22 -0700, Grant Grundler <iod00d@hp.com> said:

  Grant> For Streaming DMA, I'd rather stick with the current scheme.

Nobody said anything about changing streaming DMA.  The discussion was
purely about coherent/non-coherent allocations.

	--david

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 08:17:58AM -0500, Colin Ngam wrote:
> Hi,
> 
> The Altix PCI-X Bridge chipset expects devices that runs in
> PCI-X mode to always be 64Bit DMA Address capable(DAC).  This
> requirement is directly in conflict with both the
> pci_alloc_consistent() and dma_alloc_coherent() interfaces.

I don't (yet) see the conflict.
Can't the 32-bit address can be extended to 64-bits as
described in the dma_alloc_coherent() interface?
Are you running out of 32-bit memory?


> Anyway, to make a long story short, we like to "extend" these
> interfaces to return DMA addresses of 64 bits if and when the
> driver has the full capabality to support DAC - via it's
> DMA mask capability setting.

jbarnes@sgi.com proposed this before.  On April 23, 2002, davem wrote:
| %99 of PCI chips out there do not support DAC addressing for things
| like descriptor tables etc.  So it's not a matter of "well" it's
| a matter of "at all".

If you want the complete thread, check ia64-linux 2002 mail archives.

hth,
grant

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 12:16:37PM -0700, Grant Grundler wrote:
> Can't the 32-bit address can be extended to 64-bits as
> described in the dma_alloc_coherent() interface?
> Are you running out of 32-bit memory?

I found the answer to my question here:
https://lists.linuxia64.org/archives//linux-ia64/2002-April/003418.html

> If you want the complete thread, check ia64-linux 2002 mail archives.

/me happened to take his own advice this time...

grant

ps. "cygnus.com" is a dead domain. IIRC, rth is at redhat now. trying that.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 08:17:58 -0500
   
   Anyway, to make a long story short, we like to "extend" these
   interfaces to return DMA addresses of 64 bits if and when the
   driver has the full capabality to support DAC - via it's
   DMA mask capability setting.

All devices I am aware of do not support 64-bit addresses for their
descriptors, only 32-bit.

There is a huge difference between using a "64-bit address" and
requiring to generate a DAC cycle, the latter of which is
probably what you need and that is independant of what kind of
address pci_alloc_consistent() is giving you.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 10:05:28 -0500
   
   I actually like the flag as a parameter because it allows platform
   specific extensions to pass in platform specific capabilities that
   can be ignored by platform that does not support these capabilities.

I hate this concept.  I can't think of any attribute that all
platforms can't take advantage of in some way and we'd actually
want drivers to use.

Here be the way to private platform-specific interfaces, which is
totally against the reason we have these interfaces in the first
place.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Grant Grundler <iod00d@hp.com>
   Date: Thu, 15 May 2003 12:06:22 -0700
   
   This would help chipset specific DMA support pick "optimal"
   settings for DMA operations.

I vehemently will fight any such rediculious ideas.

The whole concept behind these portable DMA apis is to avoid
platforms specific junk, it all must be generic and apply to
everyone.

   I was thinking of redefining "direction" parameter to be "hints".
   It would continue to support Read or Write "hint" but then use
   upper bits as flags for other things (don't want to completely
   spoil my talk).

Do not mix "must" parameters such as read/write protections with
"it would be nice if" flag bits.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jesse Barnes]

On Thu, May 15, 2003 at 01:07:39PM -0700, David S. Miller wrote:
>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 08:17:58 -0500
>    
>    Anyway, to make a long story short, we like to "extend" these
>    interfaces to return DMA addresses of 64 bits if and when the
>    driver has the full capabality to support DAC - via it's
>    DMA mask capability setting.
> 
> All devices I am aware of do not support 64-bit addresses for their
> descriptors, only 32-bit.

PCI-X devices are required to support 64 bit addrs.  They'll only
generate SAC if the top 32 bits are 0, but our platform is incapable
of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we
need high order bits for PCI-X devices).

> There is a huge difference between using a "64-bit address" and
> requiring to generate a DAC cycle, the latter of which is
> probably what you need and that is independant of what kind of
> address pci_alloc_consistent() is giving you.

Unfortunately, we need both for PCI-X on Altix.  How about this:
drivers that support PCI-X cards are required to program the full 64
bits of a pci_alloc_consistent DMA address, the top 32 bits of which
can be zero for most platforms.  That means that the device will only
generate a SAC for most platforms, but on platforms like ours, the top
32 bits will contain useful information, so a DAC will be generated.

Thanks,
Jesse

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 01:19:40PM -0700, David S. Miller wrote:
> Do not mix "must" parameters such as read/write protections with
> "it would be nice if" flag bits.

R/W "hints" is not required for ia64/parisc IOMMUs.
AFAIK, only swiotlb requires it and ZX1 IOMMU support has a one
performance optimization which makes use of R/W hint.

In short, "must" only applies to some subset of working platforms.

But in general I agree, hints should be useful to most architectures.
Having too many platform specific hints becomes hell for driver
writers.

grant

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Matthew Wilcox]

On Thu, May 15, 2003 at 01:24:12PM -0700, Jesse Barnes wrote:
> > All devices I am aware of do not support 64-bit addresses for their
> > descriptors, only 32-bit.
> 
> PCI-X devices are required to support 64 bit addrs.  They'll only
> generate SAC if the top 32 bits are 0, but our platform is incapable
> of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we
> need high order bits for PCI-X devices).

I think you're missing Dave's point.  While the card is required to
support DAC that doesn't mean the device on the card is necessarily
capable of being handed a full 64-bit address to DMA into.

From the way he's talking, I'd surmise tg3 is a good example -- I suspect
the ring buffers only support 32-bit addresses.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 08:17:58 -0500
>
>    Anyway, to make a long story short, we like to "extend" these
>    interfaces to return DMA addresses of 64 bits if and when the
>    driver has the full capabality to support DAC - via it's
>    DMA mask capability setting.
>
> All devices I am aware of do not support 64-bit addresses for their
> descriptors, only 32-bit.
>
> There is a huge difference between using a "64-bit address" and
> requiring to generate a DAC cycle, the latter of which is
> probably what you need and that is independant of what kind of
> address pci_alloc_consistent() is giving you.

David,

I am quite confuse with what you are saying.

pci_alloc_consistent(), as documented must return a 32bits DMA in the
dma mapped address in "dma_addr_t".

Right now, on our platform, the Upper 32 Bits of this address is always 0s
and the Lower 32 Bits has the IOMMU mapped DMA address.  When this
address is given to the card, SAC is generated because the Upper 32 Bits
are 0s - if the whole 64bits are given to the card.

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA
address that  has the Upper 32 bits non 0s.  When the driver gives this
address to the card, it will generate a DAC for the DMA.

From what I understand from the pci_alloc_consistant() interface, we cannot
return a "dma_addr_t" that has the Upper 32bits NON 0s.  Is that TRUE?  If
it is, our platform cannot deal with this restriction .. therefore we are
requesting
for this extension.

Thanks.

colin

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

James Bottomley wrote:

> On Thu, 2003-05-15 at 17:14, Colin Ngam wrote:
> > Right now, on our platform, the Upper 32 Bits of this address is always 0s
> > and the Lower 32 Bits has the IOMMU mapped DMA address.  When this
> > address is given to the card, SAC is generated because the Upper 32 Bits
> > are 0s - if the whole 64bits are given to the card.
> >
> > Now, for PCI-X devices, we need to generate an IOMMU mapped DMA
> > address that  has the Upper 32 bits non 0s.  When the driver gives this
> > address to the card, it will generate a DAC for the DMA.
> >
> > >From what I understand from the pci_alloc_consistant() interface, we cannot
> > return a "dma_addr_t" that has the Upper 32bits NON 0s.  Is that TRUE?  If
> > it is, our platform cannot deal with this restriction .. therefore we are
> > requesting
> > for this extension.
>
> OK, now you've confused me.  I thought you were saying that your PCI
> hardware generates two type of addresses: one for DAC that *doesn't* go
> through the IOMMU and one for SAC that does?

Yes, you are correct .. we have the capability to generate a SAC or DAC address
in PCI Mode only.

>
>
> In which case, SAC should be able to go through the IOMMU to reach
> anwhere in memory, and thus pci_alloc_consistent should work for you
> now.
>
> Or are you saying it's more efficient to go via the DAC non-iommu cycle
> and you'd like to do that preferentially if you could (which is a
> feature request, not a hard requirement)?

However, in PCI-X mode, we do not have the capability to generate SAC addresses.
And yes, you are also correct that it is very much efficient on our architecture
to generate DAC for all(PCI/PCI-X)

So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.

Hope that is clear ..

Thanks.

colin

>
>
> James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[James Bottomley]

On Thu, 2003-05-15 at 17:14, Colin Ngam wrote:
> Right now, on our platform, the Upper 32 Bits of this address is always 0s
> and the Lower 32 Bits has the IOMMU mapped DMA address.  When this
> address is given to the card, SAC is generated because the Upper 32 Bits
> are 0s - if the whole 64bits are given to the card.
> 
> Now, for PCI-X devices, we need to generate an IOMMU mapped DMA
> address that  has the Upper 32 bits non 0s.  When the driver gives this
> address to the card, it will generate a DAC for the DMA.
> 
> >From what I understand from the pci_alloc_consistant() interface, we cannot
> return a "dma_addr_t" that has the Upper 32bits NON 0s.  Is that TRUE?  If
> it is, our platform cannot deal with this restriction .. therefore we are
> requesting
> for this extension.

OK, now you've confused me.  I thought you were saying that your PCI
hardware generates two type of addresses: one for DAC that *doesn't* go
through the IOMMU and one for SAC that does?

In which case, SAC should be able to go through the IOMMU to reach
anwhere in memory, and thus pci_alloc_consistent should work for you
now.

Or are you saying it's more efficient to go via the DAC non-iommu cycle
and you'd like to do that preferentially if you could (which is a
feature request, not a hard requirement)?

James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[James Bottomley]

On Thu, 2003-05-15 at 17:44, Colin Ngam wrote:
> However, in PCI-X mode, we do not have the capability to generate SAC addresses.
> And yes, you are also correct that it is very much efficient on our architecture
> to generate DAC for all(PCI/PCI-X)
> 
> So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
> 
> Hope that is clear ..

Actually, no.  This seems to imply that your PCI-X cards can *never*
access the first 4GB of memory (because they can't use an IOMMU and they
have to generate a DAC cycle with non zero in bits 63--32).

James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

James Bottomley wrote:

> On Thu, 2003-05-15 at 17:44, Colin Ngam wrote:
> > However, in PCI-X mode, we do not have the capability to generate SAC addresses.
> > And yes, you are also correct that it is very much efficient on our architecture
> > to generate DAC for all(PCI/PCI-X)
> >
> > So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
> >
> > Hope that is clear ..
>
> Actually, no.  This seems to imply that your PCI-X cards can *never*
> access the first 4GB of memory (because they can't use an IOMMU and they
> have to generate a DAC cycle with non zero in bits 63--32).

Actually, our system does not have Physical Memory Address smaller than
48Bits - give or take couple bits ...  Hence, 32Bits Mapped DMA address
maps to 48Bits System Physical Address.

64Bits Mapped DMA address maps directly to System Physical Memory
with Bits 63-49 containing transfer attributes like hardware support
for byte swapping .. if needed(e.g. Big Endian Cards).

Thanks.

colin

>
>
> James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Thu, 15 May 2003 13:24:12 -0700
   
   How about this: drivers that support PCI-X cards are required to
   program the full 64 bits of a pci_alloc_consistent DMA address, the
   top 32 bits of which can be zero for most platforms.

How about this: you put the device attributes where they belong,
in the device struct.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 17:14:11 -0500
>
>    Now, for PCI-X devices, we need to generate an IOMMU mapped DMA
>    address that  has the Upper 32 bits non 0s.
>
> Then add a consistent_dma_mask attribute to the device struct,
> problem solved.

Hi David,

I do not understand what you are saying .. how would a consistent_dma_mask
attribute in the device struct helps pci_alloc_consistent() to return a dma
hanlde
that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits
are not constants, but can vary greatly, and is part of the 64 bit Physical
System
Memory Address?

Thanks.

colin

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Matthew Wilcox <willy@debian.org>
   Date: Thu, 15 May 2003 22:29:18 +0100
   
   From the way he's talking,

My main point is that device attributes belong IN THE DEVICE
not in some umpteenth new parameter to the DMA interfaces.

Just put consistent_dma_mask into the device struct next to
the existing dma_mask, voila problem solved.

I have two major problems with the generic DMA interfaces
and how they are being handled in general:

1) Their use is entirely inappropriate in unary-bus devices.
   There is no way someone can justify some PCI driver continually
   asking itself whether it is a PCI device or not on each DMA
   API call.

2) Additions to the parameters in the interfaces is not given
   enough thought.  Each new parameter you add is a discrete failure
   of the API.  It was built to be simple, and new parameters do
   nothing but add complexity.

   I cannot even fathom that someone would suggest to pass device
   attributes into the DMA interfaces.  That is just garbage and
   unnecessary bloat.

Finally, I see nobody tackling the real hard bugs we have in the DMA
API.  These should be fixed first before adding random DMA behaviorial
attributes.  For example, pci_dma_sync_*() sucks because it only syncs
in one direction, device to cpu, there is nothing available to sync
things back from cpu-->device.  This prevents MIPS from using the
devices that need to do syncs without unmaps (eepro100 is but one
example, USB host controller drivers want this too).

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 18:25:54 -0500
>
>    Actually, our system does not have Physical Memory Address smaller than
>    48Bits - give or take couple bits ...  Hence, 32Bits Mapped DMA address
>    maps to 48Bits System Physical Address.
>
> If you support mapping 32-bit DMA bus spaces to the full 48-bit
> physical memory spaces, then why do you even need >4GB addressing?
>
> Saying that PCI-X devices can only generate DAC cycles is bogus,

I did not say that.  it is very obvious that PCI-X devices can generate
either SAC or DAC.

What I did say is that, on our platform, in PCI-X mode, our PCI-X
Bridge Chipset do NOT support 32-Bit DMA Bus Space.

This does not break any PCI-X specs.

However, the pci_alloc_consistent() interface is documented to ONLY
return a 32-Bit DMA Bus Space .. we like this to be extended so that
PCI-X device drivers or PCI drivers can consumed 64-Bit DMA Bus
Space, if they have the capability.

Thanks.

colin

>
> I have PCI-X tg3 cards in my systems and they function just fine.
> Or does this rule only apply when the device is on a PCI-X bus?

>
>
> This all sounds seriously broken to me, I thought the issue was that
> a) the addresses could not be translated, it was direct phys-->virt
> and b) you can't guarentee that the physical address of memory
> allocated on any particular node is <4GB.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 17:14:11 -0500
   
   Now, for PCI-X devices, we need to generate an IOMMU mapped DMA
   address that  has the Upper 32 bits non 0s.

Then add a consistent_dma_mask attribute to the device struct,
problem solved.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: James Bottomley <James.Bottomley@HansenPartnership.com>
   Date: 15 May 2003 17:47:04 -0500
   
   OK, now you've confused me.

Their hardware platform cannot guarentee low-32bit memory allocations
to devices because each node can have physical memory starting at
>4GB etc etc.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 17:44:57 -0500
   
   However, in PCI-X mode, we do not have the capability to generate
   SAC addresses.

The device can generate addresses that have the top-32bits 0 though.

You problem is rather that you cannot guarentee to be able to allocate
physical memory which is <4GB on each and every node.

That is very different from a "PCI-X issue".  It is a "this NUMA
platform" issue.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: James Bottomley <James.Bottomley@HansenPartnership.com>
>    Date: 15 May 2003 17:47:04 -0500
>
>    OK, now you've confused me.
>
> Their hardware platform cannot guarentee low-32bit memory allocations
> to devices because each node can have physical memory starting at
> >4GB etc etc.

Hi,

Something else I do not understand .. in the PCI world, what does
Physical Memory Address got to do with 32Bit Mapped DMA
Bus Address?

Not only do we not guarantee that we cannot generate 32 Bit Memory
Allocations .. we plain old do not have Memory Addresses smaller
below 32Bits .. well, below 48 Bits for that matter .. I have
not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.

Thanks.

colin

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:25:54 -0500
   
   Actually, our system does not have Physical Memory Address smaller than
   48Bits - give or take couple bits ...  Hence, 32Bits Mapped DMA address
   maps to 48Bits System Physical Address.

If you support mapping 32-bit DMA bus spaces to the full 48-bit
physical memory spaces, then why do you even need >4GB addressing?

Saying that PCI-X devices can only generate DAC cycles is bogus,
I have PCI-X tg3 cards in my systems and they function just fine.
Or does this rule only apply when the device is on a PCI-X bus?

This all sounds seriously broken to me, I thought the issue was that
a) the addresses could not be translated, it was direct phys-->virt
and b) you can't guarentee that the physical address of memory
allocated on any particular node is <4GB.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 18:32:31 -0500
>
>    I do not understand what you are saying .. how would a consistent_dma_mask
>    attribute in the device struct helps pci_alloc_consistent() to return a dma
>    hanlde that can have the Upper 32 Bits Non 0's, and, moreover,
>    these Upper 32 Bits are not constants, but can vary greatly, and is
>    part of the 64 bit Physical System Memory Address?
>
> The mask says that the consistent address "can" have the upper
> 32-bits non-zero.  Then you define a consistent_dma_attrs bitmask
> that can define things like "requires >4GB address" etc.
>
> Again, all of this is device attribute stuff and not an issue
> concerning the DMA api calls.

Hi David,

Implementation details aside, you are agreeing that the pci_alloc_consistent()
interface can return an address > 4GB and we can document as such?

Thanks.

colin

>
>
> But I am ever further confused, how do things like tg3 cards even
> function TODAY on any PCI-X systems.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:32:31 -0500
   
   I do not understand what you are saying .. how would a consistent_dma_mask
   attribute in the device struct helps pci_alloc_consistent() to return a dma
   hanlde that can have the Upper 32 Bits Non 0's, and, moreover,
   these Upper 32 Bits are not constants, but can vary greatly, and is
   part of the 64 bit Physical System Memory Address?
   
The mask says that the consistent address "can" have the upper
32-bits non-zero.  Then you define a consistent_dma_attrs bitmask
that can define things like "requires >4GB address" etc.

Again, all of this is device attribute stuff and not an issue
concerning the DMA api calls.

But I am ever further confused, how do things like tg3 cards even
function TODAY on any PCI-X systems.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 18:45:42 -0500
>
>    Something else I do not understand .. in the PCI world, what does
>    Physical Memory Address got to do with 32Bit Mapped DMA
>    Bus Address?
>
> Many platforms direct map, that is what I assumed was happening
> here and causing the problems.
>
>    I have not seen any of the drivers broken yet - e.g. qlXXX, acenic,
>    tg3 etc.
>
> It only works because these devices happen to allow >32-bit DMA
> addresses for descriptors.  You will find eventually that many
> comodity pieces of PCI-X hardware will opt to only support 32-bit
> DMA addressing for descriptor tables and whatnot.

Hi David,

It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support
DAC.
It did not say that it cannot support SAC or should not support SAC,
but it has to support DAC.

Thanks.

colin

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:40:22 -0500
   
   What I did say is that, on our platform, in PCI-X mode, our PCI-X
   Bridge Chipset do NOT support 32-Bit DMA Bus Space.

Sounds like you can't use a whole swath of devices then.
What an entirely unfortunate design decision.

Anyways, now that I understand the problem, merely adding
the consistent_dma_mask attribute to the device struct will
allow drivers to indicate this capability just fine.
The driver goes:

	if (pci_set_consistent_dma_mask(pdev, ~(u64)0))
		goto probe_error;

and then the platform pci_alloc_consistent() can do whatever is
appropriate.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:45:42 -0500

   Something else I do not understand .. in the PCI world, what does
   Physical Memory Address got to do with 32Bit Mapped DMA
   Bus Address?
   
Many platforms direct map, that is what I assumed was happening
here and causing the problems.

   I have not seen any of the drivers broken yet - e.g. qlXXX, acenic,
   tg3 etc.
   
It only works because these devices happen to allow >32-bit DMA
addresses for descriptors.  You will find eventually that many
comodity pieces of PCI-X hardware will opt to only support 32-bit
DMA addressing for descriptor tables and whatnot.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:51:35 -0500
   
   Implementation details aside, you are agreeing that the
   pci_alloc_consistent() interface can return an address > 4GB and we
   can document as such?

I can return an address that matches the requirements specified by
dev->consistent_dma_mask which is set by default to 0xffffffff and
must be changed by a device driver if it wants to use a non-default
value.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 18:57:09 -0500
   
   It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have
   to support DAC.

Having to support DAC cycles does not mean that it must support full
64-bit DMA addresses in it's descriptors.

A cost sensitive PCI-X device may wish to use 32-bit addressing in
it's descriptors in order to keep pin counts down etc.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jesse Barnes]

On Thu, May 15, 2003 at 04:27:33PM -0700, David S. Miller wrote:
>    From: Jesse Barnes <jbarnes@sgi.com>
>    Date: Thu, 15 May 2003 13:24:12 -0700
>    
>    How about this: drivers that support PCI-X cards are required to
>    program the full 64 bits of a pci_alloc_consistent DMA address, the
>    top 32 bits of which can be zero for most platforms.
> 
> How about this: you put the device attributes where they belong,
> in the device struct.

That's fine, it'll work for us.  I just wanted to make sure that
dma_alloc_consistent could return a 64 bit DMA addr.  Having a
consistent_dma_mask will allow us to do that for devices that support
it.  We'll have to switch devices that can't into PCI mode for them to
operate correctly on our hw.

Also, are you saying that tg3 doesn't support all 64 bits of the DMA
address?  If so, I'm not sure how it's working on our platform either,
unless we're forcing it into PCI mode; I'll have to check.

Thanks,
Jesse

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[James Bottomley]

On Thu, 2003-05-15 at 19:17, David S. Miller wrote:
> Having to support DAC cycles does not mean that it must support full
> 64-bit DMA addresses in it's descriptors.
> 
> A cost sensitive PCI-X device may wish to use 32-bit addressing in
> it's descriptors in order to keep pin counts down etc.

This is certainly a property of the aic79xx driver (only using 32 bit
descriptor tables).  It has this nice code in the driver for doing it:

aic79xx.c:

	/*
	 * Although we can dma data above 4GB, our
	 * "consistent" memory is below 4GB for
	 * space efficiency reasons (only need a 4byte
	 * address).  For this reason, we have to reset
	 * our dma mask when doing allocations.
	 */
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
	*vaddr = pci_alloc_consistent(ahd->dev_softc,
				      dmat->maxsize, &map->bus_addr);
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc,
				     ahd->platform_data->hw_dma_mask);

And it comes in PCI-X flavours.

James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Thu, 15 May 2003 17:29:05 -0700
   
   Also, are you saying that tg3 doesn't support all 64 bits of the DMA
   address?  If so, I'm not sure how it's working on our platform either,
   unless we're forcing it into PCI mode; I'll have to check.

I believe tg3 does, but some cost-sensitive PCI-X devices might decide
not to.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

James Bottomley wrote:

> On Thu, 2003-05-15 at 19:17, David S. Miller wrote:
> > Having to support DAC cycles does not mean that it must support full
> > 64-bit DMA addresses in it's descriptors.
> >
> > A cost sensitive PCI-X device may wish to use 32-bit addressing in
> > it's descriptors in order to keep pin counts down etc.
>
> This is certainly a property of the aic79xx driver (only using 32 bit
> descriptor tables).  It has this nice code in the driver for doing it:
>
> aic79xx.c:
>
>         /*
>          * Although we can dma data above 4GB, our
>          * "consistent" memory is below 4GB for
>          * space efficiency reasons (only need a 4byte
>          * address).  For this reason, we have to reset
>          * our dma mask when doing allocations.
>          */
>         if (ahd->dev_softc != NULL)
>                 ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
>         *vaddr = pci_alloc_consistent(ahd->dev_softc,
>                                       dmat->maxsize, &map->bus_addr);
>         if (ahd->dev_softc != NULL)
>                 ahd_pci_set_dma_mask(ahd->dev_softc,
>                                      ahd->platform_data->hw_dma_mask);
>
> And it comes in PCI-X flavours.

Yes agreed.  And I think this is fine.  But let's not have the interface
prevents 64bits drivers that wants to do DAC.  that's all I am
asking ..

Thanks.

colin

>
>
> James

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: James Bottomley <James.Bottomley@HansenPartnership.com>
   Date: 15 May 2003 19:31:20 -0500

   	if (ahd->dev_softc != NULL)
   		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
   	*vaddr = pci_alloc_consistent(ahd->dev_softc,
   				      dmat->maxsize, &map->bus_addr);

Funny how whoever wrote this didn't understand that pci_set_dma_mask()
has zero effect on pci_alloc_consistent() behavior.

Someone please tell Justin to fix this broken bit of code.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 18:57:09 -0500
>
>    It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have
>    to support DAC.
>
> Having to support DAC cycles does not mean that it must support full
> 64-bit DMA addresses in it's descriptors.

Hi David,

I do not believe it is within spec to perform DAC when the
Upper 32bits are 0s.  Current usage of pci_alloc_consistent(),
even if you want to do DAC, you cannot, because the interface
is defined to return only 32bits worth of addresses.

However, I think we are beyond this point now.

Thanks.

colin

>
>
> A cost sensitive PCI-X device may wish to use 32-bit addressing in
> it's descriptors in order to keep pin counts down etc.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jesse Barnes]

On Thu, May 15, 2003 at 07:31:20PM -0500, James Bottomley wrote:
> > A cost sensitive PCI-X device may wish to use 32-bit addressing in
> > it's descriptors in order to keep pin counts down etc.
> 
> This is certainly a property of the aic79xx driver (only using 32 bit
> descriptor tables).  It has this nice code in the driver for doing it:
> 
> aic79xx.c:
> 
> 	/*
> 	 * Although we can dma data above 4GB, our
> 	 * "consistent" memory is below 4GB for
> 	 * space efficiency reasons (only need a 4byte
> 	 * address).  For this reason, we have to reset
> 	 * our dma mask when doing allocations.
> 	 */
> 	if (ahd->dev_softc != NULL)
> 		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
> 	*vaddr = pci_alloc_consistent(ahd->dev_softc,
> 				      dmat->maxsize, &map->bus_addr);
> 	if (ahd->dev_softc != NULL)
> 		ahd_pci_set_dma_mask(ahd->dev_softc,
> 				     ahd->platform_data->hw_dma_mask);
> 
> And it comes in PCI-X flavours.

Ah, ok (set_dma_mask vs. alloc_consistent bugs aside).  I think there
was some confusion about the fact that the PCI-X spec. says that cards
must support DAC vs. them actually supporting 64 bit DMA addresses.
They all have to do the former, but not the latter.  Also, using an
IOMMU is different than using direct mapping.  Our hw can do both for
PCI cards that only handle 32 bit DMA addresses.  IOMMU mapping allows
you to map pages anywhere in memory, while direct mapping gives you a
static 2GB window into system memory.

Hope this clears things up a little.

Thanks,
Jesse

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

"David S. Miller" wrote:

>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 19:36:55 -0500
>
>    I do not believe it is within spec to perform DAC when the
>    Upper 32bits are 0s.
>
> That is an illogical requirement in that it means PCI-X mode is
> illegal in the presence of such addresses.  It means that PCI-X
> devices cannot work in x86 systems with <4GB of memory which is
> obviously not true.

No that is not the case.

If you have an address that only has the Lower 32bits set, your card
do a SAC and not a DAC.

SAC is not prohibited in PCI-X mode.  But DAC has to be supported by the card.

Someone please correct me if I misread.

Thanks.

colin

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Colin Ngam <cngam@sgi.com>
   Date: Thu, 15 May 2003 19:36:55 -0500
   
   I do not believe it is within spec to perform DAC when the
   Upper 32bits are 0s.

That is an illogical requirement in that it means PCI-X mode is
illegal in the presence of such addresses.  It means that PCI-X
devices cannot work in x86 systems with <4GB of memory which is
obviously not true.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 04:53:30PM -0700, David S. Miller wrote:
> But I am ever further confused, how do things like tg3 cards even
> function TODAY on any PCI-X systems.

They work just fine on most platforms with 32-bit addresses.
The issue is with what the SGI platform can support, not what
the PCI-X devices can support.

BTW, adding consistent_dma_mask to struct pci_dev and a
corresponding pci_set_consistent_dma_mask() is a good idea.
I think it would simplify our SBA code a bit.

thanks,
grant

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Grant Grundler <iod00d@hp.com>
   Date: Thu, 15 May 2003 18:20:51 -0700
   
   BTW, adding consistent_dma_mask to struct pci_dev and a
   corresponding pci_set_consistent_dma_mask() is a good idea.
   I think it would simplify our SBA code a bit.
   
Great, I'm glad this is resolved. :-)

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 06:32:31PM -0500, Colin Ngam wrote:
> I do not understand what you are saying .. how would a consistent_dma_mask
> attribute in the device struct helps pci_alloc_consistent() to return a dma
> hanlde that can have the Upper 32 Bits Non 0's...

Colin,
davem has offered a counter proposal that makes way more sense
that adding parameters to the dma_*() API. It won't change existing
code or existing behaviors. And yes, Documentation/DMA_mapping.txt
(API documentation) has to change to reflect that.

Device Drivers SGI cares about will have to be modified to call
pci_set_consistent_dma_mask() before calling pci_alloc_consistent().

hth,
grant

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Colin Ngam]

Grant Grundler wrote:

> On Thu, May 15, 2003 at 06:32:31PM -0500, Colin Ngam wrote:
> > I do not understand what you are saying .. how would a consistent_dma_mask
> > attribute in the device struct helps pci_alloc_consistent() to return a dma
> > hanlde that can have the Upper 32 Bits Non 0's...
>
> Colin,
> davem has offered a counter proposal that makes way more sense
> that adding parameters to the dma_*() API. It won't change existing
> code or existing behaviors. And yes, Documentation/DMA_mapping.txt
> (API documentation) has to change to reflect that.
>
> Device Drivers SGI cares about will have to be modified to call
> pci_set_consistent_dma_mask() before calling pci_alloc_consistent().
>
> hth,
> grant

Hi Grant,

That would be very fine :-)

Thanks.

colin

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Thu, 2003-05-15 at 18:39, David S. Miller wrote:
> My main point is that device attributes belong IN THE DEVICE
> not in some umpteenth new parameter to the DMA interfaces.
> 
> Just put consistent_dma_mask into the device struct next to
> the existing dma_mask, voila problem solved.
> 
> I have two major problems with the generic DMA interfaces
> and how they are being handled in general:
> 
> 1) Their use is entirely inappropriate in unary-bus devices.
>    There is no way someone can justify some PCI driver continually
>    asking itself whether it is a PCI device or not on each DMA
>    API call.

Where's that in the API?  The PA-RISC implementation copes with about
four different buses and nowhere does it have a switch trying to
ascertain which bus it's talking about (it uses the dev->platform_data
field to cache the relevant bus attributes).

> 2) Additions to the parameters in the interfaces is not given
>    enough thought.  Each new parameter you add is a discrete failure
>    of the API.  It was built to be simple, and new parameters do
>    nothing but add complexity.
> 
>    I cannot even fathom that someone would suggest to pass device
>    attributes into the DMA interfaces.  That is just garbage and
>    unnecessary bloat.

You mean like xadding gfp flags to the allocator?

> Finally, I see nobody tackling the real hard bugs we have in the DMA
> API.  These should be fixed first before adding random DMA behaviorial
> attributes.  For example, pci_dma_sync_*() sucks because it only syncs
> in one direction, device to cpu, there is nothing available to sync
> things back from cpu-->device.  This prevents MIPS from using the
> devices that need to do syncs without unmaps (eepro100 is but one
> example, USB host controller drivers want this too).

OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want?  what
am I missing?

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: James Bottomley <James.Bottomley@HansenPartnership.com>
   Date: 15 May 2003 21:05:12 -0500
   
   > Finally, I see nobody tackling the real hard bugs we have in the DMA
   > API.  These should be fixed first before adding random DMA behaviorial
   > attributes.  For example, pci_dma_sync_*() sucks because it only syncs
   > in one direction, device to cpu, there is nothing available to sync
   > things back from cpu-->device.  This prevents MIPS from using the
   > devices that need to do syncs without unmaps (eepro100 is but one
   > example, USB host controller drivers want this too).
   
   OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want?  what
   am I missing?
   
You don't understand, a driver does:

	pci_dma_sync_single(... buf);

This transfers ownership of the buffer from device back to the CPU.
It guarentees that the CPU will see uptodate contents of the buffer
and that it may make modifications to 'buf'.

Then it looks at 'buf', and decides that it wants to give the buffer
back to the device.  What does it call to transfer ownership back from
CPU to device?

Answer: No such API call exists.

This is a big problem for implementations that need to flush CPU
caches in order to implement the cpu<-->device buffer ownership
transfer properly.

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

> You don't understand, a driver does:
> 
> 	pci_dma_sync_single(... buf);
> 
> This transfers ownership of the buffer from device back to the CPU.
> It guarentees that the CPU will see uptodate contents of the buffer
> and that it may make modifications to 'buf'.
> 
> Then it looks at 'buf', and decides that it wants to give the buffer
> back to the device.  What does it call to transfer ownership back from
> CPU to device?
> 
> Answer: No such API call exists.
> 
> This is a big problem for implementations that need to flush CPU
> caches in order to implement the cpu<-->device buffer ownership
> transfer properly.

It was my impression that a CPU cache invalidate cycle relinquishes the
cache line (and thus leaves the line free to be owned by the device). 
The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient?  I
grant that CPU does auto ownership, so if the CPU ever touches the data
again the cache line is pulled back in again.

Is the issue that we have no way of preventing the CPU from pulling in
the cache line if it is referenced?

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: James Bottomley <James.Bottomley@HansenPartnership.com>
   Date: 15 May 2003 21:26:14 -0500
   
   It was my impression that a CPU cache invalidate cycle relinquishes the
   cache line (and thus leaves the line free to be owned by the device). 
   The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient?  I
   grant that CPU does auto ownership, so if the CPU ever touches the data
   again the cache line is pulled back in again.
   
What flushes the cpu cache after the cpu writes to 'buf'
so that the device sees what the CPU wrote there?  Consider
copy-back L2 caches like MIPS has.

   Is the issue that we have no way of preventing the CPU from pulling in
   the cache line if it is referenced?
   
The "issue" is that we need a pci_dma_sync_*_to_device() call added.
We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu()
to make all of the naming consistent and update all the drivers.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Thu, May 15, 2003 at 01:07:39PM -0700, David S. Miller wrote:
>    From: Colin Ngam <cngam@sgi.com>
>    Date: Thu, 15 May 2003 08:17:58 -0500
>    
>    Anyway, to make a long story short, we like to "extend" these
>    interfaces to return DMA addresses of 64 bits if and when the
>    driver has the full capabality to support DAC - via it's
>    DMA mask capability setting.
> 
> All devices I am aware of do not support 64-bit addresses for their
> descriptors, only 32-bit.

The Qlogic and LSI Logic SCSI and FC devices support 64 bit addresses
for their descriptors.  Also the old Adaptec (now JNI) 1160.  I do
not believe this is unusual.  Certainly all PCI-X devices should support
64 bit addresses for descriptors.

The Tigon3 supports 64 bit addresses for descriptors.

> There is a huge difference between using a "64-bit address" and
> requiring to generate a DAC cycle, the latter of which is
> probably what you need and that is independant of what kind of
> address pci_alloc_consistent() is giving you.

Not independent.  Because if pci_alloc_consistent gives an address
in which any of bits 32-63 are set, then the adapter will use a DAC
to access the descriptor.  I guess you're saying that some adapters
may support more than 32 bits but fewer than 64.  Such devices
would not work on our platform.  Which PCI-X devices do not support
the full 64 bit address space? I don't know of any.

jeremy

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[David S. Miller]

   From: Jeremy Higdon <jeremy@sgi.com>
   Date: Thu, 15 May 2003 23:47:55 -0700

Please read further into your mailbox before replying,
all of the misunderstandings and issues have been resolved
already.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Thu, May 15, 2003 at 11:55:47PM -0700, David S. Miller wrote:
>    From: Jeremy Higdon <jeremy@sgi.com>
>    Date: Thu, 15 May 2003 23:47:55 -0700
> 
> Please read further into your mailbox before replying,
> all of the misunderstandings and issues have been resolved
> already.

I did.  I didn't see that issue addressed.  I guess someone did
educate you, perhaps in a message I did not see.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Thu, May 15, 2003 at 06:24:21PM -0500, James Bottomley wrote:
> On Thu, 2003-05-15 at 17:44, Colin Ngam wrote:
> > However, in PCI-X mode, we do not have the capability to generate SAC addresses.
> > And yes, you are also correct that it is very much efficient on our architecture
> > to generate DAC for all(PCI/PCI-X)
> > 
> > So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
> > 
> > Hope that is clear ..
> 
> Actually, no.  This seems to imply that your PCI-X cards can *never*
> access the first 4GB of memory (because they can't use an IOMMU and they
> have to generate a DAC cycle with non zero in bits 63--32).
> 
> James

I don't know if it was clear or not in the ensuing replies, but we
don't present direct physical addresses to PCI-X cards.  Instead it's an
physical address with various bits ORed in at the top to set routing
and attribute information.  So if you want to DMA to physical address
0x1000 in the system, the PCI address might be 0x9100000000001000.

If it was already clear, sorry for the duplication.

jeremy

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Thu, May 15, 2003 at 07:32:26PM -0700, David S. Miller wrote:
>    From: James Bottomley <James.Bottomley@HansenPartnership.com>
>    Date: 15 May 2003 21:26:14 -0500
>    
>    It was my impression that a CPU cache invalidate cycle relinquishes the
>    cache line (and thus leaves the line free to be owned by the device). 
>    The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient?  I
>    grant that CPU does auto ownership, so if the CPU ever touches the data
>    again the cache line is pulled back in again.
>    
> What flushes the cpu cache after the cpu writes to 'buf'
> so that the device sees what the CPU wrote there?  Consider
> copy-back L2 caches like MIPS has.

Isn't that what dma_cache_wback_inv does?  It flushes cpu cache to memory
so that it is visible to devices.  Is there some additional semantic
desired?

jeremy

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Jeremy Higdon <jeremy@sgi.com>
   Date: Fri, 16 May 2003 00:48:12 -0700
   
   Isn't that what dma_cache_wback_inv does?

That function is not part of the DMA api, and no portable
driver should use this dma_cache_wback_inv() thing.

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Thu, May 15, 2003 at 07:31:20PM -0500, James Bottomley wrote:
> On Thu, 2003-05-15 at 19:17, David S. Miller wrote:
> > Having to support DAC cycles does not mean that it must support full
> > 64-bit DMA addresses in it's descriptors.
> > 
> > A cost sensitive PCI-X device may wish to use 32-bit addressing in
> > it's descriptors in order to keep pin counts down etc.
> 
> This is certainly a property of the aic79xx driver (only using 32 bit
> descriptor tables).  It has this nice code in the driver for doing it:

This is a driver choice, rather than an adapter limitation, right?
Or is the adapter limited to 32 bit addresses for map->bus_addr (the
PCI-X flavor)?

> aic79xx.c:
> 
> 	/*
> 	 * Although we can dma data above 4GB, our
> 	 * "consistent" memory is below 4GB for
> 	 * space efficiency reasons (only need a 4byte
> 	 * address).  For this reason, we have to reset
> 	 * our dma mask when doing allocations.
> 	 */
> 	if (ahd->dev_softc != NULL)
> 		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
> 	*vaddr = pci_alloc_consistent(ahd->dev_softc,
> 				      dmat->maxsize, &map->bus_addr);
> 	if (ahd->dev_softc != NULL)
> 		ahd_pci_set_dma_mask(ahd->dev_softc,
> 				     ahd->platform_data->hw_dma_mask);
> 
> And it comes in PCI-X flavours.
> 
> James
> 

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Thu, 2003-05-15 at 21:32, David S. Miller wrote:
> The "issue" is that we need a pci_dma_sync_*_to_device() call added.
> We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu()
> to make all of the naming consistent and update all the drivers.

OK, I get it now (I was thinking really in terms of the old
writeback/invalidate interface where you had to know how to place them).

You mean we need an interface that is called when the CPU hands off the
region to the device and again when the CPU reclaims it (so we always
get the writeback or invalidate in the correct place).

How about just adding an ownership attribute to dma_sync..?  It would
either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?

Of course, retrofitting something like this is going to be fun.

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Fri, May 16, 2003 at 12:36:15PM -0500, James Bottomley wrote:
> You mean we need an interface that is called when the CPU hands off the
> region to the device and again when the CPU reclaims it (so we always
> get the writeback or invalidate in the correct place).

yes

> How about just adding an ownership attribute to dma_sync..?  It would
> either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?

I prefer davem's suggestion of a different function call (*_to_cpu and
*_to_dev) rather than making the same function do completely different
things based on parameters. Keeps API documentation and resulting
implementations for each smaller.

> Of course, retrofitting something like this is going to be fun.

It's not so bad.
It's really only needed for non-coherent machines.
Folks who care about those platforms probably already know where
those calls belong.

grant

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Fri, 2003-05-16 at 12:59, Grant Grundler wrote:
> On Fri, May 16, 2003 at 12:36:15PM -0500, James Bottomley wrote:
> > You mean we need an interface that is called when the CPU hands off the
> > region to the device and again when the CPU reclaims it (so we always
> > get the writeback or invalidate in the correct place).
> 
> yes
> 
> > How about just adding an ownership attribute to dma_sync..?  It would
> > either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?
> 
> I prefer davem's suggestion of a different function call (*_to_cpu and
> *_to_dev) rather than making the same function do completely different
> things based on parameters. Keeps API documentation and resulting
> implementations for each smaller.

Hmm, another API vs adding properties.  I'm easy on this.  However, the
fact that data direction is an attribute does give me a preference for
having owner as an attribute too.

> > Of course, retrofitting something like this is going to be fun.
> 
> It's not so bad.
> It's really only needed for non-coherent machines.
> Folks who care about those platforms probably already know where
> those calls belong.

Well, it applies to partially coherent machines too.  However, the point
of the API is to make it so that the driver writer doesn't need to
understand where the calls go,  So the API encapsulates all the platform
quirks in a simple set of rules of thumb.

Thus:

    map region
    
    Use region in driver
    
    dma_sync ... DMA_OWNER_DEVICE  .. and direction
    
    device uses region (driver may not touch it)
    
    dma_sync ... DMA_OWNER_CPU .. and direction
    
    driver accesses region again (may repeat back to use region in
    driver)
    
    unmap region
    
That's unambiguous and simple to articulate.  The platform
implementation does the necessary magic.

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Russell King]

On Fri, May 16, 2003 at 01:13:07PM -0500, James Bottomley wrote:
> Well, it applies to partially coherent machines too.  However, the point
> of the API is to make it so that the driver writer doesn't need to
> understand where the calls go,  So the API encapsulates all the platform
> quirks in a simple set of rules of thumb.
> 
> Thus:
> 
>     map region
>     
>     Use region in driver
>     
>     dma_sync ... DMA_OWNER_DEVICE  .. and direction
>     
>     device uses region (driver may not touch it)
>     
>     dma_sync ... DMA_OWNER_CPU .. and direction
>     
>     driver accesses region again (may repeat back to use region in
>     driver)
>     
>     unmap region

Isn't the common use (eg, net devices):

	buffer passed from lower levels

	map buffer

	(device owns buffer)

	device uses buffer

	unmap buffer

	(cpu owns buffer)

	free buffer

So, to use your example:

	(cpu owns region)

	map

	(device owns region)

	dma_sync...cpu

	(cpu owns region)

	dma_sync...device

	(device owns region)

	unmap

	(cpu owns region)

Your suggestion seems to add in two extra sync calls (and thus extra code
in the common case.)

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

[Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Thu, May 15, 2003 at 11:47:55PM -0700, Jeremy Higdon wrote:
> Certainly all PCI-X devices should support
> 64 bit addresses for descriptors.

I finally found the bit in the PCI-X spec hat mentions this:
(Section 1.8 "Bus Width")
| The following PCI-X requirements are different from conventional PCI:
| 1. All devices that initiate memory transactions must be capable of
|    generating 64-bit addresses.
| 2. Each device includes a configuration status bit that indicates
|    the width of that device's interface.

PCI-X also has a different command set.
It's definitely NOT just a PCI bus speed up.

thanks,
grant

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Fri, 2003-05-16 at 13:18, Russell King wrote:
> Your suggestion seems to add in two extra sync calls (and thus extra code
> in the common case.)

Yes, sorry, needlessly complex example.

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jes Sorensen]

>>>>> "David" = David S Miller <davem@redhat.com> writes:

David> Anyways, now that I understand the problem, merely adding the
David> consistent_dma_mask attribute to the device struct will allow
David> drivers to indicate this capability just fine.  The driver
David> goes:

David> 	if (pci_set_consistent_dma_mask(pdev, ~(u64)0)) goto
David> probe_error;

David> and then the platform pci_alloc_consistent() can do whatever is
David> appropriate.

Hi Dave,

How does this patch look to you? It adds the generic code for setting
the mask, it will then be up to Colin to fix the SGI SN2 specific code
to honor it.

I also added a patch to the tg3 driver to set the mask as it seems to
work just dandy with 64 bit consistent addresses.

Tested on ia64, but I see no reason why it should fail elsewhere.

Cheers,
Jes

diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/net/tg3.c linux-2.5.69-pci/drivers/net/tg3.c
--- linux-2.5.69-030509-vanilla/drivers/net/tg3.c	Sun May  4 19:53:31 2003
+++ linux-2.5.69-pci/drivers/net/tg3.c	Fri May 16 15:42:55 2003
@@ -6743,6 +6743,12 @@
 	/* Configure DMA attributes. */
 	if (!pci_set_dma_mask(pdev, (u64) 0xffffffffffffffff)) {
 		pci_using_dac = 1;
+		if (pci_set_consistent_dma_mask(pdev,
+						(u64) 0xffffffffffffffff)) {
+			printk(KERN_ERR PFX "Unable to obtain 64 bit DMA "
+			       "for consistent allocations\n");
+			goto err_out_free_res;
+		}
 	} else {
 		err = pci_set_dma_mask(pdev, (u64) 0xffffffff);
 		if (err) {
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/pci/pci.c linux-2.5.69-pci/drivers/pci/pci.c
--- linux-2.5.69-030509-vanilla/drivers/pci/pci.c	Sun May  4 19:53:08 2003
+++ linux-2.5.69-pci/drivers/pci/pci.c	Fri May 16 15:43:44 2003
@@ -701,6 +701,17 @@
 	return 0;
 }
 
+int
+pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask)
+{
+	if (!pci_dma_supported(dev, mask))
+		return -EIO;
+
+	dev->consistent_dma_mask = mask;
+
+	return 0;
+}
+
 static int __devinit pci_init(void)
 {
 	struct pci_dev *dev;
@@ -751,6 +762,7 @@
 EXPORT_SYMBOL(pci_clear_mwi);
 EXPORT_SYMBOL(pci_set_dma_mask);
 EXPORT_SYMBOL(pci_dac_set_dma_mask);
+EXPORT_SYMBOL(pci_set_consistent_dma_mask);
 EXPORT_SYMBOL(pci_assign_resource);
 EXPORT_SYMBOL(pci_find_parent_resource);
 
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/pci/probe.c linux-2.5.69-pci/drivers/pci/probe.c
--- linux-2.5.69-030509-vanilla/drivers/pci/probe.c	Sun May  4 19:53:35 2003
+++ linux-2.5.69-pci/drivers/pci/probe.c	Fri May 16 15:44:40 2003
@@ -502,6 +502,7 @@
 	/* Assume 32-bit PCI; let 64-bit PCI cards (which are far rarer)
 	   set this higher, assuming the system even supports it.  */
 	dev->dma_mask = 0xffffffff;
+	dev->consistent_dma_mask = 0xffffffff;
 	if (pci_setup_device(dev) < 0) {
 		kfree(dev);
 		return NULL;
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/include/linux/pci.h linux-2.5.69-pci/include/linux/pci.h
--- linux-2.5.69-030509-vanilla/include/linux/pci.h	Sun May  4 19:53:14 2003
+++ linux-2.5.69-pci/include/linux/pci.h	Fri May 16 15:42:55 2003
@@ -390,6 +390,11 @@
 					   or supports 64-bit transfers.  */
 	struct list_head pools;		/* pci_pools tied to this device */
 
+	u64		consistent_dma_mask;/* Like dma_mask, but for
+					       pci_alloc_consistent mappings as
+					       not all hardware supports
+					       64 bit addresses for consistent
+					       allocations such descriptors. */
 	u32             current_state;  /* Current operating state. In ACPI-speak,
 					   this is D0-D3, D0 being fully functional,
 					   and D3 being off. */
@@ -614,6 +619,7 @@
 void pci_clear_mwi(struct pci_dev *dev);
 int pci_set_dma_mask(struct pci_dev *dev, u64 mask);
 int pci_dac_set_dma_mask(struct pci_dev *dev, u64 mask);
+int pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask);
 int pci_assign_resource(struct pci_dev *dev, int i);
 
 /* Power management related routines */

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Jes Sorensen <jes@wildopensource.com>
   Date: 16 May 2003 17:02:36 -0400

   How does this patch look to you? It adds the generic code for setting
   the mask, it will then be up to Colin to fix the SGI SN2 specific code
   to honor it.
   
   I also added a patch to the tg3 driver to set the mask as it seems to
   work just dandy with 64 bit consistent addresses.
   
   Tested on ia64, but I see no reason why it should fail elsewhere.
   
I'd hate to say this but we probably need a
pci_consistent_dma_supported().

But this can wait and your patch is fine, EXCEPT the missing
documentation update :-)

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

On Fri, May 16, 2003 at 02:06:27PM -0700, David S. Miller wrote:
> 
> But this can wait and your patch is fine, EXCEPT the missing
> documentation update :-)

From my reading of the Tigon driver, test DMA relies on a 32 bit address
from dma_alloc_consistent, whereas other consistent allocations seem to
be fine with a 64 bit address.

Do you have any problem with a patch to drivers/net/tg3.c to enable
the test DMA to be 64 bit?

--- /usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20	2003-05-16 15:37:45.000000000 -0700
+++ linux/drivers/net/tg3.c	2003-05-16 15:35:40.000000000 -0700
@@ -6401,7 +6401,7 @@
 	tw32(FTQ_RESET, 0);
 
 	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;
 	test_desc.nic_mbuf = 0x00002100;
 	test_desc.len = size;

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jeremy Higdon]

> --- /usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20	2003-05-16 15:37:45.000000000 -0700
> +++ linux/drivers/net/tg3.c	2003-05-16 15:35:40.000000000 -0700
> @@ -6401,7 +6401,7 @@
>  	tw32(FTQ_RESET, 0);
>  
>  	/* pci_alloc_consistent gives only non-DAC addresses */
> -	test_desc.addr_hi = 0;
> +	test_desc.addr_hi = ((u64) buf_dma >> 32);
>  	test_desc.addr_lo = buf_dma & 0xffffffff;
>  	test_desc.nic_mbuf = 0x00002100;
>  	test_desc.len = size;
> 

I guess this is better:
--- /usr/tmp/TmpDir.15906-0/linux/drivers/net/tg3.c_1.20	2003-05-16 15:41:19.000000000 -0700
+++ linux/drivers/net/tg3.c	2003-05-16 15:39:13.000000000 -0700
@@ -6400,8 +6400,7 @@
 	tw32(BUFMGR_MODE, 0);
 	tw32(FTQ_RESET, 0);
 
-	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;
 	test_desc.nic_mbuf = 0x00002100;
 	test_desc.len = size;

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Jeremy Higdon <jeremy@classic.engr.sgi.com>
   Date: Fri, 16 May 2003 16:40:25 -0700 (PDT)
   
   I guess this is better:

I have no problems with this, Jes can add it to his diff.

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Grant Grundler]

On Fri, May 16, 2003 at 04:40:25PM -0700, Jeremy Higdon wrote:
> -	/* pci_alloc_consistent gives only non-DAC addresses */
> -	test_desc.addr_hi = 0;
> +	test_desc.addr_hi = ((u64) buf_dma >> 32);
>  	test_desc.addr_lo = buf_dma & 0xffffffff;

Just curious, is the "& 0xffffffff" really necessary?
I was expecting "(u32) buf_dma" to be sufficient.

grant

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Grant Grundler <iod00d@hp.com>
   Date: Fri, 16 May 2003 21:26:10 -0700

   On Fri, May 16, 2003 at 04:40:25PM -0700, Jeremy Higdon wrote:
   > -	/* pci_alloc_consistent gives only non-DAC addresses */
   > -	test_desc.addr_hi = 0;
   > +	test_desc.addr_hi = ((u64) buf_dma >> 32);
   >  	test_desc.addr_lo = buf_dma & 0xffffffff;
   
   Just curious, is the "& 0xffffffff" really necessary?
   I was expecting "(u32) buf_dma" to be sufficient.

Not necessary, just a stylistic choice.

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Jes Sorensen]

>>>>> "David" = David S Miller <davem@redhat.com> writes:

David>    From: Jes Sorensen <jes@wildopensource.com> Date: 16 May
David> 2003 17:02:36 -0400

>    How does this patch look to you? It adds the generic code
> for setting the mask, it will then be up to Colin to fix the
> SGI SN2 specific code to honor it.
   
>    I also added a patch to the tg3 driver to set the mask as it
> seems to work just dandy with 64 bit consistent addresses.
   
David> I'd hate to say this but we probably need a
David> pci_consistent_dma_supported().

Hi Dave,

I thought about this one, but from the system's point of view, I don't
see a difference between being able to do 64 bit DMA to consistent
allocations and 64 bit DMA to dynamic allocations. I'd say this is a
host adapter/driver problem for descriptor limitations as you pointed
out.

I am not strongly biased on this one, it just seems redundant to me.
I'll be happy to add it to the patch if you prefer that.

David> But this can wait and your patch is fine, EXCEPT the missing
David> documentation update :-)

I have an updated patch nearly ready, including fancy documentation
updates and Jeremy's fix to tg3.c. I will post this to linux-kernel
shortly.

Cheers,
Jes

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[Colin Ngam]

Jes Sorensen wrote:

> >>>>> "David" = David S Miller <davem@redhat.com> writes:
>
> David>    From: Jes Sorensen <jes@wildopensource.com> Date: 16 May
> David> 2003 17:02:36 -0400
>
> >    How does this patch look to you? It adds the generic code
> > for setting the mask, it will then be up to Colin to fix the
> > SGI SN2 specific code to honor it.
>
> >    I also added a patch to the tg3 driver to set the mask as it
> > seems to work just dandy with 64 bit consistent addresses.
>
> David> I'd hate to say this but we probably need a
> David> pci_consistent_dma_supported().
>
> Hi Dave,
>
> I thought about this one, but from the system's point of view, I don't
> see a difference between being able to do 64 bit DMA to consistent
> allocations and 64 bit DMA to dynamic allocations. I'd say this is a
> host adapter/driver problem for descriptor limitations as you pointed
> out.

If we define consistent DMA capability as a capability that is restricted and
controlled by the driver/adapter, yes, the system DMA capability, which is
what pci_dma_supported() returns, is all we need.  However, if we define
consistent DMA capability as a capability that the system/platform can
restrict, and it can be different than what pci_dma_supported(), then , yes
we will need pci_consistent_dma_supported().

I am rather bias towards treating this as a device driver/adapter restriction.
Unless,
it is also expected by this pci_alloc_consistent() interface to ensure that
actual
physical memory allocated is between 0-4G .. which on our platform is not
possible as we do not have such memory ranges.  I do not understand this to
be the case.  The requirement for a 32bits DMA handle to perform SAC has
nothing to to with the actual location of the Physical memory,  when this DMA
Handle is presented to the IO Chipset for either a DMA read or a DMA write on
the PCI Bus.

I am bias but both ways will work, with pci_consistent_dma_supported() providing

an option for platforms(not driver/adapter) that do restrict consistent DMA
capability.  Is this a valid system/platform restriction?  If so, we will need
pci_consistent_dma_supported().  Otherwise, no, we do not need this new
interface.

Thanks.

colin



>
>
> I am not strongly biased on this one, it just seems redundant to me.
> I'll be happy to add it to the patch if you prefer that.
>
> David> But this can wait and your patch is fine, EXCEPT the missing
> David> documentation update :-)
>
> I have an updated patch nearly ready, including fancy documentation
> updates and Jeremy's fix to tg3.c. I will post this to linux-kernel
> shortly.
>
> Cheers,
> Jes

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Fri, 2003-05-16 at 13:18, Russell King wrote:
> Isn't the common use (eg, net devices):
> 
> 	buffer passed from lower levels
> 
> 	map buffer
> 
> 	(device owns buffer)
> 
> 	device uses buffer
> 
> 	unmap buffer
> 
> 	(cpu owns buffer)
> 
> 	free buffer
> 
> So, to use your example:
> 
> 	(cpu owns region)
> 
> 	map
> 
> 	(device owns region)
> 
> 	dma_sync...cpu
> 
> 	(cpu owns region)
> 
> 	dma_sync...device
> 
> 	(device owns region)
> 
> 	unmap
> 
> 	(cpu owns region)

A grep through the tree implies that you (for arm) and me (for parisc)
are the only current implementors of the dma_ API.

The only consumers seem to be:  lasi_82596, 53c700 and sym53c8xx_comm.h
(all of which I did anyway), and usb/core/usb.c

How would you feel about making this API change?  I take it you have a
preference for using _to_cpu and _to_device call variants?

James

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent Interfaces.

[Russell King]

On Sat, May 17, 2003 at 01:36:01PM -0500, James Bottomley wrote:
> How would you feel about making this API change?

No problem.

> I take it you have a preference for using _to_cpu and _to_device
> call variants?

Thinking about it some, these could be confused with the direction of
DMA.

Would "dma_sync_for_device" and "dma_sync_for_cpu" fit better?

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[David S. Miller]

   From: Jes Sorensen <jes@wildopensource.com>
   Date: 17 May 2003 12:44:52 -0400
   
   I thought about this one, but from the system's point of view, I don't
   see a difference between being able to do 64 bit DMA to consistent
   allocations and 64 bit DMA to dynamic allocations. I'd say this is a
   host adapter/driver problem for descriptor limitations as you pointed
   out.

You're probably right, we'll see ;-)

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

[James Bottomley]

On Sat, 2003-05-17 at 16:09, Russell King wrote:
> On Sat, May 17, 2003 at 01:36:01PM -0500, James Bottomley wrote:
> > How would you feel about making this API change?
> 
> No problem.
> 
> > I take it you have a preference for using _to_cpu and _to_device
> > call variants?
> 
> Thinking about it some, these could be confused with the direction of
> DMA.
> 
> Would "dma_sync_for_device" and "dma_sync_for_cpu" fit better?

That's still a bit ambiguous.

I suppose dma_sync_ownership_to_device etc. would be descriptive but too
cumbersome...?

James