[Linux-ia64] Re: PCI DAC routines for SN

[Linux-ia64] Re: PCI DAC routines for SN

[David Mosberger]

>>>>> On Mon, 22 Apr 2002 15:31:56 -0700, Jesse Barnes <jbarnes@sgi.com> said:

  Jesse> Here's an implementation of the PCI DAC API for SN, along
  Jesse> with some fixes to the existing implementation.  If this
  Jesse> looks ok, I'd like it to get integrated into the next
  Jesse> release, otherwise I'll fix it up.  Please take a look.

You do realize that there it is never necessary to use this API for
ia64 linux?  dma_addr_t is already 64 bits.

	--david

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

Except that pci_alloc_consistent is supposed to return a 32 bit
address...

Jesse

On Mon, Apr 22, 2002 at 03:34:38PM -0700, David Mosberger wrote:
> >>>>> On Mon, 22 Apr 2002 15:31:56 -0700, Jesse Barnes <jbarnes@sgi.com> said:
> 
>   Jesse> Here's an implementation of the PCI DAC API for SN, along
>   Jesse> with some fixes to the existing implementation.  If this
>   Jesse> looks ok, I'd like it to get integrated into the next
>   Jesse> release, otherwise I'll fix it up.  Please take a look.
> 
> You do realize that there it is never necessary to use this API for
> ia64 linux?  dma_addr_t is already 64 bits.
> 
> 	--david

[Linux-ia64] Re: PCI DAC routines for SN

[David Mosberger]

>>>>> On Mon, 22 Apr 2002 15:39:46 -0700, Jesse Barnes <jbarnes@sgi.com> said:

  Jesse> Except that pci_alloc_consistent is supposed to return a 32
  Jesse> bit address...

Oh man, what a mess!  Have you checked with Dave Miller?  I suspect
he might not like this.  I'm not terribly fond of it either as it
makes it even more confusing as to when to use the DAC interface.  I'd
rather see a separate "alloc_consistent" routine, which doesn't isn't
on <4GB addresses.

	--david

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Mon, Apr 22, 2002 at 04:07:33PM -0700, David Mosberger wrote:
> Oh man, what a mess!  Have you checked with Dave Miller?  I suspect
> he might not like this.  I'm not terribly fond of it either as it

He was the one that implemented it I thought.  His assertion was that
since most chipsets don't handle 64 bit coherent allocations well, the
consistent interface should be forced to return a 32 bit address (is
that right Dave?).  I don't mind that as long as we add a DAC
consistent call, otherwise I'd like to leave it up to the platform
(i.e. ia64/sn could return a 64 bit address and sparc64 could do 32).

> makes it even more confusing as to when to use the DAC interface.  I'd
> rather see a separate "alloc_consistent" routine, which doesn't isn't
> on <4GB addresses.

I agree, Cc'ing Dave Miller.

Thanks,
Jesse

Re: [Linux-ia64] Re: PCI DAC routines for SN

[Grant Grundler]

Jesse Barnes wrote:
> Except that pci_alloc_consistent is supposed to return a 32 bit
> address...

And why is that a problem?
Doesn't (or couldn't) swiotlb allocate memory from a "DMA ZONE"?
(I've only worked on real IOMMU support for parisc-linux)

or do you have a driver/system config that might exhaust available
32-bit address space?

BTW, I agree the DAC support and SAC for pci_alloc_consistent()
is, uhm, not very clear.

thanks,
grant

Re: [Linux-ia64] Re: PCI DAC routines for SN

[Grant Grundler]

Jesse Barnes wrote:
> He was the one that implemented it I thought.  His assertion was that
> since most chipsets don't handle 64 bit coherent allocations well, the
> consistent interface should be forced to return a 32 bit address (is
> that right Dave?).

Right - HP recently changed ZX1 support based on the same assertion.
And for the drivers I'm aware of (eg tg3, sym53c8xx), Dave is right.

...
> otherwise I'd like to leave it up to the platform
> (i.e. ia64/sn could return a 64 bit address and sparc64 could do 32).

This is not feasible. The driver/device will use/support the same number
of address bits regardless of architecture. If the architecture provides
fewer bits than the driver/device supports, no problem obviously.

I'm curious to see what happens with the pci_dac_alloc_consistent() idea.

grant

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Mon, 22 Apr 2002 16:40:14 -0700

   On Mon, Apr 22, 2002 at 04:07:33PM -0700, David Mosberger wrote:
   > Oh man, what a mess!  Have you checked with Dave Miller?  I suspect
   > he might not like this.  I'm not terribly fond of it either as it
   
   He was the one that implemented it I thought.  His assertion was that
   since most chipsets don't handle 64 bit coherent allocations well, the
   consistent interface should be forced to return a 32 bit address (is
   that right Dave?).  I don't mind that as long as we add a DAC
   consistent call, otherwise I'd like to leave it up to the platform
   (i.e. ia64/sn could return a 64 bit address and sparc64 could do 32).
   
%99 of PCI chips out there do not support DAC addressing for things
like descriptor tables etc.  So it's not a matter of "well" it's
a matter of "at all".

Therefore pci_alloc_consistent MUST provide SAC only addressing.

I was seeing patches where people would set the DMA mask for the
pci_dev around pci_alloc_consistent calls in order to accomplish
getting SAC addresses.  That is exactly the kind of crap I was
trying to avoid.

Therefore, as per the API specification
(Documentation/DMA-mapping.txt) and reality, it's unacceptable
for pci_alloc_consistent() to return anything other than SAC
addresses (or something more constrained, if the DMA mask indicates
this, for example for devices with ISA addressing limitations).

I think it is unreasonable to add a special DAC alloc consistent
call.

Is this needed because you bozos don't have any physical memory below
4GB on some weird ia64 system ___AND___ you lack a PCI IOMMU in the
controllers again?  This is getting rediculious if so, and I really
want to avoid crapping up the PCI DMA interfaces just because the ia64
PCI hardware folks keep making stupid design decisions.

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Tue, Apr 23, 2002 at 09:04:10PM -0700, David S. Miller wrote:
> %99 of PCI chips out there do not support DAC addressing for things
> like descriptor tables etc.  So it's not a matter of "well" it's
> a matter of "at all".

Right.  But some platforms _can_ do DAC addressing for regions that
need to be coherent.  Seems like we shouldn't penalize good hardware
just because there's a lot of bad hardware out there.

> Therefore pci_alloc_consistent MUST provide SAC only addressing.
> 
> I was seeing patches where people would set the DMA mask for the
> pci_dev around pci_alloc_consistent calls in order to accomplish
> getting SAC addresses.  That is exactly the kind of crap I was
> trying to avoid.

Why would they need to do that?  If the driver sets its dma_mask to 64
bits, why can't the platform choose to return a 64 bit DMA address?
Obviously, if their device only supports 32 bits, then they'd set
their dma_mask accordingly and only get SAC addresses back from
alloc_consistent.  I'm obviously not understaning something...

> Therefore, as per the API specification
> (Documentation/DMA-mapping.txt) and reality, it's unacceptable
> for pci_alloc_consistent() to return anything other than SAC
> addresses (or something more constrained, if the DMA mask indicates
> this, for example for devices with ISA addressing limitations).
> 
> I think it is unreasonable to add a special DAC alloc consistent
> call.

Why?  It seems like a simple addition, even if not all platforms can
handle it.  Our platform won't work optimally if we don't either
change pci_alloc_consistent, or provide some other call to get 64 bit
coherent regions.

> Is this needed because you bozos don't have any physical memory below
> 4GB on some weird ia64 system ___AND___ you lack a PCI IOMMU in the
> controllers again?  This is getting rediculious if so, and I really
> want to avoid crapping up the PCI DMA interfaces just because the ia64
> PCI hardware folks keep making stupid design decisions.

Not at all.  SGI Origin hardware has PCI bridges that can coherently
access 64 bit DMA regions.  It can also map a limited number of 32 bit
addresses into arbitrary 4 GB memory windows of system memory.  The
number of mappings is somewhat limited however, since most devices are
expected to use 64 bit addresses directly.

Thanks,
Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Tue, 23 Apr 2002 22:49:48 -0700

   > Therefore pci_alloc_consistent MUST provide SAC only addressing.
   > 
   > I was seeing patches where people would set the DMA mask for the
   > pci_dev around pci_alloc_consistent calls in order to accomplish
   > getting SAC addresses.  That is exactly the kind of crap I was
   > trying to avoid.
   
   Why would they need to do that?  If the driver sets its dma_mask to 64
   bits, why can't the platform choose to return a 64 bit DMA address?
   Obviously, if their device only supports 32 bits, then they'd set
   their dma_mask accordingly and only get SAC addresses back from
   alloc_consistent.  I'm obviously not understaning something...
   
NO!  They were doing this to workaround the IA64 broken behavior.

They wanted "dma_mask = DAC, but give me SAC for consistent
allocations" (ie. what the code should do automatically for them as
per Documentation/DMA-mapping.txt) and they were messing with the
dma_mask around the pci_alloc_consistent() calls because of IA64
not following the API.

   > Is this needed because you bozos don't have any physical memory below
   > 4GB on some weird ia64 system ___AND___ you lack a PCI IOMMU in the
   > controllers again?  This is getting rediculious if so, and I really
   > want to avoid crapping up the PCI DMA interfaces just because the ia64
   > PCI hardware folks keep making stupid design decisions.
   
   Not at all.  SGI Origin hardware has PCI bridges that can coherently
   access 64 bit DMA regions.  It can also map a limited number of 32 bit
   addresses into arbitrary 4 GB memory windows of system memory.  The
   number of mappings is somewhat limited however, since most devices are
   expected to use 64 bit addresses directly.
   
Bad expectation, most devices cannot do this.  Like I suspected, bad
hardware design decisions are behind this.

I keep my position that there is no legitimate need to add this
interface besides perhaps broken hardware.

4GB of SAC addressible RAM is enough to satisfy even the most
convoluted use of consisten allocations.

Furthermore, and the important part you are ignoring, using DAC for
consistent allocations will incur a performance penalty because every
address cycle emitted by the device for a transaction will require
more cycles than if SAC addresses were only used for consistent
memory.

You have totally failed to consider this issue.  All you can think
about is "my system can do it, why can't I have the interface" without
once considering if that interface makes any sense to use at all even
if you can do it.

Re: [Linux-ia64] Re: PCI DAC routines for SN

[Grant Grundler]

Jesse Barnes wrote:
> Right.  But some platforms _can_ do DAC addressing for regions that
> need to be coherent.  Seems like we shouldn't penalize good hardware
> just because there's a lot of bad hardware out there.

Can you be more specific about which HW you are talking about?
(ie which PCI device can do 64-bit DAC to coherent regions)

thanks,
grant

Re: [Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Grant Grundler <grundler@cup.hp.com>
   Date: Wed, 24 Apr 2002 09:13:15 -0700

   Jesse Barnes wrote:
   > Right.  But some platforms _can_ do DAC addressing for regions that
   > need to be coherent.  Seems like we shouldn't penalize good hardware
   > just because there's a lot of bad hardware out there.
   
   Can you be more specific about which HW you are talking about?
   (ie which PCI device can do 64-bit DAC to coherent regions)
   
I remember that the qla2200's can...

Re: [Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Grant Grundler <grundler@cup.hp.com>
   Date: Tue, 23 Apr 2002 14:11:37 -0700
   
   I'm curious to see what happens with the pci_dac_alloc_consistent() idea.

I'm deep space 9'ing it currently.  It makes no sense
especially from a performance standpoint.

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

After an enlightening discussion with Mr. Miller, my patch stands as
is.  Does it look good enough to integrate into the next release?  If
not, what should I clean up?

Thanks a lot,
Jesse

On Mon, Apr 22, 2002 at 04:07:33PM -0700, David Mosberger wrote:
> >>>>> On Mon, 22 Apr 2002 15:39:46 -0700, Jesse Barnes <jbarnes@sgi.com> said:
> 
>   Jesse> Except that pci_alloc_consistent is supposed to return a 32
>   Jesse> bit address...
> 
> Oh man, what a mess!  Have you checked with Dave Miller?  I suspect
> he might not like this.  I'm not terribly fond of it either as it
> makes it even more confusing as to when to use the DAC interface.  I'd
> rather see a separate "alloc_consistent" routine, which doesn't isn't
> on <4GB addresses.
> 
> 	--david

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Tue, Apr 23, 2002 at 11:43:45PM -0700, David S. Miller wrote:
>    > Eh? You tell me that first.  You totally ignore my performance points
>    > and this makes it very difficult to discuss this issue with you.
>    
>    I understand that the performance will be lower, but out of curiosity,
>    do you know by how much?  Have you tried running (say) qlogicfc with
>    32 and 64 bit consistent addresses to measure the difference?
>    
> Every descriptor fetch will use two address cycles instead of
> one, I mean, do the math.

Ok, there are _many_ cards that can handle 64 bit coherent pointers,
including qlogic 1040, 1240, 1080, 1280, 12160, 2100, 2200, 2300,
2310, 2342, adaptec/jni 1160 (and followons, presumably).  The extra
address cycle is *rarely* an issue under most loads (e.g. on a 64k
scsi i/o, using SAC saves you 2 or 3 cycles out of 8000).  The
advantage of using 64 bit mappings is that you save 32 bit mappings
for cards that really need them.

So the penalty for using DAC comes down to ~.0375% for a 64k I/O, and
the benefit is that you don't have to use 32 bit mappings (which can
be scarce on some platforms).

In short, please reconsider your opposition to adding a
pci_dac_page_to_consistent type call.  It's an easy thing to add...

Thanks,
Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Wed, 24 Apr 2002 16:13:08 -0700
   
   So the penalty for using DAC comes down to ~.0375% for a 64k I/O, and
   the benefit is that you don't have to use 32 bit mappings (which can
   be scarce on some platforms).

No some, _your_ platform.   It is the only one with this problem.

You only need this 4GB memory for consistent memory, not
for anything else!  %99 of the kernel is going to use memory
for pages and data, not descriptors.

You can use zones and whatnot to make sure this isn't a problem at
all.

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Wed, 24 Apr 2002 17:11:19 -0700
   
   So if performance isn't an obstacle to implementing a new call, and
   I've shown that there are platforms where 64 bit consistent mappings
   are better to use than 32 bit mappings when possible, why can't we add
   it?  Or maybe just a seperate consistent_mask that drivers could set
   to indicate they support the operation and let the platform take care
   of it?
   
Where is the platform where it is a problem to keep around
enough of the _4_ __GIGABYTES__ of consistent memory available
at bootup?

If you can't keep around enough consistent memory within the
lower 4GB, you have other problems.

Devices use, EACH, something like a few kilobytes of consistent
memory for descriptors.  If you are going to tell me this is
a big deal, you have to be kidding me.

Why don't you tell me what the real problem is?

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Wed, Apr 24, 2002 at 04:53:31PM -0700, David S. Miller wrote:
>    From: Jesse Barnes <jbarnes@sgi.com>
>    Date: Wed, 24 Apr 2002 16:13:08 -0700
>    
>    So the penalty for using DAC comes down to ~.0375% for a 64k I/O, and
>    the benefit is that you don't have to use 32 bit mappings (which can
>    be scarce on some platforms).
> 
> No some, _your_ platform.   It is the only one with this problem.

Right, the platform that I'd like to see work well.

> You only need this 4GB memory for consistent memory, not
> for anything else!  %99 of the kernel is going to use memory
> for pages and data, not descriptors.
> 
> You can use zones and whatnot to make sure this isn't a problem at
> all.

I think you're confusing our platform with generic ia64 platforms.
I'm not worried about allocating in the low 4GB or whatever, I'm
concerned about using 32 bit mappings needlessly.  Of course, generic
ia64 boxes would also benefit...

So if performance isn't an obstacle to implementing a new call, and
I've shown that there are platforms where 64 bit consistent mappings
are better to use than 32 bit mappings when possible, why can't we add
it?  Or maybe just a seperate consistent_mask that drivers could set
to indicate they support the operation and let the platform take care
of it?

Thanks,
Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Wed, 24 Apr 2002 17:21:19 -0700

   On Wed, Apr 24, 2002 at 05:08:08PM -0700, David S. Miller wrote:
   > Devices use, EACH, something like a few kilobytes of consistent
   > memory for descriptors.
   
   I thought I did.  The issue isn't keeping 4gb around, it's using up
   the 32 bit mappings in the bridge for 64 bit capable cards.

Reread what I've quoted, these cards use almost nothing.

Consistent allocations account for maybe %2 to %6 of the total IOMMU
mappings on a fully loaded sparc64 bus.  It doesn't even show up on
the radar.

I've rolled the numbers before, I know it isn't a problem.

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Wed, Apr 24, 2002 at 05:08:08PM -0700, David S. Miller wrote:
>    So if performance isn't an obstacle to implementing a new call, and
>    I've shown that there are platforms where 64 bit consistent mappings
>    are better to use than 32 bit mappings when possible, why can't we add
>    it?  Or maybe just a seperate consistent_mask that drivers could set
>    to indicate they support the operation and let the platform take care
>    of it?
>    
> Where is the platform where it is a problem to keep around
> enough of the _4_ __GIGABYTES__ of consistent memory available
> at bootup?
> 
> If you can't keep around enough consistent memory within the
> lower 4GB, you have other problems.
> 
> Devices use, EACH, something like a few kilobytes of consistent
> memory for descriptors.  If you are going to tell me this is
> a big deal, you have to be kidding me.
> 
> Why don't you tell me what the real problem is?

I thought I did.  The issue isn't keeping 4gb around, it's using up
the 32 bit mappings in the bridge for 64 bit capable cards.

Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Wed, Apr 24, 2002 at 05:17:32PM -0700, David S. Miller wrote:
> Reread what I've quoted, these cards use almost nothing.
> 
> Consistent allocations account for maybe %2 to %6 of the total IOMMU
> mappings on a fully loaded sparc64 bus.  It doesn't even show up on
> the radar.
> 
> I've rolled the numbers before, I know it isn't a problem.

Ok, so there's no performance argument against adding the call.
You're just saying that since there are enough 32 bit mappings to go
around, using 64 bit mappings is silly.  Right?

Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Wed, 24 Apr 2002 17:36:52 -0700

   Ok, so there's no performance argument against adding the call.
   You're just saying that since there are enough 32 bit mappings to go
   around, using 64 bit mappings is silly.  Right?

We don't add things to Linux that we "might need" we add things
we actually need.  It is a slight performance problem too.

I don't think the hardware folks would be happy to hear that you're
willing to sacrifice real hardware cycles just for some DAC nirvana
or whatever it is you have for this issue.

[Linux-ia64] Re: PCI DAC routines for SN

[David S. Miller]

   From: Jesse Barnes <jbarnes@sgi.com>
   Date: Wed, 24 Apr 2002 18:01:59 -0700

   I fully understand and agree with that, and I don't propose adding the
   call just for the fun of it.  It's just that most of the time, it's
   easier software-wise to use the 64 bit consistent mappings if your
   device can support it (e.g. generic ia64, ia64/sn).
   
It isn't easier, in fact it's a lot harder.

The DMA api is confusing enough, and you want to add yet another
interface for device driver author Joe Blow to have to learn?

   I don't think the performance impact of those extra cycles would be
   measurable, but I'd be happy to see any numbers you might have to the
   contrary.
   
You already showed me the numbers.

Hardware folks don't make hardware so you can waste cycles, every
one counts.  I guess this is a fundamental disagreement in approach
between you and me.  So let's just agree to disagree.

You have to show me a significant advantage to add a completely new
API to the mix.  It has to be important enough that it is worth making
every driver author learn the new interface.

This basically boils down to the fact that it must do one of two
things:

1) It must make performance phenominally better.

   In the DAC consistent case, it actaully will decrease bus
   utilization slightly, we agree on this.

2) It must make some DMA performance optimization possible which
   currently is not possible at all.

   I do not believe this is the case.  You've tried to argue that
   SAC consistent memory is precious for mappings, and I've tried
   to show you that the consistent part of the mappings used by
   devices won't even show up on the radar.

We are at an impasse' and I really doubt this is going to change.
All you can do by arguing further is just make me type the same
things a few more times, and I'd like to avoid that, I type enough
already. :-)

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Wed, Apr 24, 2002 at 05:43:18PM -0700, David S. Miller wrote:
> We don't add things to Linux that we "might need" we add things
> we actually need.  It is a slight performance problem too.

I fully understand and agree with that, and I don't propose adding the
call just for the fun of it.  It's just that most of the time, it's
easier software-wise to use the 64 bit consistent mappings if your
device can support it (e.g. generic ia64, ia64/sn).

> I don't think the hardware folks would be happy to hear that you're
> willing to sacrifice real hardware cycles just for some DAC nirvana
> or whatever it is you have for this issue.

I don't think the performance impact of those extra cycles would be
measurable, but I'd be happy to see any numbers you might have to the
contrary.

Jesse

[Linux-ia64] Re: PCI DAC routines for SN

[Jesse Barnes]

On Wed, Apr 24, 2002 at 06:00:23PM -0700, David S. Miller wrote:
> The DMA api is confusing enough, and you want to add yet another
> interface for device driver author Joe Blow to have to learn?

What if it's just a simple flag (or mask like dma_mask) that the
driver optionally sets to true (or 64 bits)?  That way, the drivers
wouldn't have to change at all and the default behaviour would be
identical to the current API?

Of course, this only make sense if you acknowledge that (2) is at
least of some small concern (small, simple fix for a small, simple
problem).

> This basically boils down to the fact that it must do one of two
> things:
> 
> 1) It must make performance phenominally better.
> 
>    In the DAC consistent case, it actaully will decrease bus
>    utilization slightly, we agree on this.
> 
> 2) It must make some DMA performance optimization possible which
>    currently is not possible at all.
> 
>    I do not believe this is the case.  You've tried to argue that
>    SAC consistent memory is precious for mappings, and I've tried
>    to show you that the consistent part of the mappings used by
>    devices won't even show up on the radar.
> 
> We are at an impasse' and I really doubt this is going to change.
> All you can do by arguing further is just make me type the same
> things a few more times, and I'd like to avoid that, I type enough
> already. :-)

I know the feeling.  At any rate, thanks for the lively discussion.
It's been educational.

Jesse