DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Shawn Jin]

Hi,

The page table 'consistent_pte' covers the uncached DMA consistent
allocation space. Its size is only one page, each page has 512 PTEs.
That means only 2MB memory are available for DMA. For some
applications this is not enough. So how to eliminate this limitation?

I noticed that there are two configuration macros related to this
issue: CONFIG_CONSISTENT_START and CONFIG_CONSISTENT_SIZE. The current
value for CONFIG_CONSISTENT_SIZE is 2MB, which is exactly able to be
covered by one page. I guess that's why consistent_pte is initialized
to just one page. So simply changing CONFIG_CONSISTENT_SIZE cannot
remove the 2MB limitation. dma_alloc_init() has to be modified to
initialize consisten_pte to some pages which are enough to cover
CONFIG_CONSISTENT_SIZE memory space. Right?

Does increasing CONSISTENT_SIZE have any side effects?

Thanks,
-Shawn.

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Eugene Surovegin]

On Wed, May 18, 2005 at 05:54:39PM -0700, Shawn Jin wrote:
> The page table 'consistent_pte' covers the uncached DMA consistent
> allocation space. Its size is only one page, each page has 512 PTEs.
> That means only 2MB memory are available for DMA. For some
> applications this is not enough. So how to eliminate this limitation?

No, _all_ physical memory is available for DMA. "consistent" pool is 
used for small non-cached allocations, e.g. buffer descriptors, etc. 
Don't use it for actual data buffers.

-- 
Eugene

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Shawn Jin]

> No, _all_ physical memory is available for DMA. "consistent" pool is
> used for small non-cached allocations, e.g. buffer descriptors, etc.
> Don't use it for actual data buffers.

We used to pci_alloc_consistent() to allocate DMA buffers on 2.4.
There was no problem to allocate more than 2MB buffer. The
implementation of pci_alloc_consistent() on 2.6 differs from that on
2.4, which calls dma_alloc_coherent(). Following this, I traced down
to 'consistent_pte', which is the root cause of 2MB limitation.

Then what's your recommendation to allocate a big chunk of memory for
pci device? From this perspective, 2.6 is not compatible with 2.4,
which I think, is pretty bad.

Thanks,
-Shawn.

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Eugene Surovegin]

On Wed, May 18, 2005 at 06:52:52PM -0700, Shawn Jin wrote:
> Then what's your recommendation to allocate a big chunk of memory for
> pci device? 

My recommendation - don't do this. Why do you need to allocate this 
big chunk of consistent memory in the first place? You can do DMA 
_without_ allocating "consistent" memory. In fact, this is how 
virtually all devices work in Linux. For more info about DMA API - 
look at Documentation/DMA-API.txt.

Technically, you can make consistent pool bigger, if you really insist 
on using this approach.

-- 
Eugene

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Shawn Jin]

> My recommendation - don't do this. Why do you need to allocate this
> big chunk of consistent memory in the first place? You can do DMA
> _without_ allocating "consistent" memory. In fact, this is how
> virtually all devices work in Linux. For more info about DMA API -
> look at Documentation/DMA-API.txt.

The driver stack we've been developing (already 4.0) uses consistent
memory a lot for DMA. The stack is for many kinds of high performance
storage IO e.g. iSCSI, FC. It works fine on 2.4.x because there is no
such 2MB consistent pool limitation.

> Technically, you can make consistent pool bigger, if you really insist
> on using this approach.

I want to understand the motivation and the rationale of choosing
'consistent_pte' on 2.6.x, what impact there would be to increase the
consistent pool, and so on. Some pointers to articles, posts are more
helpful.

Thanks a lot,
-Shawn.

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Matt Porter]

On Fri, May 20, 2005 at 11:12:54AM -0700, Shawn Jin wrote:
> > My recommendation - don't do this. Why do you need to allocate this
> > big chunk of consistent memory in the first place? You can do DMA
> > _without_ allocating "consistent" memory. In fact, this is how
> > virtually all devices work in Linux. For more info about DMA API -
> > look at Documentation/DMA-API.txt.
> 
> The driver stack we've been developing (already 4.0) uses consistent
> memory a lot for DMA. The stack is for many kinds of high performance
> storage IO e.g. iSCSI, FC. It works fine on 2.4.x because there is no
> such 2MB consistent pool limitation.

Why are you using consistent memory for your DMA buffers?
 
> > Technically, you can make consistent pool bigger, if you really insist
> > on using this approach.
> 
> I want to understand the motivation and the rationale of choosing
> 'consistent_pte' on 2.6.x, what impact there would be to increase the
> consistent pool, and so on. Some pointers to articles, posts are more
> helpful.
 
The rationale on moving to the new implementation in 2.6 was to fix a
critical bug.  The bug could not be fixed with a band-aid solution, but
required an allocation system with different semantics. After considering
a few possibilities the basic framework from ARM was selected.

The bug that was fixed was the inability to allocate consistent memory
from an interrupt context. This is allowed by the PCI DMA API and the
more general DMA API and is used by many drivers.  In 2.4, these drivers
would simply BUG() on PPC4xx/8xx.

The difference with this implementation is that it doesn't allow for
arbitrarily huge consistent allocations by default.  However, I carefully
placed a comment at the top to point people to the advanced setup
menu where they can tune the consistent pool to be larger than the
2MB default.  You simply have to understand the memory map of your
board port when setting this option.  You can break things as with
other advanced setup menu options.

-Matt

Re: DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

[Shawn Jin]

First of all thanks for your attention. :)

> Why are you using consistent memory for your DMA buffers?

Both CPU and IO controller (e.g. FC controller) have to be aware of
changes the other party makes on the data buffer. Some architectures
guarantee this coherence through bus snooping, which ppc440 is
unfortunately lack of. The simplest solution is to use consistent
memory.. On 2.4.x to some extent we don't need worry about how much
consistent memory is available. So the developers used consistent
mappings a lot instead of streaming DMA mappings. We would have been
more conservative to use consistent mappings had we known the
limitation earlier.

> The difference with this implementation is that it doesn't allow for
> arbitrarily huge consistent allocations by default.  However, I carefully
> placed a comment at the top to point people to the advanced setup
> menu where they can tune the consistent pool to be larger than the
> 2MB default.  You simply have to understand the memory map of your

I noticed your comment about the advanced settings:
CONFIG_CONSISTENT_SIZE and _START. However simply increasing the value
of CONSISTENT_SIZE won't get rid of the limiation. Right? The root
cause is in consistent_pte, which is only one page. It may be good to
have consistent_pte have more than one pages according to the
CONSISTENT_SIZE. Is it a useful fix?

> board port when setting this option.  You can break things as with
> other advanced setup menu options.

OK. Have to be very careful here.

Best regards,
-Shawn.