30 bits DMA and ppc

30 bits DMA and ppc

[Benjamin Herrenschmidt]

Hi !

I was thinking about the best ways to deal with the 30 bits limitation
of the Broadcom chips for DMA on PowerMacs. No emergency there, we still
need to have a working bcm43xx driver and the developpement can be done
with <= 1Gb of RAM, but it will have to be addressed at one point as we
are getting closer to something that is useable :)

I will not implement something like GFP_DMA, I think. It just sucks too
much...

However, what I can do is have the architecture code reserve a pool of
memory at boot if the machine main RAM is bigger than 1Gb, to use for
bounce-buffering. On the G5 with more than 2Gb, this is even easier
since I already have to blow away a 16Mb page for use by the IOMMU, but
the IOMMU only uses 2Mb in there, so I have about 14Mb that I could
re-use for that. On 32 bits machine, I can just reserve something early
during boot.

Now, how to actually make use of that pool. One way is to hack something
specific inside the bcm43xx driver. The pool can then be easily cut in
regions: the descriptor rings buffers, and 2 pools, one for Rx and one
for Tx. The allocation inside of those pools can be done as simple ring
buffer too due to the inherently ordered processing of packets.

However, the above would require arch specific hacks, and would only
work for one card in the system (too bad if you plug a cardbus one).

Another possibility that might be more interesting is to use swiotlb.
This is a somewhat generic bounce-buffering implementation of the DMA
mapping routines that is used by ia64 and x86_64 when no IOMMU is
available. It will automatically do nothing if the address fits the DMA
mask so it shouldn't add much overhead to other drivers and would "make
things work" transparently. In addition, for G5s with more than 2Gb of
RAM (which have an iommu), I could modify the iommu code to take into
account the DMA mask when allocating DMA virtual space. (The later would
have a slight risk of failure, but I doubt it will happen in practice,
as it would mean one has more than 1Gb of pending DMA at a given point
in time).

I tend to prefer the later solution ...

Anybody with strong disagreement with using swiotlb on PPC ? The choice
of wether to allocate RAM for it at boot and how much can be done per
platform and based on the amount of real RAM (for example, on pmac, I
suppose I would only allocate some if the physical RAM is more than 1Gb
since that's the limitation of those Broadcom chips and I don't see any
other potential user for it).

Besides, it may end up being useful for some crappy embedded stuffs (no
name here :)

Ben.
 

Re: [Bcm43xx-dev] 30 bits DMA and ppc

[Michael Buesch]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On Sunday 30 October 2005 05:03, Benjamin Herrenschmidt wrote:
[snip]
> However, the above would require arch specific hacks, and would only
> work for one card in the system (too bad if you plug a cardbus one).
[snap]

I think we should not need to modify the driver.
Drivers already set the DMA mask. As far as I can see, this mask
is currently (mostly) ignored on PPC and i386 (at least).
As far as I can see, PPC ignores it completely and i386 allocates in the
GFP_DMA region, if DMA is limited.
So I would say, it should be possible to use this mask
with some bounce buffers, as you suggested. That sounds like a good
solution to me. It's worth a try.

-- 
Greetings Michael.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 30 bits DMA and ppc

[Olof Johansson]

On Sun, Oct 30, 2005 at 03:03:55PM +1100, Benjamin Herrenschmidt wrote:

> However, what I can do is have the architecture code reserve a pool of
> memory at boot if the machine main RAM is bigger than 1Gb, to use for
> bounce-buffering. On the G5 with more than 2Gb, this is even easier
> since I already have to blow away a 16Mb page for use by the IOMMU, but
> the IOMMU only uses 2Mb in there, so I have about 14Mb that I could
> re-use for that. On 32 bits machine, I can just reserve something early
> during boot.

Keep in mind that those 16MB are cache inhibited. Not sure you'd want
that for the bounce buffer. And you can't map the same page as cacheable
or you'll end up in inconsistent state. I guess you could remap the 14MB
as 4K cacheable pages somewhere else.

> Now, how to actually make use of that pool. One way is to hack something
> specific inside the bcm43xx driver. The pool can then be easily cut in
> regions: the descriptor rings buffers, and 2 pools, one for Rx and one
> for Tx. The allocation inside of those pools can be done as simple ring
> buffer too due to the inherently ordered processing of packets.
> 
> However, the above would require arch specific hacks, and would only
> work for one card in the system (too bad if you plug a cardbus one).
> 
> Another possibility that might be more interesting is to use swiotlb.
> This is a somewhat generic bounce-buffering implementation of the DMA
> mapping routines that is used by ia64 and x86_64 when no IOMMU is
> available. It will automatically do nothing if the address fits the DMA
> mask so it shouldn't add much overhead to other drivers and would "make
> things work" transparently. In addition, for G5s with more than 2Gb of
> RAM (which have an iommu), I could modify the iommu code to take into
> account the DMA mask when allocating DMA virtual space. (The later would
> have a slight risk of failure, but I doubt it will happen in practice,
> as it would mean one has more than 1Gb of pending DMA at a given point
> in time).

Some of the Infiniband and Myrinet adapters like to map as much as they
possibly can. I'm not sure what the likeliness of them being used on a
machine at the same time as one of these crippled devices is though.
Besides, they usually back off a bit from allocating everything in the
system, so there should be some room.

> I tend to prefer the later solution ...

Sounds reasonable to me too. I guess time will tell how hairy it gets,
implementation-wise. The implementation could also be nicely abstracted
away and isolated thanks to Stephen's per-device-dma-ops stuff.


-Olof

Re: 30 bits DMA and ppc

[Benjamin Herrenschmidt]

> Keep in mind that those 16MB are cache inhibited. Not sure you'd want
> that for the bounce buffer. And you can't map the same page as cacheable
> or you'll end up in inconsistent state. I guess you could remap the 14MB
> as 4K cacheable pages somewhere else.

Euh... I think that's exactly what we do :) We _unmap_ the 16Mb page
from the linear mapping, and we remap a part of it using ioremap() (thus
as 4k pages) in the DART driver... The remaining bits are thus not
mapped at all, there is no problem using __ioremap() to get a cacheable
mapping there.

> Some of the Infiniband and Myrinet adapters like to map as much as they
> possibly can. I'm not sure what the likeliness of them being used on a
> machine at the same time as one of these crippled devices is though.

Why would this be a problem ? The infiniband driver would hopefully have
a sane dma_mask, and thus it's mapping requests wouldn't hit the swiotlb
code path.
. 
> Besides, they usually back off a bit from allocating everything in the
> system, so there should be some room.
> 
> > I tend to prefer the later solution ...
> 
> Sounds reasonable to me too. I guess time will tell how hairy it gets,
> implementation-wise. The implementation could also be nicely abstracted
> away and isolated thanks to Stephen's per-device-dma-ops stuff.

Yes, though that's not strictly necessary. The dma_mask should be enough
to tell us wether to use the normal code path or the swiotlb one. So if
swiotlb is enabled, it could just be added to the normal code path for
the non-iommu case. For the iommu case, I'm not sure. I think we don't
need bounce buffering. I could fairly easily have the DART code limit
allocations to a given DMA mask. The TCE one may have more issues since
the DMA window of a given slot may not fit the requirements, but in that
case, it's probably just a matter of failing all mapping requests.

Ben.

Re: 30 bits DMA and ppc

[Olof Johansson]

On Mon, Oct 31, 2005 at 08:14:18AM +1100, Benjamin Herrenschmidt wrote:
> 
> > Keep in mind that those 16MB are cache inhibited. Not sure you'd want
> > that for the bounce buffer. And you can't map the same page as cacheable
> > or you'll end up in inconsistent state. I guess you could remap the 14MB
> > as 4K cacheable pages somewhere else.
> 
> Euh... I think that's exactly what we do :) We _unmap_ the 16Mb page
> from the linear mapping, and we remap a part of it using ioremap() (thus
> as 4k pages) in the DART driver... The remaining bits are thus not
> mapped at all, there is no problem using __ioremap() to get a cacheable
> mapping there.

Sure, that would work.

> > Some of the Infiniband and Myrinet adapters like to map as much as they
> > possibly can. I'm not sure what the likeliness of them being used on a
> > machine at the same time as one of these crippled devices is though.
> 
> Why would this be a problem ? The infiniband driver would hopefully have
> a sane dma_mask, and thus it's mapping requests wouldn't hit the swiotlb
> code path.

Not a problem, I was just thinking out loud. IOMMU pressure might be
higher on these systems than the average one, but there should still be
enough room.

> > Sounds reasonable to me too. I guess time will tell how hairy it gets,
> > implementation-wise. The implementation could also be nicely abstracted
> > away and isolated thanks to Stephen's per-device-dma-ops stuff.
> 
> Yes, though that's not strictly necessary. The dma_mask should be enough
> to tell us wether to use the normal code path or the swiotlb one. So if
> swiotlb is enabled, it could just be added to the normal code path for
> the non-iommu case.

The non-iommu case might want to do that for other devices as well,
i.e. 32-bit limited ones on 64-bit machines.

> For the iommu case, I'm not sure. I think we don't
> need bounce buffering. I could fairly easily have the DART code limit
> allocations to a given DMA mask. The TCE one may have more issues since
> the DMA window of a given slot may not fit the requirements, but in that
> case, it's probably just a matter of failing all mapping requests.

Yep, the table is already split once, and I'm not sure in retrospect how
useful that split was anyway, it can maybe go away. Switching it around
shouldn't be a big issue.


-Olof

Re: 30 bits DMA and ppc

[Benjamin Herrenschmidt]

> The non-iommu case might want to do that for other devices as well,
> i.e. 32-bit limited ones on 64-bit machines.

Yup, that would be useful with some 3rd party bridges that don't have an
iommu ... :)

> Yep, the table is already split once, and I'm not sure in retrospect how
> useful that split was anyway, it can maybe go away. Switching it around
> shouldn't be a big issue.

No real need to split permanently... the search could be restricted to a
given range easily.

Ben.

Re: 30 bits DMA and ppc

[Olof Johansson]

On Mon, Oct 31, 2005 at 08:41:54AM +1100, Benjamin Herrenschmidt wrote:
> > Yep, the table is already split once, and I'm not sure in retrospect how
> > useful that split was anyway, it can maybe go away. Switching it around
> > shouldn't be a big issue.
> 
> No real need to split permanently... the search could be restricted to a
> given range easily.

Yeah, that's what it does now too, if it can't find anything in the
"right" part of the table, it'll try the other. So other devices would
use the same section if the "regular" table got full.


-Olof

exception vectors

[Ingmar]

Hi all,

I am trying to overwrite the exception vector space of an ibook G4 :).

I have set up (for every exception) a small piece of code, that's a prefix of a
handler to be called. My problem is, that writing the “small” chunks of code to
the exception vector space gives no problem(so it seams) but writing all the
pieces of code as one chunk gives a exception [dsi, dsisr 0x42000000], this
indicates a store problem.

I have tried different modes of copying, mmu on/off, also chanced the WING bit,
switched the exception prefix on.

I don't believe putting the exception vectors to there “place” one by one is the
right way, in the Linux kernel the kernel get relocated and the code comes into
place. I have taken this as an example, unfortunately to to result :(..

 - Am I overlooking something?
 - Is the a “standard way” to overwrite the exception vector space of a powerpc?

ThankZ,
  Ingmar

Re: exception vectors

[Benjamin Herrenschmidt]

On Mon, 2005-10-31 at 01:35 +0100, Ingmar wrote:
> Hi all,
> 
> I am trying to overwrite the exception vector space of an ibook G4 :).
> 
> I have set up (for every exception) a small piece of code, that's a prefix of a
> handler to be called. My problem is, that writing the “small” chunks of code to
> the exception vector space gives no problem(so it seams) but writing all the
> pieces of code as one chunk gives a exception [dsi, dsisr 0x42000000], this
> indicates a store problem.
> 
> I have tried different modes of copying, mmu on/off, also chanced the WING bit,
> switched the exception prefix on.
> 
> I don't believe putting the exception vectors to there “place” one by one is the
> right way, in the Linux kernel the kernel get relocated and the code comes into
> place. I have taken this as an example, unfortunately to to result :(..
> 
>  - Am I overlooking something?
>  - Is the a “standard way” to overwrite the exception vector space of a powerpc?

Well, you typically do this with the MMU disabled and making sure you
don't take an exception while copying over them...

Ben.

Re: exception vectors

[Hollis Blanchard]

On Oct 30, 2005, at 6:35 PM, Ingmar wrote:
>
> I am trying to overwrite the exception vector space of an ibook G4 :).

Why?

> I have set up (for every exception) a small piece of code, that's a=20
> prefix of a
> handler to be called. My problem is, that writing the =93small=94 =
chunks=20
> of code to
> the exception vector space gives no problem(so it seams) but writing=20=

> all the
> pieces of code as one chunk gives a exception [dsi, dsisr 0x42000000],=20=

> this
> indicates a store problem.

Code please. You're just calling memcpy?

> I have tried different modes of copying, mmu on/off, also chanced the=20=

> WING bit,
> switched the exception prefix on.

You could not have gotten a DSI if you disabled the MMU...

> I don't believe putting the exception vectors to there =93place=94 one =
by=20
> one is the
> right way, in the Linux kernel the kernel get relocated and the code=20=

> comes into
> place. I have taken this as an example, unfortunately to to result =
:(..

So you are not trying to overwrite Linux's functioning exception=20
handlers, but rather trying to write your own OS?

>  - Am I overlooking something?
>  - Is the a =93standard way=94 to overwrite the exception vector space =
of=20
> a powerpc?

Do you think this is a common task? :)

-Hollis

Re: exception vectors

[Ingmar]

On Sun, 30 Oct 2005 20:28:49 -0600, Hollis Blanchard wrote
> On Oct 30, 2005, at 6:35 PM, Ingmar wrote:
> >
> > I am trying to overwrite the exception vector space of an ibook G4 :).
> 
> Why?
> 

For my thesis project I am porting the Minix (v3) OS.

> > I have set up (for every exception) a small piece of code, that's a 
> > prefix of a
> > handler to be called. My problem is, that writing the small chunks 
> > of code to
> > the exception vector space gives no problem(so it seams) but writing 
> > all the
> > pieces of code as one chunk gives a exception [dsi, dsisr 0x42000000], 
> > this
> > indicates a store problem.
> 
> Code please. You're just calling memcpy?

I have done different methods (lots of code),
 - as a test just setting the memory, with memset,
 - custom copying word by word, flushing the cache while going,
 - flushing the cache afterwards,

the results are not always what I excected, the memset works if I don't set to
much like from 0x0 – 0x200. The copying would be the complete block.

> > I have tried different modes of copying, mmu on/off, also chanced the 
> > WING bits,
> > switched the exception prefix on.
> 
> You could not have gotten a DSI if you disabled the MMU...

Oke, than this is one I am sure of now, thanks :)

> > I don't believe putting the exception vectors to there place one by 
> > one is the
> > right way, in the Linux kernel the kernel get relocated and the code 
> > comes into
> > place. I have taken this as an example, unfortunately to to result :(..
> 
> So you are not trying to overwrite Linux's functioning exception 
> handlers, but rather trying to write your own OS?

Yes :).

> >  - Am I overlooking something?
> >  - Is the a standard way to overwrite the exception vector space of 
> > a powerpc?
> 
> Do you think this is a common task? :)

No the task maybe not, but the method could be,
 - I am sure (now) that the mmu must be off,
 - copying must be done with flushing the caches,

> -Hollis

Thanks
  Ingmar.

Re: exception vectors

[Olof Johansson]

On Mon, Oct 31, 2005 at 10:58:53AM +0100, Ingmar wrote:
> On Sun, 30 Oct 2005 20:28:49 -0600, Hollis Blanchard wrote
> > On Oct 30, 2005, at 6:35 PM, Ingmar wrote:
> > >
> > > I am trying to overwrite the exception vector space of an ibook G4 :).
> > 
> > Why?
> > 
> 
> For my thesis project I am porting the Minix (v3) OS.

Sounds like using qemu or some other emulator to do the PPC port first
might be an easier choice, and make debugging easier for you during
early bringup.


-Olof

Re: exception vectors

[Ingmar]

> > > > I am trying to overwrite the exception vector space of an ibook G4 :).
> > > 
> > > Why?
> > > 
> > 
> > For my thesis project I am porting the Minix (v3) OS.
> 
> Sounds like using qemu or some other emulator to do the PPC port first
> might be an easier choice, and make debugging easier for you during
> early bringup.

I was going to try that, before I got the ibook. Then I got the ibook and I
wanted to get my hand dirty. Nothing beats the real thing :)

But I have figured it out, I have the code working...

It are more or less (just) two things:
 - mmu off,
 - make sure u copy through the cache.

And of course make sure your secondary code is working as it should, then a emu
would help :).

Thanks for the reactions.
  Ingmar