Thoughts about DBDMA and cache coherency

Thoughts about DBDMA and cache coherency

[Benjamin Herrenschmidt]

Hi all !

I'm sure I'm wrong but I would like someone to point me to my error:

It appears that Apple's Darwin drivers use to flushe the cache of the
memory occupied by a DBDMA descriptor bloc before using it. This would
mean that the memory is not cache coherent when seen from the PCI bus,
and when you think about it, it looks logical: Writes issued from the PCI
to memory are coherent since snooped by the CPU and will force it to
reload the cache (I'm thinking about the 750 with backside cache, but
this may apply to other implementations depending on the bridge), but
when a device reads a piece of memory, i don't see the bridge asking the
CPU to flush this cache range before the read completes. I didn't see any
setting in Grackle to force this and in the case of the G3, the cache is
not handled by the bridge anyway but by the CPU itself...

That would mean that we need to flush the range occupied by DBDMA
descriptors, but also any buffers used by DBDMA when outputing via a
DBDMA channel.

Under MacOS, we use an OS function called PrepareMemoryForIO that will
take care of making pages resident, do appropriate cache flushing and
returns physical page tables for ranges of system memory. We can also,
for small buffers, use LockMemoryContiguous which will ... disable
caching for the selected pages.

Either I missed something big, either linux fails to do so and may have
unreliable writes to DBDMA devices all the time (looks like it would
crash a lot more often than it does, I must be wrong somewhere).

Someone has the solution ?

-- 
           E-Mail: <mailto:bh40@calva.net>
BenH.      Web   : <http://calvaweb.calvacom.fr/bh40/>





[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

Re: Thoughts about DBDMA and cache coherency

[Paul Mackerras]

Benjamin Herrenschmidt <bh40@calva.net> wrote:

> It appears that Apple's Darwin drivers use to flushe the cache of the
> memory occupied by a DBDMA descriptor bloc before using it. This would

The NuBus powermacs don't maintain cache coherency between memory and
the nubus AFAIK, but the PCI powermacs do.

> mean that the memory is not cache coherent when seen from the PCI bus,
> and when you think about it, it looks logical: Writes issued from the PCI
> to memory are coherent since snooped by the CPU and will force it to
> reload the cache (I'm thinking about the 750 with backside cache, but
> this may apply to other implementations depending on the bridge), but
> when a device reads a piece of memory, i don't see the bridge asking the
> CPU to flush this cache range before the read completes. I didn't see any

No, what happens (on the 60x bus) is that the read is snooped by the
cache controller, which asserts a `backoff and retry' signal.  The
PCI bridge then terminates the transaction, the cache controller gets
the 60x bus and writes the cache line out to memory, then the PCI
bridge retries the read and gets the right data from memory.

One small wrinkle is that for all this to work, the bridge has to
assert the GBL (global) signal for all its accesses.  On some
Starmaxes, we weren't getting the cache coherency maintained, but
someone (Harry Eaton, from memory) found a bit to set in the bridge's
configuration space which turned on the cache coherency - most likely
this bit controlled whether the GBL signal was asserted or not.

> That would mean that we need to flush the range occupied by DBDMA
> descriptors, but also any buffers used by DBDMA when outputing via a
> DBDMA channel.

Fortunately we don't need to, on PCI powermacs at least.  As I said,
nubus powermacs are a whole 'nother can of worms, as are 68k macs.

> Either I missed something big, either linux fails to do so and may have
> unreliable writes to DBDMA devices all the time (looks like it would
> crash a lot more often than it does, I must be wrong somewhere).

It would crash a lot more often.  Things just basically wouldn't work.

Paul.

[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

Re: Thoughts about DBDMA and cache coherency

[Benjamin Herrenschmidt]

On Fri, Mar 19, 1999, Paul Mackerras <paulus@cs.anu.edu.au> wrote:

>No, what happens (on the 60x bus) is that the read is snooped by the
>cache controller, which asserts a `backoff and retry' signal.  The
>PCI bridge then terminates the transaction, the cache controller gets
>the 60x bus and writes the cache line out to memory, then the PCI
>bridge retries the read and gets the right data from memory.

Ok, thanks for the clarification.

>One small wrinkle is that for all this to work, the bridge has to
>assert the GBL (global) signal for all its accesses.  On some
>Starmaxes, we weren't getting the cache coherency maintained, but
>someone (Harry Eaton, from memory) found a bit to set in the bridge's
>configuration space which turned on the cache coherency - most likely
>this bit controlled whether the GBL signal was asserted or not.

I beleive MacOS can run without it (MacOS probably assumes cache coherency
is not here since it makes DMA space uncachable).

>> That would mean that we need to flush the range occupied by DBDMA
>> descriptors, but also any buffers used by DBDMA when outputing via a
>> DBDMA channel.
>
>Fortunately we don't need to, on PCI powermacs at least.  As I said,
>nubus powermacs are a whole 'nother can of worms, as are 68k macs.

Hum, asun will have fun making them work... and I too since I promised I would
help fixing head.S and kernel mappings...

>> Either I missed something big, either linux fails to do so and may have
>> unreliable writes to DBDMA devices all the time (looks like it would
>> crash a lot more often than it does, I must be wrong somewhere).
>
>It would crash a lot more often.  Things just basically wouldn't work.

Yep

-- 
           E-Mail: <mailto:bh40@calva.net>
BenH.      Web   : <http://calvaweb.calvacom.fr/bh40/>





[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

egcs 1.1.2 big improvement over egcs 1.1.1 !

[Kevin B. Hendricks]

To Franz Sirl, Gary Thomas, and Anyone else involved with egcs development
on LinuxPPC,

THANK YOU!

I just finished recompiling the JDK 1.2 with Gary's egcs 1.1.2 rpm and for
the first time since egcs 1.0-2e, a build of the JDK will actually work
without much fuss.  None of the egcs 1.1 or 1.1.1 rpms was ever able to
perform this feat without having lots of silent compiler errors which were
next to impossible to track down.

THANK YOU!   THANK YOU!

If any of you are having trouble with projects compiled with egcs 1.1.1, I
highly recommend trying egcs 1.1.2.  The only problems I have found with it
are that the haifa scheduler sometimes changes the sequence enough to mess
up monitor/lock and unlock code but that can be worked around by not using
haifa for those files.

egcs 1.1.2 is a definite improvement!

Kevin



[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

Re: egcs 1.1.2 big improvement over egcs 1.1.1 !

[Franz Sirl]

At 18:02 20.03.99 , Kevin B. Hendricks wrote:
>If any of you are having trouble with projects compiled with egcs 1.1.1, I
>highly recommend trying egcs 1.1.2.  The only problems I have found with it
>are that the haifa scheduler sometimes changes the sequence enough to mess
>up monitor/lock and unlock code but that can be worked around by not using
>haifa for those files.

Can you boil down this bug to a testcase that exit(0)'s if the code runs 
OK, or abort()'s if there is a bug?
And I have still two egcs patches on my "maybe apply for R5 release" list, 
you can try them, if you are interested.

Franz.


[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

smbfs and smbmount and Paul's 2.2.1 kernel

[Kevin B. Hendricks]

Hi,

In order to prove the Linuxppc port of the JDK is correct I need to mount NT
filesystems on my LinuxPPC system using smbmount and smbumount which are part of
the smbfs rpm (Sun's JDK on an NT box provides a reference system to run the JCK
against).

To make smbfs work (I grabbed the src rpm and built it) I need to compile SMB
_FS into the kernel.  I grabbed Paul's latest kernel source and dot.config file
and noticed there was no support for SMB included in his kernel by default.

Before I go and roll my own, has anyone played with mounting NT filesystems
using smbmount and smbumount commands?  Is this stable on linuxppc?  Are there
any gotchas I should look out for with this filesystem?

Thanks,

Kevin

[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]

Re: smbfs and smbmount and Paul's 2.2.1 kernel

[Benjamin Herrenschmidt]

On Mon, Mar 22, 1999, Kevin B. Hendricks <kbhend@business.wm.edu> wrote:

>To make smbfs work (I grabbed the src rpm and built it) I need to compile SMB
>_FS into the kernel.  I grabbed Paul's latest kernel source and dot.config
>file
>and noticed there was no support for SMB included in his kernel by default.
>
>Before I go and roll my own, has anyone played with mounting NT filesystems
>using smbmount and smbumount commands?  Is this stable on linuxppc?  Are
there
>any gotchas I should look out for with this filesystem?

I use it regulary, compiled as a module, and it seems to work (but
displays tons of messages in the active console). I had to compile recent
version of smbmount (from a 2.0 samba package, If I remember correctly),
the ones bundled with R4 didn't work with the latest kernel smbfs.

-- 
           E-Mail: <mailto:bh40@calva.net>
BenH.      Web   : <http://calvaweb.calvacom.fr/bh40/>





[[ This message was sent via the linuxppc-dev mailing list.  Replies are ]]
[[ not  forced  back  to the list, so be sure to Cc linuxppc-dev if your ]]
[[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting.   ]]