* 83xx: Marking or Allocating Pages as Cache-Inhibited
@ 2009-03-06 4:32 Ben Menchaca
2009-03-06 5:22 ` Liu Dave-R63238
0 siblings, 1 reply; 17+ messages in thread
From: Ben Menchaca @ 2009-03-06 4:32 UTC (permalink / raw)
To: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]
I am working on a Freescale 8314e design, and the embedded device is
configured as a PCI-e endpoint running a 2.6.27-5 kernel. For context, we
have written a kernel module which, among other things, uses the RDMA/WDMA
engine in the PCI-e IP block. On the host side, these DMAs are coherent.
However, on the embedded side, things are quite a bit less rosy; we must
manually flush/invalidate cache lines for WDMA/RDMAs to occur successfully.
After speaking with (several) FAEs at Freescale, we believe there is a
configuration issue that is the cause, but we have yet to have anyone
successfully point to it.
Disabling the data cache altogether resolves the issue entirely, but of
course, also completely tanks performance. As a temporary workaround, I
would like to simply mark the pages (obtained currently via
dma_alloc_coherent) involved as cache-inhibited. I have attempted to do
this via some snippets remaining in fec.c (va_to_pte, uncache_pte to set
_PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is almost
certainly braindead; va_to_pte is not a part of the 83xx source, as far as I
can tell; 8xx only.
A quick pointer in the correct direction for marking pages as
cache-inhibited on a 2.6.27-5 kernel would be appreciated, or if my approach
to a workaround is flawed, a pointer to the correct way would be great.
Ben Menchaca
[-- Attachment #2: Type: text/html, Size: 1394 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 4:32 83xx: Marking or Allocating Pages as Cache-Inhibited Ben Menchaca
@ 2009-03-06 5:22 ` Liu Dave-R63238
2009-03-06 5:27 ` Liu Dave-R63238
0 siblings, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-06 5:22 UTC (permalink / raw)
To: Ben Menchaca, linuxppc-dev
Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP]
and PEX_RDMA_CTRL[SNOOP]?
What is the freq settings? CORE/CSB bus.
Thanks, Dave
________________________________
From: linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org
[mailto:linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org] On =
Behalf
Of Ben Menchaca
Sent: Friday, March 06, 2009 12:33 PM
To: linuxppc-dev@ozlabs.org
Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
=09
=09
I am working on a Freescale 8314e design, and the embedded
device is configured as a PCI-e endpoint running a 2.6.27-5 kernel. For
context, we have written a kernel module which, among other things, uses
the RDMA/WDMA engine in the PCI-e IP block. On the host side, these
DMAs are coherent. However, on the embedded side, things are quite a
bit less rosy; we must manually flush/invalidate cache lines for
WDMA/RDMAs to occur successfully. After speaking with (several) FAEs at
Freescale, we believe there is a configuration issue that is the cause,
but we have yet to have anyone successfully point to it. =20
=09
Disabling the data cache altogether resolves the issue entirely,
but of course, also completely tanks performance. As a temporary
workaround, I would like to simply mark the pages (obtained currently
via dma_alloc_coherent) involved as cache-inhibited. I have attempted
to do this via some snippets remaining in fec.c (va_to_pte, uncache_pte
to set _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
almost certainly braindead; va_to_pte is not a part of the 83xx source,
as far as I can tell; 8xx only.
=09
A quick pointer in the correct direction for marking pages as
cache-inhibited on a 2.6.27-5 kernel would be appreciated, or if my
approach to a workaround is flawed, a pointer to the correct way would
be great.
=09
Ben Menchaca
=09
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:22 ` Liu Dave-R63238
@ 2009-03-06 5:27 ` Liu Dave-R63238
2009-03-06 5:38 ` Ben Menchaca
0 siblings, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-06 5:27 UTC (permalink / raw)
To: Liu Dave-R63238, Ben Menchaca, linuxppc-dev
and what settings is DMA description bit 3?
> -----Original Message-----
> From: linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org=20
> [mailto:linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca; linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>=20
> Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and=20
> PEX_RDMA_CTRL[SNOOP]?
>=20
> What is the freq settings? CORE/CSB bus.
>=20
> Thanks, Dave
>=20
> ________________________________
>=20
> From: linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu=3Dfreescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009 12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
> =09
> =09
> I am working on a Freescale 8314e design, and the=20
> embedded device is configured as a PCI-e endpoint running a=20
> 2.6.27-5 kernel. For context, we have written a kernel=20
> module which, among other things, uses the RDMA/WDMA engine=20
> in the PCI-e IP block. On the host side, these DMAs are=20
> coherent. However, on the embedded side, things are quite a=20
> bit less rosy; we must manually flush/invalidate cache lines=20
> for WDMA/RDMAs to occur successfully. After speaking with=20
> (several) FAEs at Freescale, we believe there is a=20
> configuration issue that is the cause, but we have yet to=20
> have anyone successfully point to it. =20
> =09
> Disabling the data cache altogether resolves the issue=20
> entirely, but of course, also completely tanks performance. =20
> As a temporary workaround, I would like to simply mark the=20
> pages (obtained currently via dma_alloc_coherent) involved as=20
> cache-inhibited. I have attempted to do this via some=20
> snippets remaining in fec.c (va_to_pte, uncache_pte to set=20
> _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is=20
> almost certainly braindead; va_to_pte is not a part of the=20
> 83xx source, as far as I can tell; 8xx only.
> =09
> A quick pointer in the correct direction for marking=20
> pages as cache-inhibited on a 2.6.27-5 kernel would be=20
> appreciated, or if my approach to a workaround is flawed, a=20
> pointer to the correct way would be great.
> =09
> Ben Menchaca
> =09
>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>=20
>=20
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:27 ` Liu Dave-R63238
@ 2009-03-06 5:38 ` Ben Menchaca
2009-03-06 5:40 ` Liu Dave-R63238
2009-03-06 5:49 ` Liu Dave-R63238
0 siblings, 2 replies; 17+ messages in thread
From: Ben Menchaca @ 2009-03-06 5:38 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]
1. BAT2 in linux is set to WIMG=0010, and covers all 64M
2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
5. The DMA descriptor (randomly chosen when I hit a trigger...just ignore
the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
> and what settings is DMA description bit 3?
>
> > -----Original Message-----
> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
> freescale.com@ozlabs.org]
> > On Behalf Of Liu Dave-R63238
> > Sent: Friday, March 06, 2009 1:22 PM
> > To: Ben Menchaca; linuxppc-dev@ozlabs.org
> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
> >
> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
> > PEX_RDMA_CTRL[SNOOP]?
> >
> > What is the freq settings? CORE/CSB bus.
> >
> > Thanks, Dave
> >
> > ________________________________
> >
> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
> freescale.com@ozlabs.org]
> > On Behalf Of Ben Menchaca
> > Sent: Friday, March 06, 2009 12:33 PM
> > To: linuxppc-dev@ozlabs.org
> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
> >
> >
> > I am working on a Freescale 8314e design, and the
> > embedded device is configured as a PCI-e endpoint running a
> > 2.6.27-5 kernel. For context, we have written a kernel
> > module which, among other things, uses the RDMA/WDMA engine
> > in the PCI-e IP block. On the host side, these DMAs are
> > coherent. However, on the embedded side, things are quite a
> > bit less rosy; we must manually flush/invalidate cache lines
> > for WDMA/RDMAs to occur successfully. After speaking with
> > (several) FAEs at Freescale, we believe there is a
> > configuration issue that is the cause, but we have yet to
> > have anyone successfully point to it.
> >
> > Disabling the data cache altogether resolves the issue
> > entirely, but of course, also completely tanks performance.
> > As a temporary workaround, I would like to simply mark the
> > pages (obtained currently via dma_alloc_coherent) involved as
> > cache-inhibited. I have attempted to do this via some
> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
> > almost certainly braindead; va_to_pte is not a part of the
> > 83xx source, as far as I can tell; 8xx only.
> >
> > A quick pointer in the correct direction for marking
> > pages as cache-inhibited on a 2.6.27-5 kernel would be
> > appreciated, or if my approach to a workaround is flawed, a
> > pointer to the correct way would be great.
> >
> > Ben Menchaca
> >
> >
> > _______________________________________________
> > Linuxppc-dev mailing list
> > Linuxppc-dev@ozlabs.org
> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
> >
> >
>
[-- Attachment #2: Type: text/html, Size: 5138 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:38 ` Ben Menchaca
@ 2009-03-06 5:40 ` Liu Dave-R63238
2009-03-06 6:10 ` Ben Menchaca
2009-03-06 5:49 ` Liu Dave-R63238
1 sibling, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-06 5:40 UTC (permalink / raw)
To: Ben Menchaca; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3629 bytes --]
what is the value of ACR register?
________________________________
From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 1:38 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited
1. BAT2 in linux is set to WIMG=0010, and covers all 64M
2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the
DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
5. The DMA descriptor (randomly chosen when I hit a
trigger...just ignore the size...) contains 0002AFF3 at offset 0, so
nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
and what settings is DMA description bit 3?
> -----Original Message-----
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca; linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or Allocating Pages as
Cache-Inhibited
>
> Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP]
and
> PEX_RDMA_CTRL[SNOOP]?
>
> What is the freq settings? CORE/CSB bus.
>
> Thanks, Dave
>
> ________________________________
>
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009 12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or Allocating Pages as
Cache-Inhibited
>
>
> I am working on a Freescale 8314e design, and
the
> embedded device is configured as a PCI-e endpoint
running a
> 2.6.27-5 kernel. For context, we have written a
kernel
> module which, among other things, uses the RDMA/WDMA
engine
> in the PCI-e IP block. On the host side, these DMAs
are
> coherent. However, on the embedded side, things are
quite a
> bit less rosy; we must manually flush/invalidate cache
lines
> for WDMA/RDMAs to occur successfully. After speaking
with
> (several) FAEs at Freescale, we believe there is a
> configuration issue that is the cause, but we have yet
to
> have anyone successfully point to it.
>
> Disabling the data cache altogether resolves the
issue
> entirely, but of course, also completely tanks
performance.
> As a temporary workaround, I would like to simply mark
the
> pages (obtained currently via dma_alloc_coherent)
involved as
> cache-inhibited. I have attempted to do this via some
> snippets remaining in fec.c (va_to_pte, uncache_pte to
set
> _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but
this is
> almost certainly braindead; va_to_pte is not a part of
the
> 83xx source, as far as I can tell; 8xx only.
>
> A quick pointer in the correct direction for
marking
> pages as cache-inhibited on a 2.6.27-5 kernel would be
> appreciated, or if my approach to a workaround is
flawed, a
> pointer to the correct way would be great.
>
> Ben Menchaca
>
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>
[-- Attachment #2: Type: text/html, Size: 6497 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:38 ` Ben Menchaca
2009-03-06 5:40 ` Liu Dave-R63238
@ 2009-03-06 5:49 ` Liu Dave-R63238
2009-03-08 15:19 ` Timur Tabi
1 sibling, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-06 5:49 UTC (permalink / raw)
To: Liu Dave-R63238, Ben Menchaca; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4004 bytes --]
could you try to set '1' to DMA description bit3?
________________________________
From: Liu Dave-R63238
Sent: Friday, March 06, 2009 1:41 PM
To: 'Ben Menchaca'
Cc: linuxppc-dev@ozlabs.org
Subject: RE: 83xx: Marking or Allocating Pages as
Cache-Inhibited
what is the value of ACR register?
________________________________
From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 1:38 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited
1. BAT2 in linux is set to WIMG=0010, and covers all
64M
2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54):
0x1020
3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation
of the DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
5. The DMA descriptor (randomly chosen when I hit a
trigger...just ignore the size...) contains 0002AFF3 at offset 0, so
nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
and what settings is DMA description bit 3?
> -----Original Message-----
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca; linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or Allocating Pages
as Cache-Inhibited
>
> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and
> PEX_RDMA_CTRL[SNOOP]?
>
> What is the freq settings? CORE/CSB bus.
>
> Thanks, Dave
>
> ________________________________
>
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009 12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or Allocating
Pages as Cache-Inhibited
>
>
> I am working on a Freescale 8314e
design, and the
> embedded device is configured as a PCI-e
endpoint running a
> 2.6.27-5 kernel. For context, we have written
a kernel
> module which, among other things, uses the
RDMA/WDMA engine
> in the PCI-e IP block. On the host side,
these DMAs are
> coherent. However, on the embedded side,
things are quite a
> bit less rosy; we must manually
flush/invalidate cache lines
> for WDMA/RDMAs to occur successfully. After
speaking with
> (several) FAEs at Freescale, we believe there
is a
> configuration issue that is the cause, but we
have yet to
> have anyone successfully point to it.
>
> Disabling the data cache altogether
resolves the issue
> entirely, but of course, also completely tanks
performance.
> As a temporary workaround, I would like to
simply mark the
> pages (obtained currently via
dma_alloc_coherent) involved as
> cache-inhibited. I have attempted to do this
via some
> snippets remaining in fec.c (va_to_pte,
uncache_pte to set
> _PAGE_NO_CACHE, flush_tlb_page, then
unmap_pte), but this is
> almost certainly braindead; va_to_pte is not a
part of the
> 83xx source, as far as I can tell; 8xx only.
>
> A quick pointer in the correct direction
for marking
> pages as cache-inhibited on a 2.6.27-5 kernel
would be
> appreciated, or if my approach to a workaround
is flawed, a
> pointer to the correct way would be great.
>
> Ben Menchaca
>
>
>
_______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
>
https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>
[-- Attachment #2: Type: text/html, Size: 7380 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:40 ` Liu Dave-R63238
@ 2009-03-06 6:10 ` Ben Menchaca
2009-03-06 6:30 ` Liu Dave-R63238
0 siblings, 1 reply; 17+ messages in thread
From: Ben Menchaca @ 2009-03-06 6:10 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3969 bytes --]
I can look at ACR morning...although I can say with a fair amount of
certainty that I have not changed it from the POR value.
I will try enabling No Snoop for CSB in the descriptor (bit 3, yes?)...this
seems a bit counterintuitive to me.
What is the hope regarding these two? Some combination I am not seeing?
On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
> what is the value of ACR register?
>
> ------------------------------
> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
> *Sent:* Friday, March 06, 2009 1:38 PM
> *To:* Liu Dave-R63238
> *Cc:* linuxppc-dev@ozlabs.org
> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>
> 1. BAT2 in linux is set to WIMG=0010, and covers all 64M
> 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
> 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
> 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
> 5. The DMA descriptor (randomly chosen when I hit a trigger...just ignore
> the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>
> Core is 400MHz, and CSB is 133MHz.
>
> - Ben
>
> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>
>> and what settings is DMA description bit 3?
>>
>> > -----Original Message-----
>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
>> freescale.com@ozlabs.org]
>> > On Behalf Of Liu Dave-R63238
>> > Sent: Friday, March 06, 2009 1:22 PM
>> > To: Ben Menchaca; linuxppc-dev@ozlabs.org
>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>> >
>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>> > PEX_RDMA_CTRL[SNOOP]?
>> >
>> > What is the freq settings? CORE/CSB bus.
>> >
>> > Thanks, Dave
>> >
>> > ________________________________
>> >
>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
>> freescale.com@ozlabs.org]
>> > On Behalf Of Ben Menchaca
>> > Sent: Friday, March 06, 2009 12:33 PM
>> > To: linuxppc-dev@ozlabs.org
>> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>> >
>> >
>> > I am working on a Freescale 8314e design, and the
>> > embedded device is configured as a PCI-e endpoint running a
>> > 2.6.27-5 kernel. For context, we have written a kernel
>> > module which, among other things, uses the RDMA/WDMA engine
>> > in the PCI-e IP block. On the host side, these DMAs are
>> > coherent. However, on the embedded side, things are quite a
>> > bit less rosy; we must manually flush/invalidate cache lines
>> > for WDMA/RDMAs to occur successfully. After speaking with
>> > (several) FAEs at Freescale, we believe there is a
>> > configuration issue that is the cause, but we have yet to
>> > have anyone successfully point to it.
>> >
>> > Disabling the data cache altogether resolves the issue
>> > entirely, but of course, also completely tanks performance.
>> > As a temporary workaround, I would like to simply mark the
>> > pages (obtained currently via dma_alloc_coherent) involved as
>> > cache-inhibited. I have attempted to do this via some
>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>> > almost certainly braindead; va_to_pte is not a part of the
>> > 83xx source, as far as I can tell; 8xx only.
>> >
>> > A quick pointer in the correct direction for marking
>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>> > appreciated, or if my approach to a workaround is flawed, a
>> > pointer to the correct way would be great.
>> >
>> > Ben Menchaca
>> >
>> >
>> > _______________________________________________
>> > Linuxppc-dev mailing list
>> > Linuxppc-dev@ozlabs.org
>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>> >
>> >
>>
>
>
[-- Attachment #2: Type: text/html, Size: 6902 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 6:10 ` Ben Menchaca
@ 2009-03-06 6:30 ` Liu Dave-R63238
2009-03-06 16:12 ` Ben Menchaca
0 siblings, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-06 6:30 UTC (permalink / raw)
To: Ben Menchaca; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4553 bytes --]
Did you enable the descriptor bit 3 to have a try?
________________________________
From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 2:10 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited
I can look at ACR morning...although I can say with a fair
amount of certainty that I have not changed it from the POR value.
I will try enabling No Snoop for CSB in the descriptor (bit 3,
yes?)...this seems a bit counterintuitive to me.
What is the hope regarding these two? Some combination I am not
seeing?
On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
what is the value of ACR register?
________________________________
From: Ben Menchaca
[mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 1:38 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages
as Cache-Inhibited
1. BAT2 in linux is set to WIMG=0010, and
covers all 64M
2. PEX_DEVICE_CONTROL in PCI-E Config Space
(0x54): 0x1020
3. PEX_xDMA_CTRL is set to 0x00000401 at the
initiation of the DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
5. The DMA descriptor (randomly chosen when I
hit a trigger...just ignore the size...) contains 0002AFF3 at offset 0,
so nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
and what settings is DMA description
bit 3?
> -----Original Message-----
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca;
linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and
> PEX_RDMA_CTRL[SNOOP]?
>
> What is the freq settings? CORE/CSB
bus.
>
> Thanks, Dave
>
> ________________________________
>
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009
12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
>
> I am working on a Freescale
8314e design, and the
> embedded device is configured as a
PCI-e endpoint running a
> 2.6.27-5 kernel. For context, we have
written a kernel
> module which, among other things, uses
the RDMA/WDMA engine
> in the PCI-e IP block. On the host
side, these DMAs are
> coherent. However, on the embedded
side, things are quite a
> bit less rosy; we must manually
flush/invalidate cache lines
> for WDMA/RDMAs to occur successfully.
After speaking with
> (several) FAEs at Freescale, we
believe there is a
> configuration issue that is the cause,
but we have yet to
> have anyone successfully point to it.
>
> Disabling the data cache
altogether resolves the issue
> entirely, but of course, also
completely tanks performance.
> As a temporary workaround, I would
like to simply mark the
> pages (obtained currently via
dma_alloc_coherent) involved as
> cache-inhibited. I have attempted to
do this via some
> snippets remaining in fec.c
(va_to_pte, uncache_pte to set
> _PAGE_NO_CACHE, flush_tlb_page, then
unmap_pte), but this is
> almost certainly braindead; va_to_pte
is not a part of the
> 83xx source, as far as I can tell; 8xx
only.
>
> A quick pointer in the correct
direction for marking
> pages as cache-inhibited on a 2.6.27-5
kernel would be
> appreciated, or if my approach to a
workaround is flawed, a
> pointer to the correct way would be
great.
>
> Ben Menchaca
>
>
>
_______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
>
https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>
[-- Attachment #2: Type: text/html, Size: 8562 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 6:30 ` Liu Dave-R63238
@ 2009-03-06 16:12 ` Ben Menchaca
2009-03-06 16:30 ` Ben Menchaca
2009-03-07 21:51 ` Benjamin Herrenschmidt
0 siblings, 2 replies; 17+ messages in thread
From: Ben Menchaca @ 2009-03-06 16:12 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4687 bytes --]
Testing now...it looks like it (almost) works, though! Why does setting
no-snoop cause snooping to work? More on the effect on setting that bit in
a few minutes...need more testing.
ACR is 0x00030300.
- Ben
On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
> Did you enable the descriptor bit 3 to have a try?
>
> ------------------------------
> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
> *Sent:* Friday, March 06, 2009 2:10 PM
>
> *To:* Liu Dave-R63238
> *Cc:* linuxppc-dev@ozlabs.org
> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>
> I can look at ACR morning...although I can say with a fair amount of
> certainty that I have not changed it from the POR value.
>
> I will try enabling No Snoop for CSB in the descriptor (bit 3, yes?)...this
> seems a bit counterintuitive to me.
>
> What is the hope regarding these two? Some combination I am not seeing?
>
>
> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>
>> what is the value of ACR register?
>>
>> ------------------------------
>> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
>> *Sent:* Friday, March 06, 2009 1:38 PM
>> *To:* Liu Dave-R63238
>> *Cc:* linuxppc-dev@ozlabs.org
>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>
>> 1. BAT2 in linux is set to WIMG=0010, and covers all 64M
>> 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
>> 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
>> 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
>> 5. The DMA descriptor (randomly chosen when I hit a trigger...just ignore
>> the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>>
>> Core is 400MHz, and CSB is 133MHz.
>>
>> - Ben
>>
>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>>
>>> and what settings is DMA description bit 3?
>>>
>>> > -----Original Message-----
>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
>>> freescale.com@ozlabs.org]
>>> > On Behalf Of Liu Dave-R63238
>>> > Sent: Friday, March 06, 2009 1:22 PM
>>> > To: Ben Menchaca; linuxppc-dev@ozlabs.org
>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>> >
>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>>> > PEX_RDMA_CTRL[SNOOP]?
>>> >
>>> > What is the freq settings? CORE/CSB bus.
>>> >
>>> > Thanks, Dave
>>> >
>>> > ________________________________
>>> >
>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>=
>>> freescale.com@ozlabs.org]
>>> > On Behalf Of Ben Menchaca
>>> > Sent: Friday, March 06, 2009 12:33 PM
>>> > To: linuxppc-dev@ozlabs.org
>>> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>> >
>>> >
>>> > I am working on a Freescale 8314e design, and the
>>> > embedded device is configured as a PCI-e endpoint running a
>>> > 2.6.27-5 kernel. For context, we have written a kernel
>>> > module which, among other things, uses the RDMA/WDMA engine
>>> > in the PCI-e IP block. On the host side, these DMAs are
>>> > coherent. However, on the embedded side, things are quite a
>>> > bit less rosy; we must manually flush/invalidate cache lines
>>> > for WDMA/RDMAs to occur successfully. After speaking with
>>> > (several) FAEs at Freescale, we believe there is a
>>> > configuration issue that is the cause, but we have yet to
>>> > have anyone successfully point to it.
>>> >
>>> > Disabling the data cache altogether resolves the issue
>>> > entirely, but of course, also completely tanks performance.
>>> > As a temporary workaround, I would like to simply mark the
>>> > pages (obtained currently via dma_alloc_coherent) involved as
>>> > cache-inhibited. I have attempted to do this via some
>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>>> > almost certainly braindead; va_to_pte is not a part of the
>>> > 83xx source, as far as I can tell; 8xx only.
>>> >
>>> > A quick pointer in the correct direction for marking
>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>>> > appreciated, or if my approach to a workaround is flawed, a
>>> > pointer to the correct way would be great.
>>> >
>>> > Ben Menchaca
>>> >
>>> >
>>> > _______________________________________________
>>> > Linuxppc-dev mailing list
>>> > Linuxppc-dev@ozlabs.org
>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>> >
>>> >
>>>
>>
>>
>
[-- Attachment #2: Type: text/html, Size: 8684 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 16:12 ` Ben Menchaca
@ 2009-03-06 16:30 ` Ben Menchaca
2009-03-09 2:30 ` Liu Dave-R63238
2009-03-09 2:56 ` Liu Dave-R63238
2009-03-07 21:51 ` Benjamin Herrenschmidt
1 sibling, 2 replies; 17+ messages in thread
From: Ben Menchaca @ 2009-03-06 16:30 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 5496 bytes --]
Thank you for your help! That bit resolved all of the RDMA/WDMA coherency
issues on the CSB side...except:
We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters) to
the Host. This region is also not behaving correctly with respect to
coherency on SOME hosts; again, disabling our cache makes it work correctly
on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54)
to 0x2010 (sorry about the endianness below). We thought that CLEARING the
no-snoop bit here would indicate that snooping was required for this
region...is this a similar issue?
- Ben
On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.menchaca@gmail.com>wrote:
> Testing now...it looks like it (almost) works, though! Why does setting
> no-snoop cause snooping to work? More on the effect on setting that bit in
> a few minutes...need more testing.
>
> ACR is 0x00030300.
>
> - Ben
>
>
> On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>
>> Did you enable the descriptor bit 3 to have a try?
>>
>> ------------------------------
>> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
>> *Sent:* Friday, March 06, 2009 2:10 PM
>>
>> *To:* Liu Dave-R63238
>> *Cc:* linuxppc-dev@ozlabs.org
>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>
>> I can look at ACR morning...although I can say with a fair amount of
>> certainty that I have not changed it from the POR value.
>>
>> I will try enabling No Snoop for CSB in the descriptor (bit 3,
>> yes?)...this seems a bit counterintuitive to me.
>>
>> What is the hope regarding these two? Some combination I am not seeing?
>>
>>
>> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>>
>>> what is the value of ACR register?
>>>
>>> ------------------------------
>>> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
>>> *Sent:* Friday, March 06, 2009 1:38 PM
>>> *To:* Liu Dave-R63238
>>> *Cc:* linuxppc-dev@ozlabs.org
>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>
>>> 1. BAT2 in linux is set to WIMG=0010, and covers all 64M
>>> 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
>>> 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
>>> 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
>>> 5. The DMA descriptor (randomly chosen when I hit a trigger...just
>>> ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>>>
>>>
>>> Core is 400MHz, and CSB is 133MHz.
>>>
>>> - Ben
>>>
>>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>>>
>>>> and what settings is DMA description bit 3?
>>>>
>>>> > -----Original Message-----
>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>
>>>> =freescale.com@ozlabs.org]
>>>> > On Behalf Of Liu Dave-R63238
>>>> > Sent: Friday, March 06, 2009 1:22 PM
>>>> > To: Ben Menchaca; linuxppc-dev@ozlabs.org
>>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>> >
>>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>>>> > PEX_RDMA_CTRL[SNOOP]?
>>>> >
>>>> > What is the freq settings? CORE/CSB bus.
>>>> >
>>>> > Thanks, Dave
>>>> >
>>>> > ________________________________
>>>> >
>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>
>>>> =freescale.com@ozlabs.org]
>>>> > On Behalf Of Ben Menchaca
>>>> > Sent: Friday, March 06, 2009 12:33 PM
>>>> > To: linuxppc-dev@ozlabs.org
>>>> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>> >
>>>> >
>>>> > I am working on a Freescale 8314e design, and the
>>>> > embedded device is configured as a PCI-e endpoint running a
>>>> > 2.6.27-5 kernel. For context, we have written a kernel
>>>> > module which, among other things, uses the RDMA/WDMA engine
>>>> > in the PCI-e IP block. On the host side, these DMAs are
>>>> > coherent. However, on the embedded side, things are quite a
>>>> > bit less rosy; we must manually flush/invalidate cache lines
>>>> > for WDMA/RDMAs to occur successfully. After speaking with
>>>> > (several) FAEs at Freescale, we believe there is a
>>>> > configuration issue that is the cause, but we have yet to
>>>> > have anyone successfully point to it.
>>>> >
>>>> > Disabling the data cache altogether resolves the issue
>>>> > entirely, but of course, also completely tanks performance.
>>>> > As a temporary workaround, I would like to simply mark the
>>>> > pages (obtained currently via dma_alloc_coherent) involved as
>>>> > cache-inhibited. I have attempted to do this via some
>>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>>>> > almost certainly braindead; va_to_pte is not a part of the
>>>> > 83xx source, as far as I can tell; 8xx only.
>>>> >
>>>> > A quick pointer in the correct direction for marking
>>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>>>> > appreciated, or if my approach to a workaround is flawed, a
>>>> > pointer to the correct way would be great.
>>>> >
>>>> > Ben Menchaca
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Linuxppc-dev mailing list
>>>> > Linuxppc-dev@ozlabs.org
>>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
[-- Attachment #2: Type: text/html, Size: 10299 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 16:12 ` Ben Menchaca
2009-03-06 16:30 ` Ben Menchaca
@ 2009-03-07 21:51 ` Benjamin Herrenschmidt
2009-03-08 3:15 ` Ben Menchaca
1 sibling, 1 reply; 17+ messages in thread
From: Benjamin Herrenschmidt @ 2009-03-07 21:51 UTC (permalink / raw)
To: Ben Menchaca; +Cc: linuxppc-dev, Liu Dave-R63238
On Fri, 2009-03-06 at 10:12 -0600, Ben Menchaca wrote:
> Testing now...it looks like it (almost) works, though! Why does
> setting no-snoop cause snooping to work? More on the effect on
> setting that bit in a few minutes...need more testing.
Maybe they got the documentation for that bit backward ? :-)
Cheers,
Ben.
> ACR is 0x00030300.
>
> - Ben
>
> On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
> <DaveLiu@freescale.com> wrote:
> Did you enable the descriptor bit 3 to have a try?
>
>
> ______________________________________________________
> From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
>
> Sent: Friday, March 06, 2009 2:10 PM
>
>
> To: Liu Dave-R63238
> Cc: linuxppc-dev@ozlabs.org
> Subject: Re: 83xx: Marking or Allocating Pages as
> Cache-Inhibited
>
>
>
>
>
> I can look at ACR morning...although I can say with a
> fair amount of certainty that I have not changed it
> from the POR value.
>
> I will try enabling No Snoop for CSB in the descriptor
> (bit 3, yes?)...this seems a bit counterintuitive to
> me.
>
> What is the hope regarding these two? Some
> combination I am not seeing?
>
>
> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238
> <DaveLiu@freescale.com> wrote:
> what is the value of ACR register?
>
>
> ______________________________________
> From: Ben Menchaca
> [mailto:ben.menchaca@gmail.com]
> Sent: Friday, March 06, 2009 1:38 PM
> To: Liu Dave-R63238
> Cc: linuxppc-dev@ozlabs.org
> Subject: Re: 83xx: Marking or
> Allocating Pages as Cache-Inhibited
>
>
>
>
> 1. BAT2 in linux is set to WIMG=0010,
> and covers all 64M
> 2. PEX_DEVICE_CONTROL in PCI-E Config
> Space (0x54): 0x1020
> 3. PEX_xDMA_CTRL is set to 0x00000401
> at the initiation of the DMA.
> 4. OWAR0 is set to 0xFFFFF005, so
> NSNP is 0.
> 5. The DMA descriptor (randomly
> chosen when I hit a trigger...just
> ignore the size...) contains 0002AFF3
> at offset 0, so nosnoops are
> cleared.
>
> Core is 400MHz, and CSB is 133MHz.
>
> - Ben
>
> On Thu, Mar 5, 2009 at 11:27 PM, Liu
> Dave-R63238 <DaveLiu@freescale.com>
> wrote:
> and what settings is DMA
> description bit 3?
>
> > -----Original Message-----
> > From: linuxppc-dev-bounces
> +daveliu=freescale.com@ozlabs.org
> > [mailto:linuxppc-dev-bounces
> +daveliu=freescale.com@ozlabs.org]
>
>
> > On Behalf Of Liu
> Dave-R63238
> > Sent: Friday, March 06, 2009
> 1:22 PM
> > To: Ben Menchaca;
> linuxppc-dev@ozlabs.org
> > Subject: RE: 83xx: Marking
> or Allocating Pages as
> Cache-Inhibited
> >
> > Did you enable the snoop bit
> at PEX_WDMA_CTRL[SNOOP] and
> > PEX_RDMA_CTRL[SNOOP]?
> >
> > What is the freq settings?
> CORE/CSB bus.
> >
> > Thanks, Dave
> >
> >
> ________________________________
> >
> > From:
> linuxppc-dev-bounces
> +daveliu=freescale.com@ozlabs.org
> > [mailto:linuxppc-dev-bounces
> +daveliu=freescale.com@ozlabs.org]
> > On Behalf Of Ben Menchaca
> > Sent: Friday, March
> 06, 2009 12:33 PM
> > To:
> linuxppc-dev@ozlabs.org
> > Subject: 83xx: Marking
> or Allocating Pages as
> Cache-Inhibited
> >
> >
> > I am working on a
> Freescale 8314e design, and
> the
> > embedded device is
> configured as a PCI-e endpoint
> running a
> > 2.6.27-5 kernel. For
> context, we have written a
> kernel
> > module which, among other
> things, uses the RDMA/WDMA
> engine
> > in the PCI-e IP block. On
> the host side, these DMAs are
> > coherent. However, on the
> embedded side, things are
> quite a
> > bit less rosy; we must
> manually flush/invalidate
> cache lines
> > for WDMA/RDMAs to occur
> successfully. After speaking
> with
> > (several) FAEs at Freescale,
> we believe there is a
> > configuration issue that is
> the cause, but we have yet to
> > have anyone successfully
> point to it.
> >
> > Disabling the data
> cache altogether resolves the
> issue
> > entirely, but of course,
> also completely tanks
> performance.
> > As a temporary workaround, I
> would like to simply mark the
> > pages (obtained currently
> via dma_alloc_coherent)
> involved as
> > cache-inhibited. I have
> attempted to do this via some
> > snippets remaining in fec.c
> (va_to_pte, uncache_pte to set
> > _PAGE_NO_CACHE,
> flush_tlb_page, then
> unmap_pte), but this is
> > almost certainly braindead;
> va_to_pte is not a part of the
> > 83xx source, as far as I can
> tell; 8xx only.
> >
> > A quick pointer in the
> correct direction for marking
> > pages as cache-inhibited on
> a 2.6.27-5 kernel would be
> > appreciated, or if my
> approach to a workaround is
> flawed, a
> > pointer to the correct way
> would be great.
> >
> > Ben Menchaca
> >
> >
>
> >
> _______________________________________________
> > Linuxppc-dev mailing list
> > Linuxppc-dev@ozlabs.org
> >
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
> >
> >
>
>
>
>
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-07 21:51 ` Benjamin Herrenschmidt
@ 2009-03-08 3:15 ` Ben Menchaca
0 siblings, 0 replies; 17+ messages in thread
From: Ben Menchaca @ 2009-03-08 3:15 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev, Liu Dave-R63238
[-- Attachment #1: Type: text/plain, Size: 762 bytes --]
On Sat, Mar 7, 2009 at 3:51 PM, Benjamin Herrenschmidt <
benh@kernel.crashing.org> wrote:
> On Fri, 2009-03-06 at 10:12 -0600, Ben Menchaca wrote:
> > Testing now...it looks like it (almost) works, though! Why does
> > setting no-snoop cause snooping to work? More on the effect on
> > setting that bit in a few minutes...need more testing.
>
> Maybe they got the documentation for that bit backward ? :-)
For posterity...it does appear that this is the case. I don't have a bus
analyzer to watch the transaction, but a JTAG trigger caught the update
happening if and only if bit 3 was set in the (R|W)DMA descriptor. My FAE
at FS said he is watching this thread, so hopefully some doc errata can be
generated so others can avoid my confusion :-).
- Ben
[-- Attachment #2: Type: text/html, Size: 1091 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 5:49 ` Liu Dave-R63238
@ 2009-03-08 15:19 ` Timur Tabi
2009-03-09 2:33 ` Liu Dave-R63238
0 siblings, 1 reply; 17+ messages in thread
From: Timur Tabi @ 2009-03-08 15:19 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: Ben Menchaca, linuxppc-dev
On Fri, Mar 6, 2009 at 12:49 AM, Liu Dave-R63238 <DaveLiu@freescale.com> wrote:
> could you try to set '1' to DMA description bit3?
Dave,
I'm looking at the 8315e reference manual. What is "DMA description bit 3"?
--
Timur Tabi
Linux kernel developer at Freescale
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 16:30 ` Ben Menchaca
@ 2009-03-09 2:30 ` Liu Dave-R63238
2009-03-09 2:56 ` Liu Dave-R63238
1 sibling, 0 replies; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-09 2:30 UTC (permalink / raw)
To: Ben Menchaca; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6343 bytes --]
Hi Ben,
The second issue. you told me "some hosts" has problem,
and some hosts worked well.
what is the problem-hosts?
The issue seems like the hosts did set the NO SNOOP attribute
bit at TLP.
The PEX_DEVICE_CONTROL is standard PCI configuration space
register, it controls the behavior of the initiator's transaction.
For 8315, it is outbound, not inbound transaction.
Thanks, Dave
________________________________
From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
Sent: Saturday, March 07, 2009 12:30 AM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited
Thank you for your help! That bit resolved all of the RDMA/WDMA
coherency issues on the CSB side...except:
We expose a 1MB region of memory from CSB via a BAR (BAR1, if it
matters) to the Host. This region is also not behaving correctly with
respect to coherency on SOME hosts; again, disabling our cache makes it
work correctly on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to 0x2010 (sorry about the endianness below). We
thought that CLEARING the no-snoop bit here would indicate that snooping
was required for this region...is this a similar issue?
- Ben
On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca
<ben.menchaca@gmail.com> wrote:
Testing now...it looks like it (almost) works, though!
Why does setting no-snoop cause snooping to work? More on the effect on
setting that bit in a few minutes...need more testing.
ACR is 0x00030300.
- Ben
On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
Did you enable the descriptor bit 3 to have a
try?
________________________________
From: Ben Menchaca
[mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 2:10 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited
I can look at ACR morning...although I
can say with a fair amount of certainty that I have not changed it from
the POR value.
I will try enabling No Snoop for CSB in
the descriptor (bit 3, yes?)...this seems a bit counterintuitive to me.
What is the hope regarding these two?
Some combination I am not seeing?
On Thu, Mar 5, 2009 at 11:40 PM, Liu
Dave-R63238 <DaveLiu@freescale.com> wrote:
what is the value of ACR register?
________________________________
From: Ben Menchaca
[mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 1:38 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited
1. BAT2 in linux is set to WIMG=0010,
and covers all 64M
2. PEX_DEVICE_CONTROL in PCI-E Config
Space (0x54): 0x1020
3. PEX_xDMA_CTRL is set to 0x00000401
at the initiation of the DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP
is 0.
5. The DMA descriptor (randomly chosen
when I hit a trigger...just ignore the size...) contains 0002AFF3 at
offset 0, so nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu
Dave-R63238 <DaveLiu@freescale.com> wrote:
and what settings is DMA description
bit 3?
> -----Original Message-----
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca;
linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and
> PEX_RDMA_CTRL[SNOOP]?
>
> What is the freq settings? CORE/CSB
bus.
>
> Thanks, Dave
>
> ________________________________
>
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009
12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
>
> I am working on a Freescale
8314e design, and the
> embedded device is configured as a
PCI-e endpoint running a
> 2.6.27-5 kernel. For context, we have
written a kernel
> module which, among other things, uses
the RDMA/WDMA engine
> in the PCI-e IP block. On the host
side, these DMAs are
> coherent. However, on the embedded
side, things are quite a
> bit less rosy; we must manually
flush/invalidate cache lines
> for WDMA/RDMAs to occur successfully.
After speaking with
> (several) FAEs at Freescale, we
believe there is a
> configuration issue that is the cause,
but we have yet to
> have anyone successfully point to it.
>
> Disabling the data cache
altogether resolves the issue
> entirely, but of course, also
completely tanks performance.
> As a temporary workaround, I would
like to simply mark the
> pages (obtained currently via
dma_alloc_coherent) involved as
> cache-inhibited. I have attempted to
do this via some
> snippets remaining in fec.c
(va_to_pte, uncache_pte to set
> _PAGE_NO_CACHE, flush_tlb_page, then
unmap_pte), but this is
> almost certainly braindead; va_to_pte
is not a part of the
> 83xx source, as far as I can tell; 8xx
only.
>
> A quick pointer in the correct
direction for marking
> pages as cache-inhibited on a 2.6.27-5
kernel would be
> appreciated, or if my approach to a
workaround is flawed, a
> pointer to the correct way would be
great.
>
> Ben Menchaca
>
>
>
_______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
>
https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>
[-- Attachment #2: Type: text/html, Size: 14731 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-08 15:19 ` Timur Tabi
@ 2009-03-09 2:33 ` Liu Dave-R63238
0 siblings, 0 replies; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-09 2:33 UTC (permalink / raw)
To: Tabi Timur-B04825; +Cc: Ben Menchaca, linuxppc-dev
Timur,
See the section 14.7.1 DMA Description Format at page 14-112
of MPC8315ERM rev.1.
> -----Original Message-----
> From: timur.tabi@gmail.com [mailto:timur.tabi@gmail.com] On=20
> Behalf Of Tabi Timur-B04825
> Sent: Sunday, March 08, 2009 11:20 PM
> To: Liu Dave-R63238
> Cc: Ben Menchaca; linuxppc-dev@ozlabs.org
> Subject: Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>=20
> On Fri, Mar 6, 2009 at 12:49 AM, Liu Dave-R63238=20
> <DaveLiu@freescale.com> wrote:
> > could you try to set '1' to DMA description bit3?
>=20
> Dave,
>=20
> I'm looking at the 8315e reference manual. What is "DMA=20
> description bit 3"?
>=20
> --
> Timur Tabi
> Linux kernel developer at Freescale
>=20
>=20
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-06 16:30 ` Ben Menchaca
2009-03-09 2:30 ` Liu Dave-R63238
@ 2009-03-09 2:56 ` Liu Dave-R63238
2009-03-09 3:27 ` Ben Menchaca
1 sibling, 1 reply; 17+ messages in thread
From: Liu Dave-R63238 @ 2009-03-09 2:56 UTC (permalink / raw)
To: Ben Menchaca; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6343 bytes --]
Hi Ben,
The second issue. you told me "some hosts" has problem,
and some hosts worked well.
what is the problem-hosts?
The issue seems like the hosts did set the NO SNOOP attribute
bit at TLP.
The PEX_DEVICE_CONTROL is standard PCI configuration space
register, it controls the behavior of the initiator's transaction.
For 8315, it is outbound, not inbound transaction.
Thanks, Dave
________________________________
From: Ben Menchaca [mailto:ben.menchaca@gmail.com]
Sent: Saturday, March 07, 2009 12:30 AM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited
Thank you for your help! That bit resolved all of the RDMA/WDMA
coherency issues on the CSB side...except:
We expose a 1MB region of memory from CSB via a BAR (BAR1, if it
matters) to the Host. This region is also not behaving correctly with
respect to coherency on SOME hosts; again, disabling our cache makes it
work correctly on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to 0x2010 (sorry about the endianness below). We
thought that CLEARING the no-snoop bit here would indicate that snooping
was required for this region...is this a similar issue?
- Ben
On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca
<ben.menchaca@gmail.com> wrote:
Testing now...it looks like it (almost) works, though!
Why does setting no-snoop cause snooping to work? More on the effect on
setting that bit in a few minutes...need more testing.
ACR is 0x00030300.
- Ben
On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
<DaveLiu@freescale.com> wrote:
Did you enable the descriptor bit 3 to have a
try?
________________________________
From: Ben Menchaca
[mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 2:10 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited
I can look at ACR morning...although I
can say with a fair amount of certainty that I have not changed it from
the POR value.
I will try enabling No Snoop for CSB in
the descriptor (bit 3, yes?)...this seems a bit counterintuitive to me.
What is the hope regarding these two?
Some combination I am not seeing?
On Thu, Mar 5, 2009 at 11:40 PM, Liu
Dave-R63238 <DaveLiu@freescale.com> wrote:
what is the value of ACR register?
________________________________
From: Ben Menchaca
[mailto:ben.menchaca@gmail.com]
Sent: Friday, March 06, 2009 1:38 PM
To: Liu Dave-R63238
Cc: linuxppc-dev@ozlabs.org
Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited
1. BAT2 in linux is set to WIMG=0010,
and covers all 64M
2. PEX_DEVICE_CONTROL in PCI-E Config
Space (0x54): 0x1020
3. PEX_xDMA_CTRL is set to 0x00000401
at the initiation of the DMA.
4. OWAR0 is set to 0xFFFFF005, so NSNP
is 0.
5. The DMA descriptor (randomly chosen
when I hit a trigger...just ignore the size...) contains 0002AFF3 at
offset 0, so nosnoops are cleared.
Core is 400MHz, and CSB is 133MHz.
- Ben
On Thu, Mar 5, 2009 at 11:27 PM, Liu
Dave-R63238 <DaveLiu@freescale.com> wrote:
and what settings is DMA description
bit 3?
> -----Original Message-----
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Liu Dave-R63238
> Sent: Friday, March 06, 2009 1:22 PM
> To: Ben Menchaca;
linuxppc-dev@ozlabs.org
> Subject: RE: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and
> PEX_RDMA_CTRL[SNOOP]?
>
> What is the freq settings? CORE/CSB
bus.
>
> Thanks, Dave
>
> ________________________________
>
> From:
linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com@ozlabs.org]
> On Behalf Of Ben Menchaca
> Sent: Friday, March 06, 2009
12:33 PM
> To: linuxppc-dev@ozlabs.org
> Subject: 83xx: Marking or
Allocating Pages as Cache-Inhibited
>
>
> I am working on a Freescale
8314e design, and the
> embedded device is configured as a
PCI-e endpoint running a
> 2.6.27-5 kernel. For context, we have
written a kernel
> module which, among other things, uses
the RDMA/WDMA engine
> in the PCI-e IP block. On the host
side, these DMAs are
> coherent. However, on the embedded
side, things are quite a
> bit less rosy; we must manually
flush/invalidate cache lines
> for WDMA/RDMAs to occur successfully.
After speaking with
> (several) FAEs at Freescale, we
believe there is a
> configuration issue that is the cause,
but we have yet to
> have anyone successfully point to it.
>
> Disabling the data cache
altogether resolves the issue
> entirely, but of course, also
completely tanks performance.
> As a temporary workaround, I would
like to simply mark the
> pages (obtained currently via
dma_alloc_coherent) involved as
> cache-inhibited. I have attempted to
do this via some
> snippets remaining in fec.c
(va_to_pte, uncache_pte to set
> _PAGE_NO_CACHE, flush_tlb_page, then
unmap_pte), but this is
> almost certainly braindead; va_to_pte
is not a part of the
> 83xx source, as far as I can tell; 8xx
only.
>
> A quick pointer in the correct
direction for marking
> pages as cache-inhibited on a 2.6.27-5
kernel would be
> appreciated, or if my approach to a
workaround is flawed, a
> pointer to the correct way would be
great.
>
> Ben Menchaca
>
>
>
_______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
>
https://ozlabs.org/mailman/listinfo/linuxppc-dev
>
>
[-- Attachment #2: Type: text/html, Size: 14763 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
2009-03-09 2:56 ` Liu Dave-R63238
@ 2009-03-09 3:27 ` Ben Menchaca
0 siblings, 0 replies; 17+ messages in thread
From: Ben Menchaca @ 2009-03-09 3:27 UTC (permalink / raw)
To: Liu Dave-R63238; +Cc: linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 7201 bytes --]
The only problem host found so far is a newer Asus 970FX motherboard,
regardless of OS.
We are seeing that some time after LTSSM finishes, but long before OS load,
our PEX_DEVICE_CONTROL register is changed. On the "working" motherboards,
NO_SNOOP is enabled; if I read the spec correctly, this means that TLPs with
no-snoop are permitted. On the non-working motherboards, NO_SNOOP is
disabled; this is supposed to mean that TLPs with no-snoop are not permitted
(again, if I understand correctly). Is it possible that there has been
another misinterpretation of this bit? Something regarding the generation
of snoops on CSB when it is cleared?
This bit could be a complete red herring; it was one of the few differences
in the config space, however, so it was my best guess.
- Ben
On Sun, Mar 8, 2009 at 9:56 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
> Hi Ben,
>
> The second issue. you told me "some hosts" has problem,
> and some hosts worked well.
>
> what is the problem-hosts?
>
> The issue seems like the hosts did set the NO SNOOP attribute
> bit at TLP.
>
> The PEX_DEVICE_CONTROL is standard PCI configuration space
> register, it controls the behavior of the initiator's transaction.
> For 8315, it is outbound, not inbound transaction.
>
> Thanks, Dave
>
> ------------------------------
> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
> *Sent:* Saturday, March 07, 2009 12:30 AM
> *To:* Liu Dave-R63238
> *Cc:* linuxppc-dev@ozlabs.org
> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>
> Thank you for your help! That bit resolved all of the RDMA/WDMA coherency
> issues on the CSB side...except:
>
> We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters)
> to the Host. This region is also not behaving correctly with respect to
> coherency on SOME hosts; again, disabling our cache makes it work correctly
> on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54)
> to 0x2010 (sorry about the endianness below). We thought that CLEARING
> the no-snoop bit here would indicate that snooping was required for this
> region...is this a similar issue?
>
> - Ben
>
> On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.menchaca@gmail.com>wrote:
>
>> Testing now...it looks like it (almost) works, though! Why does setting
>> no-snoop cause snooping to work? More on the effect on setting that bit in
>> a few minutes...need more testing.
>>
>> ACR is 0x00030300.
>>
>> - Ben
>>
>>
>> On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>>
>>> Did you enable the descriptor bit 3 to have a try?
>>>
>>> ------------------------------
>>> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
>>> *Sent:* Friday, March 06, 2009 2:10 PM
>>>
>>> *To:* Liu Dave-R63238
>>> *Cc:* linuxppc-dev@ozlabs.org
>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>
>>> I can look at ACR morning...although I can say with a fair amount of
>>> certainty that I have not changed it from the POR value.
>>>
>>> I will try enabling No Snoop for CSB in the descriptor (bit 3,
>>> yes?)...this seems a bit counterintuitive to me.
>>>
>>> What is the hope regarding these two? Some combination I am not seeing?
>>>
>>>
>>> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu@freescale.com>wrote:
>>>
>>>> what is the value of ACR register?
>>>>
>>>> ------------------------------
>>>> *From:* Ben Menchaca [mailto:ben.menchaca@gmail.com]
>>>> *Sent:* Friday, March 06, 2009 1:38 PM
>>>> *To:* Liu Dave-R63238
>>>> *Cc:* linuxppc-dev@ozlabs.org
>>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>
>>>> 1. BAT2 in linux is set to WIMG=0010, and covers all 64M
>>>> 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
>>>> 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
>>>> 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0.
>>>> 5. The DMA descriptor (randomly chosen when I hit a trigger...just
>>>> ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>>>>
>>>>
>>>> Core is 400MHz, and CSB is 133MHz.
>>>>
>>>> - Ben
>>>>
>>>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu@freescale.com
>>>> > wrote:
>>>>
>>>>> and what settings is DMA description bit 3?
>>>>>
>>>>> > -----Original Message-----
>>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>>>> > [mailto:linuxppc-dev-bounces+daveliu<linuxppc-dev-bounces%2Bdaveliu>
>>>>> =freescale.com@ozlabs.org]
>>>>> > On Behalf Of Liu Dave-R63238
>>>>> > Sent: Friday, March 06, 2009 1:22 PM
>>>>> > To: Ben Menchaca; linuxppc-dev@ozlabs.org
>>>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>> >
>>>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>>>>> > PEX_RDMA_CTRL[SNOOP]?
>>>>> >
>>>>> > What is the freq settings? CORE/CSB bus.
>>>>> >
>>>>> > Thanks, Dave
>>>>> >
>>>>> > ________________________________
>>>>> >
>>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com@ozlabs.org
>>>>> > [mailto:linuxppc-dev-bounces+daveliu<linuxppc-dev-bounces%2Bdaveliu>
>>>>> =freescale.com@ozlabs.org]
>>>>> > On Behalf Of Ben Menchaca
>>>>> > Sent: Friday, March 06, 2009 12:33 PM
>>>>> > To: linuxppc-dev@ozlabs.org
>>>>> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>> >
>>>>> >
>>>>> > I am working on a Freescale 8314e design, and the
>>>>> > embedded device is configured as a PCI-e endpoint running a
>>>>> > 2.6.27-5 kernel. For context, we have written a kernel
>>>>> > module which, among other things, uses the RDMA/WDMA engine
>>>>> > in the PCI-e IP block. On the host side, these DMAs are
>>>>> > coherent. However, on the embedded side, things are quite a
>>>>> > bit less rosy; we must manually flush/invalidate cache lines
>>>>> > for WDMA/RDMAs to occur successfully. After speaking with
>>>>> > (several) FAEs at Freescale, we believe there is a
>>>>> > configuration issue that is the cause, but we have yet to
>>>>> > have anyone successfully point to it.
>>>>> >
>>>>> > Disabling the data cache altogether resolves the issue
>>>>> > entirely, but of course, also completely tanks performance.
>>>>> > As a temporary workaround, I would like to simply mark the
>>>>> > pages (obtained currently via dma_alloc_coherent) involved as
>>>>> > cache-inhibited. I have attempted to do this via some
>>>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>>>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>>>>> > almost certainly braindead; va_to_pte is not a part of the
>>>>> > 83xx source, as far as I can tell; 8xx only.
>>>>> >
>>>>> > A quick pointer in the correct direction for marking
>>>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>>>>> > appreciated, or if my approach to a workaround is flawed, a
>>>>> > pointer to the correct way would be great.
>>>>> >
>>>>> > Ben Menchaca
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > Linuxppc-dev mailing list
>>>>> > Linuxppc-dev@ozlabs.org
>>>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
[-- Attachment #2: Type: text/html, Size: 14998 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2009-03-09 3:27 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-03-06 4:32 83xx: Marking or Allocating Pages as Cache-Inhibited Ben Menchaca
2009-03-06 5:22 ` Liu Dave-R63238
2009-03-06 5:27 ` Liu Dave-R63238
2009-03-06 5:38 ` Ben Menchaca
2009-03-06 5:40 ` Liu Dave-R63238
2009-03-06 6:10 ` Ben Menchaca
2009-03-06 6:30 ` Liu Dave-R63238
2009-03-06 16:12 ` Ben Menchaca
2009-03-06 16:30 ` Ben Menchaca
2009-03-09 2:30 ` Liu Dave-R63238
2009-03-09 2:56 ` Liu Dave-R63238
2009-03-09 3:27 ` Ben Menchaca
2009-03-07 21:51 ` Benjamin Herrenschmidt
2009-03-08 3:15 ` Ben Menchaca
2009-03-06 5:49 ` Liu Dave-R63238
2009-03-08 15:19 ` Timur Tabi
2009-03-09 2:33 ` Liu Dave-R63238
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.