MPC8641D PEX: programming OWBAR in Endpoint mode?

MPC8641D PEX: programming OWBAR in Endpoint mode?

[david.hagood]

I am trying to get the PCIe interfaces of a Freescale MPC8641D working in
endpoint mode (i.e. as a PCI device rather than a PCI root complex host).

I can get the device to show up on the host's PCI bus, I can program the
inbound ATMUs such that the BARS are updated when the host (re-)scans
them, but I cannot for the life of me get the PPC's Outbound ATMUS to
work.

When I attempt to program them, I can program ALL the registers EXCEPT the
OWBAR - which steadfastly remains 0 no matter what I write to it.

As a result, when I attempt to bus master out from the PPC to the PCIe
address spaces via the outbound ATMUs, I get a bus fault on the PPC as
there is no device at the address I am accessing.

I've double-checked the LAWs to make sure the PEX is mapped into local
space, I've put the OWBAR in that address space, I've tried different
outbound ATMUs, and NOTHING works. Not ATMU0, not ATMU1, etc.

I've been trying to work with our Freescale rep, but I am getting nowhere
on that front.

Does anybody have any suggestions on what I might be doing wrong? I mean,
it looks like it should be a simple out_be32(addr_of_OWBAR,value), just
like all the other accesses to the ATMU registers that seem to be working
(as in, I read back what I wrote).

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[Chen, Tiejun]

> -----Original Message-----
> From:=20
> linuxppc-dev-bounces+tiejun.chen=3Dwindriver.com@lists.ozlabs.or
> g=20
> [mailto:linuxppc-dev-bounces+tiejun.chen=3Dwindriver.com@lists.o
zlabs.org] On Behalf Of david.hagood@gmail.com
> Sent: Wednesday, September 22, 2010 11:56 PM
> To: linuxppc-dev@ozlabs.org
> Subject: MPC8641D PEX: programming OWBAR in Endpoint mode?
>=20
> I am trying to get the PCIe interfaces of a Freescale=20
> MPC8641D working in endpoint mode (i.e. as a PCI device=20
> rather than a PCI root complex host).
>=20

As far as I recalled on 4xx the u-boot support for dynamic configuration
of PCIe ports for the targets equipped with PCIe interfaces. Often this
is done via the "pcie_mode" environement variable on the u-boot prompt.
On there you can set to "EP" or "RP" for endpoint or rootpoint mode.

But for Freescale MPC86xx this should be set accordin to the external
configure pin.

> I can get the device to show up on the host's PCI bus, I can=20

This only ensure you can access the PCIe configure space.

> program the inbound ATMUs such that the BARS are updated when=20
> the host (re-)scans them, but I cannot for the life of me get

What value are configured to IntBound REGs?
=20
> the PPC's Outbound ATMUS to work.
>=20
> When I attempt to program them, I can program ALL the=20
> registers EXCEPT the OWBAR - which steadfastly remains 0 no=20
> matter what I write to it.
>=20

How do you configure OWS of PEXOWAR?

I means you still access that if OWS is match the whole target memory
size even when '0' is as the internal platform address.

> As a result, when I attempt to bus master out from the PPC to=20
> the PCIe address spaces via the outbound ATMUs, I get a bus=20
> fault on the PPC as there is no device at the address I am accessing.
>=20
> I've double-checked the LAWs to make sure the PEX is mapped=20
> into local space, I've put the OWBAR in that address space,=20
> I've tried different outbound ATMUs, and NOTHING works. Not=20
> ATMU0, not ATMU1, etc.
>=20
> I've been trying to work with our Freescale rep, but I am=20
> getting nowhere on that front.
>=20
> Does anybody have any suggestions on what I might be doing=20
> wrong? I mean, it looks like it should be a simple=20
> out_be32(addr_of_OWBAR,value), just like all the other=20
> accesses to the ATMU registers that seem to be working (as=20
> in, I read back what I wrote).

Out_be32 should be fine for atmu REGs. And also you can refe to the
function, setup_pci_atmu & setup_one_atmu, on the file,
arch/powerpc/sysdev/fsl_pci.c, to know how to access atmu REGs. Often
you should disable them, configure then enable/invoke atmu antry as
normal configuring sequent.

Additionally I'm a bit afraid your initial phase :) As you know PCIe
would be used as RC mode on Freescale PowerPC kernel. So I don't know if
you also drop this path on your kernel to conflict each other :)=20

Cheers
Tiejun

>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>=20

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[David Hagood]

On Thu, 2010-09-23 at 05:21 +0200, Chen, Tiejun wrote:
> > I can get the device to show up on the host's PCI bus, I can 
> 
> This only ensure you can access the PCIe configure space.
Not quite: I can also read the BARs that I program, and the memory
behind them on the PPC.
> 
> > program the inbound ATMUs such that the BARS are updated when 
> > the host (re-)scans them, but I cannot for the life of me get
> 
> What value are configured to IntBound REGs?
I can program them at run time via sysfs on the PPC's side, so there is
no single set of values. However, I am pointing them at the PPC's RAM
space, and as I stated above, I can read the PPC's RAM from the host
side via the BARs.

> How do you configure OWS of PEXOWAR?
> 
> I means you still access that if OWS is match the whole target memory
> size even when '0' is as the internal platform address.
As I understand it, not if the OWS is not correctly mapped on the PPC
side - the PEX outbound ATMU's OWBAR must be mapped to a region of the
PPCs address space that is also mapped to PEX in the LAW. The LAW does
NOT indicate that PPC address 0 is mapped to the PEX.

> 
> Out_be32 should be fine for atmu REGs. And also you can refe to the
> function, setup_pci_atmu & setup_one_atmu, on the file,
> arch/powerpc/sysdev/fsl_pci.c, to know how to access atmu REGs. Often
> you should disable them, configure then enable/invoke atmu antry as
> normal configuring sequent.
I have tried disabling the outbound ATMU when I program it, with no
change.
I have looked at the functions you mention, and that is a part of my
confusion, as they aren't doing anything different than I am.
> 
> Additionally I'm a bit afraid your initial phase :) As you know PCIe
> would be used as RC mode on Freescale PowerPC kernel. So I don't know if
> you also drop this path on your kernel to conflict each other :) 
I have tried doing this under a kernel built without PCI support with no
change.

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[Chen, Tiejun]

> -----Original Message-----
> From: David Hagood [mailto:david.hagood@gmail.com]=20
> Sent: Thursday, September 23, 2010 7:11 PM
> To: Chen, Tiejun; linuxppc-dev@ozlabs.org
> Subject: RE: MPC8641D PEX: programming OWBAR in Endpoint mode?
>=20
> On Thu, 2010-09-23 at 05:21 +0200, Chen, Tiejun wrote:
> > > I can get the device to show up on the host's PCI bus, I can
> >=20
> > This only ensure you can access the PCIe configure space.
> Not quite: I can also read the BARs that I program, and the=20
> memory behind them on the PPC.

Absolutely.=20

> >=20
> > > program the inbound ATMUs such that the BARS are updated when the=20
> > > host (re-)scans them, but I cannot for the life of me get
> >=20
> > What value are configured to IntBound REGs?
> I can program them at run time via sysfs on the PPC's side,=20
> so there is no single set of values. However, I am pointing=20
> them at the PPC's RAM space, and as I stated above, I can=20
> read the PPC's RAM from the host side via the BARs.

I read your email again and something hint me. I notice you clarify you
already condigure InBound successfully. Right? If so I'm a bit confused.
For PCIe EP mode PEXIWBARs are not implemented in the memory-mapped
space. If you read any PEXIWBAR these registers always return zero
regardless of writing any value at first.=20

You only can program 4 inbound BARs by type 0 configure action like
normal PCIe device.

>=20
> > How do you configure OWS of PEXOWAR?
> >=20
> > I means you still access that if OWS is match the whole=20
> target memory=20
> > size even when '0' is as the internal platform address.
> As I understand it, not if the OWS is not correctly mapped on=20
> the PPC side - the PEX outbound ATMU's OWBAR must be mapped=20
> to a region of the PPCs address space that is also mapped to=20
> PEX in the LAW. The LAW does NOT indicate that PPC address 0=20
> is mapped to the PEX.

If there is no any law entry for PCIe the kernel should trap machine
check when you access PCIe space. And as my above comment I'm afraid you
mix up InBound and OutBound on EP mode? So you always read zero from
your so-called OutBound? I means that should be PEXIWBAR in fact. I'm
not sure but you can check this.

>=20
> >=20
> > Out_be32 should be fine for atmu REGs. And also you can refe to the=20
> > function, setup_pci_atmu & setup_one_atmu, on the file,=20
> > arch/powerpc/sysdev/fsl_pci.c, to know how to access atmu=20
> REGs. Often=20
> > you should disable them, configure then enable/invoke atmu antry as=20
> > normal configuring sequent.
> I have tried disabling the outbound ATMU when I program it,=20
> with no change.
> I have looked at the functions you mention, and that is a=20
> part of my confusion, as they aren't doing anything different=20
> than I am.

I only means you can refer how to access these registers.

> >=20
> > Additionally I'm a bit afraid your initial phase :) As you=20
> know PCIe=20
> > would be used as RC mode on Freescale PowerPC kernel. So I=20
> don't know=20
> > if you also drop this path on your kernel to conflict each other :)
> I have tried doing this under a kernel built without PCI=20
> support with no change.

Good.

Tiejun

>=20
>=20
>=20

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[david.hagood]

>> -----Original Message-----
 via the BARs.
>
> I read your email again and something hint me. I notice you clarify you
> already condigure InBound successfully.

I am programming BOTH the inbound ATMUs to make PPC memory available to
the root complex, AND programming outbound ATMUs to enable the PPC to bus
master to the root complex's memory space on PCIe.

I am NOT attempting to program the IWBARs - as you noted, they get
programmed by the root complex via PCI config operations.

>
>  And as my above comment I'm afraid you
> mix up InBound and OutBound on EP mode?

No, I am NOT confusing the two - that is why I am being VERY EXPLICIT
about accessing the OUTBOUND ATMUs.

The only reason I mention the inbound ATMUs is to demonstrate that the
physical layer is working.

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[Chen, Tiejun]

> -----Original Message-----
> From: david.hagood@gmail.com [mailto:david.hagood@gmail.com]=20
> Sent: Thursday, September 23, 2010 10:44 PM
> To: Chen, Tiejun
> Cc: David Hagood; linuxppc-dev@ozlabs.org
> Subject: RE: MPC8641D PEX: programming OWBAR in Endpoint mode?
>=20
> >> -----Original Message-----
>  via the BARs.
> >
> > I read your email again and something hint me. I notice you clarify=20
> > you already condigure InBound successfully.
>=20
> I am programming BOTH the inbound ATMUs to make PPC memory=20
> available to the root complex, AND programming outbound ATMUs=20
> to enable the PPC to bus master to the root complex's memory=20
> space on PCIe.
>=20

Right but this should be done for RC mode, not for EP mode we're
discussing.

Tiejun

> I am NOT attempting to program the IWBARs - as you noted,=20
> they get programmed by the root complex via PCI config operations.
>=20
> >
> >  And as my above comment I'm afraid you mix up InBound and=20
> OutBound on=20
> > EP mode?
>=20
> No, I am NOT confusing the two - that is why I am being VERY=20
> EXPLICIT about accessing the OUTBOUND ATMUs.
>=20
> The only reason I mention the inbound ATMUs is to demonstrate=20
> that the physical layer is working.
>=20
>=20
>=20

RE: MPC8641D PEX: programming OWBAR in Endpoint mode?

[David Hagood]

On Fri, 2010-09-24 at 07:09 +0200, Chen, Tiejun wrote:

> 
> Right but this should be done for RC mode, not for EP mode we're
> discussing.
> 
> Tiejun

According to the Freescale documentation, outbound is just as valid for
endpoint as for root complex - indeed, to generate MSIs from software
REQUIRES programming an outbound ATMU to access the host's APIC.

Moreover, ANY PCI endpoint SHOULD be able to do bus master access, and
that is done by the outbound ATMUs.

Re: MPC8641D PEX: programming OWBAR in Endpoint mode?

[tiejun.chen]

David Hagood wrote:
> On Fri, 2010-09-24 at 07:09 +0200, Chen, Tiejun wrote:
> 
>> Right but this should be done for RC mode, not for EP mode we're
>> discussing.
>>
>> Tiejun
> 
> According to the Freescale documentation, outbound is just as valid for
> endpoint as for root complex - indeed, to generate MSIs from software

Yes.

As a summary you have no any issue to access InBound/BAR/LAW. And actually I'm
always strange that even you cannot configure InBound/BAR/LAW properly, you also
should be allowed to write OutBound REGs. And I remember there is only one
requirement to OutBound, and that is the window address should be aligned based
on the size from OWS. Are you sure?

And did you try to configure OutBound REGS when RC mode? If so I'm afraid we may
miss some errata on EP.

And do you have any response from Freescale?

Cheers
Tiejun

> REQUIRES programming an outbound ATMU to access the host's APIC.
> 
> Moreover, ANY PCI endpoint SHOULD be able to do bus master access, and
> that is done by the outbound ATMUs.
> 
> 
> 

Re: MPC8641D PEX: programming OWBAR in Endpoint mode?

[David Hagood]

On Sat, 2010-09-25 at 17:46 +0800, tiejun.chen wrote:
> As a summary you have no any issue to access InBound/BAR/LAW.

Correct: full access to the inbound.
> And I remember there is only one
> requirement to OutBound, and that is the window address should be aligned based
> on the size from OWS. Are you sure?
Yes. As my first test I've been using 4k and 64k sizes, aligned on 16M
boundaries.
> 
> And did you try to configure OutBound REGS when RC mode? If so I'm afraid we may
> miss some errata on EP.
In RC mode the built-in drivers take over.

Re: MPC8641D PEX: programming OWBAR in Endpoint mode?

[tiejun.chen]

David Hagood wrote:
> On Sat, 2010-09-25 at 17:46 +0800, tiejun.chen wrote:
>> As a summary you have no any issue to access InBound/BAR/LAW.
> 
> Correct: full access to the inbound.
>> And I remember there is only one
>> requirement to OutBound, and that is the window address should be aligned based
>> on the size from OWS. Are you sure?
> Yes. As my first test I've been using 4k and 64k sizes, aligned on 16M
> boundaries.
>> And did you try to configure OutBound REGS when RC mode? If so I'm afraid we may
>> miss some errata on EP.
> In RC mode the built-in drivers take over.

Can you confirm the build-in driver can write/read OutBound correctly on your
target?

-Tiejun

> 
> 
>