FW: P4080 device tree problems with fsl dpaa ...

FW: P4080 device tree problems with fsl dpaa ...

[Robert Sciuk]

-----Original Message-----
From: Robert Sciuk=20
Sent: Friday, October 14, 2011 5:27 PM
To: 'devicetree-discuss@lists.ozlabs.org'
Subject: P4080 device tree problems with fsl dpaa ...

I've just joined the list, and I hope that this is not an inappropriate =
question, but I'm looking for some direction with respect to device =
trees, and the fsl, dpaa Ethernet drivers.

I'm wondering if anyone has had any experience with the FreeScale DPAA =
drivers for the 1g dtsec interface.  We are getting interfaces defined, =
and the tx count increases, but we are not seeing packets on the "wire".

...
[    0.911592] Freescale FM module (Oct 13 2011:14:41:07)
[    0.916745] cpu6/6: fsl_mac: FSL FMan MAC API based driver ()
[    0.923077] cpu6/6: fsl_mac: ffe4e0000.ethernet: FMan dTSEC version: =
0x08240101
[    0.930403] cpu6/6: fsl_mac: ffe4e0000.ethernet: FMan MAC address: =
00:a0:a9:be:ef:10
...
[    1.015863] cpu6/6: fsl_dpa: FSL DPAA Ethernet driver ()
[    1.021446] cpu6/6: fsl_dpa: ethernet.23: =
dpaa_eth.c:1684:dpa_bp_create() eth%d: Using private BM buffer pools
[    1.032263] cpu6/6: Using dynamic RX QM frame queues
[    1.037242] cpu6/6: Using dynamic TX QM frame queues
[    1.042263] cpu6/6: > WARNING (FM) =
[/export2/rd2/dev/robsci1/Work/gold/wr4linux-layer/obj/hotwire1/hotwire1/=
build/linux/drivers/n
et/dpa/NetCommSw/Peripherals/FM/fm.c:911 FmGetSetPortParams]:
[    1.059138] cpu6/6: FIFO size enlarged to 11008
[    1.063670] cpu6/6:
[    1.066672] cpu6/6: fsl_dpa: ethernet.23: =
dpaa_eth.c:2327:dpaa_oh_probe() no OH port bindings on node =
/fsl,dpaa/ethernet@0
[    1.077806] cpu6/6: fsl_dpa: ethernet.24: =
dpaa_eth.c:1684:dpa_bp_create() eth%d: Using private BM buffer pools
[    1.087826] cpu6/6: Using dynamic RX QM frame queues
[    1.092798] cpu6/6: Using dynamic TX QM frame queues
...

Our device tree defines the top level Ethernet as:
ethernet@0 {
 compatible =3D "fsl,p4080-dpa-ethernet", "fsl,dpa-ethernet";
 fsl,qman-channel =3D <0x13>;=20
 fsl,fman-mac =3D <0x50>;
};

And the mac and phys are defined as follows:

ethernet@e0000 {
 cell-index =3D <0x0>;
 compatible =3D "fsl,p4080-fman-1g-mac", "fsl,fman-1g-mac";
 reg =3D <0xe0000 0x1000>;
 fsl,port-handles =3D <0x22 0x23>;
 tbi-handle =3D <0x24>;
 phy-handle =3D <0x25>;
 phy-connection-type =3D "sgmii";
 ptimer-handle =3D <0x26>;
 linux,phandle =3D <0x50>;
};
mdio@e1120 {
 #address-cells =3D <0x1>;
 #size-cells =3D <0x0>;
 compatible =3D "fsl,fman-mdio";
 reg =3D <0xe1120 0xee0>;
 interrupts =3D <0x64 0x1 0x0 0x0>;
 gpios =3D <0x27 0x0 0x0 0x27 0x1 0x0>;
 linux,phandle =3D <0x28>;
 tbi-phy@8 {
  reg =3D <0x8>;
  device_type =3D "tbi-phy";
  linux,phandle =3D <0x24>;
 };
 phy0: ethernet-phy@0 {
  device_type=3D"ethernet-phy";
  compatible=3D"broadcom,BCM5482";
  reg =3D <0>;
  linux,phandle =3D <0x25>;
 };
};


I have no idea what an OH binding is, what it might look like, and what =
it entails, but I think that it might be a significant factor in our not =
seeing a working interface.  Has anyone any experience with dpaa device =
trees, and configuration?  Any pointers?  Any docs? Shots in the dark?=20

Robert Sciuk
Senior Designer, R&D.
905.738.3741 xt 22621

Re: FW: P4080 device tree problems with fsl dpaa ...

[Thomas De Schampheleire]

Hi Robert,

On Fri, Oct 14, 2011 at 11:57 PM, Robert Sciuk <robert.sciuk@exfo.com> wrot=
e:
>
>
> -----Original Message-----
> From: Robert Sciuk
> Sent: Friday, October 14, 2011 5:27 PM
> To: 'devicetree-discuss@lists.ozlabs.org'
> Subject: P4080 device tree problems with fsl dpaa ...
>
> I've just joined the list, and I hope that this is not an inappropriate q=
uestion, but I'm looking for some direction with respect to device trees, a=
nd the fsl, dpaa Ethernet drivers.
>
> I'm wondering if anyone has had any experience with the FreeScale DPAA dr=
ivers for the 1g dtsec interface. =A0We are getting interfaces defined, and=
 the tx count increases, but we are not seeing packets on the "wire".
>
> ...
> [ =A0 =A00.911592] Freescale FM module (Oct 13 2011:14:41:07)
> [ =A0 =A00.916745] cpu6/6: fsl_mac: FSL FMan MAC API based driver ()
> [ =A0 =A00.923077] cpu6/6: fsl_mac: ffe4e0000.ethernet: FMan dTSEC versio=
n: 0x08240101
> [ =A0 =A00.930403] cpu6/6: fsl_mac: ffe4e0000.ethernet: FMan MAC address:=
 00:a0:a9:be:ef:10
> ...
> [ =A0 =A01.015863] cpu6/6: fsl_dpa: FSL DPAA Ethernet driver ()
> [ =A0 =A01.021446] cpu6/6: fsl_dpa: ethernet.23: dpaa_eth.c:1684:dpa_bp_c=
reate() eth%d: Using private BM buffer pools
> [ =A0 =A01.032263] cpu6/6: Using dynamic RX QM frame queues
> [ =A0 =A01.037242] cpu6/6: Using dynamic TX QM frame queues
> [ =A0 =A01.042263] cpu6/6: > WARNING (FM) [/export2/rd2/dev/robsci1/Work/=
gold/wr4linux-layer/obj/hotwire1/hotwire1/build/linux/drivers/n
> et/dpa/NetCommSw/Peripherals/FM/fm.c:911 FmGetSetPortParams]:
> [ =A0 =A01.059138] cpu6/6: FIFO size enlarged to 11008
> [ =A0 =A01.063670] cpu6/6:
> [ =A0 =A01.066672] cpu6/6: fsl_dpa: ethernet.23: dpaa_eth.c:2327:dpaa_oh_=
probe() no OH port bindings on node /fsl,dpaa/ethernet@0
> [ =A0 =A01.077806] cpu6/6: fsl_dpa: ethernet.24: dpaa_eth.c:1684:dpa_bp_c=
reate() eth%d: Using private BM buffer pools
> [ =A0 =A01.087826] cpu6/6: Using dynamic RX QM frame queues
> [ =A0 =A01.092798] cpu6/6: Using dynamic TX QM frame queues
> ...
>
> Our device tree defines the top level Ethernet as:
> ethernet@0 {
> =A0compatible =3D "fsl,p4080-dpa-ethernet", "fsl,dpa-ethernet";
> =A0fsl,qman-channel =3D <0x13>;
> =A0fsl,fman-mac =3D <0x50>;
> };
>
> And the mac and phys are defined as follows:
>
> ethernet@e0000 {
> =A0cell-index =3D <0x0>;
> =A0compatible =3D "fsl,p4080-fman-1g-mac", "fsl,fman-1g-mac";
> =A0reg =3D <0xe0000 0x1000>;
> =A0fsl,port-handles =3D <0x22 0x23>;
> =A0tbi-handle =3D <0x24>;
> =A0phy-handle =3D <0x25>;
> =A0phy-connection-type =3D "sgmii";
> =A0ptimer-handle =3D <0x26>;
> =A0linux,phandle =3D <0x50>;
> };
> mdio@e1120 {
> =A0#address-cells =3D <0x1>;
> =A0#size-cells =3D <0x0>;
> =A0compatible =3D "fsl,fman-mdio";
> =A0reg =3D <0xe1120 0xee0>;
> =A0interrupts =3D <0x64 0x1 0x0 0x0>;
> =A0gpios =3D <0x27 0x0 0x0 0x27 0x1 0x0>;
> =A0linux,phandle =3D <0x28>;
> =A0tbi-phy@8 {
> =A0reg =3D <0x8>;
> =A0device_type =3D "tbi-phy";
> =A0linux,phandle =3D <0x24>;
> =A0};
> =A0phy0: ethernet-phy@0 {
> =A0device_type=3D"ethernet-phy";
> =A0compatible=3D"broadcom,BCM5482";
> =A0reg =3D <0>;
> =A0linux,phandle =3D <0x25>;
> =A0};
> };
>
>
> I have no idea what an OH binding is, what it might look like, and what i=
t entails, but I think that it might be a significant factor in our not see=
ing a working interface. =A0Has anyone any experience with dpaa device tree=
s, and configuration? =A0Any pointers? =A0Any docs? Shots in the dark?

We are using a device tree on p4080 with a dpaa configuration, yes.
I also don't know about the OH bindings, but I have a vague memory of
us having that message too, and it not being a real problem.

What I do know is that the device tree is very easy to get wrong, and
that it should match your hardware precisely. I'm not using the same
configuration as you are (there are no MDIO devices between our phy
and dTSEC) so the device tree looks a little different.

Are you using the reference design or is this custom?
Did you have a look at the device tree in the linux kernel sources?
(arch/powerpc/boot/dts/p4080.dts). Does this configuration match
yours? In that configuration, there are separate nodes of type
p4080ds-mdio, inside the main mdio node.

Is your phy configured correctly? Are there any boot messages about
this? Can you check its registers?
The same for the mdio.

Some generic pointers:
* If you set CONFIG_PROC_DEVICETREE in the kernel config, you can read
out the device-tree at runtime in /proc/device-tree. This helps in
verifying whether the nodes you programmed actually appear in the
running system.

* The /sys/devices directory can also be useful in identifying
potential problems for unfound devices (which does not appear to be
your case).

* Can you step through the code, e.g. with a JTAG debugger? Try
following the transmit path in the dpaa_eth driver. Maybe something
errors out prematurely.

Best regards,
Thomas

RE: FW: P4080 device tree problems with fsl dpaa ...

[Robert Sciuk]

> -----Original Message-----
> From: patrickdepinguin@gmail.com [mailto:patrickdepinguin@gmail.com] =
On
> Behalf Of Thomas De Schampheleire
> Sent: Monday, October 17, 2011 5:01 AM
> To: Robert Sciuk
> Cc: linuxppc-dev@lists.ozlabs.org
> Subject: Re: FW: P4080 device tree problems with fsl dpaa ...
>=20
> Hi Robert,

[my stuff snipped]
>=20
> > I have no idea what an OH binding is, what it might look like, and
> what it entails, but I think that it might be a significant factor in
> our not seeing a working interface. =A0Has anyone any experience with
> dpaa device trees, and configuration? =A0Any pointers? =A0Any docs? =
Shots
> in the dark?
>=20
> We are using a device tree on p4080 with a dpaa configuration, yes.
> I also don't know about the OH bindings, but I have a vague memory of
> us having that message too, and it not being a real problem.
>=20

Other indicators concur with your assessment, and suggest that I back =
those nodes out as well. =20


> What I do know is that the device tree is very easy to get wrong, and
> that it should match your hardware precisely. I'm not using the same
> configuration as you are (there are no MDIO devices between our phy
> and dTSEC) so the device tree looks a little different.

Would it be possible to have a look at the relevant portions of your dev =
tree?



>=20
> Are you using the reference design or is this custom?

This is a custom design, and is an ATCA blade.


> Did you have a look at the device tree in the linux kernel sources?
> (arch/powerpc/boot/dts/p4080.dts). Does this configuration match
> yours? In that configuration, there are separate nodes of type
> p4080ds-mdio, inside the main mdio node.

We actually used the 36 bit tree (we plan to ship the boards with 16G =
initially, and up to 32G eventually).  I've also re-worked the tree from =
the LTIB (System Builder) in an attempt to get this to work.  Our design =
differs significantly from the P4080DS in that there exists no MUX =
between the bus and phy ... and there is no PIXIS FPGA to do all the =
clocking and startup stuff.=20

I've modified u-boot to assign mac addresses based upon our OUI, and the =
MACs all seem to be configured properly coming out of firmware, the =
ifconfig on the Linux side recognizes all of our 6 interfaces ...


>=20
> Is your phy configured correctly? Are there any boot messages about
> this? Can you check its registers?
> The same for the mdio.

There is some question about the TBI initialization, and we are tracing =
that as we speak ...

>=20
> Some generic pointers:
> * If you set CONFIG_PROC_DEVICETREE in the kernel config, you can read
> out the device-tree at runtime in /proc/device-tree. This helps in
> verifying whether the nodes you programmed actually appear in the
> running system.

Ack.

>=20
> * The /sys/devices directory can also be useful in identifying
> potential problems for unfound devices (which does not appear to be
> your case).
>=20
=20
Ack.  Also using /proc/interrupts ...


> * Can you step through the code, e.g. with a JTAG debugger? Try
> following the transmit path in the dpaa_eth driver. Maybe something
> errors out prematurely.
>=20

With some headaches ... using codewarrior, and getting assembly level =
tracing ...

> Best regards,
> Thomas

Re: FW: P4080 device tree problems with fsl dpaa ...

[Thomas De Schampheleire]

Hi Robert,

On Mon, Oct 17, 2011 at 8:34 PM, Robert Sciuk <robert.sciuk@exfo.com> wrote=
:
>
>
>> -----Original Message-----
>> From: patrickdepinguin@gmail.com [mailto:patrickdepinguin@gmail.com] On
>> Behalf Of Thomas De Schampheleire
>> Sent: Monday, October 17, 2011 5:01 AM
>> To: Robert Sciuk
>> Cc: linuxppc-dev@lists.ozlabs.org
>> Subject: Re: FW: P4080 device tree problems with fsl dpaa ...
>>
>> Hi Robert,
>
> [my stuff snipped]
>>
>> > I have no idea what an OH binding is, what it might look like, and
>> what it entails, but I think that it might be a significant factor in
>> our not seeing a working interface. =A0Has anyone any experience with
>> dpaa device trees, and configuration? =A0Any pointers? =A0Any docs? Shot=
s
>> in the dark?
>>
>> We are using a device tree on p4080 with a dpaa configuration, yes.
>> I also don't know about the OH bindings, but I have a vague memory of
>> us having that message too, and it not being a real problem.
>>
>
> Other indicators concur with your assessment, and suggest that I back tho=
se nodes out as well.
>
>
>> What I do know is that the device tree is very easy to get wrong, and
>> that it should match your hardware precisely. I'm not using the same
>> configuration as you are (there are no MDIO devices between our phy
>> and dTSEC) so the device tree looks a little different.
>
> Would it be possible to have a look at the relevant portions of your dev =
tree?
>
>

Sure.
The following snippets are for two SGMII devices on the second FMAN:

First the ports:

// RX
                        fman1_rx2: port@8a000 {
                                compatible =3D
"fsl,p4080-fman-port-1g-rx", "fsl,fman-port-1g-rx";
                                reg =3D <0x8a000 0x1000>;
                                fsl,liodn =3D <0x2d>;
         // Logical Input/Output Device Number used by PAMU
                                cell-index =3D <2>;
         // 1G index
                        };
                        fman1_rx3: port@8b000 {
                                compatible =3D
"fsl,p4080-fman-port-1g-rx", "fsl,fman-port-1g-rx";
                                reg =3D <0x8b000 0x1000>;
                                fsl,liodn =3D <0x2e>;
         // Logical Input/Output Device Number used by PAMU
                                cell-index =3D <3>;
         // 1G index
                        };
// TX
                        fman1_tx2: port@aa000 {
                                compatible =3D
"fsl,p4080-fman-port-1g-tx", "fsl,fman-port-1g-tx";
                                reg =3D <0xaa000 0x1000>;
                                fsl,qman-channel-id =3D <0x63>;
         // Direct-HW-Portal 1: SP3
                                cell-index =3D <2>;
         // 1G index
                        };
                        fman1_tx3: port@ab000 {
                                compatible =3D
"fsl,p4080-fman-port-1g-tx", "fsl,fman-port-1g-tx";
                                reg =3D <0xab000 0x1000>;
                                fsl,qman-channel-id =3D <0x64>;
         // Direct-HW-Portal 1: SP4
                                cell-index =3D <3>;
         // 1G index
                        };

Then the TSECs:
                        enet7: ethernet@e4000 { // dTSEC3
                                compatible =3D "fsl,p4080-fman-1g-mac",
"fsl,fman-1g-mac";
                                reg =3D <0xe4000 0x1000>;
                                fixed-link =3D <7 1 1000 0 0>;
         // Use fixed settings because there is no MDIO bus between
SOC and PHY
                                local-mac-address =3D [00 BA 0B AB 00 0A];
                                fsl,port-handles =3D <&fman1_rx2
&fman1_tx2>;     // RX + TX port
                                phy-connection-type =3D "sgmii";
         // 1Gb
                                cell-index =3D <2>;
                        };
                        enet8: ethernet@e6000 { // dTSEC4
                                compatible =3D "fsl,p4080-fman-1g-mac",
"fsl,fman-1g-mac";
                                reg =3D <0xe6000 0x1000>;
                                fixed-link =3D <8 1 1000 0 0>;
         // Use fixed settings because there is no MDIO bus between
SOC and PHY
                                local-mac-address =3D [00 BA 0B AB 00 0B];
                                fsl,port-handles =3D <&fman1_rx3
&fman1_tx3>;     // RX + TX port
                                phy-connection-type =3D "sgmii";
         // 1Gb
                                cell-index =3D <3>;
                        };

The ethernet devices in the fsl,dpaa node:
                ethernet@7 {
                        compatible =3D "fsl,p4080-dpa-ethernet",
"fsl,dpa-ethernet";
                        fsl,qman-channel =3D <&qportal0>;
                        fsl,fman-mac =3D <&enet7>;
                };
                ethernet@8 {
                        compatible =3D "fsl,p4080-dpa-ethernet",
"fsl,dpa-ethernet";
                        fsl,qman-channel =3D <&qportal0>;
                        fsl,fman-mac =3D <&enet8>;


>
>>
>> Are you using the reference design or is this custom?
>
> This is a custom design, and is an ATCA blade.
>
>
>> Did you have a look at the device tree in the linux kernel sources?
>> (arch/powerpc/boot/dts/p4080.dts). Does this configuration match
>> yours? In that configuration, there are separate nodes of type
>> p4080ds-mdio, inside the main mdio node.
>
> We actually used the 36 bit tree (we plan to ship the boards with 16G ini=
tially, and up to 32G eventually). =A0I've also re-worked the tree from the=
 LTIB (System Builder) in an attempt to get this to work. =A0Our design dif=
fers significantly from the P4080DS in that there exists no MUX between the=
 bus and phy ... and there is no PIXIS FPGA to do all the clocking and star=
tup stuff.

I have to admit that I'm not an export at the DPAA part either. My
colleague who is is on holiday, unfortunately.

In our case, we needed the 'fixed-link' settings because of the
absence of an MDIO, and I also initially made the mistake of
registering the devices on the wrong FMAN.
For RX, I also had some problems receiving frames correctly. Setting
the source MAC address correctly fixed that problem.
But you seem to have problems with TX...

>
> I've modified u-boot to assign mac addresses based upon our OUI, and the =
MACs all seem to be configured properly coming out of firmware, the ifconfi=
g on the Linux side recognizes all of our 6 interfaces ...
>
>
>>
>> Is your phy configured correctly? Are there any boot messages about
>> this? Can you check its registers?
>> The same for the mdio.
>
> There is some question about the TBI initialization, and we are tracing t=
hat as we speak ...
>
>>
>> Some generic pointers:
>> * If you set CONFIG_PROC_DEVICETREE in the kernel config, you can read
>> out the device-tree at runtime in /proc/device-tree. This helps in
>> verifying whether the nodes you programmed actually appear in the
>> running system.
>
> Ack.
>
>>
>> * The /sys/devices directory can also be useful in identifying
>> potential problems for unfound devices (which does not appear to be
>> your case).
>>
>
> Ack. =A0Also using /proc/interrupts ...
>
>
>> * Can you step through the code, e.g. with a JTAG debugger? Try
>> following the transmit path in the dpaa_eth driver. Maybe something
>> errors out prematurely.
>>
>
> With some headaches ... using codewarrior, and getting assembly level tra=
cing ...

Also, did you check in the dTSEC/FMAN/BMAN for relevant counter
registers? Do you find proof that the packets make it as far as the
dTSEC? (then maybe there is a problem in the link between dTSEC and
PHY).
Are there counters in the PHY that can indicate that packets or no
packets are received?

Which version of the kernel and associated Freescale patches are you using?

How and what type of packets are you sending? Raw MAC frames, IP? Are
you using custom code, or can you try things like ping?
If you are using IP, I trust you checked the Linux configuration (like
assigning a valid IP address to the interface and making sure the
routing table is correct).

Best regards,
Thomas

RE: FW: P4080 device tree problems with fsl dpaa ... RESOLVED.

[Robert Sciuk]

[snip]

>=20
> How and what type of packets are you sending? Raw MAC frames, IP? Are
> you using custom code, or can you try things like ping?
> If you are using IP, I trust you checked the Linux configuration (like
> assigning a valid IP address to the interface and making sure the
> routing table is correct).
>=20
> Best regards,
> Thomas

Thomas,

I would like to thank you for your very kind assistance on the DPAA =
problem.  We traced the problem to the connection between MAC and PHY, =
and we have resolved it by instrumenting code and comparing the TBI and =
PHY initialization from u-boot to the Linux DPA code, and replacing some =
of the "magic numbers" we found in the driver with "magic numbers" we =
know to work.  OUCH!

We spent an inordinate amount of effort in trying to understand the =
undocumented code/tree, but I am convinced that the device-tree approach =
to embedded boards is a very valuable approach, and will ultimately =
solve many more problems than it creates short term. =20

Again, your information was of great assistance, and I look forward to =
being able to pay forward the same kind of assistance in the future.

Cheers,
Rob Sciuk.

Re: FW: P4080 device tree problems with fsl dpaa ... RESOLVED.

[Thomas De Schampheleire]

Hi Robert,

On Tue, Oct 18, 2011 at 11:38 PM, Robert Sciuk <robert.sciuk@exfo.com> wrot=
e:
>
>
> [snip]
>
>>
>> How and what type of packets are you sending? Raw MAC frames, IP? Are
>> you using custom code, or can you try things like ping?
>> If you are using IP, I trust you checked the Linux configuration (like
>> assigning a valid IP address to the interface and making sure the
>> routing table is correct).
>>
>> Best regards,
>> Thomas
>
> Thomas,
>
> I would like to thank you for your very kind assistance on the DPAA probl=
em. =A0We traced the problem to the connection between MAC and PHY, and we =
have resolved it by instrumenting code and comparing the TBI and PHY initia=
lization from u-boot to the Linux DPA code, and replacing some of the "magi=
c numbers" we found in the driver with "magic numbers" we know to work. =A0=
OUCH!
>
> We spent an inordinate amount of effort in trying to understand the undoc=
umented code/tree, but I am convinced that the device-tree approach to embe=
dded boards is a very valuable approach, and will ultimately solve many mor=
e problems than it creates short term.
>
> Again, your information was of great assistance, and I look forward to be=
ing able to pay forward the same kind of assistance in the future.

I'm very glad you found the issue (I was running out of ideas ;-) and
that I was able to help you in some way.
Thanks for posting back.

Good luck with the rest of your project!

Best regards,
Thomas

>
> Cheers,
> Rob Sciuk.
>
>