[PATCH] tqma6: Fix DDR configuration

[PATCH] tqma6: Fix DDR configuration

[Miquel Raynal]

Initially investigating a Linux network issue causing a lot of drop and
poor network performances on a custom system based on a TQMA6A module
(based on an iMX6Q), [1st link below].

I eventually correlated my observations with a contention at the NIC
level when in concurrency with the graphics pipeline. Troubleshooting
this in the kernel lead to disabling DMA bursts accesses made by the IPU
in order to avoid triggering the QoS at the interconnect level, reducing
from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
worked on my setup but not on others, which still suffered from
abnormally high drop rates even with this "fix".

After looking a while into TQ Systems BSP I figured out a number of
differences in recent U-Boot out-of-tree patches they had in their
repository [3rd link]. Parsing the differences one after the other lead
me to this final solution.

The reset pad of the DDR controller was apparently misconfigured, Bit
18-19 picturing the "DDR select field". The current value b11 is
reserved. The only defined value as of version 6 of the iMX6Q manual was
b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
spotted after changing this configuration but all issues tracked thus
far just vanished. All previous fixes have been proven irrelevant. Just
clearing this field solved all our network issues and the drop rate as
measured by iperf3 felt back to 0%.

Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
---
 board/tq/tqma6/tqma6q.cfg | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg
index a49489aed3f..a345c4de93d 100644
--- a/board/tq/tqma6/tqma6q.cfg
+++ b/board/tq/tqma6/tqma6q.cfg
@@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
 DATA 4, MX6_IOM_DRAM_CAS, 0x00008030
 DATA 4, MX6_IOM_DRAM_RAS, 0x00008030
 DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
-DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030
+DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
 DATA 4, MX6_IOM_DRAM_SDCKE0, 0x00003000
 DATA 4, MX6_IOM_DRAM_SDCKE1, 0x00000000
 DATA 4, MX6_IOM_DRAM_SDBA2, 0x00000000
-- 
2.34.1

Re: [PATCH] tqma6: Fix DDR configuration

[Fabio Estevam]

On Fri, Nov 17, 2023 at 1:50 PM Miquel Raynal <miquel.raynal@bootlin.com> wrote:
>
> Initially investigating a Linux network issue causing a lot of drop and
> poor network performances on a custom system based on a TQMA6A module
> (based on an iMX6Q), [1st link below].
>
> I eventually correlated my observations with a contention at the NIC
> level when in concurrency with the graphics pipeline. Troubleshooting
> this in the kernel lead to disabling DMA bursts accesses made by the IPU
> in order to avoid triggering the QoS at the interconnect level, reducing
> from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> worked on my setup but not on others, which still suffered from
> abnormally high drop rates even with this "fix".
>
> After looking a while into TQ Systems BSP I figured out a number of
> differences in recent U-Boot out-of-tree patches they had in their
> repository [3rd link]. Parsing the differences one after the other lead
> me to this final solution.
>
> The reset pad of the DDR controller was apparently misconfigured, Bit
> 18-19 picturing the "DDR select field". The current value b11 is
> reserved. The only defined value as of version 6 of the iMX6Q manual was
> b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> spotted after changing this configuration but all issues tracked thus
> far just vanished. All previous fixes have been proven irrelevant. Just
> clearing this field solved all our network issues and the drop rate as
> measured by iperf3 felt back to 0%.
>
> Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
> Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
> Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>

Yes, better not to write to reserved fields:

Reviewed-by: Fabio Estevam <festevam@gmail.com>

Re: [PATCH] tqma6: Fix DDR configuration

[Fabio Estevam]

On Fri, Nov 17, 2023 at 1:50 PM Miquel Raynal <miquel.raynal@bootlin.com> wrote:
>
> Initially investigating a Linux network issue causing a lot of drop and
> poor network performances on a custom system based on a TQMA6A module
> (based on an iMX6Q), [1st link below].
>
> I eventually correlated my observations with a contention at the NIC
> level when in concurrency with the graphics pipeline. Troubleshooting
> this in the kernel lead to disabling DMA bursts accesses made by the IPU
> in order to avoid triggering the QoS at the interconnect level, reducing
> from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> worked on my setup but not on others, which still suffered from
> abnormally high drop rates even with this "fix".
>
> After looking a while into TQ Systems BSP I figured out a number of
> differences in recent U-Boot out-of-tree patches they had in their
> repository [3rd link]. Parsing the differences one after the other lead
> me to this final solution.
>
> The reset pad of the DDR controller was apparently misconfigured, Bit
> 18-19 picturing the "DDR select field". The current value b11 is
> reserved. The only defined value as of version 6 of the iMX6Q manual was
> b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> spotted after changing this configuration but all issues tracked thus
> far just vanished. All previous fixes have been proven irrelevant. Just
> clearing this field solved all our network issues and the drop rate as
> measured by iperf3 felt back to 0%.
>
> Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
> Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
> Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>

Applied, thanks.

Re: [PATCH] tqma6: Fix DDR configuration

[Miquel Raynal]

Hi Fabio,

festevam@gmail.com wrote on Tue, 12 Dec 2023 13:11:53 -0300:

> On Fri, Nov 17, 2023 at 1:50 PM Miquel Raynal <miquel.raynal@bootlin.com> wrote:
> >
> > Initially investigating a Linux network issue causing a lot of drop and
> > poor network performances on a custom system based on a TQMA6A module
> > (based on an iMX6Q), [1st link below].
> >
> > I eventually correlated my observations with a contention at the NIC
> > level when in concurrency with the graphics pipeline. Troubleshooting
> > this in the kernel lead to disabling DMA bursts accesses made by the IPU
> > in order to avoid triggering the QoS at the interconnect level, reducing
> > from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> > worked on my setup but not on others, which still suffered from
> > abnormally high drop rates even with this "fix".
> >
> > After looking a while into TQ Systems BSP I figured out a number of
> > differences in recent U-Boot out-of-tree patches they had in their
> > repository [3rd link]. Parsing the differences one after the other lead
> > me to this final solution.
> >
> > The reset pad of the DDR controller was apparently misconfigured, Bit
> > 18-19 picturing the "DDR select field". The current value b11 is
> > reserved. The only defined value as of version 6 of the iMX6Q manual was
> > b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> > spotted after changing this configuration but all issues tracked thus
> > far just vanished. All previous fixes have been proven irrelevant. Just
> > clearing this field solved all our network issues and the drop rate as
> > measured by iperf3 felt back to 0%.
> >
> > Link: https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> > Link: https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
> > Link: https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
> > Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>  
> 
> Applied, thanks.

Thanks a lot. Any chances you would have access to additional
information to explain how the network vs video pipeline
interact without this patch? I am eager to understand.

Cheers,
Miquèl

Re: [PATCH] tqma6: Fix DDR configuration

[Fabio Estevam]

Hi Miquel,

On Tue, Dec 12, 2023 at 1:21 PM Miquel Raynal <miquel.raynal@bootlin.com> wrote:

> Thanks a lot. Any chances you would have access to additional
> information to explain how the network vs video pipeline
> interact without this patch? I am eager to understand.

Unfortunately, I don't have additional information on this topic.

I think someone at NXP who is close to the i.MX6 DDR controller design
would be able to explain it.

I am adding Peng Fan on Cc just in case. Or maybe you need to enter a
ticket to the NXP folks.

Regards,

Fabio Estevam

Re: [PATCH] tqma6: Fix DDR configuration

[Markus Niebel]

Hello Miquel,

> Initially investigating a Linux network issue causing a lot of drop and
> poor network performances on a custom system based on a TQMA6A module
> (based on an iMX6Q), [1st link below].
> 
> I eventually correlated my observations with a contention at the NIC
> level when in concurrency with the graphics pipeline. Troubleshooting
> this in the kernel lead to disabling DMA bursts accesses made by the IPU
> in order to avoid triggering the QoS at the interconnect level, reducing
> from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> worked on my setup but not on others, which still suffered from
> abnormally high drop rates even with this "fix".
> 
> After looking a while into TQ Systems BSP I figured out a number of
> differences in recent U-Boot out-of-tree patches they had in their
> repository [3rd link]. Parsing the differences one after the other lead
> me to this final solution.
> 
> The reset pad of the DDR controller was apparently misconfigured, Bit
> 18-19 picturing the "DDR select field". The current value b11 is
> reserved. The only defined value as of version 6 of the iMX6Q manual was
> b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> spotted after changing this configuration but all issues tracked thus
> far just vanished. All previous fixes have been proven irrelevant. Just
> clearing this field solved all our network issues and the drop rate as
> measured by iperf3 felt back to 0%.
> 
> Link: 
> https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> 
> Link: 
> https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
> 
> Link: 
> https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
> 
> Signed-off-by: Miquel Raynal <
> miquel.raynal@bootlin.com
> >
> ---
>  board/tq/tqma6/tqma6q.cfg | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg
> index a49489aed3f..a345c4de93d 100644
> --- a/board/tq/tqma6/tqma6q.cfg
> +++ b/board/tq/tqma6/tqma6q.cfg
> @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
>  DATA 4, MX6_IOM_DRAM_CAS, 0x00008030
>  DATA 4, MX6_IOM_DRAM_RAS, 0x00008030
>  DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
> -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030
> +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030

Thank you for pointing this out. Originally this error came from an
older/ancient reference manual. Sorry that we missed to bring this
upstream. We will send the changes for DCD data in the next days.

>  DATA 4, MX6_IOM_DRAM_SDCKE0, 0x00003000
>  DATA 4, MX6_IOM_DRAM_SDCKE1, 0x00000000
>  DATA 4, MX6_IOM_DRAM_SDBA2, 0x00000000
> -- 
> 2.34.1

Markus

-- 
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/

Re: [PATCH] tqma6: Fix DDR configuration

[Miquel Raynal]

Hi Markus,

Markus.Niebel@ew.tq-group.com wrote on Thu, 23 Nov 2023 14:04:43 +0100:

> Hello Miquel,
> 
> > Initially investigating a Linux network issue causing a lot of drop and
> > poor network performances on a custom system based on a TQMA6A module
> > (based on an iMX6Q), [1st link below].
> > 
> > I eventually correlated my observations with a contention at the NIC
> > level when in concurrency with the graphics pipeline. Troubleshooting
> > this in the kernel lead to disabling DMA bursts accesses made by the IPU
> > in order to avoid triggering the QoS at the interconnect level, reducing
> > from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> > worked on my setup but not on others, which still suffered from
> > abnormally high drop rates even with this "fix".
> > 
> > After looking a while into TQ Systems BSP I figured out a number of
> > differences in recent U-Boot out-of-tree patches they had in their
> > repository [3rd link]. Parsing the differences one after the other lead
> > me to this final solution.
> > 
> > The reset pad of the DDR controller was apparently misconfigured, Bit
> > 18-19 picturing the "DDR select field". The current value b11 is
> > reserved. The only defined value as of version 6 of the iMX6Q manual was
> > b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> > spotted after changing this configuration but all issues tracked thus
> > far just vanished. All previous fixes have been proven irrelevant. Just
> > clearing this field solved all our network issues and the drop rate as
> > measured by iperf3 felt back to 0%.
> > 
> > Link: 
> > https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> > 
> > Link: 
> > https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.html
> > 
> > Link: 
> > https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c28467b85485911dad3da6bc
> > 
> > Signed-off-by: Miquel Raynal <
> > miquel.raynal@bootlin.com  
> > >  
> > ---
> >  board/tq/tqma6/tqma6q.cfg | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg
> > index a49489aed3f..a345c4de93d 100644
> > --- a/board/tq/tqma6/tqma6q.cfg
> > +++ b/board/tq/tqma6/tqma6q.cfg
> > @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
> >  DATA 4, MX6_IOM_DRAM_CAS, 0x00008030
> >  DATA 4, MX6_IOM_DRAM_RAS, 0x00008030
> >  DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
> > -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030
> > +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030  
> 
> Thank you for pointing this out. Originally this error came from an
> older/ancient reference manual. Sorry that we missed to bring this
> upstream. We will send the changes for DCD data in the next days.

No problem, I'm glad this can now be solved. By any chance, could you
point to the relevant location of the manual (ddr or imx6 ?) explaining
what this is actually about? Because I failed to bring any real
explanation to my observations so far.

Thanks,
Miquèl

Re: [PATCH] tqma6: Fix DDR configuration

[Alexander Stein]

Hi Miquel,

Am Donnerstag, 23. November 2023, 14:12:33 CET schrieb Miquel Raynal:
> Hi Markus,
> 
> Markus.Niebel@ew.tq-group.com wrote on Thu, 23 Nov 2023 14:04:43 +0100:
> > Hello Miquel,
> > 
> > > Initially investigating a Linux network issue causing a lot of drop and
> > > poor network performances on a custom system based on a TQMA6A module
> > > (based on an iMX6Q), [1st link below].
> > > 
> > > I eventually correlated my observations with a contention at the NIC
> > > level when in concurrency with the graphics pipeline. Troubleshooting
> > > this in the kernel lead to disabling DMA bursts accesses made by the IPU
> > > in order to avoid triggering the QoS at the interconnect level, reducing
> > > from 50 to 10% the drop rate on eth0, [2nd link below]. The solution
> > > worked on my setup but not on others, which still suffered from
> > > abnormally high drop rates even with this "fix".
> > > 
> > > After looking a while into TQ Systems BSP I figured out a number of
> > > differences in recent U-Boot out-of-tree patches they had in their
> > > repository [3rd link]. Parsing the differences one after the other lead
> > > me to this final solution.
> > > 
> > > The reset pad of the DDR controller was apparently misconfigured, Bit
> > > 18-19 picturing the "DDR select field". The current value b11 is
> > > reserved. The only defined value as of version 6 of the iMX6Q manual was
> > > b00 "DDR3 and LPDDR2 mode". In practice no register difference has been
> > > spotted after changing this configuration but all issues tracked thus
> > > far just vanished. All previous fixes have been proven irrelevant. Just
> > > clearing this field solved all our network issues and the drop rate as
> > > measured by iperf3 felt back to 0%.
> > > 
> > > Link:
> > > https://lore.kernel.org/netdev/20231012193410.3d1812cf@xps-13/
> > > 
> > > Link:
> > > https://lists.freedesktop.org/archives/dri-devel/2023-October/428251.htm
> > > l
> > > 
> > > Link:
> > > https://github.com/tq-systems/u-boot-tqmaxx/commit/15eb6abbefbf6916c2846
> > > 7b85485911dad3da6bc
> > > 
> > > Signed-off-by: Miquel Raynal <
> > > miquel.raynal@bootlin.com
> > > 
> > > ---
> > > 
> > >  board/tq/tqma6/tqma6q.cfg | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/board/tq/tqma6/tqma6q.cfg b/board/tq/tqma6/tqma6q.cfg
> > > index a49489aed3f..a345c4de93d 100644
> > > --- a/board/tq/tqma6/tqma6q.cfg
> > > +++ b/board/tq/tqma6/tqma6q.cfg
> > > @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
> > > 
> > >  DATA 4, MX6_IOM_DRAM_CAS, 0x00008030
> > >  DATA 4, MX6_IOM_DRAM_RAS, 0x00008030
> > >  DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
> > > 
> > > -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030
> > > +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030
> > 
> > Thank you for pointing this out. Originally this error came from an
> > older/ancient reference manual. Sorry that we missed to bring this
> > upstream. We will send the changes for DCD data in the next days.
> 
> No problem, I'm glad this can now be solved. By any chance, could you
> point to the relevant location of the manual (ddr or imx6 ?) explaining
> what this is actually about? Because I failed to bring any real
> explanation to my observations so far.

It's a bit hidden but the comment above that list indicates these settings are 
iomuxc configurations. In this case the register 
IOMUXC_SW_PAD_CTL_PAD_DRAM_RESET. See i.MX6Q RM Rev.6 05/2020 section 
36.4.347.
Your patch configures the field "DDR Select Field" from reserved3 to 
DDR3_LPDDR2. It seems this field has to be set to 0 in every case.

Thanks again and best regards,
Alexander


> Thanks,
> Miquèl


-- 
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/

Re: [PATCH] tqma6: Fix DDR configuration

[Miquel Raynal]

Hi Alexander,

> > > > --- a/board/tq/tqma6/tqma6q.cfg
> > > > +++ b/board/tq/tqma6/tqma6q.cfg
> > > > @@ -36,7 +36,7 @@ DATA 4, MX6_IOM_DRAM_SDCLK_1, 0x00008030
> > > > 
> > > >  DATA 4, MX6_IOM_DRAM_CAS, 0x00008030
> > > >  DATA 4, MX6_IOM_DRAM_RAS, 0x00008030
> > > >  DATA 4, MX6_IOM_GRP_ADDDS, 0x00000030
> > > > 
> > > > -DATA 4, MX6_IOM_DRAM_RESET, 0x000C3030
> > > > +DATA 4, MX6_IOM_DRAM_RESET, 0x00003030  
> > > 
> > > Thank you for pointing this out. Originally this error came from an
> > > older/ancient reference manual. Sorry that we missed to bring this
> > > upstream. We will send the changes for DCD data in the next days.  
> > 
> > No problem, I'm glad this can now be solved. By any chance, could you
> > point to the relevant location of the manual (ddr or imx6 ?) explaining
> > what this is actually about? Because I failed to bring any real
> > explanation to my observations so far.  
> 
> It's a bit hidden but the comment above that list indicates these settings are 
> iomuxc configurations. In this case the register 
> IOMUXC_SW_PAD_CTL_PAD_DRAM_RESET. See i.MX6Q RM Rev.6 05/2020 section 
> 36.4.347.
> Your patch configures the field "DDR Select Field" from reserved3 to 
> DDR3_LPDDR2. It seems this field has to be set to 0 in every case.

Yes, that's also what I get from reading the iMX6Q manual. But I'm
sorry I don't understand why a runtime change of a reset pin
configuration can be so impacting. It's not like it only affects the
power-on sequence because I can reproduce the strange QoS/NIC issues
with a devmem once the kernel has started. I  believe there is a bit
more than just a pad configuration behind this field.

Thanks,
Miquèl