[PATCH] ARM: mvebu: fix overlap of Crypto SRAM with PCIe memory window

[olof@lixom.net (Olof Johansson)]

On Thu, Mar 10, 2016 at 01:56:38PM +0100, Gregory CLEMENT wrote:
> Hi Thomas,
>  
>  On mar., mars 08 2016, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:
> 
> > When the Crypto SRAM mappings were added to the Device Tree files
> > describing the Armada XP boards in commit c466d997bb16 ("ARM: mvebu:
> > define crypto SRAM ranges for all armada-xp boards"), the fact that
> > those mappings were overlaping with the PCIe memory aperture was
> > overlooked. Due to this, we currently have for all Armada XP platforms
> > a situation that looks like this:
> >
> > Memory mapping on Armada XP boards with internal registers at
> > 0xf1000000:
> >
> >  - 0x00000000 -> 0xf0000000	3.75G 	RAM
> >  - 0xf0000000 -> 0xf1000000	16M	NOR flashes (AXP GP / AXP DB)
> >  - 0xf1000000 -> 0xf1100000	1M	internal registers
> >  - 0xf8000000 -> 0xffe0000	126M	PCIe memory aperture
> >  - 0xf8100000 -> 0xf8110000	64KB	Crypto SRAM #0	=> OVERLAPS WITH PCIE !
> >  - 0xf8110000 -> 0xf8120000	64KB	Crypto SRAM #1	=> OVERLAPS WITH PCIE !
> >  - 0xffe00000 -> 0xfff00000	1M	PCIe I/O aperture
> >  - 0xfff0000  -> 0xffffffff	1M	BootROM
> >
> > The overlap means that when PCIe devices are added, depending on their
> > memory window needs, they might or might not be mapped into the
> > physical address space. Indeed, they will not be mapped if the area
> > allocated in the PCIe memory aperture by the PCI core overlaps with
> > one of the Crypto SRAM. Typically, a Intel IGB PCIe NIC that needs 8MB
> > of PCIe memory will see its PCIe memory window allocated from
> > 0xf80000000 for 8MB, which overlaps with the Crypto SRAM windows. Due
> > to this, the PCIe window is not created, and any attempt to access the
> > PCIe window makes the kernel explode:
> >
> > [    3.302213] igb: Copyright (c) 2007-2014 Intel Corporation.
> > [    3.307841] pci 0000:00:09.0: enabling device (0140 -> 0143)
> > [    3.313539] mvebu_mbus: cannot add window '4:f8', conflicts with another window
> > [    3.320870] mvebu-pcie soc:pcie-controller: Could not create MBus window at [mem 0xf8000000-0xf87fffff]: -22
> > [    3.330811] Unhandled fault: external abort on non-linefetch (0x1008) at 0xf08c0018
> >
> > This problem does not occur on Armada 370 boards, because we use the
> > following memory mapping (for boards that have internal registers at
> > 0xf1000000):
> >
> >  - 0x00000000 -> 0xf0000000	3.75G 	RAM
> >  - 0xf0000000 -> 0xf1000000	16M	NOR flashes (AXP GP / AXP DB)
> >  - 0xf1000000 -> 0xf1100000	1M	internal registers
> >  - 0xf1100000 -> 0xf1110000	64KB	Crypto SRAM #0 => OK !
> >  - 0xf8000000 -> 0xffe0000	126M	PCIe memory
> >  - 0xffe00000 -> 0xfff00000	1M	PCIe I/O
> >  - 0xfff0000  -> 0xffffffff	1M	BootROM
> >
> > Obviously, the solution is to align the location of the Crypto SRAM
> > mappings of Armada XP to be similar with the ones on Armada 370, i.e
> > have them between the "internal registers" area and the beginning of
> > the PCIe aperture.
> >
> > However, we have a special case with the OpenBlocks AX3-4 platform,
> > which has a 128 MB NOR flash. Currently, this NOR flash is mapped from
> > 0xf0000000 to 0xf8000000. This is possible because on OpenBlocks
> > AX3-4, the internal registers are not at 0xf1000000. And this explains
> > why the Crypto SRAM mappings were not configured at the same place on
> > Armada XP.
> >
> > Hence, the solution is two-fold:
> >
> >  (1) Move the NOR flash mapping on Armada XP OpenBlocks AX3-4 from
> >      0xe8000000 to 0xf0000000. This frees the 0xf0000000 ->
> >      0xf80000000 space.
> >
> >  (2) Move the Crypto SRAM mappings on Armada XP to be similar to
> >      Armada 370 (except of course that Armada XP has two Crypto SRAM
> >      and not one).
> >
> > After this patch, the memory mapping on Armada XP boards with
> > registers at 0xf1 is:
> >
> >  - 0x00000000 -> 0xf0000000	3.75G 	RAM
> >  - 0xf0000000 -> 0xf1000000	16M	NOR flashes (AXP GP / AXP DB)
> >  - 0xf1000000 -> 0xf1100000	1M	internal registers
> >  - 0xf1100000 -> 0xf1110000	64KB	Crypto SRAM #0
> >  - 0xf1110000 -> 0xf1120000	64KB	Crypto SRAM #1
> >  - 0xf8000000 -> 0xffe0000	126M	PCIe memory
> >  - 0xffe00000 -> 0xfff00000	1M	PCIe I/O
> >  - 0xfff0000  -> 0xffffffff	1M	BootROM
> >
> > And the memory mapping for the special case of the OpenBlocks AX3-4
> > (internal registers at 0xd0000000, NOR of 128 MB):
> >
> >  - 0x00000000 -> 0xc0000000	3G 	RAM
> >  - 0xd0000000 -> 0xd1000000	1M	internal registers
> >  - 0xe800000  -> 0xf0000000	128M	NOR flash
> >  - 0xf1100000 -> 0xf1110000	64KB	Crypto SRAM #0
> >  - 0xf1110000 -> 0xf1120000	64KB	Crypto SRAM #1
> >  - 0xf8000000 -> 0xffe0000	126M	PCIe memory
> >  - 0xffe00000 -> 0xfff00000	1M	PCIe I/O
> >  - 0xfff0000  -> 0xffffffff	1M	BootROM
> >
> > Fixes: c466d997bb16 ("ARM: mvebu: define crypto SRAM ranges for all armada-xp boards")
> > Reported-by: Phil Sutter <phil@nwl.cc>
> > Cc: Phil Sutter <phil@nwl.cc>
> > Cc: <stable@kernel.org>
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
> Acked-by: Gregory CLEMENT <gregory.clement@free-electrons.com>
> 
> Arnd, Olof,
> 
> it would be nice having this fix applied on v4.5-rc. Could you applied
> this patch directly, or do you want pull request for it?

Applied to fixes. Are you 150% sure you don't need some bake time on this patch
to avoid surprises?


-Olof