From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lokesh Vutla Subject: Re: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory Date: Thu, 15 Oct 2015 20:44:49 +0530 Message-ID: <561FC2E9.9050805@ti.com> References: <1444778000-22537-1-git-send-email-tony@atomide.com> <561DD014.10403@ti.com> <20151014160245.GA10113@atomide.com> <561F5C0C.6080106@ti.com> <20151015140551.GM10113@atomide.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20151015140551.GM10113@atomide.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=m.gmane.org@lists.infradead.org To: Tony Lindgren , "Woodruff, Richard" Cc: Dmitry Lifshitz , Enric Balletbo Serra , "Dr. H. Nikolaus Schaller" , Grazvydas Ignotas , Benoit Cousson , Javier Martinez Canillas , linux-omap@vger.kernel.org, Robert Nelson , Marek Belisko , linux-arm-kernel@lists.infradead.org List-Id: linux-omap@vger.kernel.org +Richard On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote: > * Lokesh Vutla [151015 01:02]: >> Hi Tony, >> >> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote: >>> * Lokesh Vutla [151013 20:53]: >>>> Hi Tony, >>>> >>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote: >>>>> On boards with more than 2GB of RAM booting goes wrong with things not working >>>>> and we're getting lots of l3 warnings: >>>>> >>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384() >>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access >>>>> ... >>>>> [] (scsi_add_host_with_dma) from [] (ata_scsi_add_hosts+0x5c/0x18c) >>>>> [] (ata_scsi_add_hosts) from [] (ata_host_register+0x150/0x2cc) >>>>> [] (ata_host_register) from [] (ata_host_activate+0xd4/0x124) >>>>> [] (ata_host_activate) from [] (ahci_host_activate+0x5c/0x194) >>>>> [] (ahci_host_activate) from [] (ahci_platform_init_host+0x1f0/0x3f0) >>>>> [] (ahci_platform_init_host) from [] (ahci_probe+0x70/0x98) >>>>> [] (ahci_probe) from [] (platform_drv_probe+0x54/0xb4) >>>>> >>>>> Let's fix the issue by enabling ZONE_DMA for LPAE. >>>> >>>> May I know on which platform you have reproduced this? >>> >>> This is on the 4GB version of isee igepv5 GEP0050-RB10 >>> >>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432 >>> >>>> Just wondering what other changes you made for booting a OMAP5+ based >>>> board with more than 2GB. >>> >>> Just the minimal dts changes I posted yesterday to use shared dtsi file >>> for omap5-uevm variants. Then boot with something like this in the kernel >>> cmdline: >>> >>> mem=2032M@0x80000000 mem=2048M@0x300000000 >> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the >> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from >> 0x2_0000_0000.(or am I missing something?) >> >> Any chance you tried doing memtest for all this memory? > > Seems to work just fine, it's been up since I posted $subject patch., I tried > a kernel compile on it already earlier and just started memtester 3900M which > seems to only capable of testing 2924MB though. > >> IIUC, It is not allowed to use the mix of L3 address mapping and MA >> physical address mapping. TRM specifies that >> "It is expected that the OS uses either the lower 2-GiB space and the >> lower aliased address of emif(a) and (b), or the continuous 8-GiB space >> and the upper aliasing of the emif(a) and (b) for all EMIF accesses." > > I think that depends on the chip selects that can be used. The above > is probably for the case where all 8GB was to be used. Richard, Do you agree with this? > >> I tried a similar 4GB setup and observed few stuff: >> - Need to have ARM_PV_FIXUP enabled something similar that is done in >> Keystone inorder to shift to higher address space.[1] [2] > > Seems that's optional for 4GB.. But probably a must for 8GB :) > >> - All the peripherals other than MPU, can access only 2GB. So we need to >> restrict the DMA and need an addresses translation. Along with this >> patch [3] [4] are needed. > > Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll > post an updated patch with the 2GB limit for dma_zone_size after some > more tests. > > For the dma-ranges, looks like I have not hit that one yet. Which drivers > did you run into problems without that? This is needed only when you completely shift to higher address space. > > Regards, > > Tony > >> [1] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b >> [2] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac >> [3] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d >> [4] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8 > From mboxrd@z Thu Jan 1 00:00:00 1970 From: a0131933@ti.com (Lokesh Vutla) Date: Thu, 15 Oct 2015 20:44:49 +0530 Subject: [PATCH] ARM: OMAP2+: Fix oops with LPAE and more than 2GB of memory In-Reply-To: <20151015140551.GM10113@atomide.com> References: <1444778000-22537-1-git-send-email-tony@atomide.com> <561DD014.10403@ti.com> <20151014160245.GA10113@atomide.com> <561F5C0C.6080106@ti.com> <20151015140551.GM10113@atomide.com> Message-ID: <561FC2E9.9050805@ti.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org +Richard On Thursday 15 October 2015 07:35 PM, Tony Lindgren wrote: > * Lokesh Vutla [151015 01:02]: >> Hi Tony, >> >> On Wednesday 14 October 2015 09:32 PM, Tony Lindgren wrote: >>> * Lokesh Vutla [151013 20:53]: >>>> Hi Tony, >>>> >>>> On Wednesday 14 October 2015 04:43 AM, Tony Lindgren wrote: >>>>> On boards with more than 2GB of RAM booting goes wrong with things not working >>>>> and we're getting lots of l3 warnings: >>>>> >>>>> WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x260/0x384() >>>>> 44000000.ocp:L3 Custom Error: MASTER MMC6 TARGET DMM1 (Idle): Data Access in User mode during Functional access >>>>> ... >>>>> [] (scsi_add_host_with_dma) from [] (ata_scsi_add_hosts+0x5c/0x18c) >>>>> [] (ata_scsi_add_hosts) from [] (ata_host_register+0x150/0x2cc) >>>>> [] (ata_host_register) from [] (ata_host_activate+0xd4/0x124) >>>>> [] (ata_host_activate) from [] (ahci_host_activate+0x5c/0x194) >>>>> [] (ahci_host_activate) from [] (ahci_platform_init_host+0x1f0/0x3f0) >>>>> [] (ahci_platform_init_host) from [] (ahci_probe+0x70/0x98) >>>>> [] (ahci_probe) from [] (platform_drv_probe+0x54/0xb4) >>>>> >>>>> Let's fix the issue by enabling ZONE_DMA for LPAE. >>>> >>>> May I know on which platform you have reproduced this? >>> >>> This is on the 4GB version of isee igepv5 GEP0050-RB10 >>> >>> https://isee.biz/products/igep-processor-boards/igepv5-omap5432 >>> >>>> Just wondering what other changes you made for booting a OMAP5+ based >>>> board with more than 2GB. >>> >>> Just the minimal dts changes I posted yesterday to use shared dtsi file >>> for omap5-uevm variants. Then boot with something like this in the kernel >>> cmdline: >>> >>> mem=2032M at 0x80000000 mem=2048M at 0x300000000 >> Is it the upper 2GB memory starts from 0x3_0000_0000? Looking at the >> Table 4-4. "AXI Access memory map" in TRM, it tells it starts from >> 0x2_0000_0000.(or am I missing something?) >> >> Any chance you tried doing memtest for all this memory? > > Seems to work just fine, it's been up since I posted $subject patch., I tried > a kernel compile on it already earlier and just started memtester 3900M which > seems to only capable of testing 2924MB though. > >> IIUC, It is not allowed to use the mix of L3 address mapping and MA >> physical address mapping. TRM specifies that >> "It is expected that the OS uses either the lower 2-GiB space and the >> lower aliased address of emif(a) and (b), or the continuous 8-GiB space >> and the upper aliasing of the emif(a) and (b) for all EMIF accesses." > > I think that depends on the chip selects that can be used. The above > is probably for the case where all 8GB was to be used. Richard, Do you agree with this? > >> I tried a similar 4GB setup and observed few stuff: >> - Need to have ARM_PV_FIXUP enabled something similar that is done in >> Keystone inorder to shift to higher address space.[1] [2] > > Seems that's optional for 4GB.. But probably a must for 8GB :) > >> - All the peripherals other than MPU, can access only 2GB. So we need to >> restrict the DMA and need an addresses translation. Along with this >> patch [3] [4] are needed. > > Yes there's no memory accessible beyond 2GB for the DMA hardware. I'll > post an updated patch with the 2GB limit for dma_zone_size after some > more tests. > > For the dma-ranges, looks like I have not hit that one yet. Which drivers > did you run into problems without that? This is needed only when you completely shift to higher address space. > > Regards, > > Tony > >> [1] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/be7d2d7e18f473e6078cd7c0b25ec3c58f40736b >> [2] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/dd0a564ee565405bec5e9f58f0c97fb6714110ac >> [3] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/985ed401e04926cb88b23646271dc96900abc50d >> [4] >> https://git.ti.com/~lokesh/ti-linux-kernel/lokesh-ti-linux-kernel/commit/c1f7568c4cc6f3146b7137783bd403967aabd0d8 >