[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[Thierry Reding]

On Mon, Feb 11, 2019 at 10:04:37AM +0000, Tristan Bastian wrote:
> 
> 
> 
> Thierry Reding – Mon, 11. February 2019 10:38
> > On Mon, Feb 11, 2019 at 09:20:33AM +0000, Tristan Bastian wrote:
> > > 
> > > Thierry Reding – Mon, 11. February 2019 9:52
> > > > On Sun, Feb 10, 2019 at 08:53:11PM +0100, Tristan Bastian wrote:
> > > > > Thierry, do you have any news on this?
> > > > > 
> > > > > I don't think, that google is going to push an updated version of
> > coreboot
> > > > > to each nyan-big device..
> > > > > So reverting this patch at least for the nyan devices would be the best
> > I
> > > > > think..
> > > > 
> > > > Yes, I agree. I had a brief chat with Rob Herring about this and he too
> > > > agrees that we should revert it for now. I'll make it a manual revert
> > > > and add a comment to the device tree node that will hopefully avoid any
> > > > future janitorial "cleanups" of this sort.
> > > 
> > > great news! :)
> > > 
> > > > > BTW: I'm now running u-boot natively and it seems like u-boot always has
> > a
> > > > > problem with the memory..
> > > > > If u-boot is used chainloaded to coreboot it is only getting 2GB of
> > memory
> > > > > and running u-boot natively also just gives me 2GB..
> > > > > I've tested that with a kernel with "ARM: tegra: Fix unit_address_vs_reg
> > DTC
> > > > > warnings for /memory" reverted and also on a kernel, before this patch
> > was
> > > > > applied..
> > > > 
> > > > It's possible that U-Boot doesn't support LPAE and therefore may not be
> > > > able to use more than the 2 GiB of memory.
> > > 
> > > So I at least enabled LPAE in u-boot with "CONFIG_ARMV7_LPAE=y" and
> > > this was for some reason also needed to get some output on the
> > > display..
> > > I'm not sure why LPAE needs to activated in u-boot for display output
> > > on the nyan-big..
> > > Without LPAE enabled u-boot was still working, and booted linux, but
> > > u-boot didn't display anything on screen, linux then did..
> > 
> > Yeah, that's surprising. Perhaps without LPAE U-Boot thinks there's not
> > enough memory for the framebuffer? There should be plenty, so maybe
> > there is something else going on here.
> > 
> > > > However, U-Boot should be
> > > > able to tell the kernel exactly how much memory the system has and pass
> > > > that on via device tree. That still does work, right?
> > > 
> > > It seems like this is not working..
> > > And this seems to be the case with both, u-boot chainloaded and
> > > running u-boot natively..
> > > 
> > > I've used these scripts for flashing:
> > > github.com/NVIDIA/tegra-uboot-flasher-scripts
> > > And used the norrin device since it seems like it is the nyan-big dev
> > > board?
> > > But going with the norrin device config should not be the issue here
> > > since the problem also exists when chainloading u-boot, right?
> > 
> > It could be a problem. The memory bank configuration is stored in what's
> > called a BCT along with a bunch of other parameters that define what the
> > memory controller needs to access the given memory chips. So if you've
> > flashed a BCT that's for a board with only 2 GiB of memory you would end
> > up with a system that can't address any more than that. It's somewhat
> > surprising that memory accesses work at all with a BCT that's for
> > different memory chips, but sometimes you can get lucky.
> > 
> > You may want to try reflashing with the right BCT. The simplest would
> > probably be to duplicate the cbootimage configuration for Norrin and
> > substitute the relevant bits by what you have from the Chromebook flash
> > utilities. Looks like the BCT in really the only thing you can replace
> > there. Make sure to replace it with the one that matches your Chromebook
> > and it should give you the right memory bank configuration.
> 
> Is there a way to check if the correct bct for my 4GB model was used?
> What I tried was to edit the existing bct config file of norrin from here:
> https://github.com/NVIDIA/cbootimage-configs/blob/master/tegra124/nvidia/norrin/PM370_Hynix_2GB_H5TC4G63AFR_PBA_924MHz_01212014.bct.cfg
> by replacing all entries for each memory bank with the entries from here:
> https://github.com/coreboot/coreboot/blob/master/src/mainboard/google/nyan_big/bct/sdram-hynix-4GB-792.inc
> 
> I also added the SDRAM[0], SDRAM[1], SDRAM[2], SDRAM[3] each time in front of each line..
> This also booted for me, but also just leaves me with only 2Gb of my 4GB..
> 
> And since this problem also occurs, when chainloading u-boot, this must be a problem with u-boot, I think..

Not necessarily. U-Boot reads the memory size from EMC registers and
those EMC registers are programmed based on the contents of the BCT. So
if the BCT is wrong, then it's going to be wrong irrespective of what
bootloader you use.

That said, I does indeed look as if U-Boot only supports 2 GiB of memory
on 32-bit Tegra. arch/arm/mach-tegra/board.c contains the relevant code.
If you look at dram_init() it uses query_sdram_size() to get the size of
RAM. That will allow to convey up to 4 GiB of RAM, but then the code in
board2.c (see dram_init_banksize()) only uses a single bank on 32-bit
Tegra (CONFIG_PHYS_64BIT is only set for 64-bit Tegra).

According to the comments for dram_init_bank_size() we can't store the
second bank because the .start field is 32 bits wide and therefore can't
store the 0x100000000 start address for the bank.

It sounds to me like ARMV7_LPAE should perhaps select PHYS_64BIT to make
sure all the appropriate variables can store addresses beyond the 32-bit
boundary. It's one thing to not support > 2 GiB of memory in U-Boot, but
it's an entirely different story if we can't even pass > GiB of memory
to the kernel if they exist.

Stephen, do you remember if there was anything preventing us from going
64-bit for phys_addr_t and phys_size_t on LPAE enabled systems?

Cc Tom for visibility and U-Boot since this is a U-Boot bug.

Thierry

> Tristan
> 
> > > Simon Glass used his servo board to enter the tegrarcm mode:
> > > patchwork.ozlabs.org/patch/1019003/
> > > 
> > > I had to erase the internal SPI flash in order to boot into the
> > > tegrarcm mode.. 
> > > Do you have some instructions or internal documentation to tell me, if
> > > some of the unpopulated buttons on the nyan-big mainboard can be used
> > > to enter the tegrarcm state?
> > 
> > You should be able to boot into recovery mode with the "enterrcm"
> > command in U-Boot. I'm not aware of any buttons on the mainboard that
> > can be used for this, unfortunately.
> > 
> > Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20190211/c94ed6dd/attachment.sig>

[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[Tristan Bastian]

Am 11.02.19 um 11:48 schrieb Thierry Reding:
> On Mon, Feb 11, 2019 at 10:04:37AM +0000, Tristan Bastian wrote:
>>
>>
>> Thierry Reding – Mon, 11. February 2019 10:38
>>> On Mon, Feb 11, 2019 at 09:20:33AM +0000, Tristan Bastian wrote:
>>>> Thierry Reding – Mon, 11. February 2019 9:52
>>>>> On Sun, Feb 10, 2019 at 08:53:11PM +0100, Tristan Bastian wrote:
>>>>>> Thierry, do you have any news on this?
>>>>>>
>>>>>> I don't think, that google is going to push an updated version of
>>> coreboot
>>>>>> to each nyan-big device..
>>>>>> So reverting this patch at least for the nyan devices would be the best
>>> I
>>>>>> think..
>>>>> Yes, I agree. I had a brief chat with Rob Herring about this and he too
>>>>> agrees that we should revert it for now. I'll make it a manual revert
>>>>> and add a comment to the device tree node that will hopefully avoid any
>>>>> future janitorial "cleanups" of this sort.
>>>> great news! :)
>>>>
>>>>>> BTW: I'm now running u-boot natively and it seems like u-boot always has
>>> a
>>>>>> problem with the memory..
>>>>>> If u-boot is used chainloaded to coreboot it is only getting 2GB of
>>> memory
>>>>>> and running u-boot natively also just gives me 2GB..
>>>>>> I've tested that with a kernel with "ARM: tegra: Fix unit_address_vs_reg
>>> DTC
>>>>>> warnings for /memory" reverted and also on a kernel, before this patch
>>> was
>>>>>> applied..
>>>>> It's possible that U-Boot doesn't support LPAE and therefore may not be
>>>>> able to use more than the 2 GiB of memory.
>>>> So I at least enabled LPAE in u-boot with "CONFIG_ARMV7_LPAE=y" and
>>>> this was for some reason also needed to get some output on the
>>>> display..
>>>> I'm not sure why LPAE needs to activated in u-boot for display output
>>>> on the nyan-big..
>>>> Without LPAE enabled u-boot was still working, and booted linux, but
>>>> u-boot didn't display anything on screen, linux then did..
>>> Yeah, that's surprising. Perhaps without LPAE U-Boot thinks there's not
>>> enough memory for the framebuffer? There should be plenty, so maybe
>>> there is something else going on here.
>>>
>>>>> However, U-Boot should be
>>>>> able to tell the kernel exactly how much memory the system has and pass
>>>>> that on via device tree. That still does work, right?
>>>> It seems like this is not working..
>>>> And this seems to be the case with both, u-boot chainloaded and
>>>> running u-boot natively..
>>>>
>>>> I've used these scripts for flashing:
>>>> github.com/NVIDIA/tegra-uboot-flasher-scripts
>>>> And used the norrin device since it seems like it is the nyan-big dev
>>>> board?
>>>> But going with the norrin device config should not be the issue here
>>>> since the problem also exists when chainloading u-boot, right?
>>> It could be a problem. The memory bank configuration is stored in what's
>>> called a BCT along with a bunch of other parameters that define what the
>>> memory controller needs to access the given memory chips. So if you've
>>> flashed a BCT that's for a board with only 2 GiB of memory you would end
>>> up with a system that can't address any more than that. It's somewhat
>>> surprising that memory accesses work at all with a BCT that's for
>>> different memory chips, but sometimes you can get lucky.
>>>
>>> You may want to try reflashing with the right BCT. The simplest would
>>> probably be to duplicate the cbootimage configuration for Norrin and
>>> substitute the relevant bits by what you have from the Chromebook flash
>>> utilities. Looks like the BCT in really the only thing you can replace
>>> there. Make sure to replace it with the one that matches your Chromebook
>>> and it should give you the right memory bank configuration.
>> Is there a way to check if the correct bct for my 4GB model was used?
>> What I tried was to edit the existing bct config file of norrin from here:
>> https://github.com/NVIDIA/cbootimage-configs/blob/master/tegra124/nvidia/norrin/PM370_Hynix_2GB_H5TC4G63AFR_PBA_924MHz_01212014.bct.cfg
>> by replacing all entries for each memory bank with the entries from here:
>> https://github.com/coreboot/coreboot/blob/master/src/mainboard/google/nyan_big/bct/sdram-hynix-4GB-792.inc
>>
>> I also added the SDRAM[0], SDRAM[1], SDRAM[2], SDRAM[3] each time in front of each line..
>> This also booted for me, but also just leaves me with only 2Gb of my 4GB..
>>
>> And since this problem also occurs, when chainloading u-boot, this must be a problem with u-boot, I think..
> Not necessarily. U-Boot reads the memory size from EMC registers and
> those EMC registers are programmed based on the contents of the BCT. So
> if the BCT is wrong, then it's going to be wrong irrespective of what
> bootloader you use.
>
> That said, I does indeed look as if U-Boot only supports 2 GiB of memory
> on 32-bit Tegra. arch/arm/mach-tegra/board.c contains the relevant code.
> If you look at dram_init() it uses query_sdram_size() to get the size of
> RAM. That will allow to convey up to 4 GiB of RAM, but then the code in
> board2.c (see dram_init_banksize()) only uses a single bank on 32-bit
> Tegra (CONFIG_PHYS_64BIT is only set for 64-bit Tegra).

I've enabled "CONFIG_PHYS_64BIT" for testing and it seems to be at least 
helping.
I'm getting a bunch of compile warnings about casting pointers from 
integers of different size..

But I'm getting 4GB of memory now if I boot via an extlinux.conf file. :)
Unfortunately this makes some other things unusable, like efi booting is 
no longer working for me..
I'm getting the following error when trying to efi boot grub2:

free magic is broken at 0xe59ff3f8: 0xffffffff

And if I try to boot again (without resetting) via efi I'm getting:

fdt_root: FDT_ERR_BADMAGIC
ERROR: root node setup failed
- must RESET the board to recover

Tristan

> According to the comments for dram_init_bank_size() we can't store the
> second bank because the .start field is 32 bits wide and therefore can't
> store the 0x100000000 start address for the bank.
>
> It sounds to me like ARMV7_LPAE should perhaps select PHYS_64BIT to make
> sure all the appropriate variables can store addresses beyond the 32-bit
> boundary. It's one thing to not support > 2 GiB of memory in U-Boot, but
> it's an entirely different story if we can't even pass > GiB of memory
> to the kernel if they exist.
>
> Stephen, do you remember if there was anything preventing us from going
> 64-bit for phys_addr_t and phys_size_t on LPAE enabled systems?
>
> Cc Tom for visibility and U-Boot since this is a U-Boot bug.
>
> Thierry
>
>> Tristan
>>
>>>> Simon Glass used his servo board to enter the tegrarcm mode:
>>>> patchwork.ozlabs.org/patch/1019003/
>>>>
>>>> I had to erase the internal SPI flash in order to boot into the
>>>> tegrarcm mode..
>>>> Do you have some instructions or internal documentation to tell me, if
>>>> some of the unpopulated buttons on the nyan-big mainboard can be used
>>>> to enter the tegrarcm state?
>>> You should be able to boot into recovery mode with the "enterrcm"
>>> command in U-Boot. I'm not aware of any buttons on the mainboard that
>>> can be used for this, unfortunately.
>>>
>>> Thierry

[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[Stephen Warren]

On 2/11/19 3:48 AM, Thierry Reding wrote:
> On Mon, Feb 11, 2019 at 10:04:37AM +0000, Tristan Bastian wrote:
>>
>>
>>
>> Thierry Reding – Mon, 11. February 2019 10:38
>>> On Mon, Feb 11, 2019 at 09:20:33AM +0000, Tristan Bastian wrote:
>>>>
>>>> Thierry Reding – Mon, 11. February 2019 9:52
>>>>> On Sun, Feb 10, 2019 at 08:53:11PM +0100, Tristan Bastian wrote:
>>>>>> Thierry, do you have any news on this?
>>>>>>
>>>>>> I don't think, that google is going to push an updated version of
>>> coreboot
>>>>>> to each nyan-big device..
>>>>>> So reverting this patch at least for the nyan devices would be the best
>>> I
>>>>>> think..
>>>>>
>>>>> Yes, I agree. I had a brief chat with Rob Herring about this and he too
>>>>> agrees that we should revert it for now. I'll make it a manual revert
>>>>> and add a comment to the device tree node that will hopefully avoid any
>>>>> future janitorial "cleanups" of this sort.
>>>>
>>>> great news! :)
>>>>
>>>>>> BTW: I'm now running u-boot natively and it seems like u-boot always has
>>> a
>>>>>> problem with the memory..
>>>>>> If u-boot is used chainloaded to coreboot it is only getting 2GB of
>>> memory
>>>>>> and running u-boot natively also just gives me 2GB..
>>>>>> I've tested that with a kernel with "ARM: tegra: Fix unit_address_vs_reg
>>> DTC
>>>>>> warnings for /memory" reverted and also on a kernel, before this patch
>>> was
>>>>>> applied..
>>>>>
>>>>> It's possible that U-Boot doesn't support LPAE and therefore may not be
>>>>> able to use more than the 2 GiB of memory.
>>>>
>>>> So I at least enabled LPAE in u-boot with "CONFIG_ARMV7_LPAE=y" and
>>>> this was for some reason also needed to get some output on the
>>>> display..
>>>> I'm not sure why LPAE needs to activated in u-boot for display output
>>>> on the nyan-big..
>>>> Without LPAE enabled u-boot was still working, and booted linux, but
>>>> u-boot didn't display anything on screen, linux then did..
>>>
>>> Yeah, that's surprising. Perhaps without LPAE U-Boot thinks there's not
>>> enough memory for the framebuffer? There should be plenty, so maybe
>>> there is something else going on here.
>>>
>>>>> However, U-Boot should be
>>>>> able to tell the kernel exactly how much memory the system has and pass
>>>>> that on via device tree. That still does work, right?
>>>>
>>>> It seems like this is not working..
>>>> And this seems to be the case with both, u-boot chainloaded and
>>>> running u-boot natively..
>>>>
>>>> I've used these scripts for flashing:
>>>> github.com/NVIDIA/tegra-uboot-flasher-scripts
>>>> And used the norrin device since it seems like it is the nyan-big dev
>>>> board?
>>>> But going with the norrin device config should not be the issue here
>>>> since the problem also exists when chainloading u-boot, right?
>>>
>>> It could be a problem. The memory bank configuration is stored in what's
>>> called a BCT along with a bunch of other parameters that define what the
>>> memory controller needs to access the given memory chips. So if you've
>>> flashed a BCT that's for a board with only 2 GiB of memory you would end
>>> up with a system that can't address any more than that. It's somewhat
>>> surprising that memory accesses work at all with a BCT that's for
>>> different memory chips, but sometimes you can get lucky.
>>>
>>> You may want to try reflashing with the right BCT. The simplest would
>>> probably be to duplicate the cbootimage configuration for Norrin and
>>> substitute the relevant bits by what you have from the Chromebook flash
>>> utilities. Looks like the BCT in really the only thing you can replace
>>> there. Make sure to replace it with the one that matches your Chromebook
>>> and it should give you the right memory bank configuration.
>>
>> Is there a way to check if the correct bct for my 4GB model was used?
>> What I tried was to edit the existing bct config file of norrin from here:
>> https://github.com/NVIDIA/cbootimage-configs/blob/master/tegra124/nvidia/norrin/PM370_Hynix_2GB_H5TC4G63AFR_PBA_924MHz_01212014.bct.cfg
>> by replacing all entries for each memory bank with the entries from here:
>> https://github.com/coreboot/coreboot/blob/master/src/mainboard/google/nyan_big/bct/sdram-hynix-4GB-792.inc
>>
>> I also added the SDRAM[0], SDRAM[1], SDRAM[2], SDRAM[3] each time in front of each line..
>> This also booted for me, but also just leaves me with only 2Gb of my 4GB..
>>
>> And since this problem also occurs, when chainloading u-boot, this must be a problem with u-boot, I think..
> 
> Not necessarily. U-Boot reads the memory size from EMC registers and
> those EMC registers are programmed based on the contents of the BCT. So
> if the BCT is wrong, then it's going to be wrong irrespective of what
> bootloader you use.
> 
> That said, I does indeed look as if U-Boot only supports 2 GiB of memory
> on 32-bit Tegra. arch/arm/mach-tegra/board.c contains the relevant code.
> If you look at dram_init() it uses query_sdram_size() to get the size of
> RAM. That will allow to convey up to 4 GiB of RAM, but then the code in
> board2.c (see dram_init_banksize()) only uses a single bank on 32-bit
> Tegra (CONFIG_PHYS_64BIT is only set for 64-bit Tegra).
> 
> According to the comments for dram_init_bank_size() we can't store the
> second bank because the .start field is 32 bits wide and therefore can't
> store the 0x100000000 start address for the bank.
> 
> It sounds to me like ARMV7_LPAE should perhaps select PHYS_64BIT to make
> sure all the appropriate variables can store addresses beyond the 32-bit
> boundary. It's one thing to not support > 2 GiB of memory in U-Boot, but
> it's an entirely different story if we can't even pass > GiB of memory
> to the kernel if they exist.
> 
> Stephen, do you remember if there was anything preventing us from going
> 64-bit for phys_addr_t and phys_size_t on LPAE enabled systems?

We probably just never tried since we never needed it and/or LPAE wasn't 
available (in U-Boot SW) when it initially might have been useful. I 
would not be surprised if some of the low-level Tegra code in U-Boot 
conflates 64-bit support with SoC being >= T210, so things might not be 
as simple as enabling CONFIG_PHYS_64BIT, but you never know...

[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[Marcel Ziswiler]

Hi there

On Mon, 2019-02-11 at 15:13 -0700, Stephen Warren wrote:
> On 2/11/19 3:48 AM, Thierry Reding wrote:
> > On Mon, Feb 11, 2019 at 10:04:37AM +0000, Tristan Bastian wrote:
> > > 
> > > 
> > > Thierry Reding – Mon, 11. February 2019 10:38
> > > > On Mon, Feb 11, 2019 at 09:20:33AM +0000, Tristan Bastian
> > > > wrote:
> > > > > Thierry Reding – Mon, 11. February 2019 9:52
> > > > > > On Sun, Feb 10, 2019 at 08:53:11PM +0100, Tristan Bastian
> > > > > > wrote:
> > > > > > > Thierry, do you have any news on this?
> > > > > > > 
> > > > > > > I don't think, that google is going to push an updated
> > > > > > > version of
> > > > coreboot
> > > > > > > to each nyan-big device..
> > > > > > > So reverting this patch at least for the nyan devices
> > > > > > > would be the best
> > > > I
> > > > > > > think..
> > > > > > 
> > > > > > Yes, I agree. I had a brief chat with Rob Herring about
> > > > > > this and he too
> > > > > > agrees that we should revert it for now. I'll make it a
> > > > > > manual revert
> > > > > > and add a comment to the device tree node that will
> > > > > > hopefully avoid any
> > > > > > future janitorial "cleanups" of this sort.
> > > > > 
> > > > > great news! :)
> > > > > 
> > > > > > > BTW: I'm now running u-boot natively and it seems like u-
> > > > > > > boot always has
> > > > a
> > > > > > > problem with the memory..
> > > > > > > If u-boot is used chainloaded to coreboot it is only
> > > > > > > getting 2GB of
> > > > memory
> > > > > > > and running u-boot natively also just gives me 2GB..
> > > > > > > I've tested that with a kernel with "ARM: tegra: Fix
> > > > > > > unit_address_vs_reg
> > > > DTC
> > > > > > > warnings for /memory" reverted and also on a kernel,
> > > > > > > before this patch
> > > > was
> > > > > > > applied..
> > > > > > 
> > > > > > It's possible that U-Boot doesn't support LPAE and
> > > > > > therefore may not be
> > > > > > able to use more than the 2 GiB of memory.
> > > > > 
> > > > > So I at least enabled LPAE in u-boot with
> > > > > "CONFIG_ARMV7_LPAE=y" and
> > > > > this was for some reason also needed to get some output on
> > > > > the
> > > > > display..
> > > > > I'm not sure why LPAE needs to activated in u-boot for
> > > > > display output
> > > > > on the nyan-big..
> > > > > Without LPAE enabled u-boot was still working, and booted
> > > > > linux, but
> > > > > u-boot didn't display anything on screen, linux then did..
> > > > 
> > > > Yeah, that's surprising. Perhaps without LPAE U-Boot thinks
> > > > there's not
> > > > enough memory for the framebuffer? There should be plenty, so
> > > > maybe
> > > > there is something else going on here.
> > > > 
> > > > > > However, U-Boot should be
> > > > > > able to tell the kernel exactly how much memory the system
> > > > > > has and pass
> > > > > > that on via device tree. That still does work, right?
> > > > > 
> > > > > It seems like this is not working..
> > > > > And this seems to be the case with both, u-boot chainloaded
> > > > > and
> > > > > running u-boot natively..
> > > > > 
> > > > > I've used these scripts for flashing:
> > > > > github.com/NVIDIA/tegra-uboot-flasher-scripts
> > > > > And used the norrin device since it seems like it is the
> > > > > nyan-big dev
> > > > > board?
> > > > > But going with the norrin device config should not be the
> > > > > issue here
> > > > > since the problem also exists when chainloading u-boot,
> > > > > right?
> > > > 
> > > > It could be a problem. The memory bank configuration is stored
> > > > in what's
> > > > called a BCT along with a bunch of other parameters that define
> > > > what the
> > > > memory controller needs to access the given memory chips. So if
> > > > you've
> > > > flashed a BCT that's for a board with only 2 GiB of memory you
> > > > would end
> > > > up with a system that can't address any more than that. It's
> > > > somewhat
> > > > surprising that memory accesses work at all with a BCT that's
> > > > for
> > > > different memory chips, but sometimes you can get lucky.
> > > > 
> > > > You may want to try reflashing with the right BCT. The simplest
> > > > would
> > > > probably be to duplicate the cbootimage configuration for
> > > > Norrin and
> > > > substitute the relevant bits by what you have from the
> > > > Chromebook flash
> > > > utilities. Looks like the BCT in really the only thing you can
> > > > replace
> > > > there. Make sure to replace it with the one that matches your
> > > > Chromebook
> > > > and it should give you the right memory bank configuration.
> > > 
> > > Is there a way to check if the correct bct for my 4GB model was
> > > used?
> > > What I tried was to edit the existing bct config file of norrin
> > > from here:
> > > https://github.com/NVIDIA/cbootimage-configs/blob/master/tegra124/nvidia/norrin/PM370_Hynix_2GB_H5TC4G63AFR_PBA_924MHz_01212014.bct.cfg
> > > by replacing all entries for each memory bank with the entries
> > > from here:
> > > https://github.com/coreboot/coreboot/blob/master/src/mainboard/google/nyan_big/bct/sdram-hynix-4GB-792.inc
> > > 
> > > I also added the SDRAM[0], SDRAM[1], SDRAM[2], SDRAM[3] each time
> > > in front of each line..
> > > This also booted for me, but also just leaves me with only 2Gb of
> > > my 4GB..
> > > 
> > > And since this problem also occurs, when chainloading u-boot,
> > > this must be a problem with u-boot, I think..
> > 
> > Not necessarily. U-Boot reads the memory size from EMC registers
> > and
> > those EMC registers are programmed based on the contents of the
> > BCT. So
> > if the BCT is wrong, then it's going to be wrong irrespective of
> > what
> > bootloader you use.
> > 
> > That said, I does indeed look as if U-Boot only supports 2 GiB of
> > memory
> > on 32-bit Tegra. arch/arm/mach-tegra/board.c contains the relevant
> > code.
> > If you look at dram_init() it uses query_sdram_size() to get the
> > size of
> > RAM. That will allow to convey up to 4 GiB of RAM, but then the
> > code in
> > board2.c (see dram_init_banksize()) only uses a single bank on 32-
> > bit
> > Tegra (CONFIG_PHYS_64BIT is only set for 64-bit Tegra).
> > 
> > According to the comments for dram_init_bank_size() we can't store
> > the
> > second bank because the .start field is 32 bits wide and therefore
> > can't
> > store the 0x100000000 start address for the bank.
> > 
> > It sounds to me like ARMV7_LPAE should perhaps select PHYS_64BIT to
> > make
> > sure all the appropriate variables can store addresses beyond the
> > 32-bit
> > boundary. It's one thing to not support > 2 GiB of memory in U-
> > Boot, but
> > it's an entirely different story if we can't even pass > GiB of
> > memory
> > to the kernel if they exist.
> > 
> > Stephen, do you remember if there was anything preventing us from
> > going
> > 64-bit for phys_addr_t and phys_size_t on LPAE enabled systems?
> 
> We probably just never tried since we never needed it and/or LPAE
> wasn't 
> available (in U-Boot SW) when it initially might have been useful. I 
> would not be surprised if some of the low-level Tegra code in U-Boot 
> conflates 64-bit support with SoC being >= T210, so things might not
> be 
> as simple as enabling CONFIG_PHYS_64BIT, but you never know...

Interesting discussion but I have to admit that I also only tested LPAE
once on the 4GB TK1 Chromebook running stock boot loader.

Unfortunately, the 4GB Apalis TK1 SKU got cancelled down the road so I
never got to actually really needing any of this in U-Boot. Sorry, not
being of much help here...

Cheers

Marcel

[U-Boot] [REGRESSION] tegra124: nyan-big: LPAE not working

[Peter Geis]

On 2/11/2019 5:13 PM, Stephen Warren wrote:
> On 2/11/19 3:48 AM, Thierry Reding wrote:
>> On Mon, Feb 11, 2019 at 10:04:37AM +0000, Tristan Bastian wrote:
>>>
>>>
>>>
>>> Thierry Reding – Mon, 11. February 2019 10:38
>>>> On Mon, Feb 11, 2019 at 09:20:33AM +0000, Tristan Bastian wrote:
>>>>>
>>>>> Thierry Reding – Mon, 11. February 2019 9:52
>>>>>> On Sun, Feb 10, 2019 at 08:53:11PM +0100, Tristan Bastian wrote:
>>>>>>> Thierry, do you have any news on this?
>>>>>>>
>>>>>>> I don't think, that google is going to push an updated version of
>>>> coreboot
>>>>>>> to each nyan-big device..
>>>>>>> So reverting this patch at least for the nyan devices would be 
>>>>>>> the best
>>>> I
>>>>>>> think..
>>>>>>
>>>>>> Yes, I agree. I had a brief chat with Rob Herring about this and 
>>>>>> he too
>>>>>> agrees that we should revert it for now. I'll make it a manual revert
>>>>>> and add a comment to the device tree node that will hopefully 
>>>>>> avoid any
>>>>>> future janitorial "cleanups" of this sort.
>>>>>
>>>>> great news! :)
>>>>>
>>>>>>> BTW: I'm now running u-boot natively and it seems like u-boot 
>>>>>>> always has
>>>> a
>>>>>>> problem with the memory..
>>>>>>> If u-boot is used chainloaded to coreboot it is only getting 2GB of
>>>> memory
>>>>>>> and running u-boot natively also just gives me 2GB..
>>>>>>> I've tested that with a kernel with "ARM: tegra: Fix 
>>>>>>> unit_address_vs_reg
>>>> DTC
>>>>>>> warnings for /memory" reverted and also on a kernel, before this 
>>>>>>> patch
>>>> was
>>>>>>> applied..
>>>>>>
>>>>>> It's possible that U-Boot doesn't support LPAE and therefore may 
>>>>>> not be
>>>>>> able to use more than the 2 GiB of memory.
>>>>>
>>>>> So I at least enabled LPAE in u-boot with "CONFIG_ARMV7_LPAE=y" and
>>>>> this was for some reason also needed to get some output on the
>>>>> display..
>>>>> I'm not sure why LPAE needs to activated in u-boot for display output
>>>>> on the nyan-big..
>>>>> Without LPAE enabled u-boot was still working, and booted linux, but
>>>>> u-boot didn't display anything on screen, linux then did..
>>>>
>>>> Yeah, that's surprising. Perhaps without LPAE U-Boot thinks there's not
>>>> enough memory for the framebuffer? There should be plenty, so maybe
>>>> there is something else going on here.
>>>>
>>>>>> However, U-Boot should be
>>>>>> able to tell the kernel exactly how much memory the system has and 
>>>>>> pass
>>>>>> that on via device tree. That still does work, right?
>>>>>
>>>>> It seems like this is not working..
>>>>> And this seems to be the case with both, u-boot chainloaded and
>>>>> running u-boot natively..
>>>>>
>>>>> I've used these scripts for flashing:
>>>>> github.com/NVIDIA/tegra-uboot-flasher-scripts
>>>>> And used the norrin device since it seems like it is the nyan-big dev
>>>>> board?
>>>>> But going with the norrin device config should not be the issue here
>>>>> since the problem also exists when chainloading u-boot, right?
>>>>
>>>> It could be a problem. The memory bank configuration is stored in 
>>>> what's
>>>> called a BCT along with a bunch of other parameters that define what 
>>>> the
>>>> memory controller needs to access the given memory chips. So if you've
>>>> flashed a BCT that's for a board with only 2 GiB of memory you would 
>>>> end
>>>> up with a system that can't address any more than that. It's somewhat
>>>> surprising that memory accesses work at all with a BCT that's for
>>>> different memory chips, but sometimes you can get lucky.
>>>>
>>>> You may want to try reflashing with the right BCT. The simplest would
>>>> probably be to duplicate the cbootimage configuration for Norrin and
>>>> substitute the relevant bits by what you have from the Chromebook flash
>>>> utilities. Looks like the BCT in really the only thing you can replace
>>>> there. Make sure to replace it with the one that matches your 
>>>> Chromebook
>>>> and it should give you the right memory bank configuration.
>>>
>>> Is there a way to check if the correct bct for my 4GB model was used?
>>> What I tried was to edit the existing bct config file of norrin from 
>>> here:
>>> https://github.com/NVIDIA/cbootimage-configs/blob/master/tegra124/nvidia/norrin/PM370_Hynix_2GB_H5TC4G63AFR_PBA_924MHz_01212014.bct.cfg 
>>>
>>> by replacing all entries for each memory bank with the entries from 
>>> here:
>>> https://github.com/coreboot/coreboot/blob/master/src/mainboard/google/nyan_big/bct/sdram-hynix-4GB-792.inc 
>>>
>>>
>>> I also added the SDRAM[0], SDRAM[1], SDRAM[2], SDRAM[3] each time in 
>>> front of each line..
>>> This also booted for me, but also just leaves me with only 2Gb of my 
>>> 4GB..
>>>
>>> And since this problem also occurs, when chainloading u-boot, this 
>>> must be a problem with u-boot, I think..
>>
>> Not necessarily. U-Boot reads the memory size from EMC registers and
>> those EMC registers are programmed based on the contents of the BCT. So
>> if the BCT is wrong, then it's going to be wrong irrespective of what
>> bootloader you use.
>>
>> That said, I does indeed look as if U-Boot only supports 2 GiB of memory
>> on 32-bit Tegra. arch/arm/mach-tegra/board.c contains the relevant code.
>> If you look at dram_init() it uses query_sdram_size() to get the size of
>> RAM. That will allow to convey up to 4 GiB of RAM, but then the code in
>> board2.c (see dram_init_banksize()) only uses a single bank on 32-bit
>> Tegra (CONFIG_PHYS_64BIT is only set for 64-bit Tegra).
>>
>> According to the comments for dram_init_bank_size() we can't store the
>> second bank because the .start field is 32 bits wide and therefore can't
>> store the 0x100000000 start address for the bank.
>>
>> It sounds to me like ARMV7_LPAE should perhaps select PHYS_64BIT to make
>> sure all the appropriate variables can store addresses beyond the 32-bit
>> boundary. It's one thing to not support > 2 GiB of memory in U-Boot, but
>> it's an entirely different story if we can't even pass > GiB of memory
>> to the kernel if they exist.
>>
>> Stephen, do you remember if there was anything preventing us from going
>> 64-bit for phys_addr_t and phys_size_t on LPAE enabled systems?
> 
> We probably just never tried since we never needed it and/or LPAE wasn't 
> available (in U-Boot SW) when it initially might have been useful. I 
> would not be surprised if some of the low-level Tegra code in U-Boot 
> conflates 64-bit support with SoC being >= T210, so things might not be 
> as simple as enabling CONFIG_PHYS_64BIT, but you never know...

Good Evening,

I have finally built mainline u-boot for the rk3328-roc-cc board, and 
have actually run into this issue as well.
With the downstream 2017 u-boot, the board correctly reports 4GB to the 
kernel.
With mainline, no matter what I try, the kernel only ever sees 2GB.
I don't see any significant differences between the spin-up code from 
the downstream u-boot to the mainline u-boot.

I am starting to wonder if this is a u-boot regression, but since most 
embedded SBCs don't use 4GB of ram it does not crop up often.
Of course I may just be missing something, since this is my first foray 
into u-boot.