memory hole close to 4GB on qemu risc-v

memory hole close to 4GB on qemu risc-v

[Wu, Fei]

Hi All,

I am working on enabling PCIe passthrough on qemu risc-v, in order for
the guest to access the host x86 pci resource directly, ram on guest
won't cover this range, so if guest has 4GB ram, two ranges are created:
  1. 2G-3G
  2. 4G-7G

u-boot is not able to handle this but reports:

Reserving 869k for U-Boot at: ff725000
initcall: 0000000080216f94
Reserving 8320k for malloc() at: fef05000
initcall: 0000000080216ee8
Unhandled exception: Store/AMO access fault

EP: 0000000080201a38 RA: 0000000080216f10 TVAL: 00000000fef04f90

It looks like fdtdec_setup_mem_size_base() in u-boot sets ram_base/size
without considering all memory nodes, and board_get_usable_ram_top()
returns 4G-1 that causes gd->relocaddr set to the wrong address which
has no ram backed. More details can be found here:
  https://patchew.org/QEMU/20230731015317.1026996-1-fei2.wu@intel.com/


Can we add support for this multi-range ram setting? Any comments?

Thanks,
Fei.

Re: memory hole close to 4GB on qemu risc-v

[Wu, Fei]

On 8/7/2023 3:50 PM, Wu, Fei wrote:
> Hi All,
> 
> I am working on enabling PCIe passthrough on qemu risc-v, in order for
> the guest to access the host x86 pci resource directly, ram on guest
> won't cover this range, so if guest has 4GB ram, two ranges are created:
>   1. 2G-3G
>   2. 4G-7G
> 
> u-boot is not able to handle this but reports:
> 
> Reserving 869k for U-Boot at: ff725000
> initcall: 0000000080216f94
> Reserving 8320k for malloc() at: fef05000
> initcall: 0000000080216ee8
> Unhandled exception: Store/AMO access fault
> 
> EP: 0000000080201a38 RA: 0000000080216f10 TVAL: 00000000fef04f90
> 
> It looks like fdtdec_setup_mem_size_base() in u-boot sets ram_base/size
> without considering all memory nodes, and board_get_usable_ram_top()
> returns 4G-1 that causes gd->relocaddr set to the wrong address which
> has no ram backed. More details can be found here:
>   https://patchew.org/QEMU/20230731015317.1026996-1-fei2.wu@intel.com/
> 
> 
> Can we add support for this multi-range ram setting? Any comments?
> 
The test command is:
qemu-system-riscv64 -machine virt -nographic -m 4G -smp 4 \
  -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf \
  -kernel /home/wufei/nvme/u-boot/u-boot.bin \
  -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
  -drive
file=ubuntu-22.04.1-preinstalled-server-riscv64+unmatched.img,if=virtio,format=raw

Here is a quick fix, is it good?

diff --git a/arch/riscv/cpu/generic/dram.c b/arch/riscv/cpu/generic/dram.c
index 44e11bd56c..fb53a57b4e 100644
--- a/arch/riscv/cpu/generic/dram.c
+++ b/arch/riscv/cpu/generic/dram.c
@@ -13,7 +13,7 @@ DECLARE_GLOBAL_DATA_PTR;

 int dram_init(void)
 {
-       return fdtdec_setup_mem_size_base();
+       return fdtdec_setup_mem_size_base_lowest();
 }

 int dram_init_banksize(void)
diff --git a/configs/qemu-riscv64_smode_defconfig
b/configs/qemu-riscv64_smode_defconfig
index 1d0f021ade..de08a49dab 100644
--- a/configs/qemu-riscv64_smode_defconfig
+++ b/configs/qemu-riscv64_smode_defconfig
@@ -1,6 +1,6 @@
 CONFIG_RISCV=y
 CONFIG_SYS_MALLOC_LEN=0x800000
-CONFIG_NR_DRAM_BANKS=1
+CONFIG_NR_DRAM_BANKS=2
 CONFIG_HAS_CUSTOM_SYS_INIT_SP_ADDR=y
 CONFIG_CUSTOM_SYS_INIT_SP_ADDR=0x80200000
 CONFIG_ENV_SIZE=0x20000

Thanks,
Fei.