[PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Avnish Chouhan]

Change RMA size from 512 MB to 768 MB which will result 
in more RMA at boot time for PowerPC. When PowerPC LPAR use/uses vTPM,
Secure Boot or FADump, the 512 MB RMA memory is not sufficient for 
booting. With this 512 MB RMA, GRUB2 run out of memory and unable to 
load the necessary. Sometimes even usage of CDROM which requires more
memory for installation along with the options mentioned above troubles
the boot memory and result in boot failures. Increasing the RMA size 
will resolves multiple out of memory issues observed in PowerPC. 

Failure details:

1. GRUB2

kern/ieee1275/init.c:550: mm requested region of size 8513000, flags 1
kern/ieee1275/init.c:563: Cannot satisfy allocation and retain minimum runtime
space
kern/ieee1275/init.c:550: mm requested region of size 8513000, flags 0
kern/ieee1275/init.c:563: Cannot satisfy allocation and retain minimum runtime
space
kern/file.c:215: Closing `/ppc/ppc64/initrd.img' ...
kern/disk.c:297: Closing
`ieee1275//vdevice/v-scsi
@30000067/disk@8300000000000000'...
kern/disk.c:311: Closing
`ieee1275//vdevice/v-scsi
@30000067/disk@8300000000000000' succeeded.
kern/file.c:225: Closing `/ppc/ppc64/initrd.img' failed with 3.
kern/file.c:148: Opening `/ppc/ppc64/initrd.img' succeeded.
error: ../../grub-core/kern/mm.c:552:out of memory.

2. Kernel

[    0.777633] List of all partitions:
[    0.777639] No filesystem could mount root, tried:
[    0.777640]
[    0.777649] Kernel panic - not syncing: VFS: Unable to mount root fs on "" or unknown-block(0,0)
[    0.777658] CPU: 17 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.11.0-0.rc4.20.el10.ppc64le #1
[    0.777669] Hardware name: IBM,9009-22A POWER9 (architected) 0x4e0202 0xf000005 of:IBM,FW950.B0 (VL950_149) hv:phyp pSeries
[    0.777678] Call Trace:
[    0.777682] [c000000003db7b60] [c000000001119714] dump_stack_lvl+0x88/0xc4 (unreliable)
[    0.777700] [c000000003db7b90] [c00000000016c274] panic+0x174/0x460
[    0.777711] [c000000003db7c30] [c00000000200631c] mount_root_generic+0x320/0x354
[    0.777724] [c000000003db7d00] [c0000000020066f8] prepare_namespace+0x27c/0x2f4
[    0.777735] [c000000003db7d90] [c000000002005824] kernel_init_freeable+0x254/0x294
[    0.777747] [c000000003db7df0] [c00000000001131c] kernel_init+0x30/0x1c4
[    0.777757] [c000000003db7e50] [c00000000000debc] ret_from_kernel_user_thread+0x14/0x1c
[    0.777768] --- interrupt: 0 at 0x0
[    0.784238] pstore: backend (nvram) writing error (-1)
[    0.790447] Rebooting in 10 seconds..

Signed-off-by: Avnish Chouhan <avnish@linux.ibm.com>
---
Change logs:

v2:
 - Added GRUB2 debug logs and Kernel traces.

---
 arch/powerpc/kernel/prom_init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c
index fbb68fc28ed3..c42fd5a826c0 100644
--- a/arch/powerpc/kernel/prom_init.c
+++ b/arch/powerpc/kernel/prom_init.c
@@ -1061,7 +1061,7 @@ static const struct ibm_arch_vec ibm_architecture_vec_template __initconst = {
 		.virt_base = cpu_to_be32(0xffffffff),
 		.virt_size = cpu_to_be32(0xffffffff),
 		.load_base = cpu_to_be32(0xffffffff),
-		.min_rma = cpu_to_be32(512),		/* 512MB min RMA */
+		.min_rma = cpu_to_be32(768),		/* 768MB min RMA */
 		.min_load = cpu_to_be32(0xffffffff),	/* full client load */
 		.min_rma_percent = 0,	/* min RMA percentage of total RAM */
 		.max_pft_size = 48,	/* max log_2(hash table size) */
-- 
2.43.5

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Michael Ellerman]

Avnish Chouhan <avnish@linux.ibm.com> writes:
> Change RMA size from 512 MB to 768 MB which will result 
> in more RMA at boot time for PowerPC.

Did you consider just increasing it to 1GB?

It's possible there's some folks running LPARs with less than 1GB, but
they are unlikely to continue doing so by the time this change trickles
into distros. To be supported modern RHEL requires 2GB minimum RAM
anyway.

Can you also describe the behaviour users will see when they configure
an LPAR with less than 768MB of RAM.

cheers

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Avnish Chouhan]

Hi Michael,
Hope you're doing wonderful!

Thank you so much for your response. I have checked on your queries. 
Please find the findings below:

1. Did you consider just increasing it to 1GB?

We have observed in our recent Out Of Memory issues, a shortage of 
around 50-60 MBs space in RMA in current issues. So we decided to 
increase the RMA by 256 MBs. Please give me couple of days, I'm 
analyzing this 1 GB change and update you on it soon.

2. an LPAR with less than 768MB of RAM

I have analyzed the multiple RAM scenarios. The behavior seems similar 
regardless of RMA size 512 or 768 MBs, as the RMA region is used by PFW 
and GRUB2 for booting. Even if GRUB is able to load the kernel for 
booting, the machines isn't booting and behaving well in low amount of 
RAM. We observe kernel panics, mostly "Out of memory: Killed 
process...." when RAM is less than 3 GBs. The different RAM configs (via 
HMC LPAR properties) and behaviors are given below:

i. RAM (3 GBs)
    System boot fine when RAM is minimum 3 GBs (It does depend on system 
config as well).
ii. RMA (512 MBs)
     With RAM as 512 MB, the system fails to boot with firmware error (eg 
B2006006).
iii. RAM (768 MB and 1 GB)
      With RAM as 768 MB and 1 GB, System boot with kernel panic as 
"Kernel panic - not syncing: System is    deadlocked on memory".
iv. RAM (2 GBs)
     System does boot fine, but abnormal behavior after the boot. I 
observed system panic in one scenario while doing a reboot. "Out of 
memory: Killed process 167....."


Regards,
Avnish Chouhan


On 2024-12-07 07:28, Michael Ellerman wrote:
> Avnish Chouhan <avnish@linux.ibm.com> writes:
>> Change RMA size from 512 MB to 768 MB which will result
>> in more RMA at boot time for PowerPC.
> 
> Did you consider just increasing it to 1GB?
> 
> It's possible there's some folks running LPARs with less than 1GB, but
> they are unlikely to continue doing so by the time this change trickles
> into distros. To be supported modern RHEL requires 2GB minimum RAM
> anyway.
> 
> Can you also describe the behaviour users will see when they configure
> an LPAR with less than 768MB of RAM.
> 
> cheers

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Sourabh Jain]

Hello Michael,

On 07/12/24 07:28, Michael Ellerman wrote:
> Avnish Chouhan <avnish@linux.ibm.com> writes:
>> Change RMA size from 512 MB to 768 MB which will result
>> in more RMA at boot time for PowerPC.
> Did you consider just increasing it to 1GB?

I see an impact of setting RMA to 1GB on fadump. Here’s how:

The minimum reservation recommended by us for fadump is 768MB.
In this case, the boot_mem_top in fadump.c will be 768MB, and the
memory reservation for fadump will start from a 768MB offset.

Now, let’s say the production kernel crashes, and the firmware copies
the 0–768MB region to the fadump reserved area and boots the fadump
kernel to capture the dump.

If RMA is set to 1GB, it will allow GRUB to use memory up to 1GB. In such
a case, there is a possibility that GRUB could corrupt the fadump 
reserved area.

Thanks,
Sourabh Jain

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Madhavan Srinivasan]

On Fri, 06 Dec 2024 12:25:45 +0530, Avnish Chouhan wrote:
> Change RMA size from 512 MB to 768 MB which will result
> in more RMA at boot time for PowerPC. When PowerPC LPAR use/uses vTPM,
> Secure Boot or FADump, the 512 MB RMA memory is not sufficient for
> booting. With this 512 MB RMA, GRUB2 run out of memory and unable to
> load the necessary. Sometimes even usage of CDROM which requires more
> memory for installation along with the options mentioned above troubles
> the boot memory and result in boot failures. Increasing the RMA size
> will resolves multiple out of memory issues observed in PowerPC.
> 
> [...]

Applied to powerpc/next.

[1/1] powerpc: increase MIN RMA size for CAS negotiation
      https://git.kernel.org/powerpc/c/ae908b87b6bb32c170e9baf5858f2a7553cacc06

Thanks

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Sourabh Jain]

Hello Maddy,

On 22/01/25 18:13, Madhavan Srinivasan wrote:
> On Fri, 06 Dec 2024 12:25:45 +0530, Avnish Chouhan wrote:
>> Change RMA size from 512 MB to 768 MB which will result
>> in more RMA at boot time for PowerPC. When PowerPC LPAR use/uses vTPM,
>> Secure Boot or FADump, the 512 MB RMA memory is not sufficient for
>> booting. With this 512 MB RMA, GRUB2 run out of memory and unable to
>> load the necessary. Sometimes even usage of CDROM which requires more
>> memory for installation along with the options mentioned above troubles
>> the boot memory and result in boot failures. Increasing the RMA size
>> will resolves multiple out of memory issues observed in PowerPC.
>>
>> [...]
> Applied to powerpc/next.
>
> [1/1] powerpc: increase MIN RMA size for CAS negotiation
>        https://git.kernel.org/powerpc/c/ae908b87b6bb32c170e9baf5858f2a7553cacc06

The above changes break the fadump additional parameter for the HASH MMU.
Here is the proposed fix:
https://lore.kernel.org/all/20250120173501.1147236-1-sourabhjain@linux.ibm.com/

The proposed fix ensures that the memory reserved for the additional 
parameter for
fadump on the HASH MMU does not overlap with GRUB.

Please share your opinion on the fixes.

Thanks,
Sourabh Jain

Re: [PATCH v2] powerpc: increase MIN RMA size for CAS negotiation

[Sourabh Jain]

Hello Maddy,


On 22/01/25 18:13, Madhavan Srinivasan wrote:
> On Fri, 06 Dec 2024 12:25:45 +0530, Avnish Chouhan wrote:
>> Change RMA size from 512 MB to 768 MB which will result
>> in more RMA at boot time for PowerPC. When PowerPC LPAR use/uses vTPM,
>> Secure Boot or FADump, the 512 MB RMA memory is not sufficient for
>> booting. With this 512 MB RMA, GRUB2 run out of memory and unable to
>> load the necessary. Sometimes even usage of CDROM which requires more
>> memory for installation along with the options mentioned above troubles
>> the boot memory and result in boot failures. Increasing the RMA size
>> will resolves multiple out of memory issues observed in PowerPC.
>>
>> [...]
> Applied to powerpc/next.
>
> [1/1] powerpc: increase MIN RMA size for CAS negotiation
>        https://git.kernel.org/powerpc/c/ae908b87b6bb32c170e9baf5858f2a7553cacc06
I posted a patch series that includes this patch along with patches to 
avoid the impact on fadump due to the `MIN_RMA` change introduced in 
this patch. 
https://lore.kernel.org/all/20250123114254.200527-1-sourabhjain@linux.ibm.com/ 
The patches in the above patch series are arranged to ensure they do not 
break `git bisect`. Previously, we tested this patch and the fadump 
changes together but sent them separately, which I now realize was not 
the best approach. If possible, please consider accepting the entire 
patch series as a whole. 
https://lore.kernel.org/all/20250123114254.200527-1-sourabhjain@linux.ibm.com/ 
We have included test cases for the above patch series and a changelog 
explaining additional details in the cover letter. Please let me know if 
you have queries regarding this. Thanks, Sourabh Jain