[RFC v2 0/2] Early use of boot service memory

[RFC v2 0/2] Early use of boot service memory

[Jerry Hoemann]

Some platform have firmware that violates the UEFI spec and access boot
service code or data segments after the system has called ExitBootServices().
The call to efi_reserve_boot_services in setup_arch is a workaround to
avoid using boot service memory until after the kernel has done
SetVirtualAddressMap.  However, this reservation fragments memory
which can cause large allocations early in boot (e.g. crash kernel)
to fail.

This patch set creates a quirk list that governs whether
efi_reserve_boot_services is called.

A new kernel parameter is added that overrides the quirk list
and allows engineers developing new platforms to test whether
they need to have the quirk applied or not.

With this information, setup_arch avoids calling efi_reserve_boot_services
when it doesn't need to, thus avoiding fragmenting memory.


ChangeLog

V1 => v2)

In version 1 the control of the workaround was only by command line argument.
In version 2, the quirk list is the primary control of the whether
the workaround is applied.  This allows for seamless integration with
tools and doesn't require any special knowledge from the user.

The command line argument is retained for platform developers to
be able to test whether their platforms require the workaround.

Jerry Hoemann (2):
  efi: Early use of boot service memory
  x86, efi: Early use of boot service memory

 Documentation/kernel-parameters.txt |  5 +++
 arch/x86/kernel/setup.c             |  5 ++-
 arch/x86/platform/efi/efi.c         | 77 ++++++++++++++++++++++++++++++++++++-
 include/linux/efi.h                 |  2 +
 4 files changed, 86 insertions(+), 3 deletions(-)

-- 
1.7.11.3

[RFC v2 1/2] efi: Early use of boot service memory

[Jerry Hoemann]

Add #define to allow for specifying that firmware doesn't reuse
boot service code or data after Exit Boot Service.

Signed-off-by: Jerry Hoemann <jerry.hoemann@hp.com>
---
 include/linux/efi.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/efi.h b/include/linux/efi.h
index 5f8f176..af085cf 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -568,6 +568,7 @@ efi_guid_unparse(efi_guid_t *guid, char *out)
 }
 
 extern void efi_init (void);
+extern void efi_quirks(void);
 extern void *efi_get_pal_addr (void);
 extern void efi_map_pal_code (void);
 extern void efi_memmap_walk (efi_freemem_callback_t callback, void *arg);
@@ -634,6 +635,7 @@ extern int __init efi_setup_pcdp_console(char *);
 #define EFI_RUNTIME_SERVICES	3	/* Can we use runtime services? */
 #define EFI_MEMMAP		4	/* Can we use EFI memory map? */
 #define EFI_64BIT		5	/* Is the firmware 64-bit? */
+#define EFI_QUIRK_RESERVE_BSCD	6	/* Reserve Boot Service Data/Code */
 
 #ifdef CONFIG_EFI
 # ifdef CONFIG_X86
-- 
1.7.11.3

[RFC v2 2/2] x86, efi: Early use of boot service memory

[Jerry Hoemann]

Some platform have firmware that violate the UEFI spec and access boot service
code or data segments after the system has called ExitBootServices().
The call to efi_reserve_boot_services is a workaround to avoid using
boot service memory until after the kernel has done SetVirtualAddressMap().
However, this reservation fragments memory which can cause
large allocations early in boot (e.g. crash kernel) to fail.

When reserve_crashkernel fails, kdump is disabled.

This patch creates a quirk list that governs when the workaround,
efi_reserve_boot_services, is called.

For all firmware released prior to 2014, the workaround will be
called unless an entry for the platform is in the quirk list saying
not to do the workaround.

For all firmware released 2014 and later,  the workaround will not
be called unless an entry for the platform is in the quirk list
saying to call the workaround.

A new kernel parameter is implemented which overrides the default
handling described above and provides an explicit method to control
whether or not the workaround is applied.   This argument allows
developers to determine on new systems whether the quirk needs
to be applied or not.

Signed-off-by: Jerry Hoemann <jerry.hoemann@hp.com>
---
 Documentation/kernel-parameters.txt |  5 +++
 arch/x86/kernel/setup.c             |  5 ++-
 arch/x86/platform/efi/efi.c         | 77 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 84 insertions(+), 3 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index fcbb736..9b2be92 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -894,6 +894,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			you are really sure that your UEFI does sane gc and
 			fulfills the spec otherwise your board may brick.
 
+	efi_quirk=	[EFI; X86] Manual override for EFI quirk testing.
+			Format: { "reserve" | "noreserve" }
+			"reserve": Force BootService Code/Data to be reserved.
+			"noreserve": Force BootService Code/Data to not be reserved.
+
 	eisa_irq_edge=	[PARISC,HW]
 			See header of drivers/parisc/eisa.c.
 
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index f0de629..74389f8 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -995,6 +995,9 @@ void __init setup_arch(char **cmdline_p)
 	dmi_scan_machine();
 	dmi_set_dump_stack_arch_desc();
 
+	if (efi_enabled(EFI_BOOT))
+		efi_quirks();
+
 	/*
 	 * VMware detection requires dmi to be available, so this
 	 * needs to be done after dmi_scan_machine, for the BP.
@@ -1074,7 +1077,7 @@ void __init setup_arch(char **cmdline_p)
 	 * The EFI specification says that boot service code won't be called
 	 * after ExitBootServices(). This is, in fact, a lie.
 	 */
-	if (efi_enabled(EFI_MEMMAP))
+	if (efi_enabled(EFI_MEMMAP) && efi_enabled(EFI_QUIRK_RESERVE_BSCD))
 		efi_reserve_boot_services();
 
 	/* preallocate 4k for mptable mpc */
diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
index c7e22ab..0e2c46e 100644
--- a/arch/x86/platform/efi/efi.c
+++ b/arch/x86/platform/efi/efi.c
@@ -42,6 +42,7 @@
 #include <linux/io.h>
 #include <linux/reboot.h>
 #include <linux/bcd.h>
+#include <linux/dmi.h>
 
 #include <asm/setup.h>
 #include <asm/efi.h>
@@ -107,6 +108,22 @@ static int __init setup_add_efi_memmap(char *arg)
 }
 early_param("add_efi_memmap", setup_add_efi_memmap);
 
+
+int efi_reserve_quirk = 0;
+EXPORT_SYMBOL(efi_reserve_quirk);
+
+static int __init setup_efi_quirk(char *arg)
+{
+	if (arg && strstr(arg, "noreserve"))
+		efi_reserve_quirk = -1;
+	else if (arg && strstr(arg, "reserve"))
+		efi_reserve_quirk = 1;
+
+	return 0;
+}
+early_param("efi_quirk", setup_efi_quirk);
+
+
 static bool efi_no_storage_paranoia;
 
 static int __init setup_storage_paranoia(char *arg)
@@ -422,6 +439,8 @@ static void __init print_efi_memmap(void)
 }
 #endif  /*  EFI_DEBUG  */
 
+static bool efi_boot_services_reservations;
+
 void __init efi_reserve_boot_services(void)
 {
 	void *p;
@@ -449,8 +468,10 @@ void __init efi_reserve_boot_services(void)
 			memblock_dbg("Could not reserve boot range "
 					"[0x%010llx-0x%010llx]\n",
 						start, start+size-1);
-		} else
+		} else {
 			memblock_reserve(start, size);
+			efi_boot_services_reservations = true;
+		}
 	}
 }
 
@@ -467,7 +488,7 @@ void __init efi_free_boot_services(void)
 {
 	void *p;
 
-	if (!efi_is_native())
+	if (!efi_is_native() || !efi_boot_services_reservations)
 		return;
 
 	for (p = memmap.map; p < memmap.map_end; p += memmap.desc_size) {
@@ -484,6 +505,7 @@ void __init efi_free_boot_services(void)
 			continue;
 
 		free_bootmem_late(start, size);
+		efi_boot_services_reservations = false;
 	}
 
 	efi_unmap_memmap();
@@ -704,6 +726,32 @@ static int __init efi_memmap_init(void)
 	return 0;
 }
 
+static int __init clr_reserve_bootservice_cd(const struct dmi_system_id *id)
+{
+	clear_bit(EFI_QUIRK_RESERVE_BSCD, &x86_efi_facility);
+	return 0;
+}
+
+static int __init set_reserve_bootservice_cd(const struct dmi_system_id *id)
+{
+	set_bit(EFI_QUIRK_RESERVE_BSCD, &x86_efi_facility);
+	return 0;
+}
+
+
+static const struct dmi_system_id efi_fw_quirks[] __initconst = {
+	{
+		.callback = clr_reserve_bootservice_cd,
+		.ident = "HP CB920s x1",
+		.matches = {
+			DMI_MATCH(DMI_SYS_VENDOR, "HP"),
+			DMI_MATCH(DMI_PRODUCT_NAME, "TBD HPServer x1"),
+		},
+	},
+	{}
+};
+
+
 void __init efi_init(void)
 {
 	efi_char16_t *c16;
@@ -780,6 +828,31 @@ void __init efi_init(void)
 #endif
 }
 
+void __init efi_quirks(void)
+{
+	int year;
+
+	if (dmi_get_date(DMI_BIOS_DATE, &year, NULL, NULL) && year < 2014)
+		set_bit(EFI_QUIRK_RESERVE_BSCD, &x86_efi_facility);
+
+	dmi_check_system(efi_fw_quirks);
+
+	/*
+	 * If user specifies "efi_quirk=reserve" or "efi_quirk=noreserve"
+	 * explicitly, that takes precedence over anything we figured
+	 * out above.
+	 */
+	switch (efi_reserve_quirk) {
+	case -1:
+		clear_bit(EFI_QUIRK_RESERVE_BSCD, &x86_efi_facility);
+		break;
+	case  1:
+		set_bit(EFI_QUIRK_RESERVE_BSCD, &x86_efi_facility);
+		break;
+	}
+}
+
+
 void __init efi_late_init(void)
 {
 	efi_bgrt_init();
-- 
1.7.11.3

Re: [RFC v2 2/2] x86, efi: Early use of boot service memory

[Borislav Petkov]

On Thu, Nov 21, 2013 at 02:01:26PM -0700, Jerry Hoemann wrote:
> Some platform have firmware that violate the UEFI spec and access boot service
> code or data segments after the system has called ExitBootServices().
> The call to efi_reserve_boot_services is a workaround to avoid using
> boot service memory until after the kernel has done SetVirtualAddressMap().
> However, this reservation fragments memory which can cause
> large allocations early in boot (e.g. crash kernel) to fail.
> 
> When reserve_crashkernel fails, kdump is disabled.
> 
> This patch creates a quirk list that governs when the workaround,
> efi_reserve_boot_services, is called.
> 
> For all firmware released prior to 2014, the workaround will be
> called unless an entry for the platform is in the quirk list saying
> not to do the workaround.
> 
> For all firmware released 2014 and later,  the workaround will not
> be called unless an entry for the platform is in the quirk list
> saying to call the workaround.

This is yet another quirk list which can grow uncontrolled considering
the notoriety of firmware bugs. And since detecting such spec violation
is very simple - boot Linux on the machine - we should rather disable
this by default for FW >= 2014 and make this test part of the firmware
test suite so that vendors can get a chance to fix their BIOSen.

Provided vendors do boot fwts on their validation platforms, that is.

Yo Fleming, got a better idea? :)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [RFC v2 0/2] Early use of boot service memory

[Matthew Garrett]

On Thu, Nov 21, 2013 at 02:01:24PM -0700, Jerry Hoemann wrote:
> 
> Some platform have firmware that violates the UEFI spec and access boot
> service code or data segments after the system has called ExitBootServices().
> The call to efi_reserve_boot_services in setup_arch is a workaround to
> avoid using boot service memory until after the kernel has done
> SetVirtualAddressMap.  However, this reservation fragments memory
> which can cause large allocations early in boot (e.g. crash kernel)
> to fail.

This is a problem we have to solve, but I don't think this is the right 
way to solve it. Why do we not just reattempt to perform the allocation 
immediately after we've freed the boot services regions?

-- 
Matthew Garrett | mjg59-1xO5oi07KQx4cg9Nei1l7Q@public.gmane.org

Re: [RFC v2 0/2] Early use of boot service memory

[H. Peter Anvin]

On 11/21/2013 03:07 PM, Matthew Garrett wrote:
> On Thu, Nov 21, 2013 at 02:01:24PM -0700, Jerry Hoemann wrote:
>>
>> Some platform have firmware that violates the UEFI spec and access boot
>> service code or data segments after the system has called ExitBootServices().
>> The call to efi_reserve_boot_services in setup_arch is a workaround to
>> avoid using boot service memory until after the kernel has done
>> SetVirtualAddressMap.  However, this reservation fragments memory
>> which can cause large allocations early in boot (e.g. crash kernel)
>> to fail.
> 
> This is a problem we have to solve, but I don't think this is the right 
> way to solve it. Why do we not just reattempt to perform the allocation 
> immediately after we've freed the boot services regions?
> 

Wouldn't the memory map already have gotten scrambled all to hell by then?

	-hpa

Re: [RFC v2 0/2] Early use of boot service memory

[jerry.hoemann-VXdhtT5mjnY]

On Thu, Nov 21, 2013 at 11:07:44PM +0000, Matthew Garrett wrote:
> On Thu, Nov 21, 2013 at 02:01:24PM -0700, Jerry Hoemann wrote:
> > 
> > Some platform have firmware that violates the UEFI spec and access boot
> > service code or data segments after the system has called ExitBootServices().
> > The call to efi_reserve_boot_services in setup_arch is a workaround to
> > avoid using boot service memory until after the kernel has done
> > SetVirtualAddressMap.  However, this reservation fragments memory
> > which can cause large allocations early in boot (e.g. crash kernel)
> > to fail.
> 
> This is a problem we have to solve, but I don't think this is the right 
> way to solve it. Why do we not just reattempt to perform the allocation 
> immediately after we've freed the boot services regions?


Hi Matthew,

I tried this back on 3.11 kernel and reserve_crashkernel after free of
boot services failed.  I will admit to not digging in too deeply as to
why it failed.

My earlier email:

https://lkml.org/lkml/2013/9/18/457


thanks


> 
> -- 
> Matthew Garrett | mjg59-1xO5oi07KQx4cg9Nei1l7Q@public.gmane.org

-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann-VXdhtT5mjnY@public.gmane.org
----------------------------------------------------------------------------

Re: [RFC v2 0/2] Early use of boot service memory

[Matthew Garrett]

On Thu, Nov 21, 2013 at 03:18:41PM -0800, H. Peter Anvin wrote:
> On 11/21/2013 03:07 PM, Matthew Garrett wrote:
> > This is a problem we have to solve, but I don't think this is the right 
> > way to solve it. Why do we not just reattempt to perform the allocation 
> > immediately after we've freed the boot services regions?
> > 
> 
> Wouldn't the memory map already have gotten scrambled all to hell by then?

If we couldn't map a 64MB region in low memory earlier then it's likely 
to have been because there was a 64MB or greater boot services region.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [RFC v2 0/2] Early use of boot service memory

[Matthew Garrett]

On Thu, Nov 21, 2013 at 04:31:04PM -0700, jerry.hoemann@hp.com wrote:

> I tried this back on 3.11 kernel and reserve_crashkernel after free of
> boot services failed.  I will admit to not digging in too deeply as to
> why it failed.

Hm. If the problem is fragmentation, then yeah, I can imagine this 
causing problems. In that case we could take a two-pass approach - find 
a gap that *will* be big enough, reserve everything that isn't currently 
reserved, and then reserve the rest after ExitBootServices()?

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [RFC v2 0/2] Early use of boot service memory

[jerry.hoemann-VXdhtT5mjnY]

On Thu, Nov 21, 2013 at 11:38:31PM +0000, Matthew Garrett wrote:
> On Thu, Nov 21, 2013 at 04:31:04PM -0700, jerry.hoemann-VXdhtT5mjnY@public.gmane.org wrote:
> 
> > I tried this back on 3.11 kernel and reserve_crashkernel after free of
> > boot services failed.  I will admit to not digging in too deeply as to
> > why it failed.
> 
> Hm. If the problem is fragmentation, then yeah, I can imagine this 
> causing problems. In that case we could take a two-pass approach - find 
> a gap that *will* be big enough, reserve everything that isn't currently 
> reserved, and then reserve the rest after ExitBootServices()?


Matthew,

Did you really mean EnterVirtualMode (not ExitBootServices?)

One of the difficulties i'm having is that distros and stable releases
want fixes upstream that they just back port.

ExitBootServices is called in different locations dependent upon
release.  The current use is to have linux do ExitBootServices, whereas
EBS used to be done in the boot loaders.  So, i'm concerned that 
such an approach might require legacy boot loader changes.  That would
be a difficult sell.

But,  splitting up what is done where may work.

While the crash kernel space is reserved early in boot,  i don't believe
it is written to until much later (kexec_load??)  So, i don't believe that
we'd loose any boot code/data that might be missed by firmware that tried
to access after ExitBootService.

In one of your earlier emails you mentioned the issue is that linux makes
regions NX.  That would cause problems if FW tried to execute a region
we just reserved.  Unfortunately, i'm not seeing where the kernel is
doing this for crash kernel memory.  Assuming it is making it NX, can
we defer that part?  Or if its not, do we have a problem w/ crash kernel
reservation at all?

Interesting questions, but as I don't have access to a system that has
the firmware defects encountered when efi_reserve_boot_services, it makes
it difficult to test that i don't break them.  Hence, the appealing nature
of quirks.  Don't have to worry about breaking other platforms as they
continue to operate same as before.

I can understand the desire some have to not introducing quirks.
They can be like potato chips.  You can't have just one.  :)  :)



-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann-VXdhtT5mjnY@public.gmane.org
----------------------------------------------------------------------------

Re: [RFC v2 0/2] Early use of boot service memory

[H. Peter Anvin]

On 11/21/2013 03:37 PM, Matthew Garrett wrote:
> On Thu, Nov 21, 2013 at 03:18:41PM -0800, H. Peter Anvin wrote:
>> On 11/21/2013 03:07 PM, Matthew Garrett wrote:
>>> This is a problem we have to solve, but I don't think this is the right 
>>> way to solve it. Why do we not just reattempt to perform the allocation 
>>> immediately after we've freed the boot services regions?
>>>
>>
>> Wouldn't the memory map already have gotten scrambled all to hell by then?
> 
> If we couldn't map a 64MB region in low memory earlier then it's likely 
> to have been because there was a 64MB or greater boot services region.
> 

I thought the problem was that they wanted to map a fairly large chunk
for faster kdump and fragmentation was being a problem.

	-hpa

Re: [RFC v2 0/2] Early use of boot service memory

[Matthew Garrett]

On Thu, Nov 21, 2013 at 06:05:15PM -0700, jerry.hoemann@hp.com wrote:
> On Thu, Nov 21, 2013 at 11:38:31PM +0000, Matthew Garrett wrote:
> > Hm. If the problem is fragmentation, then yeah, I can imagine this 
> > causing problems. In that case we could take a two-pass approach - find 
> > a gap that *will* be big enough, reserve everything that isn't currently 
> > reserved, and then reserve the rest after ExitBootServices()?
> 
> 
> Matthew,
> 
> Did you really mean EnterVirtualMode (not ExitBootServices?)

I think I actually meant SetVirtualAddressMap() :)

> In one of your earlier emails you mentioned the issue is that linux makes
> regions NX.  That would cause problems if FW tried to execute a region
> we just reserved.  Unfortunately, i'm not seeing where the kernel is
> doing this for crash kernel memory.  Assuming it is making it NX, can
> we defer that part?  Or if its not, do we have a problem w/ crash kernel
> reservation at all?

I don't think we explicitly do that at any point for crash kernel 
regions. The boot services regions get toggled by efi_set_executable().

> Interesting questions, but as I don't have access to a system that has
> the firmware defects encountered when efi_reserve_boot_services, it makes
> it difficult to test that i don't break them.  Hence, the appealing nature
> of quirks.  Don't have to worry about breaking other platforms as they
> continue to operate same as before.

Yeah. The problem is that some users may want kdump while still having 
broken firmware, so a solution that works for them is much more 
appealing than one which involves manually maintaining a list of 
verified systems...

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [RFC v2 0/2] Early use of boot service memory

[jerry.hoemann]

On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
> On 11/21/2013 03:37 PM, Matthew Garrett wrote:
> > On Thu, Nov 21, 2013 at 03:18:41PM -0800, H. Peter Anvin wrote:
> >> On 11/21/2013 03:07 PM, Matthew Garrett wrote:
> >>> This is a problem we have to solve, but I don't think this is the right 
> >>> way to solve it. Why do we not just reattempt to perform the allocation 
> >>> immediately after we've freed the boot services regions?
> >>>
> >>
> >> Wouldn't the memory map already have gotten scrambled all to hell by then?
> > 
> > If we couldn't map a 64MB region in low memory earlier then it's likely 
> > to have been because there was a 64MB or greater boot services region.
> > 
> 
> I thought the problem was that they wanted to map a fairly large chunk
> for faster kdump and fragmentation was being a problem.
> 
> 	-hpa


larger crash dump reservation is not so much performance as functionality.

Large systems w/ lots of IO require large crash kernel allocations for
the kernel to boot.  Then you have to worry about the OOM killer..... 

makedumpfile going to cyclic buffer has helped out greatly, but on
our new systems we're still looking at 512 MB crash kernels.



-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann@hp.com
----------------------------------------------------------------------------

Re: [RFC v2 0/2] Early use of boot service memory

[Yinghai Lu]

On Thu, Nov 21, 2013 at 5:25 PM,  <jerry.hoemann-VXdhtT5mjnY@public.gmane.org> wrote:
> On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
>
> Large systems w/ lots of IO require large crash kernel allocations for
> the kernel to boot.  Then you have to worry about the OOM killer.....

so go with crashkernel=1024M,high.

>
> makedumpfile going to cyclic buffer has helped out greatly, but on
> our new systems we're still looking at 512 MB crash kernels.

I tried 6TiB system/16 PCIE cards, kdump on RHEL 6.5 beta still does not work.
still get OOM.

Thanks

Yinghai

Re: [RFC v2 0/2] Early use of boot service memory

[H. Peter Anvin]

On 11/21/2013 05:29 PM, Yinghai Lu wrote:
> On Thu, Nov 21, 2013 at 5:25 PM,  <jerry.hoemann-VXdhtT5mjnY@public.gmane.org> wrote:
>> On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
>>
>> Large systems w/ lots of IO require large crash kernel allocations for
>> the kernel to boot.  Then you have to worry about the OOM killer.....
> 
> so go with crashkernel=1024M,high.
> 

Yes, there is no bloody excuse to hog that much low memory.

	-hpa

Re: [RFC v2 0/2] Early use of boot service memory

[Vivek Goyal]

[..]
> > makedumpfile going to cyclic buffer has helped out greatly, but on
> > our new systems we're still looking at 512 MB crash kernels.
> 
> I tried 6TiB system/16 PCIE cards, kdump on RHEL 6.5 beta still does not work.
> still get OOM.

What crashkernel= option you are using?

Interesting. So something is consuming lot of memory. How about setting
"debug_mem_level 1" in /etc/kdump.conf and regenerate initrd and retry.
This time it should output some memory usage info at various points 
during boot and that can give us some idea who is consuming how much
memory.

If some module are consuming lot of memory, then you can try "blacklist"
option in /etc/kdump.conf to disable those.

If it is not modules, then it will concern me because then either
kernel is consuming too much memory (which it should not) or for
some reason makedumpfile cyclic mode did not work for you properly.

While you are re-testing, how about also increasing debug message
level of makedumpfile. makedumpfile developers should be able to 
have a look at that. In /etc/kdump.conf, specify.

core_collector makedumpfile -c --message-level 31 -d 31

If message level 31 turns out to be too verbose, reduce it as per
makedumpfile man page.

Thanks
Vivek

Re: [RFC v2 0/2] Early use of boot service memory

[Vivek Goyal]

On Thu, Nov 21, 2013 at 05:29:01PM -0800, Yinghai Lu wrote:
> On Thu, Nov 21, 2013 at 5:25 PM,  <jerry.hoemann-VXdhtT5mjnY@public.gmane.org> wrote:
> > On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
> >
> > Large systems w/ lots of IO require large crash kernel allocations for
> > the kernel to boot.  Then you have to worry about the OOM killer.....
> 
> so go with crashkernel=1024M,high.

He trying to solve problem with older distributions on these machines and
there we do not have crashkernel=X,high option nor do we have capability
to load and boot bzImage above 896MB.

Thanks
Vivek

Re: [RFC v2 0/2] Early use of boot service memory

[Vivek Goyal]

On Thu, Nov 21, 2013 at 05:31:54PM -0800, H. Peter Anvin wrote:
> On 11/21/2013 05:29 PM, Yinghai Lu wrote:
> > On Thu, Nov 21, 2013 at 5:25 PM,  <jerry.hoemann@hp.com> wrote:
> >> On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
> >>
> >> Large systems w/ lots of IO require large crash kernel allocations for
> >> the kernel to boot.  Then you have to worry about the OOM killer.....
> > 
> > so go with crashkernel=1024M,high.
> > 
> 
> Yes, there is no bloody excuse to hog that much low memory.

Curious, why is low memory precious on x86_64? Who is going to use it?

Thanks
Vivek

Re: [RFC v2 0/2] Early use of boot service memory

[H. Peter Anvin]

Various DMA-deficient devices, at least without an iommu.

Vivek Goyal <vgoyal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>On Thu, Nov 21, 2013 at 05:31:54PM -0800, H. Peter Anvin wrote:
>> On 11/21/2013 05:29 PM, Yinghai Lu wrote:
>> > On Thu, Nov 21, 2013 at 5:25 PM,  <jerry.hoemann-VXdhtT5mjnY@public.gmane.org> wrote:
>> >> On Thu, Nov 21, 2013 at 05:12:57PM -0800, H. Peter Anvin wrote:
>> >>
>> >> Large systems w/ lots of IO require large crash kernel allocations
>for
>> >> the kernel to boot.  Then you have to worry about the OOM
>killer.....
>> > 
>> > so go with crashkernel=1024M,high.
>> > 
>> 
>> Yes, there is no bloody excuse to hog that much low memory.
>
>Curious, why is low memory precious on x86_64? Who is going to use it?
>
>Thanks
>Vivek

-- 
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

Re: [RFC v2 0/2] Early use of boot service memory

[HATAYAMA Daisuke]

(2013/11/22 11:29), Vivek Goyal wrote:
> [..]
>>> makedumpfile going to cyclic buffer has helped out greatly, but on
>>> our new systems we're still looking at 512 MB crash kernels.
>>
>> I tried 6TiB system/16 PCIE cards, kdump on RHEL 6.5 beta still does not work.
>> still get OOM.
>
> What crashkernel= option you are using?
>
> Interesting. So something is consuming lot of memory. How about setting
> "debug_mem_level 1" in /etc/kdump.conf and regenerate initrd and retry.
> This time it should output some memory usage info at various points
> during boot and that can give us some idea who is consuming how much
> memory.
>
> If some module are consuming lot of memory, then you can try "blacklist"
> option in /etc/kdump.conf to disable those.
>
> If it is not modules, then it will concern me because then either
> kernel is consuming too much memory (which it should not) or for
> some reason makedumpfile cyclic mode did not work for you properly.
>
> While you are re-testing, how about also increasing debug message
> level of makedumpfile. makedumpfile developers should be able to
> have a look at that. In /etc/kdump.conf, specify.
>
> core_collector makedumpfile -c --message-level 31 -d 31
>
> If message level 31 turns out to be too verbose, reduce it as per
> makedumpfile man page.
>

The following configuration is more flexible:

core_collector false
default shell

Then crash dump collection fails and emergency shell shows up,
where you can type a variety of commands.

If 2nd kernel keeps failing even on this configuration, it's likely
that kernel side already causes the OOM you're facing before reaching
invocation of the command specified by core_collector directive.

-- 
Thanks.
HATAYAMA, Daisuke

Re: [RFC v2 0/2] Early use of boot service memory

[jerry.hoemann]

On Fri, Nov 22, 2013 at 01:16:13AM +0000, Matthew Garrett wrote:
> On Thu, Nov 21, 2013 at 06:05:15PM -0700, jerry.hoemann@hp.com wrote:
> > On Thu, Nov 21, 2013 at 11:38:31PM +0000, Matthew Garrett wrote:
> > > Hm. If the problem is fragmentation, then yeah, I can imagine this 
> > > causing problems. In that case we could take a two-pass approach - find 
> > > a gap that *will* be big enough, reserve everything that isn't currently 
> > > reserved, and then reserve the rest after ExitBootServices()?
> > 
> > 
> 
> > Interesting questions, but as I don't have access to a system that has
> > the firmware defects encountered when efi_reserve_boot_services, it makes
> > it difficult to test that i don't break them.  Hence, the appealing nature
> > of quirks.  Don't have to worry about breaking other platforms as they
> > continue to operate same as before.
> 
> Yeah. The problem is that some users may want kdump while still having 
> broken firmware, so a solution that works for them is much more 
> appealing than one which involves manually maintaining a list of 
> verified systems...

Matthew, 

Sorry for the delay in replying.

We've worked with our firmware engineers who have made changes
to move around where certain drivers are allocating from which
has helped reduce some of the fragmentation issue we were having.
Since we are taking firmware drops from outside sources, we are
subject to regressions in this area, so i'm still interested in
this topic.

To your point, kdump may still works for the platforms that
have the broken firmware.  Much depends upon the memory and IO
configuration on the system in question.

>From prior mailing, apparently Macs are subject to the problem
that efi_reserve_boot_services was created to address.  Smaller
systems w/ less IO don't require the larger crash kernel allocations
and hence are more likely to be able to find a gap in low memory
that works.

Its the larger systems with lots of cpus, memory, and IO that
need to allocate the large crash kernels.  These types of systems
are more at risk.

Again,  my main concern is finding something that can be back ported
into legacy kernels/distros.  For distros based upon newer kernel and
associated tools, we can avoid the problem by allocating memory high.
But, we don't have that option with legacy kernels/distros.

So, if you have other suggestions, I'm willing to explore them.


Jerry

-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann@hp.com
----------------------------------------------------------------------------