Re: [PATCH 0/3] Early use of boot service memory

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/14/2013 10:44 AM, Pekka Enberg wrote:
> On Thu, Nov 14, 2013 at 8:04 PM,  <jerry.hoemann@hp.com> wrote:
>> Making this issue a quirk will be a lot more practical.  Its a small, focused
>> change whose implications are limited and more easily understood.
> 
> There's nothing practical with requiring users to pass a kernel option
> to make kdump work.  It's a workaround, sure, but it's not a proper
> fix.

And once you have to do that anyway, you might as well just do the kdump
load high...

	-hpa


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/14/2013 10:55 PM, Yinghai Lu wrote:
> 
> Why just asking distros to append ",high" in their installation
> program for 64bit by default?
> 
[...]
> 
> What is hpa's suggestion?
> 

Pretty much what you just said ;)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Pekka Enberg]

On 11/15/13 2:50 AM, jerry.hoemann@hp.com wrote:
> One already has to specify command line arguments to enable kdump.

Yes, so what?

The problem with your patch is that now to enable kdump, I have to know 
that there's a second command line option and if my firmware is "broken" 
or not.  The former is already a problem (how do I even know such a 
thing exists?) but the latter is almost impossible to solve from user 
point of view. And if I have a "broken" firmware, kdump won't work no 
matter what options I pass.

I really don't see what's practical about that.

                         Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Thu, Nov 14, 2013 at 10:59:05PM -0800, H. Peter Anvin wrote:
> On 11/14/2013 10:55 PM, Yinghai Lu wrote:
> > 
> > Why just asking distros to append ",high" in their installation
> > program for 64bit by default?
> > 
> [...]
> > 
> > What is hpa's suggestion?
> > 
> 
> Pretty much what you just said ;)

I think crashkernel=X,high is not a good default choice for distros. 
Reserving memory high reserves 72MB (or more) low memory for swiotlb. We
work hard to keep crashkernel memory amount low and currently reserve
128M by default. Now suddenly our total memory reservation will shoot
to 200 MB if we choose ,high option. That's jump of more than 50%. It
is not needed.

We can do dumping operation successfully in *less* reserved memory by
reserving memory below 4G. And hence crashkernel=,high is not a good
default.

Instead, crashkernel=X is a good default if we are ready to change
semantics a bit. If sufficient crashkernel memory is not available
in low memory area, look for it above 4G. This incurs penalty of
72M *only* if it has to and not by default on most of the systems.

And this should solve jerry's problem too on *latest* kernels. For
older kernels, we don't have ,high support. So using that is not
an option. (until and unless somebody is ready to backport everything
needed to boot old kernel above 4G).

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Yinghai Lu]

On Fri, Nov 15, 2013 at 6:07 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> On Thu, Nov 14, 2013 at 10:59:05PM -0800, H. Peter Anvin wrote:
>> On 11/14/2013 10:55 PM, Yinghai Lu wrote:
>> >
>> > Why just asking distros to append ",high" in their installation
>> > program for 64bit by default?
>> >
>> [...]
>> >
>> > What is hpa's suggestion?
>> >
>>
>> Pretty much what you just said ;)
>
> I think crashkernel=X,high is not a good default choice for distros.
> Reserving memory high reserves 72MB (or more) low memory for swiotlb. We
> work hard to keep crashkernel memory amount low and currently reserve
> 128M by default. Now suddenly our total memory reservation will shoot
> to 200 MB if we choose ,high option. That's jump of more than 50%. It
> is not needed.

If the system support intel IOMMU, we only need to that 72M for SWIOTLB
or AMD workaround.
If the user really care that for intel iommu enable system, they could use
"crashkernel=0,low" to have that 72M back.

and that 72M is under 4G instead of 896M.

so reserve 72M is not better than reserve 128M?

>
> We can do dumping operation successfully in *less* reserved memory by
> reserving memory below 4G. And hence crashkernel=,high is not a good
> default.
>
> Instead, crashkernel=X is a good default if we are ready to change
> semantics a bit. If sufficient crashkernel memory is not available
> in low memory area, look for it above 4G. This incurs penalty of
> 72M *only* if it has to and not by default on most of the systems.
>
> And this should solve jerry's problem too on *latest* kernels. For
> older kernels, we don't have ,high support. So using that is not
> an option. (until and unless somebody is ready to backport everything
> needed to boot old kernel above 4G).

that problem looks not related.

I have one system with 6TiB memory, kdump does not work even
crashkernel=512M in legacy mode. ( it only work on system with
4.5TiB).
--- first kernel can reserve the 512M under 896M, second kernel will
OOM as it load driver for every pci devices...

So why would RH guys not spend some time on optimizing your kdump initrd
build scripts and only put dump device related driver in it?

Thanks

Yinghai

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/15/2013 09:33 AM, Yinghai Lu wrote:
> 
> If the system support intel IOMMU, we only need to that 72M for SWIOTLB
> or AMD workaround.
> If the user really care that for intel iommu enable system, they could use
> "crashkernel=0,low" to have that 72M back.
> 
> and that 72M is under 4G instead of 896M.
> 
> so reserve 72M is not better than reserve 128M?
> 

Those 72M are in addition to 128M, which does add up quite a bit.
However, the presence of a working IOMMU in the system is something that
should be possible to know at setup time.

Now, this was discussed partly in the context of VMs.  I want to say, as
I have again and again: the right way to dump a VM is with hypervisor
assistance rather than an in-image dumper which is both expensive and
may be corrupted by the failure.

It would be good if the various VMs with interest in Linux would agree
on a mechanism for launching a dumper.  This can be done either inband
(on the execution of a specific hypercall, the hypervisor terminates I/O
to the guest, inserts a dumper into the address space and launches it)
or out-of-band (the hypervisor itself, or an assistant program, writes a
dump file) or as a hybrid (a new dump guest is launched with the
hypervisor-written or hypervisor-preserved crashed guest image somehow
passed to it.)

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Fri, Nov 15, 2013 at 09:33:41AM -0800, Yinghai Lu wrote:

[..]
> > I think crashkernel=X,high is not a good default choice for distros.
> > Reserving memory high reserves 72MB (or more) low memory for swiotlb. We
> > work hard to keep crashkernel memory amount low and currently reserve
> > 128M by default. Now suddenly our total memory reservation will shoot
> > to 200 MB if we choose ,high option. That's jump of more than 50%. It
> > is not needed.
> 
> If the system support intel IOMMU, we only need to that 72M for SWIOTLB
> or AMD workaround.
> If the user really care that for intel iommu enable system, they could use
> "crashkernel=0,low" to have that 72M back.
> 
> and that 72M is under 4G instead of 896M.
> 
> so reserve 72M is not better than reserve 128M?

This 72M is on top of 128M reserved. Also IOMMU support is very flaky
with kdump and in fact on most of the system it might not work. So
majority of systems will pay this cost of 72M.

> 
> >
> > We can do dumping operation successfully in *less* reserved memory by
> > reserving memory below 4G. And hence crashkernel=,high is not a good
> > default.
> >
> > Instead, crashkernel=X is a good default if we are ready to change
> > semantics a bit. If sufficient crashkernel memory is not available
> > in low memory area, look for it above 4G. This incurs penalty of
> > 72M *only* if it has to and not by default on most of the systems.
> >
> > And this should solve jerry's problem too on *latest* kernels. For
> > older kernels, we don't have ,high support. So using that is not
> > an option. (until and unless somebody is ready to backport everything
> > needed to boot old kernel above 4G).
> 
> that problem looks not related.
> 
> I have one system with 6TiB memory, kdump does not work even
> crashkernel=512M in legacy mode. ( it only work on system with
> 4.5TiB).

Recently I tested one system with 6TB of memory and dumped successfully
with 512MB reserved under 896MB. Also I have heard reports of successful
dump of 12TB system with 512MB reserved below 896MB (due to cyclic
mode of makedumpfile).

So with newer releases only reason one might want to reserve more
memory is that it might provide speed benefits. We need more testing
to quantify this.

> --- first kernel can reserve the 512M under 896M, second kernel will
> OOM as it load driver for every pci devices...
> 
> So why would RH guys not spend some time on optimizing your kdump initrd
> build scripts and only put dump device related driver in it?

Try latest Fedora and that's what we do. Now we have moved to dracut
based initramfs generation and we tell dracut that build initramfs for
host and additional dump destination and dracut builds it for those only.
I think there might be scope for further optimization, but I don't think
that's the problem any more. 

So issue remains that crashkernel=X,high is not a good default choice
because it consumes extra 72M which we don't have to.

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[jerry.hoemann]

On Thu, Nov 14, 2013 at 10:59:05PM -0800, H. Peter Anvin wrote:
> On 11/14/2013 10:55 PM, Yinghai Lu wrote:
> > 
> > Why just asking distros to append ",high" in their installation
> > program for 64bit by default?
> > 
> [...]
> > 
> > What is hpa's suggestion?
> > 
> 
> Pretty much what you just said ;)

The issue w/ efi_reserve_boot_services exists across several
versions and distros of linux.  So, I'd like to find a fix that
works across several kernel versions and distros.

the kernel and required utility code to allocate high isn't available
on distros based on pre 3.9 kernels.

While the alloc high code is a step in the right direction, it is
still green.  We are having much more problems getting crash dump
to work w/ top of tree kernels/utilities than we are having w/
distros running legacy bits.

Back porting this much larger change to multiple versions and
multiple distros isn't my first choice as its is much more work, much
more likely to destabilize distros w/ legacy kernels.

We will be passing along fixes for these other top of tree dump
issues as we find them,  but our first priority is enabling
our distro partners that happen to be using pre 3.9 based kernels.



Jerry


-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann@hp.com
----------------------------------------------------------------------------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Fri, Nov 15, 2013 at 09:40:49AM -0800, H. Peter Anvin wrote:
> On 11/15/2013 09:33 AM, Yinghai Lu wrote:
> > 
> > If the system support intel IOMMU, we only need to that 72M for SWIOTLB
> > or AMD workaround.
> > If the user really care that for intel iommu enable system, they could use
> > "crashkernel=0,low" to have that 72M back.
> > 
> > and that 72M is under 4G instead of 896M.
> > 
> > so reserve 72M is not better than reserve 128M?
> > 
> 
> Those 72M are in addition to 128M, which does add up quite a bit.
> However, the presence of a working IOMMU in the system is something that
> should be possible to know at setup time.
> 

And IOMMU support is very flaky with kdump. And IOMMU's can be turned
off at command line. And that would force one to remove crahkernel_low=0.
So change of one command line option forces change of another. It is
complicated.

Also there are very few systems which work with IOMMU on. A lot more
which work without IOMMU. We have all these DMAR issues and still nobody
has been able to address IOMMU issues properly.

> Now, this was discussed partly in the context of VMs.  I want to say, as
> I have again and again: the right way to dump a VM is with hypervisor
> assistance rather than an in-image dumper which is both expensive and
> may be corrupted by the failure.

I agree taking assistance of hypervisor should be useful.

One reason we use kdump for VM too because it makes life simple. There
is no difference in how we configure, start and manage crash dumps
in baremetal or inside VM. And in practice have not heard of lot of
failures of kdump in VM environment.

So while reliability remains a theoritical concern, in practice it
has not been a real concern and that's one reason I think we have
not seen a major push for alternative method in VM environment.

> 
> It would be good if the various VMs with interest in Linux would agree
> on a mechanism for launching a dumper.  This can be done either inband
> (on the execution of a specific hypercall, the hypervisor terminates I/O
> to the guest, inserts a dumper into the address space and launches it)
> or out-of-band (the hypervisor itself, or an assistant program, writes a
> dump file) or as a hybrid (a new dump guest is launched with the
> hypervisor-written or hypervisor-preserved crashed guest image somehow
> passed to it.)

virsh can take dumps of KVM guest, so hypervisor calling out to an
assistant program might help here.

Anyway, we will gladly use any new dump mechanism for VM once things
start working seamlessly. So till all this materializes, forcing user
to reserve that extra 72M concerns me (both in bare-metal and virtualized
environments).

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/15/2013 10:30 AM, Vivek Goyal wrote:
> 
> I agree taking assistance of hypervisor should be useful.
> 
> One reason we use kdump for VM too because it makes life simple. There
> is no difference in how we configure, start and manage crash dumps
> in baremetal or inside VM. And in practice have not heard of lot of
> failures of kdump in VM environment.
> 
> So while reliability remains a theoritical concern, in practice it
> has not been a real concern and that's one reason I think we have
> not seen a major push for alternative method in VM environment.
> 

Another reason, again, is that it doesn't sit on all that memory.

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/15/2013 10:46 AM, H. Peter Anvin wrote:
> On 11/15/2013 10:30 AM, Vivek Goyal wrote:
>>
>> I agree taking assistance of hypervisor should be useful.
>>
>> One reason we use kdump for VM too because it makes life simple. There
>> is no difference in how we configure, start and manage crash dumps
>> in baremetal or inside VM. And in practice have not heard of lot of
>> failures of kdump in VM environment.
>>
>> So while reliability remains a theoritical concern, in practice it
>> has not been a real concern and that's one reason I think we have
>> not seen a major push for alternative method in VM environment.
>>
> 
> Another reason, again, is that it doesn't sit on all that memory.
> 

This led me to a potentially interesting idea.  If we can tell the
hypervisor about which memory blocks belong to kdump, we can still use
kdump in its current form with only a few hypervisor calls thrown in.

One set of calls would mark memory ranges as belonging to kdump.  This
would (a) make them protected, and (b) tell the hypervisor that these
memory ranges will not be accessed and don't need to occupy physical RAM.

On a crash, we would them execute another hypercall to reanimate the
kdump areas.  Since this is a once-in-a-lifetime (literally) event, this
can be arbitrarily slow.

This would only require a small number of hypercalls inserted into
already existing code paths, and provide most of the benefit of
hypervisor-assisted crash dumping.

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Yinghai Lu]

On Fri, Nov 15, 2013 at 10:03 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> On Fri, Nov 15, 2013 at 09:33:41AM -0800, Yinghai Lu wrote:

>> I have one system with 6TiB memory, kdump does not work even
>> crashkernel=512M in legacy mode. ( it only work on system with
>> 4.5TiB).
>
> Recently I tested one system with 6TB of memory and dumped successfully
> with 512MB reserved under 896MB. Also I have heard reports of successful
> dump of 12TB system with 512MB reserved below 896MB (due to cyclic
> mode of makedumpfile).
>
> So with newer releases only reason one might want to reserve more
> memory is that it might provide speed benefits. We need more testing
> to quantify this.

You may need bunch of PCIe cards installed.

The system with 6TiB + 16 PCIe cards, second kernel OOM.
The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.

>
>> --- first kernel can reserve the 512M under 896M, second kernel will
>> OOM as it load driver for every pci devices...
>>
>> So why would RH guys not spend some time on optimizing your kdump initrd
>> build scripts and only put dump device related driver in it?
>
> Try latest Fedora and that's what we do. Now we have moved to dracut
> based initramfs generation and we tell dracut that build initramfs for
> host and additional dump destination and dracut builds it for those only.
> I think there might be scope for further optimization, but I don't think
> that's the problem any more.

Good. Assume that will be in RHEL 7.

>
> So issue remains that crashkernel=X,high is not a good default choice
> because it consumes extra 72M which we don't have to.

then if it falls into 896~4G, user may still need to update kexec-tools ?

Thanks

Yinghai

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[jerry.hoemann]

On Fri, Nov 15, 2013 at 02:24:25PM -0800, Yinghai Lu wrote:
> On Fri, Nov 15, 2013 at 10:03 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> > On Fri, Nov 15, 2013 at 09:33:41AM -0800, Yinghai Lu wrote:
> 
> >> I have one system with 6TiB memory, kdump does not work even
> >> crashkernel=512M in legacy mode. ( it only work on system with
> >> 4.5TiB).
> >
> > Recently I tested one system with 6TB of memory and dumped successfully
> > with 512MB reserved under 896MB. Also I have heard reports of successful
> > dump of 12TB system with 512MB reserved below 896MB (due to cyclic
> > mode of makedumpfile).
> >
> > So with newer releases only reason one might want to reserve more
> > memory is that it might provide speed benefits. We need more testing
> > to quantify this.
> 
> You may need bunch of PCIe cards installed.
> 
> The system with 6TiB + 16 PCIe cards, second kernel OOM.
> The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.

Yinghai,

Your original email said you were using "legacy mode".  Does this mean
you're not running makedumpfile in cyclic mode?  Cyclic mode makes
a *big* difference in memory foot print of makedumpfile.

thanks


Jerry


> 
> >
> >> --- first kernel can reserve the 512M under 896M, second kernel will
> >> OOM as it load driver for every pci devices...
> >>
> >> So why would RH guys not spend some time on optimizing your kdump initrd
> >> build scripts and only put dump device related driver in it?
> >
> > Try latest Fedora and that's what we do. Now we have moved to dracut
> > based initramfs generation and we tell dracut that build initramfs for
> > host and additional dump destination and dracut builds it for those only.
> > I think there might be scope for further optimization, but I don't think
> > that's the problem any more.
> 
> Good. Assume that will be in RHEL 7.
> 
> >
> > So issue remains that crashkernel=X,high is not a good default choice
> > because it consumes extra 72M which we don't have to.
> 
> then if it falls into 896~4G, user may still need to update kexec-tools ?
> 
> Thanks
> 
> Yinghai

-- 

----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  jerry.hoemann@hp.com
----------------------------------------------------------------------------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Yinghai Lu]

On Fri, Nov 15, 2013 at 2:55 PM,  <jerry.hoemann@hp.com> wrote:
>> You may need bunch of PCIe cards installed.
>>
>> The system with 6TiB + 16 PCIe cards, second kernel OOM.
>> The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.
>
> Yinghai,
>
> Your original email said you were using "legacy mode".  Does this mean
> you're not running makedumpfile in cyclic mode?  Cyclic mode makes
> a *big* difference in memory foot print of makedumpfile.

I mean: boot linux with legacy bios mode instead UEFI native boot.

Thanks

Yinghai

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Fri, Nov 15, 2013 at 11:16:25AM -0800, H. Peter Anvin wrote:
> On 11/15/2013 10:46 AM, H. Peter Anvin wrote:
> > On 11/15/2013 10:30 AM, Vivek Goyal wrote:
> >>
> >> I agree taking assistance of hypervisor should be useful.
> >>
> >> One reason we use kdump for VM too because it makes life simple. There
> >> is no difference in how we configure, start and manage crash dumps
> >> in baremetal or inside VM. And in practice have not heard of lot of
> >> failures of kdump in VM environment.
> >>
> >> So while reliability remains a theoritical concern, in practice it
> >> has not been a real concern and that's one reason I think we have
> >> not seen a major push for alternative method in VM environment.
> >>
> > 
> > Another reason, again, is that it doesn't sit on all that memory.
> > 
> 
> This led me to a potentially interesting idea.  If we can tell the
> hypervisor about which memory blocks belong to kdump, we can still use
> kdump in its current form with only a few hypervisor calls thrown in.
> 
> One set of calls would mark memory ranges as belonging to kdump.  This
> would (a) make them protected,

This sounds good. We already have arch hooks to map/unmap crash kernel
ranges, crash_map_reserved_pages() and crash_unmap_reserved_pages(). Now x86,
should be able to use these hooks to tell hypervisor to remove mappings
for certain physical certain ranges and remap these back when needed. s390
already does some magic there.

> and (b) tell the hypervisor that these
> memory ranges will not be accessed and don't need to occupy physical RAM.

I am not sure if we need to do anything here. I am assuming that most of
the crashkernel memory has not been touched and does not occupy physical
memory till crash actually happens. We probably will touch only 20-30MB
of crashkernel memory during kernel load and that should ultimately make
its way to swap at some point of time.

And if that's true, then reserving 72M extra due to crashkernel=X,high
should not be a big issue in KVM guests. It will still be an issue on
physical servers though.

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Fri, Nov 15, 2013 at 02:24:25PM -0800, Yinghai Lu wrote:
> On Fri, Nov 15, 2013 at 10:03 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> > On Fri, Nov 15, 2013 at 09:33:41AM -0800, Yinghai Lu wrote:
> 
> >> I have one system with 6TiB memory, kdump does not work even
> >> crashkernel=512M in legacy mode. ( it only work on system with
> >> 4.5TiB).
> >
> > Recently I tested one system with 6TB of memory and dumped successfully
> > with 512MB reserved under 896MB. Also I have heard reports of successful
> > dump of 12TB system with 512MB reserved below 896MB (due to cyclic
> > mode of makedumpfile).
> >
> > So with newer releases only reason one might want to reserve more
> > memory is that it might provide speed benefits. We need more testing
> > to quantify this.
> 
> You may need bunch of PCIe cards installed.
> 
> The system with 6TiB + 16 PCIe cards, second kernel OOM.
> The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.

What's the distro you are testing with? Do you have latest bits of
makeudmpfile where we use cyclic mode by default and one does not need
more reserved memory because of more physical memory present in the
box. I suspect that might be the problem in your testing environment
and old makedumpfile wil try to allocate larger memory on large
RAM machines and OOM.

[..]
> > So issue remains that crashkernel=X,high is not a good default choice
> > because it consumes extra 72M which we don't have to.
> 
> then if it falls into 896~4G, user may still need to update kexec-tools ?

Yep. But distributions control the version of kexec-tools and version
of kernel and can ship updated kexec-tools by default.

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/18/2013 07:22 AM, Vivek Goyal wrote:
> 
> And if that's true, then reserving 72M extra due to crashkernel=X,high
> should not be a big issue in KVM guests. It will still be an issue on
> physical servers though.
> 

Yes, but there it is a single instance and not a huge amount of RAM.

	-hpa


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Mon, Nov 18, 2013 at 10:29:18AM -0800, H. Peter Anvin wrote:
> On 11/18/2013 07:22 AM, Vivek Goyal wrote:
> > 
> > And if that's true, then reserving 72M extra due to crashkernel=X,high
> > should not be a big issue in KVM guests. It will still be an issue on
> > physical servers though.
> > 
> 
> Yes, but there it is a single instance and not a huge amount of RAM.

Agreed. But for some people it is. For example, we don't enable kdump
by default on fedora. Often people don't like 128MB of their laptop
memory not being used. And I have been thinking how to reduce memory
usage further so that I can enable kdump by default on Fedora.

Instead, now this 72MB increase come in picture which does not bring
us any benefit for most of the people. Only people who benefit from
it are large memory servers and everybody else (having memory more
than 4G) pays this penalty.

I rather prefer that this penalty of 72M is paid only by those who need
to have memory reservation above 4G.

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Yinghai Lu]

On Mon, Nov 18, 2013 at 7:32 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
>> You may need bunch of PCIe cards installed.
>>
>> The system with 6TiB + 16 PCIe cards, second kernel OOM.
>> The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.
>
> What's the distro you are testing with? Do you have latest bits of
> makeudmpfile where we use cyclic mode by default and one does not need
> more reserved memory because of more physical memory present in the
> box. I suspect that might be the problem in your testing environment
> and old makedumpfile wil try to allocate larger memory on large
> RAM machines and OOM.

Default RHEL 6.4.

Will check if i can enable cyclic mode.

Thanks

Yinghai

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Vivek Goyal]

On Mon, Nov 18, 2013 at 11:34:04AM -0800, Yinghai Lu wrote:
> On Mon, Nov 18, 2013 at 7:32 AM, Vivek Goyal <vgoyal@redhat.com> wrote:
> >> You may need bunch of PCIe cards installed.
> >>
> >> The system with 6TiB + 16 PCIe cards, second kernel OOM.
> >> The system with 4.5TiB + 16 PCIe cards, second kernel works with vmcore dumped.
> >
> > What's the distro you are testing with? Do you have latest bits of
> > makeudmpfile where we use cyclic mode by default and one does not need
> > more reserved memory because of more physical memory present in the
> > box. I suspect that might be the problem in your testing environment
> > and old makedumpfile wil try to allocate larger memory on large
> > RAM machines and OOM.
> 
> Default RHEL 6.4.
> 
> Will check if i can enable cyclic mode.

6.4 does not have makedumpfile cyclic mode support. 6.5 does and it
is enabled by default and no user intervention is required to enable it.

Thanks
Vivek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[H. Peter Anvin]

On 11/15/2013 10:30 AM, Vivek Goyal wrote:
> 
> And IOMMU support is very flaky with kdump. And IOMMU's can be turned
> off at command line. And that would force one to remove crahkernel_low=0.
> So change of one command line option forces change of another. It is
> complicated.
> 
> Also there are very few systems which work with IOMMU on. A lot more
> which work without IOMMU. We have all these DMAR issues and still nobody
> has been able to address IOMMU issues properly.
> 

Why do we need such a big bounce buffer for kdump swiotlb anyway?
Surely the vast majority of all dump devices don't need it, so it is
there for completeness, no?

	-hpa


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Early use of boot service memory

[Yinghai Lu]

On Mon, Nov 18, 2013 at 5:32 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 11/15/2013 10:30 AM, Vivek Goyal wrote:
>>
>> And IOMMU support is very flaky with kdump. And IOMMU's can be turned
>> off at command line. And that would force one to remove crahkernel_low=0.
>> So change of one command line option forces change of another. It is
>> complicated.
>>
>> Also there are very few systems which work with IOMMU on. A lot more
>> which work without IOMMU. We have all these DMAR issues and still nobody
>> has been able to address IOMMU issues properly.
>>
>
> Why do we need such a big bounce buffer for kdump swiotlb anyway?
> Surely the vast majority of all dump devices don't need it, so it is
> there for completeness, no?

Yes, because normal path will need that 64M+32k default.

We may reduce that amount to 16M or 18M and in second kernel let
allocate less for swiotlb.

Thanks

Yinghai

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>