visible memory seems wrong in kexec crash dump kernel

visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

Hi,

I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system with 8GB of memory.  (It's an embedded system and I can't do much about the fact that it's using older software.)

I booted the original kernel with "crashkernel=224M@32M" in the boot args.  I then loaded the crash kernel with:

./kexec -p vmlinux.kontron.strip --append="root=/dev/mtdblock0 rw rootfstype=jffs2 panic=5 1 maxcpus=1 noirqdistrib reset_devices"

Then I triggered a crash:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger


When the crash kernel came up, it had "elfcorehdr=48392K savemaxmem=8192M" in the kernel boot args, and "free" showed the system as having 7908652 KB of total memory instead of the expected 224MB.

Do I need to manually specify some options to kexec in order to preserve the memory of the original system?  I seem to recall that kexec did this automatically on x86 systems.

Thanks,
Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/11/2013 02:55 PM, Chris Friesen wrote:
> Hi,
>
> I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system
> with 8GB of memory.  (It's an embedded system and I can't do much
> about the fact that it's using older software.)

I should probably clarify this...I may be able to update kexec, I can't 
update the kernel but I can backport more recent code if necessary.

Looking at the version of kexec that I have, it seems like where x86 
uses "memmap=" to specify the memory map usable by the capture kernel, 
powerpc does something different.

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/11/2013 03:22 PM, Chris Friesen wrote:
> On 07/11/2013 02:55 PM, Chris Friesen wrote:
>> Hi,
>>
>> I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system
>> with 8GB of memory. (It's an embedded system and I can't do much
>> about the fact that it's using older software.)
>
> I should probably clarify this...I may be able to update kexec, I can't
> update the kernel but I can backport more recent code if necessary.
>
> Looking at the version of kexec that I have, it seems like where x86
> uses "memmap=" to specify the memory map usable by the capture kernel,
> powerpc does something different.

After some experimenting, it looks like in the capture kernel 
/dev/oldmem might actually refer to the memory owned by the old kernel.

However, there doesn't seem to be anything preventing me from trampling 
it from within the capture kernel--I can create a tmpfs filesystem in 
memory and write gigs of data to it even though the capture kernel is 
only supposed to have 224MB.

I think I got it to work properly once, but since then I haven't been 
able to get uncorrupted data out of /dev/oldmem.  (My original kernel 
reserves a chunk of memory for logging and passes the offset/size to the 
capture kernel via kexec kernel args.)

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Michael Ellerman]

On Thu, Jul 11, 2013 at 03:22:49PM -0600, Chris Friesen wrote:
> On 07/11/2013 02:55 PM, Chris Friesen wrote:
> >Hi,
> >
> >I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system
> >with 8GB of memory.  (It's an embedded system and I can't do much
> >about the fact that it's using older software.)
> 
> I should probably clarify this...I may be able to update kexec, I
> can't update the kernel but I can backport more recent code if
> necessary.
> 
> Looking at the version of kexec that I have, it seems like where x86
> uses "memmap=" to specify the memory map usable by the capture
> kernel, powerpc does something different.

>From memory, it's been years, on powerpc we add properties to the memory
nodes in the device tree that specify which memory is usable. The
properties are called "linux,usable-memory", and in modern kernels they
are read in early_init_dt_scan_memory().

cheers

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/11/2013 07:21 PM, Michael Ellerman wrote:
> On Thu, Jul 11, 2013 at 03:22:49PM -0600, Chris Friesen wrote:
>> On 07/11/2013 02:55 PM, Chris Friesen wrote:
>>> Hi,
>>>
>>> I'm running 2.6.34 with kexec 2.0.1 on a Freescale p5020-based system
>>> with 8GB of memory.  (It's an embedded system and I can't do much
>>> about the fact that it's using older software.)
>>
>> I should probably clarify this...I may be able to update kexec, I
>> can't update the kernel but I can backport more recent code if
>> necessary.
>>
>> Looking at the version of kexec that I have, it seems like where x86
>> uses "memmap=" to specify the memory map usable by the capture
>> kernel, powerpc does something different.
>
>  From memory, it's been years, on powerpc we add properties to the memory
> nodes in the device tree that specify which memory is usable. The
> properties are called "linux,usable-memory", and in modern kernels they
> are read in early_init_dt_scan_memory().

I turned on the instrumentation in early_init_dt_scan_memory() and got 
the following when jumping to the capture kernel:

memory scan node memory, reg size 16, data: 0 0 2 0,
  - 0 ,  200000000

That 0x200000000 matches the fact that I'm seeing 8GB of memory 
available in the recovery kernel.

If I boot the original kernel with "crashkernel=224M@32M", should I 
expect that only 224MB is marked as "linux,usable-memory" in the 
recovery kernel?

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/12/2013 03:08 PM, Chris Friesen wrote:

> I turned on the instrumentation in early_init_dt_scan_memory() and got
> the following when jumping to the capture kernel:
>
> memory scan node memory, reg size 16, data: 0 0 2 0,
> - 0 , 200000000
>
> That 0x200000000 matches the fact that I'm seeing 8GB of memory
> available in the recovery kernel.
>
> If I boot the original kernel with "crashkernel=224M@32M", should I
> expect that only 224MB is marked as "linux,usable-memory" in the
> recovery kernel?

I started looking at the kexec side of things, and I noticed something a 
bit odd.  In most places dealing with the device tree in kexec it 
accepts either "memory" or "memory@" for the memory node name.  In 
add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept 
"memory@".

Is this expected behaviour?  It seems to be the same in current git 
versions of kexec-tools.

On my system I see "/proc/device-tree/memory".

If I modify add_usable_mem_property() to also accept "/memory" then my 
recovery kernel boots up with

physicalMemorySize            = 0x10000000

which is 256MB (which is still a bit odd since I specified 224MB for the 
crashkernel).

However, it then hits the BUG() call at the end of mark_bootmem() in 
mm/bootmem.c.

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/12/2013 04:59 PM, Chris Friesen wrote:
> On 07/12/2013 03:08 PM, Chris Friesen wrote:
>
>> I turned on the instrumentation in early_init_dt_scan_memory() and got
>> the following when jumping to the capture kernel:
>>
>> memory scan node memory, reg size 16, data: 0 0 2 0,
>> - 0 , 200000000
>>
>> That 0x200000000 matches the fact that I'm seeing 8GB of memory
>> available in the recovery kernel.
>>
>> If I boot the original kernel with "crashkernel=224M@32M", should I
>> expect that only 224MB is marked as "linux,usable-memory" in the
>> recovery kernel?
>
> I started looking at the kexec side of things, and I noticed something a
> bit odd. In most places dealing with the device tree in kexec it accepts
> either "memory" or "memory@" for the memory node name. In
> add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept
> "memory@".
>
> Is this expected behaviour? It seems to be the same in current git
> versions of kexec-tools.
>
> On my system I see "/proc/device-tree/memory".
>
> If I modify add_usable_mem_property() to also accept "/memory" then my
> recovery kernel boots up with
>
> physicalMemorySize = 0x10000000
>
> which is 256MB (which is still a bit odd since I specified 224MB for the
> crashkernel).
>
> However, it then hits the BUG() call at the end of mark_bootmem() in
> mm/bootmem.c.

One final thing and I'll stop replying to myself. :)

It looks like the problem is that some board-specific freescale code was 
calling lmb_reserve() with a base address in the 4GB range.  It seems 
odd that lmb_reserve() didn't throw some kind of error when the recovery 
kernel was supposed to be limited to 224MB.

Rather than try and fix the bug, I turned off the (unneeded) config 
options related to the above lmb_reserve() calls and was able to 
successfully access the information I needed via /dev/oldmem.

The upshot is that there seems to be a number of things that could be 
improved:

1) kexec should accept "/memory" and not just "/memory@"
2) lmb_reserve() should really respect the crashkernel memory limit
3) the freescale stuff really shouldn't assume it can map things 
wherever it feels like

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Michael Ellerman]

On Sat, Jul 13, 2013 at 12:30:50AM -0600, Chris Friesen wrote:
> On 07/12/2013 04:59 PM, Chris Friesen wrote:
> >On 07/12/2013 03:08 PM, Chris Friesen wrote:
> >
> >>I turned on the instrumentation in early_init_dt_scan_memory() and got
> >>the following when jumping to the capture kernel:
> >>
> >>memory scan node memory, reg size 16, data: 0 0 2 0,
> >>- 0 , 200000000
> >>
> >>That 0x200000000 matches the fact that I'm seeing 8GB of memory
> >>available in the recovery kernel.
> >>
> >>If I boot the original kernel with "crashkernel=224M@32M", should I
> >>expect that only 224MB is marked as "linux,usable-memory" in the
> >>recovery kernel?
> >
> >I started looking at the kexec side of things, and I noticed something a
> >bit odd. In most places dealing with the device tree in kexec it accepts
> >either "memory" or "memory@" for the memory node name. In
> >add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only accept
> >"memory@".
> >
> >Is this expected behaviour? It seems to be the same in current git
> >versions of kexec-tools.
> >
> >On my system I see "/proc/device-tree/memory".
> >
> >If I modify add_usable_mem_property() to also accept "/memory" then my
> >recovery kernel boots up with
> >
> >physicalMemorySize = 0x10000000
> >
> >which is 256MB (which is still a bit odd since I specified 224MB for the
> >crashkernel).
> >
> >However, it then hits the BUG() call at the end of mark_bootmem() in
> >mm/bootmem.c.
> 
> One final thing and I'll stop replying to myself. :)
> 
> It looks like the problem is that some board-specific freescale code
> was calling lmb_reserve() with a base address in the 4GB range.  It
> seems odd that lmb_reserve() didn't throw some kind of error when
> the recovery kernel was supposed to be limited to 224MB.
> 
> Rather than try and fix the bug, I turned off the (unneeded) config
> options related to the above lmb_reserve() calls and was able to
> successfully access the information I needed via /dev/oldmem.
> 
> The upshot is that there seems to be a number of things that could
> be improved:
> 
> 1) kexec should accept "/memory" and not just "/memory@"

Yeah probably, I think folks tend to use "/memory@0" even if they only
have a single memory node. But that does sound like a bug in kexec.

> 2) lmb_reserve() should really respect the crashkernel memory limit

It's been replaced in mainline, so you'd have to check what the current
code does in that situation.

> 3) the freescale stuff really shouldn't assume it can map things
> wherever it feels like

Agreed :)

cheers

Re: visible memory seems wrong in kexec crash dump kernel

[Benjamin Herrenschmidt]

On Sun, 2013-07-14 at 14:36 +1000, Michael Ellerman wrote:
> > >Is this expected behaviour? It seems to be the same in current git
> > >versions of kexec-tools.
> > >
> > >On my system I see "/proc/device-tree/memory".
> > >
> > >If I modify add_usable_mem_property() to also accept "/memory" then
> my

This is a bug in your device-tree. The memory node should have a unit
address which corresponds to it's reg property. I know people tend to
skip it for "0" but it's bad practice.

So for memory starting at 0 it should be memory@0

Cheers,
Ben.

Re: visible memory seems wrong in kexec crash dump kernel

[Chris Friesen]

On 07/13/2013 11:26 PM, Benjamin Herrenschmidt wrote:
> On Sun, 2013-07-14 at 14:36 +1000, Michael Ellerman wrote:
>>>> Is this expected behaviour? It seems to be the same in current git
>>>> versions of kexec-tools.
>>>>
>>>> On my system I see "/proc/device-tree/memory".
>>>>
>>>> If I modify add_usable_mem_property() to also accept "/memory" then
>> my
>
> This is a bug in your device-tree. The memory node should have a unit
> address which corresponds to it's reg property. I know people tend to
> skip it for "0" but it's bad practice.
>
> So for memory starting at 0 it should be memory@0

There are a fair number of dts files in the kernel tree that don't 
specify an address for the memory node.

If the kernel accepts it without an address, it seems logical that kexec 
should as well.

Or maybe the kernel should just implicitly assume an address of zero and 
export it as such in /proc/device-tree?

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Benjamin Herrenschmidt]

On Sun, 2013-07-14 at 17:08 -0600, Chris Friesen wrote:
> > So for memory starting at 0 it should be memory@0
> 
> There are a fair number of dts files in the kernel tree that don't 
> specify an address for the memory node.
> 
> If the kernel accepts it without an address, it seems logical that kexec 
> should as well.

As long as kexec doesn't start being stupid when there are several nodes
and doesn't pick up the "first one in device-tree order" instead of the
one at 0...

I've been hit by that sort of bugs before (though not specifically in
kexec).

> Or maybe the kernel should just implicitly assume an address of zero and 
> export it as such in /proc/device-tree?

I don't want /proc/device-tree to expose something different than what's in
the actual device-tree, that would be the source for endless horrors.

We already are borderline with the occasional renaming we do in the case of
duplicate name+property...

Cheers,
Ben.

Re: visible memory seems wrong in kexec crash dump kernel

[Scott Wood]

On 07/13/2013 01:30:50 AM, Chris Friesen wrote:
> On 07/12/2013 04:59 PM, Chris Friesen wrote:
>> On 07/12/2013 03:08 PM, Chris Friesen wrote:
>>=20
>>> I turned on the instrumentation in early_init_dt_scan_memory() and =20
>>> got
>>> the following when jumping to the capture kernel:
>>>=20
>>> memory scan node memory, reg size 16, data: 0 0 2 0,
>>> - 0 , 200000000
>>>=20
>>> That 0x200000000 matches the fact that I'm seeing 8GB of memory
>>> available in the recovery kernel.
>>>=20
>>> If I boot the original kernel with "crashkernel=3D224M@32M", should I
>>> expect that only 224MB is marked as "linux,usable-memory" in the
>>> recovery kernel?
>>=20
>> I started looking at the kexec side of things, and I noticed =20
>> something a
>> bit odd. In most places dealing with the device tree in kexec it =20
>> accepts
>> either "memory" or "memory@" for the memory node name. In
>> add_usable_mem_property() in arch/ppc64/fs2dt.c it seems to only =20
>> accept
>> "memory@".
>>=20
>> Is this expected behaviour? It seems to be the same in current git
>> versions of kexec-tools.
>>=20
>> On my system I see "/proc/device-tree/memory".
>>=20
>> If I modify add_usable_mem_property() to also accept "/memory" then =20
>> my
>> recovery kernel boots up with
>>=20
>> physicalMemorySize =3D 0x10000000
>>=20
>> which is 256MB (which is still a bit odd since I specified 224MB for =20
>> the
>> crashkernel).
>>=20
>> However, it then hits the BUG() call at the end of mark_bootmem() in
>> mm/bootmem.c.
>=20
> One final thing and I'll stop replying to myself. :)
>=20
> It looks like the problem is that some board-specific freescale code =20
> was calling lmb_reserve() with a base address in the 4GB range.  It =20
> seems odd that lmb_reserve() didn't throw some kind of error when the =20
> recovery kernel was supposed to be limited to 224MB.
>=20
> Rather than try and fix the bug, I turned off the (unneeded) config =20
> options related to the above lmb_reserve() calls and was able to =20
> successfully access the information I needed via /dev/oldmem.
>=20
> The upshot is that there seems to be a number of things that could be =20
> improved:
>=20
> 1) kexec should accept "/memory" and not just "/memory@"
> 2) lmb_reserve() should really respect the crashkernel memory limit
> 3) the freescale stuff really shouldn't assume it can map things =20
> wherever it feels like

What "board-specific freescale code" are you referring to?

-Scott=

RE: visible memory seems wrong in kexec crash dump kernel

[Friesen, Christopher]

=46rom: Scott Wood [scottwood@freescale.com]
Sent: Monday, July 29, 2013 5:10 PM
To: Friesen, Christopher
Cc: Michael Ellerman; kexec@lists.infradead.org; Paul Mackerras; linu=
xppc-dev@lists.ozlabs.org; Vivek Goyal
Subject: Re: visible memory seems wrong in kexec crash dump kernel

On 07/13/2013 01:30:50 AM, Chris Friesen wrote:
> The upshot is that there seems to be a number of things that could =
be
> improved:
>
> 1) kexec should accept "/memory" and not just "/memory@"
> 2) lmb_reserve() should really respect the crashkernel memory limit
> 3) the freescale stuff really shouldn't assume it can map things
> wherever it feels like

What "board-specific freescale code" are you referring to?

-Scott


Sorry for the crappy quoting, I'm using a web outlook portal.

I've switched employers so I don't have access to the exact details a=
ny more.  The system in question was a Kontron AM4150 which uses the =
P5020.  As I recall, one of the Freescale drivers (I think it was the=
 buffer or queue manager that the network driver makes use of) was at=
tempting to call lmb_reserve() with a base address in the 4GB range e=
ven though the recovery kernel was limited to 224MB of memory.

While I've got your attention, the other thing that I found was that =
the "dpa" network driver didn't properly work in a kexec'd kernel eve=
n when given lots of memory.  It would work for a little bit and then=
 hang.

This was all on a Wind River Linux 4.3 system, which is based on 2.6.=
34.  It's possible it's been fixed in more recent kernels.  The testc=
ase to see if current kernels are still affected should be fairly sim=
ple:

1) Take a P5020 board and boot it with NFS rootfs, reserving 224MB fo=
r crashkernel.
2) Load a crashkernel into it, configured to use NFS rootfs.
3) Trigger a panic on the main kernel.
4) Ensure it loads the recovery kernel properly, that it is limited t=
o the correct amount of memory, that it has visibility of the origina=
l kernel's memory,  that it has network connectivity and is stable wh=
en copying a few gigs of data over the network.

Chris

Re: visible memory seems wrong in kexec crash dump kernel

[Scott Wood]

On 07/31/2013 11:40:05 AM, Friesen, Christopher wrote:
>=20
> From: Scott Wood [scottwood@freescale.com]
> Sent: Monday, July 29, 2013 5:10 PM
> To: Friesen, Christopher
> Cc: Michael Ellerman; kexec@lists.infradead.org; Paul Mackerras; =20
> linuxppc-dev@lists.ozlabs.org; Vivek Goyal
> Subject: Re: visible memory seems wrong in kexec crash dump kernel
>=20
> On 07/13/2013 01:30:50 AM, Chris Friesen wrote:
> > The upshot is that there seems to be a number of things that could =20
> be
> > improved:
> >
> > 1) kexec should accept "/memory" and not just "/memory@"
> > 2) lmb_reserve() should really respect the crashkernel memory limit
> > 3) the freescale stuff really shouldn't assume it can map things
> > wherever it feels like
>=20
> What "board-specific freescale code" are you referring to?
>=20
> -Scott
>=20
>=20
> Sorry for the crappy quoting, I'm using a web outlook portal.
>=20
> I've switched employers so I don't have access to the exact details =20
> any more.  The system in question was a Kontron AM4150 which uses the =20
> P5020.  As I recall, one of the Freescale drivers (I think it was the =20
> buffer or queue manager that the network driver makes use of) was =20
> attempting to call lmb_reserve() with a base address in the 4GB range =20
> even though the recovery kernel was limited to 224MB of memory.

That's not "board specific" code, and it's not even mainline Linux =20
code.  Unfortunately none of the datapath stuff is upstream, still.

> While I've got your attention, the other thing that I found was that =20
> the "dpa" network driver didn't properly work in a kexec'd kernel =20
> even when given lots of memory.  It would work for a little bit and =20
> then hang.

I'm not particularly surprised by this.  It doesn't help that there's =20
no way to do a device reset. :-(

Issues with Freescale SDK code should be reported on =20
https://community.freescale.com/, to support@freescale.com, or to your =20
FAE.

-Scott=