[PATCH 0/3 -mm] kexec jump -v8

[PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

This patchset provides an enhancement to kexec/kdump. It implements
the following features:

- Backup/restore memory used both by the original kernel and the
  kexeced kernel.

- Jumping between the original kernel and the kexeced kernel.

- Read/write memory image of the kexeced kernel in the original kernel
  and write memory image of the original kernel in the kexeced
  kernel. This can be used as a communication method between the
  kexeced kernel and the original kernel.


The features of this patchset can be used as follow:

- Kernel/system debug through making system snapshot. You can make
  system snapshot, jump back, do some thing and make another system
  snapshot.

- A simple hibernation implementation without ACPI support. You can
  kexec a hibernating kernel, save the memory image of original system
  and shutdown the system. When resuming, you boot a resuming kernel
  in memory range of kexeced kernel, restore the memory image of
  original system and jump back.

- Cooperative multi-kernel/system. With kexec jump, you can switch
  between several kernels/systems quickly without boot process except
  the first time. This appears like swap a whole kernel/system out/in.

- A general method to call program in physical mode. This can be used
  to invoke some BIOS code under Linux.

- The basis of a full kexec based hibernation implementation with ACPI
  support. The full kexec based hibernation implementation is provided
  in another patchset named kexec based hibernation.


Now, only the i386 architecture is supported. The patchset is based on
Linux kernel 2.6.24-rc5-mm1, and has been tested on IBM T42 with ACPI
on and off.


The following user-space tools can be used with kexec jump.

1. kexec-tools needs to be patched to support kexec jump. The patches
   and the precompiled kexec can be download from the following URL:
       source: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec-tools-src_git_kh8.tar.bz2
       patches: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec-tools-patches_git_kh8.tar.bz2
       binary: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec_git_kh8

2. makedumpfile with patches are used as memory image saving tool, it
   can exclude free pages from original kernel memory image file. The
   patches and the precompiled makedumpfile can be download from the
   following URL:
       source: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile-src_cvs_kh8.tar.bz2
       patches: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile-patches_cvs_kh8.tar.bz2
       binary: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile_cvs_kh8

3. A simplest memory image restoring tool named "krestore" is
   implemented. It can be downloaded from the following URL:
       source: http://khibernation.sourceforge.net/download/release_v8/krestore/krestore-src_cvs_kh8.tar.bz2
       binary: http://khibernation.sourceforge.net/download/release_v8/krestore/krestore_cvs_kh8

An initramfs image can be used as the root file system of kexeced
kernel. An initramfs image built with "BuildRoot" can be downloaded
from the following URL:
    initramfs image: http://khibernation.sourceforge.net/download/release_v8/initramfs/rootfs_cvs_kh8.gz
All user space tools above are included in the initramfs image.


Usage example of jumping between original and kexeced kernel:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool, or download the
   pre-built initramfs image, called rootfs.gz in following text.

3. Boot kernel compiled in step 1.

4. Load kernel compiled in step 1 with /sbin/kexec. If You want to use
   "krestore" tool, the --elf64-core-headers should be specified in
   command line of /sbin/kexec. The shell command line can be as
   follow:

   /sbin/kexec --load-jump-back /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --elf64-core-headers --initrd=rootfs.gz

5. Boot the kexeced kernel with following shell command line:

   /sbin/kexec -e

6. The kexeced kernel will boot as normal kexec. In kexeced kernel the
   memory image of original kernel can read via /proc/vmcore or
   /dev/oldmem, and can be written via /dev/oldmem. You can
   save/restore/modify it as you want to.

7. Prepare jumping back from kexeced kernel with following shell
   command lines:

   jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
   /sbin/kexec --load-jump-back-helper=$jump_back_entry

8. Jump back to the original kernel with following shell command line:

   /sbin/kexec -e

9. Now, you are in the original kernel again. You can read/write the
   memory image of kexeced kernel via /proc/kimgcore.

10. You can jump between the original kernel and kexeced kernel as you
    want to via the following shell command line:

    /sbin/kexec -e


Known issues:

- The suspend/resume callback of device drivers are used to put
  devices into quiescent state. This will unnecessarily (possibly
  harmfully) put devices into low power state. This is intended to be
  solved by separating device quiesce/unquiesce callback from the
  device suspend/resume callback.


ChangeLog:

v8:

- Split kexec jump patchset from kexec based hibernation patchset.

- Add writing support to kimgcore. This can be used as a communication
  method between kexeced kernel and original kernel.

- Merge various KEXEC_PRESERVE_* flags into one KEXEC_PRESERVE_CONTEXT
  because there is no need for such subtle control.

- Delete variable argument based "kernel to kernel" communication
  mechanism from basic kexec jump patchset.

v7:

- Add an interface to dump the loaded kexec_image, which may contains
  the memory image of kexeced system. This is used to accelerate kexec
  based hibernation.

- Refactor kexec jump to be a command driven programming model.

- Adjust ACPI support to mimic the ACPI support of u/swsusp.

- Use kexec_lock to do synchronization.

v6:

- Add ACPI support.

- Refactor kexec jump to be a general facility to call real mode code.

v5:

- A flag (KEXEC_JUMP_BACK) is added to indicate the loaded kernel
  image is used for jumping back. The reboot command for jumping back
  is removed. This interface is more stable (proposed by Eric
  Biederman).

- NX bit handling support for kexec is added.

- Merge machine_kexec and machine_kexec_jump, remove NO_RET attribute
  from machine_kexec.

- Passing jump back entry to kexeced kernel via kernel command line
  (parsed by user space tool via /proc/cmdline instead of
  kernel). Original corresponding boot parameter and sysfs code is
  removed.

v4:

- Two reboot command are merged back to one because the underlying
  implementation is same.

- Jumping without reserving memory is implemented. As a side effect,
  two direction jumping is implemented.

- A jump back protocol is defined and documented. The original kernel
  and kexeced kernel are more independent from each other.

- The CPU state save/restore code are merged into relocate_kernel.S.

v3:

- The reboot command LINUX_REBOOT_CMD_KJUMP is split into to two
  reboot command to reflect the different function.

- Document is added for added kernel parameters.

- /sys/kernel/kexec_jump_buf_pfn is made writable, it is used for
  memory image restoring.

- Console restoring after jumping back is implemented.

- Writing support is added for /dev/oldmem, to restore memory contents
  of hibernated system.

v2:

- The kexec jump implementation is put into the kexec/kdump framework
  instead of software suspend framework. The device and CPU state
  save/restore code of software suspend is called when needed.

- The same code path is used for both kexec a new kernel and jump back
  to original kernel.


Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Nigel Cunningham]

Hi.

Huang, Ying wrote:
> This patchset provides an enhancement to kexec/kdump. It implements
> the following features:
> 
> - Backup/restore memory used both by the original kernel and the
>   kexeced kernel.

Why the kexeced kernel as well?

[...]

> The features of this patchset can be used as follow:
> 
> - Kernel/system debug through making system snapshot. You can make
>   system snapshot, jump back, do some thing and make another system
>   snapshot.

Are you somehow recording all the filesystem changes after the first
snapshot? If not, this is pointless (you'll end up with filesystem
corruption).

[...]

> - Cooperative multi-kernel/system. With kexec jump, you can switch
>   between several kernels/systems quickly without boot process except
>   the first time. This appears like swap a whole kernel/system out/in.

How is this useful to the end user?

Regards,

Nigel

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Fri, 2007-12-21 at 19:35 +1100, Nigel Cunningham wrote:
> Hi.
> 
> Huang, Ying wrote:
> > This patchset provides an enhancement to kexec/kdump. It implements
> > the following features:
> > 
> > - Backup/restore memory used both by the original kernel and the
> >   kexeced kernel.
> 
> Why the kexeced kernel as well?

The memory range used by kexeced kernel is also the usable memory range
in original kernel. Maybe should be: backup/restore memory used by both
the original kernel and the kexeced kernel. My English is poor.

> [...]
> 
> > The features of this patchset can be used as follow:
> > 
> > - Kernel/system debug through making system snapshot. You can make
> >   system snapshot, jump back, do some thing and make another system
> >   snapshot.
> 
> Are you somehow recording all the filesystem changes after the first
> snapshot? If not, this is pointless (you'll end up with filesystem
> corruption).

This snapshot is not used for restore/resume. It is just used for
debugging. You can check the system state with these snapshots. So I
think it is useful even without recording filesystem changes.

> [...]
> 
> > - Cooperative multi-kernel/system. With kexec jump, you can switch
> >   between several kernels/systems quickly without boot process except
> >   the first time. This appears like swap a whole kernel/system out/in.
> 
> How is this useful to the end user?

I am not sure how useful is this. Maybe I can run a Redhat and a Debian
on my machine and switch between them.

Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Vivek Goyal]

On Fri, Dec 21, 2007 at 03:33:19PM +0800, Huang, Ying wrote:

[..]
> The following user-space tools can be used with kexec jump.
> 
> 1. kexec-tools needs to be patched to support kexec jump. The patches
>    and the precompiled kexec can be download from the following URL:
>        source: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec-tools-src_git_kh8.tar.bz2
>        patches: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec-tools-patches_git_kh8.tar.bz2
>        binary: http://khibernation.sourceforge.net/download/release_v8/kexec-tools/kexec_git_kh8
> 
> 2. makedumpfile with patches are used as memory image saving tool, it
>    can exclude free pages from original kernel memory image file. The
>    patches and the precompiled makedumpfile can be download from the
>    following URL:
>        source: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile-src_cvs_kh8.tar.bz2
>        patches: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile-patches_cvs_kh8.tar.bz2
>        binary: http://khibernation.sourceforge.net/download/release_v8/makedumpfile/makedumpfile_cvs_kh8
> 
> 3. A simplest memory image restoring tool named "krestore" is
>    implemented. It can be downloaded from the following URL:
>        source: http://khibernation.sourceforge.net/download/release_v8/krestore/krestore-src_cvs_kh8.tar.bz2
>        binary: http://khibernation.sourceforge.net/download/release_v8/krestore/krestore_cvs_kh8
> 
> An initramfs image can be used as the root file system of kexeced
> kernel. An initramfs image built with "BuildRoot" can be downloaded
> from the following URL:
>     initramfs image: http://khibernation.sourceforge.net/download/release_v8/initramfs/rootfs_cvs_kh8.gz
> All user space tools above are included in the initramfs image.
> 
> 
> Usage example of jumping between original and kexeced kernel:
> 
> 1. Compile and install patched kernel with following options selected:
> 
> CONFIG_X86_32=y
> CONFIG_RELOCATABLE=y
> CONFIG_KEXEC=y
> CONFIG_CRASH_DUMP=y
> CONFIG_PM=y
> 
> 2. Build an initramfs image contains kexec-tool, or download the
>    pre-built initramfs image, called rootfs.gz in following text.
> 
> 3. Boot kernel compiled in step 1.
> 
> 4. Load kernel compiled in step 1 with /sbin/kexec. If You want to use
>    "krestore" tool, the --elf64-core-headers should be specified in
>    command line of /sbin/kexec. The shell command line can be as
>    follow:
> 
>    /sbin/kexec --load-jump-back /boot/bzImage --mem-min=0x100000
>      --mem-max=0xffffff --elf64-core-headers --initrd=rootfs.gz
> 
> 5. Boot the kexeced kernel with following shell command line:
> 
>    /sbin/kexec -e
> 
> 6. The kexeced kernel will boot as normal kexec. In kexeced kernel the
>    memory image of original kernel can read via /proc/vmcore or
>    /dev/oldmem, and can be written via /dev/oldmem. You can
>    save/restore/modify it as you want to.
> 
> 7. Prepare jumping back from kexeced kernel with following shell
>    command lines:
> 
>    jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
>    /sbin/kexec --load-jump-back-helper=$jump_back_entry
> 
> 8. Jump back to the original kernel with following shell command line:
> 
>    /sbin/kexec -e
> 
> 9. Now, you are in the original kernel again. You can read/write the
>    memory image of kexeced kernel via /proc/kimgcore.
> 

Why do we need two interfaces, /proc/vmcore and /proc/kimgcore? Can't
we have just one say /proc/vmcore. Irrespective of what kernel you are
in /proc/vmcore gives you the access to the memory of kernel which was
previously booted.

Thanks
Vivek

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Wed, 2007-12-26 at 20:57 -0500, Vivek Goyal wrote:
[...]
> > 9. Now, you are in the original kernel again. You can read/write the
> >    memory image of kexeced kernel via /proc/kimgcore.
> > 
> 
> Why do we need two interfaces, /proc/vmcore and /proc/kimgcore? Can't
> we have just one say /proc/vmcore. Irrespective of what kernel you are
> in /proc/vmcore gives you the access to the memory of kernel which was
> previously booted.

In theory we can kexec another kernel even in a kexeced kernel, that is,
in kernel A kexec kernel B, and in kernel B kexec another kernel C. In
this situation, both /proc/vmcore and /proc/kimgcore has valid contents.
So I think, it may be better to keep two interfaces.

In fact, current kexec jump implementation use a dummy "jump back helper
image" in kexeced kernel to jump back to the original kernel. The "jump
back helper image" has no PT_LOAD segment, it is used to provide a
struct kimage (including control page, swap page) and entry point to
jump back.

Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Vivek Goyal]

On Thu, Dec 27, 2007 at 10:33:13AM +0800, Huang, Ying wrote:
> On Wed, 2007-12-26 at 20:57 -0500, Vivek Goyal wrote:
> [...]
> > > 9. Now, you are in the original kernel again. You can read/write the
> > >    memory image of kexeced kernel via /proc/kimgcore.
> > > 
> > 
> > Why do we need two interfaces, /proc/vmcore and /proc/kimgcore? Can't
> > we have just one say /proc/vmcore. Irrespective of what kernel you are
> > in /proc/vmcore gives you the access to the memory of kernel which was
> > previously booted.
> 
> In theory we can kexec another kernel even in a kexeced kernel, that is,
> in kernel A kexec kernel B, and in kernel B kexec another kernel C. In
> this situation, both /proc/vmcore and /proc/kimgcore has valid contents.
> So I think, it may be better to keep two interfaces.
> 

In those situations I think only one interface is better. For example, 
above will be broken if somebody kexec 4 kernels.

A-->B--->C--->D

I think better option might be if it is stack like situation. A kernel
shows you only the previous kernel's memory contents through /proc/vmcore
interface. So If I am in kernel D, I see only kernel C's memory image.
To see kernel B's memory image, one shall have to go back to kernel C.

Thanks
Vivek

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Thu, 2007-12-27 at 13:12 -0500, Vivek Goyal wrote:
> On Thu, Dec 27, 2007 at 10:33:13AM +0800, Huang, Ying wrote:
> > On Wed, 2007-12-26 at 20:57 -0500, Vivek Goyal wrote:
> > [...]
> > > > 9. Now, you are in the original kernel again. You can read/write the
> > > >    memory image of kexeced kernel via /proc/kimgcore.
> > > > 
> > > 
> > > Why do we need two interfaces, /proc/vmcore and /proc/kimgcore? Can't
> > > we have just one say /proc/vmcore. Irrespective of what kernel you are
> > > in /proc/vmcore gives you the access to the memory of kernel which was
> > > previously booted.
> > 
> > In theory we can kexec another kernel even in a kexeced kernel, that is,
> > in kernel A kexec kernel B, and in kernel B kexec another kernel C. In
> > this situation, both /proc/vmcore and /proc/kimgcore has valid contents.
> > So I think, it may be better to keep two interfaces.
> > 
> 
> In those situations I think only one interface is better. For example, 
> above will be broken if somebody kexec 4 kernels.
> 
> A-->B--->C--->D

I don't think the two interfaces will be broken if somebody kexec 4
kernels. For example, when kexec D from C, the /proc/vmcore is contents
of B, /proc/kimgcore is contents of D. To jump back from C to B, the D
is unloaded, and a jump back helper image is loaded.

> I think better option might be if it is stack like situation. A kernel
> shows you only the previous kernel's memory contents through /proc/vmcore
> interface. So If I am in kernel D, I see only kernel C's memory image.
> To see kernel B's memory image, one shall have to go back to kernel C.

Maybe it is not sufficient to only show the previous kernel's memory
contents. In kernel C, you maybe need to access the memory image of
kernel B and memory image of kernel D.

That is, /proc/vmcore is the memory image of the previous kernel,
and /proc/kimgcore is the memory image of the next kernel.

Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Vivek Goyal]

On Fri, Dec 21, 2007 at 03:33:19PM +0800, Huang, Ying wrote:
> This patchset provides an enhancement to kexec/kdump. It implements
> the following features:
> 
> - Backup/restore memory used both by the original kernel and the
>   kexeced kernel.
> 
> - Jumping between the original kernel and the kexeced kernel.
> 
> - Read/write memory image of the kexeced kernel in the original kernel
>   and write memory image of the original kernel in the kexeced
>   kernel. This can be used as a communication method between the
>   kexeced kernel and the original kernel.
> 
> 
> The features of this patchset can be used as follow:
> 
> - Kernel/system debug through making system snapshot. You can make
>   system snapshot, jump back, do some thing and make another system
>   snapshot.
> 

How do you differentiate between whether a core is resumable or not.
IOW, how do you know the generated /proc/vmcore has been generated after
a real crash hence can't be resumed (using krestore) or it has been 
generated because of hibernation/debug purposes and can be resumed?

I think you might have to add an extra ELF NOTE to vmcore which can help
decide whether kernel memory snapshot is resumable or not.

[..]
> 2. Build an initramfs image contains kexec-tool, or download the
>    pre-built initramfs image, called rootfs.gz in following text.
> 
> 3. Boot kernel compiled in step 1.
> 
> 4. Load kernel compiled in step 1 with /sbin/kexec. If You want to use
>    "krestore" tool, the --elf64-core-headers should be specified in
>    command line of /sbin/kexec. The shell command line can be as
>    follow:
> 
>    /sbin/kexec --load-jump-back /boot/bzImage --mem-min=0x100000
>      --mem-max=0xffffff --elf64-core-headers --initrd=rootfs.gz
> 

How about a different name like "--load-preserve-context". This will
just mean that kexec need to preserve the context while kexeing to 
image being loaded. Combination of --load-jump-back and
--load-jump-back-helper is becoming little confusing.


> 5. Boot the kexeced kernel with following shell command line:
> 
>    /sbin/kexec -e
> 
> 6. The kexeced kernel will boot as normal kexec. In kexeced kernel the
>    memory image of original kernel can read via /proc/vmcore or
>    /dev/oldmem, and can be written via /dev/oldmem. You can
>    save/restore/modify it as you want to.
> 

Restoring a hibernated image using /dev/oldmem should be easy and I 
think one should be able to launch it back using --load-jump-back-helper.

How do you restore already kexeced kernel? For example if I got two
kernels A and B. A is the one which will hibernate and B will be used
to store the hibernated kernel. I think as per the procedure one needs
to first boot into kernel B and then jump back to kernel A. This will
make image of B available in /proc/kimgcore. If I save /proc/kimgcore
to disk and want to jump back to it, how do I do it? I guess I need
to kexec again using --load-jump-back and not restore using krestore?


> 7. Prepare jumping back from kexeced kernel with following shell
>    command lines:
> 
>    jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
>    /sbin/kexec --load-jump-back-helper=$jump_back_entry
> 

How about decoupling entry point from --load-jump-back-helper. We can
introduce a separate option for entry point. Something like.

kexec --load-jump-back-helper --entry=$jump_back_entry

May be we can generalize the --entry so that a user can override the 
entry point of the normal kexec image using above.


> 8. Jump back to the original kernel with following shell command line:
> 
>    /sbin/kexec -e
> 
> 9. Now, you are in the original kernel again. You can read/write the
>    memory image of kexeced kernel via /proc/kimgcore.
> 
> 10. You can jump between the original kernel and kexeced kernel as you
>     want to via the following shell command line:
> 
>     /sbin/kexec -e
> 
> 
> Known issues:
> 
> - The suspend/resume callback of device drivers are used to put
>   devices into quiescent state. This will unnecessarily (possibly
>   harmfully) put devices into low power state. This is intended to be
>   solved by separating device quiesce/unquiesce callback from the
>   device suspend/resume callback.
> 
> 
> ChangeLog:
> 
> v8:
> 
> - Split kexec jump patchset from kexec based hibernation patchset.
> 
> - Add writing support to kimgcore. This can be used as a communication
>   method between kexeced kernel and original kernel.
> 

What are we communicating between two kernels using kimgcore?

I am setting up a system to test out the patches. More later...

Thanks
Vivek

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Fri, 2007-12-28 at 16:33 -0500, Vivek Goyal wrote:
> On Fri, Dec 21, 2007 at 03:33:19PM +0800, Huang, Ying wrote:
> > This patchset provides an enhancement to kexec/kdump. It implements
> > the following features:
> > 
> > - Backup/restore memory used both by the original kernel and the
> >   kexeced kernel.
> > 
> > - Jumping between the original kernel and the kexeced kernel.
> > 
> > - Read/write memory image of the kexeced kernel in the original kernel
> >   and write memory image of the original kernel in the kexeced
> >   kernel. This can be used as a communication method between the
> >   kexeced kernel and the original kernel.
> > 
> > 
> > The features of this patchset can be used as follow:
> > 
> > - Kernel/system debug through making system snapshot. You can make
> >   system snapshot, jump back, do some thing and make another system
> >   snapshot.
> > 
> 
> How do you differentiate between whether a core is resumable or not.
> IOW, how do you know the generated /proc/vmcore has been generated after
> a real crash hence can't be resumed (using krestore) or it has been 
> generated because of hibernation/debug purposes and can be resumed?
> 
> I think you might have to add an extra ELF NOTE to vmcore which can help
> decide whether kernel memory snapshot is resumable or not.

The current solution is as follow:

1. The original kernel will set %edi to jump back entry if resumable and
set %edi to 0 if not.

2. The purgatory of loaded kernel will check %edi, if it is not zero,
the string "jump_back_entry=<jump_back_entry>" will be appended to
kernel command line parameter.

3. In kexeced kernel, if there is "jump_back_entry=<jump_back_entry>"
in /proc/cmdline, the previous kernel is resumable, otherwise not.

As for ELF NOTE, in fact, the ELF NOTE does not work for resumable
kernel. Because the contents of source page and destination page is
swapped during kexec, and the kernel access the destination page
directly during parsing ELF NOTE. All memory that is swapped need to be
accessed via the backup pages map (image->head). I think these
information can be exchanged between two kernels via kernel command line
or /proc/kimgcore.

> [..]
> > 2. Build an initramfs image contains kexec-tool, or download the
> >    pre-built initramfs image, called rootfs.gz in following text.
> > 
> > 3. Boot kernel compiled in step 1.
> > 
> > 4. Load kernel compiled in step 1 with /sbin/kexec. If You want to use
> >    "krestore" tool, the --elf64-core-headers should be specified in
> >    command line of /sbin/kexec. The shell command line can be as
> >    follow:
> > 
> >    /sbin/kexec --load-jump-back /boot/bzImage --mem-min=0x100000
> >      --mem-max=0xffffff --elf64-core-headers --initrd=rootfs.gz
> > 
> 
> How about a different name like "--load-preserve-context". This will
> just mean that kexec need to preserve the context while kexeing to 
> image being loaded. Combination of --load-jump-back and
> --load-jump-back-helper is becoming little confusing.

Yes, this is better. I will change it.

> > 5. Boot the kexeced kernel with following shell command line:
> > 
> >    /sbin/kexec -e
> > 
> > 6. The kexeced kernel will boot as normal kexec. In kexeced kernel the
> >    memory image of original kernel can read via /proc/vmcore or
> >    /dev/oldmem, and can be written via /dev/oldmem. You can
> >    save/restore/modify it as you want to.
> > 
> 
> Restoring a hibernated image using /dev/oldmem should be easy and I 
> think one should be able to launch it back using --load-jump-back-helper.

Yes. I think so too. The current implementation of krestore restoring
the hibernated image using /dev/oldmem. And the hibernated image can be
launched using --load-jump-back-helper.

> How do you restore already kexeced kernel? For example if I got two
> kernels A and B. A is the one which will hibernate and B will be used
> to store the hibernated kernel. I think as per the procedure one needs
> to first boot into kernel B and then jump back to kernel A. This will
> make image of B available in /proc/kimgcore. If I save /proc/kimgcore
> to disk and want to jump back to it, how do I do it? I guess I need
> to kexec again using --load-jump-back and not restore using krestore?

The image of B is made as you said. And it can be restored as follow:

/sbin/kexec -l --args-none --flags=0x2 <kimgecore>
/sbin/kexec -e

That is, the image of B is loaded as a ordinary ELF file. A option
to /sbin/kexec named --flags are added to specify the
KEXEC_PRESERVE_CONTEXT flags for sys_kexec_load. This has been tested.

> > 7. Prepare jumping back from kexeced kernel with following shell
> >    command lines:
> > 
> >    jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
> >    /sbin/kexec --load-jump-back-helper=$jump_back_entry
> > 
> 
> How about decoupling entry point from --load-jump-back-helper. We can
> introduce a separate option for entry point. Something like.
> 
> kexec --load-jump-back-helper --entry=$jump_back_entry
> 
> May be we can generalize the --entry so that a user can override the 
> entry point of the normal kexec image using above.

Yes. This is better. I will change it.

> > 8. Jump back to the original kernel with following shell command line:
> > 
> >    /sbin/kexec -e
> > 
> > 9. Now, you are in the original kernel again. You can read/write the
> >    memory image of kexeced kernel via /proc/kimgcore.
> > 
> > 10. You can jump between the original kernel and kexeced kernel as you
> >     want to via the following shell command line:
> > 
> >     /sbin/kexec -e
> > 
> > 
> > Known issues:
> > 
> > - The suspend/resume callback of device drivers are used to put
> >   devices into quiescent state. This will unnecessarily (possibly
> >   harmfully) put devices into low power state. This is intended to be
> >   solved by separating device quiesce/unquiesce callback from the
> >   device suspend/resume callback.
> > 
> > 
> > ChangeLog:
> > 
> > v8:
> > 
> > - Split kexec jump patchset from kexec based hibernation patchset.
> > 
> > - Add writing support to kimgcore. This can be used as a communication
> >   method between kexeced kernel and original kernel.
> > 
> 
> What are we communicating between two kernels using kimgcore?

An example:

1. kernel A kexec kernel B, provides vmcoreinfo in kernel command line.
2. kernel B jump back to kernel A
3. in kernel A, the memory image of kernel B is saved via /proc/kimgcore
4. system reboot, kernel C is booted
5. kernel C load the memory image of kernel B, and jump back to kernel B

In step 5, the vmcoreinfo provided in step 1 is invalid, so a
communication method is needed to tell kernel B the new one. This can be
done via amending /proc/kimgcore. For example, change the memory of the
kernel command line of kernel B.

Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Vivek Goyal]

On Sat, Dec 29, 2007 at 10:00:44AM +0800, Huang, Ying wrote:
> On Fri, 2007-12-28 at 16:33 -0500, Vivek Goyal wrote:
> > On Fri, Dec 21, 2007 at 03:33:19PM +0800, Huang, Ying wrote:
> > > This patchset provides an enhancement to kexec/kdump. It implements
> > > the following features:
> > > 
> > > - Backup/restore memory used both by the original kernel and the
> > >   kexeced kernel.
> > > 
> > > - Jumping between the original kernel and the kexeced kernel.
> > > 
> > > - Read/write memory image of the kexeced kernel in the original kernel
> > >   and write memory image of the original kernel in the kexeced
> > >   kernel. This can be used as a communication method between the
> > >   kexeced kernel and the original kernel.
> > > 
> > > 
> > > The features of this patchset can be used as follow:
> > > 
> > > - Kernel/system debug through making system snapshot. You can make
> > >   system snapshot, jump back, do some thing and make another system
> > >   snapshot.
> > > 
> > 
> > How do you differentiate between whether a core is resumable or not.
> > IOW, how do you know the generated /proc/vmcore has been generated after
> > a real crash hence can't be resumed (using krestore) or it has been 
> > generated because of hibernation/debug purposes and can be resumed?
> > 
> > I think you might have to add an extra ELF NOTE to vmcore which can help
> > decide whether kernel memory snapshot is resumable or not.
> 
> The current solution is as follow:
> 
> 1. The original kernel will set %edi to jump back entry if resumable and
> set %edi to 0 if not.
> 
> 2. The purgatory of loaded kernel will check %edi, if it is not zero,
> the string "jump_back_entry=<jump_back_entry>" will be appended to
> kernel command line parameter.
> 
> 3. In kexeced kernel, if there is "jump_back_entry=<jump_back_entry>"
> in /proc/cmdline, the previous kernel is resumable, otherwise not.
> 

Ok. But If I copy the /proc/vmcore to disk. Then I reboot the system 
and boot back into a kernel which is supposed to resume the hibernated
image. This kernel will not have any command line option jump_back_entry.
I  need to resume using krestore tool. How will a krestore tool decide that
image one wants to restore is a core file or a hibernated image?

In this context I thought an PT_NOTE might be useful.

> As for ELF NOTE, in fact, the ELF NOTE does not work for resumable
> kernel. Because the contents of source page and destination page is
> swapped during kexec, and the kernel access the destination page
> directly during parsing ELF NOTE. All memory that is swapped need to be
> accessed via the backup pages map (image->head). I think these
> information can be exchanged between two kernels via kernel command line
> or /proc/kimgcore.
> 

Why should we exchange backup page map information between two kernels?
What data has been swapped and how to restore it back should be known
to the kernel who did it. I think the memory info and address of ELF
headers we can still pass to second kernel the same way we do for
/proc/vmcore. The only difference is that purgatory shall have to modify 
the command line to reflect the new address of ELF headers just before
jumping to new kernel (because of swapping).

We can also modify the actual ELF headers in purgatory to reflect the
right data (because of swapping) and new kernel will not have to know
anything about swapping.

So I thought of following sequence.

Lets say there is production kernel A and there is helper kernel B (B
will save the hibernated image of A).

- Boot into A.
- Load kernel B using --load-preserve-context.
- This load operation can also create the Elf headers.   
- Kexec -e will start hibernation of A. Pages shall be swapped. Control
  will be transferred to purgatory.
- Purgatory will readjust the ELF headers and command line based on 
  the swapping done and jump to new kernel.
- New kernel will retrieve the elf headers and export the memory of
  hibernated kernel through /proc/vmcore.

Hence there is no need to communicate swapped pages map between two
kernels.


> > [..]
> > > 2. Build an initramfs image contains kexec-tool, or download the
> > >    pre-built initramfs image, called rootfs.gz in following text.
> > > 
> > > 3. Boot kernel compiled in step 1.
> > > 
> > > 4. Load kernel compiled in step 1 with /sbin/kexec. If You want to use
> > >    "krestore" tool, the --elf64-core-headers should be specified in
> > >    command line of /sbin/kexec. The shell command line can be as
> > >    follow:
> > > 
> > >    /sbin/kexec --load-jump-back /boot/bzImage --mem-min=0x100000
> > >      --mem-max=0xffffff --elf64-core-headers --initrd=rootfs.gz
> > > 
> > 
> > How about a different name like "--load-preserve-context". This will
> > just mean that kexec need to preserve the context while kexeing to 
> > image being loaded. Combination of --load-jump-back and
> > --load-jump-back-helper is becoming little confusing.
> 
> Yes, this is better. I will change it.
> 
> > > 5. Boot the kexeced kernel with following shell command line:
> > > 
> > >    /sbin/kexec -e
> > > 
> > > 6. The kexeced kernel will boot as normal kexec. In kexeced kernel the
> > >    memory image of original kernel can read via /proc/vmcore or
> > >    /dev/oldmem, and can be written via /dev/oldmem. You can
> > >    save/restore/modify it as you want to.
> > > 
> > 
> > Restoring a hibernated image using /dev/oldmem should be easy and I 
> > think one should be able to launch it back using --load-jump-back-helper.
> 
> Yes. I think so too. The current implementation of krestore restoring
> the hibernated image using /dev/oldmem. And the hibernated image can be
> launched using --load-jump-back-helper.
> 
> > How do you restore already kexeced kernel? For example if I got two
> > kernels A and B. A is the one which will hibernate and B will be used
> > to store the hibernated kernel. I think as per the procedure one needs
> > to first boot into kernel B and then jump back to kernel A. This will
> > make image of B available in /proc/kimgcore. If I save /proc/kimgcore
> > to disk and want to jump back to it, how do I do it? I guess I need
> > to kexec again using --load-jump-back and not restore using krestore?
> 
> The image of B is made as you said. And it can be restored as follow:
> 
> /sbin/kexec -l --args-none --flags=0x2 <kimgecore>
> /sbin/kexec -e
> 
> That is, the image of B is loaded as a ordinary ELF file. A option
> to /sbin/kexec named --flags are added to specify the
> KEXEC_PRESERVE_CONTEXT flags for sys_kexec_load. This has been tested.

Shouldn't we be able to load this image using --load-preseve-context? Why
to create another otion --flags.

So there seems to be two different kind of load options for resuming
for two different kind of images.

- For /proc/vmcore kind of images use krestore (which will use
  --load-jump-back-helper)

- For /proc/kimgcore kind of images use normal kexec -l.

This is little confusing. How will one differentiate between two kind
of files? In fact the semantics between two kind of files are not very
clear. We need to do some kind of unification here and make the whole
thing somewhat simple.


> 
> > > 7. Prepare jumping back from kexeced kernel with following shell
> > >    command lines:
> > > 
> > >    jump_back_entry=`cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '='`
> > >    /sbin/kexec --load-jump-back-helper=$jump_back_entry
> > > 
> > 
> > How about decoupling entry point from --load-jump-back-helper. We can
> > introduce a separate option for entry point. Something like.
> > 
> > kexec --load-jump-back-helper --entry=$jump_back_entry
> > 
> > May be we can generalize the --entry so that a user can override the 
> > entry point of the normal kexec image using above.
> 
> Yes. This is better. I will change it.
> 
> > > 8. Jump back to the original kernel with following shell command line:
> > > 
> > >    /sbin/kexec -e
> > > 
> > > 9. Now, you are in the original kernel again. You can read/write the
> > >    memory image of kexeced kernel via /proc/kimgcore.
> > > 
> > > 10. You can jump between the original kernel and kexeced kernel as you
> > >     want to via the following shell command line:
> > > 
> > >     /sbin/kexec -e
> > > 
> > > 
> > > Known issues:
> > > 
> > > - The suspend/resume callback of device drivers are used to put
> > >   devices into quiescent state. This will unnecessarily (possibly
> > >   harmfully) put devices into low power state. This is intended to be
> > >   solved by separating device quiesce/unquiesce callback from the
> > >   device suspend/resume callback.
> > > 
> > > 
> > > ChangeLog:
> > > 
> > > v8:
> > > 
> > > - Split kexec jump patchset from kexec based hibernation patchset.
> > > 
> > > - Add writing support to kimgcore. This can be used as a communication
> > >   method between kexeced kernel and original kernel.
> > > 
> > 
> > What are we communicating between two kernels using kimgcore?
> 
> An example:
> 
> 1. kernel A kexec kernel B, provides vmcoreinfo in kernel command line.

I think vmcoreinfo is stored in a PT_NOTE and passed to second kernel
parses it while parsing all ELF headers? No separate ptr or anything
else is passed to second kernel on command line.

> 2. kernel B jump back to kernel A
> 3. in kernel A, the memory image of kernel B is saved via /proc/kimgcore
> 4. system reboot, kernel C is booted
> 5. kernel C load the memory image of kernel B, and jump back to kernel B
> 
> In step 5, the vmcoreinfo provided in step 1 is invalid, so a
> communication method is needed to tell kernel B the new one. This can be
> done via amending /proc/kimgcore. For example, change the memory of the
> kernel command line of kernel B.

I think modifying a kernel's data structures externally without kernel
knowing it is dangerous. 

I would think that initially we can keep it simple and not suppot
resuming. Resuming is little tough as you need to make sure kernel's
complete env has been restored back. But in this case, we are playing
with resume env, for example, changing of vmcoreinfo etc.

Capability to resume the already booted kernel (Kernel B here), creates
a coupling with the kernel it has booted with in the past (Kernel A) and
that's why you end up modifying various things once you swith to B 
from C. I think we should boot B fresh all the time to save the hibernated
image of kernel A or C (Though it will take more time for hibernation).
It keeps things simple.

If you still want to support resuming the kernel, then I would think that in
resume path kernel should re-initialize/re-construct /proc/vmcore
(parse vmcoreinfo again) etc, instead of modifying the image of loaded kernel
through /proc/kimgcore interface.

We can probably think of a setup page where all the needed data is put and
this page is used as a communication medium between two kernels (Something
like the way, a page of data is passed between boot-loader and kernel to
communicate the info like command line etc).

So in the above example, kernel C will pass some data to kernel B (on
setup page). This data will also include the pointer to new ELF headers
(which also includes vmcoreinfo PT_NOTE). Resume path of kernel B
will parse this data and re-initialize vmcore accordingly.

Thanks
Vivek

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Mon, 2007-12-31 at 14:26 -0500, Vivek Goyal wrote:
[...]
> Ok. But If I copy the /proc/vmcore to disk. Then I reboot the system 
> and boot back into a kernel which is supposed to resume the hibernated
> image. This kernel will not have any command line option jump_back_entry.
> I  need to resume using krestore tool. How will a krestore tool decide that
> image one wants to restore is a core file or a hibernated image?
> 
> In this context I thought an PT_NOTE might be useful.

In current implementation, when saving the memory image of hibernated
kernel in hibernating kernel, the jump back entry from kernel command
line should be saved at the same time, or the jump back entry from
kernel command line can be saved as entry point of ELF hibernated image
file by a modified makedumpfile. That is, 

- method 1:

jbe=`cat /proc/cmdline | tr -d ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' -f 2`
cp /proc/vmcore .
echo $jbe > kexec_jump_back_entry

- method 2:

jbe=`cat /proc/cmdline | tr -d ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' -f 2`
makedumpfile <other options> -j $jbe /proc/vmcore <image file>


Method 2 is better, because "jump back entry" is the very entry point of
the hibernated image. The vmcore implementation can be changed to set
entry point of /proc/vmcore if "kexec_jump_back_entry" kernel command
line option is present. I did not add this to kernel because it can be
done by user space tool "makedumpfile", and one of the guideline is to
keep kernel part as simple as possible.

With method 2, the ordinary "core file" and "hibernated image" can be
distinguished by the "entry point".


The "jump back entry" can be saved with PT_NOTE too. And more
information can be saved with PT_NOTE. I have ever defined another
mechanism to pass information between two kernels too. I will describe
it later in this mail.

> > As for ELF NOTE, in fact, the ELF NOTE does not work for resumable
> > kernel. Because the contents of source page and destination page is
> > swapped during kexec, and the kernel access the destination page
> > directly during parsing ELF NOTE. All memory that is swapped need to be
> > accessed via the backup pages map (image->head). I think these
> > information can be exchanged between two kernels via kernel command line
> > or /proc/kimgcore.
> > 
> 
> Why should we exchange backup page map information between two kernels?
> What data has been swapped and how to restore it back should be known
> to the kernel who did it. I think the memory info and address of ELF
> headers we can still pass to second kernel the same way we do for
> /proc/vmcore. The only difference is that purgatory shall have to modify 
> the command line to reflect the new address of ELF headers just before
> jumping to new kernel (because of swapping).
> 
> We can also modify the actual ELF headers in purgatory to reflect the
> right data (because of swapping) and new kernel will not have to know
> anything about swapping.
> 
> So I thought of following sequence.
> 
> Lets say there is production kernel A and there is helper kernel B (B
> will save the hibernated image of A).
> 
> - Boot into A.
> - Load kernel B using --load-preserve-context.
> - This load operation can also create the Elf headers.   
> - Kexec -e will start hibernation of A. Pages shall be swapped. Control
>   will be transferred to purgatory.
> - Purgatory will readjust the ELF headers and command line based on 
>   the swapping done and jump to new kernel.
> - New kernel will retrieve the elf headers and export the memory of
>   hibernated kernel through /proc/vmcore.
> 
> Hence there is no need to communicate swapped pages map between two
> kernels.

One original page will map to one backup page (the contents of original
page and backup page will be swapped). It is possible that the entries
number of map equals the number of backup pages.

If the helper kernel B uses 16M memory, there will be 4096 backup pages.
One PT_LOAD header is needed for each backup page to record the map
between the original page to the backup page. So, the memory needed for
PT_LOAD headers will be 32 * 4k = 128k.

If the size of PT_LOAD headers and the "readability" of headers
information of /proc/vmcore is not a big issue, I think this is a good
idea.

In addition to revising the ELF headers in purgatory, the /proc/kimgcore
interface can also be used to revise the ELF headers in kernel A. But
this needs to export backup pages map information to user space (another
sysfs/procfs file?).

[...]
> > The image of B is made as you said. And it can be restored as
> follow:
> > 
> > /sbin/kexec -l --args-none --flags=0x2 <kimgecore>
> > /sbin/kexec -e
> > 
> > That is, the image of B is loaded as a ordinary ELF file. A option
> > to /sbin/kexec named --flags are added to specify the
> > KEXEC_PRESERVE_CONTEXT flags for sys_kexec_load. This has been tested.
> 
> Shouldn't we be able to load this image using --load-preseve-context? Why
> to create another otion --flags.

Yes. This seems better. I will change it.

> So there seems to be two different kind of load options for resuming
> for two different kind of images.
> 
> - For /proc/vmcore kind of images use krestore (which will use
>   --load-jump-back-helper)
> 
> - For /proc/kimgcore kind of images use normal kexec -l.
> 
> This is little confusing. How will one differentiate between two kind
> of files? In fact the semantics between two kind of files are not very
> clear. We need to do some kind of unification here and make the whole
> thing somewhat simple.

I think there may be no difference between two kinds of files. Nothing
prevents hibernated image produced through /proc/vmcore loaded with
normal kexec -l. I have just tested it. This means we need not use
krestore to resume the hibernated kernel after enhancing /sbin/kexec
(because it uses too much anonymous memory). And, the resuming process
will be more smooth.

For completion (although may be not a big issue now):

Whether using "krestore" or normal "kexec -l" depends on the memory used
by current kernel (/proc/iomem) and memory in image file (PT_LOAD
headers). If all memory in image file is outside memory used by current
kernel, "krestore" should be used. If all memory in image file is inside
memory used by current kernel, normal "kexec -l" should be used. If
there is intersection set between memory in image file and memory of
current kernel, the image file can not be loaded.

> > 
> > An example:
> > 
> > 1. kernel A kexec kernel B, provides vmcoreinfo in kernel command line.
> 
> I think vmcoreinfo is stored in a PT_NOTE and passed to second kernel
> parses it while parsing all ELF headers? No separate ptr or anything
> else is passed to second kernel on command line.

In current implementation, the PT_NOTE does not work, so a kernel
command line can be used to pass the information. And it can be changed
for each resuming. easier than changing PT_NOTE.

> > 2. kernel B jump back to kernel A
> > 3. in kernel A, the memory image of kernel B is saved via /proc/kimgcore
> > 4. system reboot, kernel C is booted
> > 5. kernel C load the memory image of kernel B, and jump back to kernel B
> > 
> > In step 5, the vmcoreinfo provided in step 1 is invalid, so a
> > communication method is needed to tell kernel B the new one. This can be
> > done via amending /proc/kimgcore. For example, change the memory of the
> > kernel command line of kernel B.
> 
> I think modifying a kernel's data structures externally without kernel
> knowing it is dangerous. 

Yes. It is dangerous. This is not a good example. But, I think it is a
useful feature. For example, the "setup page" you proposed later can be
setup via amending /proc/kimgcore.

And, it is more useful for "invoking some code in physical mode". The
procedure is something as follow:

1. load some code executing in physical mode via kexec --load-preserve-context.
2. setup the parameters via amending /proc/kimgcore
3. execute the code in physical mode via kexec -e
4. get the result via reading /proc/kimgcore
5. setup another groups of parameters via amending /proc/kimgcore
...

> I would think that initially we can keep it simple and not suppot
> resuming. Resuming is little tough as you need to make sure kernel's
> complete env has been restored back. But in this case, we are playing
> with resume env, for example, changing of vmcoreinfo etc.
> 
> Capability to resume the already booted kernel (Kernel B here), creates
> a coupling with the kernel it has booted with in the past (Kernel A) and
> that's why you end up modifying various things once you swith to B 
> from C. I think we should boot B fresh all the time to save the hibernated
> image of kernel A or C (Though it will take more time for hibernation).
> It keeps things simple.
> 
> If you still want to support resuming the kernel, then I would think that in
> resume path kernel should re-initialize/re-construct /proc/vmcore
> (parse vmcoreinfo again) etc, instead of modifying the image of loaded kernel
> through /proc/kimgcore interface.
> 
> We can probably think of a setup page where all the needed data is put and
> this page is used as a communication medium between two kernels (Something
> like the way, a page of data is passed between boot-loader and kernel to
> communicate the info like command line etc).
> 
> So in the above example, kernel C will pass some data to kernel B (on
> setup page). This data will also include the pointer to new ELF headers
> (which also includes vmcoreinfo PT_NOTE). Resume path of kernel B
> will parse this data and re-initialize vmcore accordingly.

Yes. A communication mechanism is needed between two kernels. The "setup
page" and "PT_NOTE" earlier you proposed are two such mechanisms. It
seems that the "setup page" is more powerful. I have defined a mechanism
like "setup page" in my previous version. It is as follow:

1. The second half of "control page" is used as the "setup page". That
is, jump_back_entry + 0x800.
2. A magic number is provided in "setup page" to let krestore to
distinguish between "normal executable" and "hibernated iamge" by check
the magic number.
3. Other information such as vmcoreinfo can be added to "setup page".
4. Before one kernel jump to another kernel, the parameters are prepared
by current kernel.
5. One kernel can check the parameters of another kernel by
reading /proc/vmcore or /proc/kimgcore.
6. When memory image is saved in file. The parameters of hibernated
kernel can be check by reading memory location jump_back_entry + 0x800.

You can check the details of this mechanism in my previous patch with
title:

[PATCH 1/4 -mm] kexec based hibernation -v7 : kexec jump

The main issue of this mechanism is that: it is a kernel-to-kernel
communication mechanism, while Eric Biederman thinks we should use only
user-to-user communication mechanism. And he is not persuaded now.

Because kernel operations such as re-initialize/re-construct
the /proc/vmcore, etc are needed for kexec jump or resuming. I think a
"kernel-to-kernel" mechanism may be needed. But I don't know if Eric
Biederman will agree with this.

Best Regards,
Huang Ying

Re: [PATCH 0/3 -mm] kexec jump -v8

[Vivek Goyal]

On Thu, Jan 03, 2008 at 04:42:26PM +0800, Huang, Ying wrote:
> On Mon, 2007-12-31 at 14:26 -0500, Vivek Goyal wrote:
> [...]
> > Ok. But If I copy the /proc/vmcore to disk. Then I reboot the system 
> > and boot back into a kernel which is supposed to resume the hibernated
> > image. This kernel will not have any command line option jump_back_entry.
> > I  need to resume using krestore tool. How will a krestore tool decide that
> > image one wants to restore is a core file or a hibernated image?
> > 
> > In this context I thought an PT_NOTE might be useful.
> 
> In current implementation, when saving the memory image of hibernated
> kernel in hibernating kernel, the jump back entry from kernel command
> line should be saved at the same time, or the jump back entry from
> kernel command line can be saved as entry point of ELF hibernated image
> file by a modified makedumpfile. That is, 
> 
> - method 1:
> 
> jbe=`cat /proc/cmdline | tr -d ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' -f 2`
> cp /proc/vmcore .
> echo $jbe > kexec_jump_back_entry
> 
> - method 2:
> 
> jbe=`cat /proc/cmdline | tr -d ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' -f 2`
> makedumpfile <other options> -j $jbe /proc/vmcore <image file>
> 
> 
> Method 2 is better, because "jump back entry" is the very entry point of
> the hibernated image. The vmcore implementation can be changed to set
> entry point of /proc/vmcore if "kexec_jump_back_entry" kernel command
> line option is present. I did not add this to kernel because it can be
> done by user space tool "makedumpfile", and one of the guideline is to
> keep kernel part as simple as possible.
> 

I think storing the jump back entry in ELF header should be better.

> With method 2, the ordinary "core file" and "hibernated image" can be
> distinguished by the "entry point".
> 

Hmm... That's one possible way of differentiating between two kind of
files.

> 
> The "jump back entry" can be saved with PT_NOTE too. And more
> information can be saved with PT_NOTE. I have ever defined another
> mechanism to pass information between two kernels too. I will describe
> it later in this mail.
> 

Actually I meant to store a string in PT_NOTE, something like "RESUMABLE"
or "HIBERNATED" etc to differentiate between a normal core file and a core
file which can be resumed. Entry point should be stored in ELF header
only.


> > > As for ELF NOTE, in fact, the ELF NOTE does not work for resumable
> > > kernel. Because the contents of source page and destination page is
> > > swapped during kexec, and the kernel access the destination page
> > > directly during parsing ELF NOTE. All memory that is swapped need to be
> > > accessed via the backup pages map (image->head). I think these
> > > information can be exchanged between two kernels via kernel command line
> > > or /proc/kimgcore.
> > > 
> > 
> > Why should we exchange backup page map information between two kernels?
> > What data has been swapped and how to restore it back should be known
> > to the kernel who did it. I think the memory info and address of ELF
> > headers we can still pass to second kernel the same way we do for
> > /proc/vmcore. The only difference is that purgatory shall have to modify 
> > the command line to reflect the new address of ELF headers just before
> > jumping to new kernel (because of swapping).
> > 
> > We can also modify the actual ELF headers in purgatory to reflect the
> > right data (because of swapping) and new kernel will not have to know
> > anything about swapping.
> > 
> > So I thought of following sequence.
> > 
> > Lets say there is production kernel A and there is helper kernel B (B
> > will save the hibernated image of A).
> > 
> > - Boot into A.
> > - Load kernel B using --load-preserve-context.
> > - This load operation can also create the Elf headers.   
> > - Kexec -e will start hibernation of A. Pages shall be swapped. Control
> >   will be transferred to purgatory.
> > - Purgatory will readjust the ELF headers and command line based on 
> >   the swapping done and jump to new kernel.
> > - New kernel will retrieve the elf headers and export the memory of
> >   hibernated kernel through /proc/vmcore.
> > 
> > Hence there is no need to communicate swapped pages map between two
> > kernels.
> 
> One original page will map to one backup page (the contents of original
> page and backup page will be swapped). It is possible that the entries
> number of map equals the number of backup pages.
> 
> If the helper kernel B uses 16M memory, there will be 4096 backup pages.
> One PT_LOAD header is needed for each backup page to record the map
> between the original page to the backup page. So, the memory needed for
> PT_LOAD headers will be 32 * 4k = 128k.
> 
> If the size of PT_LOAD headers and the "readability" of headers
> information of /proc/vmcore is not a big issue, I think this is a good
> idea.
> 

Ok. So if backup page allocation is not contiguous then we will end
up with one PT_LOAD header per backup page, hence large number of PT_LOAD
headers. I know it is little ugly, but still think that it is a better
way.   

> In addition to revising the ELF headers in purgatory, the /proc/kimgcore
> interface can also be used to revise the ELF headers in kernel A. But
> this needs to export backup pages map information to user space (another
> sysfs/procfs file?).

This is kind of two step load process. Load image, then export backup
page info to user space and then readjust the headers through
/proc/kimgcore. I think purgatory doing it is cleaner.
 
> 
> [...]
> > > The image of B is made as you said. And it can be restored as
> > follow:
> > > 
> > > /sbin/kexec -l --args-none --flags=0x2 <kimgecore>
> > > /sbin/kexec -e
> > > 
> > > That is, the image of B is loaded as a ordinary ELF file. A option
> > > to /sbin/kexec named --flags are added to specify the
> > > KEXEC_PRESERVE_CONTEXT flags for sys_kexec_load. This has been tested.
> > 
> > Shouldn't we be able to load this image using --load-preseve-context? Why
> > to create another otion --flags.
> 
> Yes. This seems better. I will change it.
> 
> > So there seems to be two different kind of load options for resuming
> > for two different kind of images.
> > 
> > - For /proc/vmcore kind of images use krestore (which will use
> >   --load-jump-back-helper)
> > 
> > - For /proc/kimgcore kind of images use normal kexec -l.
> > 
> > This is little confusing. How will one differentiate between two kind
> > of files? In fact the semantics between two kind of files are not very
> > clear. We need to do some kind of unification here and make the whole
> > thing somewhat simple.
> 
> I think there may be no difference between two kinds of files. Nothing
> prevents hibernated image produced through /proc/vmcore loaded with
> normal kexec -l. I have just tested it. This means we need not use
> krestore to resume the hibernated kernel after enhancing /sbin/kexec
> (because it uses too much anonymous memory). And, the resuming process
> will be more smooth.
> 

Ok. So in a nutshell, any resumable image (be it is generated through
/proc/vmcore or /proc/kimgcore) can be launched in the same way  (I think
kexec --load-preserve-context). ?

In fact, if we are modifying the ELF headers in purgatory to reflect the
swapped page action, then we don't require /proc/kimgcore interface
at all?


> For completion (although may be not a big issue now):
> 
> Whether using "krestore" or normal "kexec -l" depends on the memory used
> by current kernel (/proc/iomem) and memory in image file (PT_LOAD
> headers). If all memory in image file is outside memory used by current
> kernel, "krestore" should be used. If all memory in image file is inside
> memory used by current kernel, normal "kexec -l" should be used. If
> there is intersection set between memory in image file and memory of
> current kernel, the image file can not be loaded.
> 

I think kexec should mask that difference. A user should be able to load
a resumable image either by using "kexec -l" or "kexec
--load-preserve-context" depending on whether user wants to come back to
orignal kernel in future or not. Kexec-tools should recognize the image
as resumable. Any page outside the current kernel can be written to
final location through /dev/oldmem. And any pages which overlap with the
current kernel, should be moved to destination when actual kexec 
happens (existing functionality).

This will require merging krestore and kexec user space functionality
so that resuming a hibernated image is effectively a "Kexec -l"
operation.

So a user will not worry about whether he is kexecing a fresh kernel
(bzImage or vmlinux) or resuming a already booted kernel. Kexec tools
should determine that and setup the entry point accordingly (might
require some purgatory changes to take care of transition while jumping
to resume hibernated image).


> > > 
> > > An example:
> > > 
> > > 1. kernel A kexec kernel B, provides vmcoreinfo in kernel command line.
> > 
> > I think vmcoreinfo is stored in a PT_NOTE and passed to second kernel
> > parses it while parsing all ELF headers? No separate ptr or anything
> > else is passed to second kernel on command line.
> 
> In current implementation, the PT_NOTE does not work, so a kernel
> command line can be used to pass the information. And it can be changed
> for each resuming. easier than changing PT_NOTE.
> 
> > > 2. kernel B jump back to kernel A
> > > 3. in kernel A, the memory image of kernel B is saved via /proc/kimgcore
> > > 4. system reboot, kernel C is booted
> > > 5. kernel C load the memory image of kernel B, and jump back to kernel B
> > > 
> > > In step 5, the vmcoreinfo provided in step 1 is invalid, so a
> > > communication method is needed to tell kernel B the new one. This can be
> > > done via amending /proc/kimgcore. For example, change the memory of the
> > > kernel command line of kernel B.
> > 
> > I think modifying a kernel's data structures externally without kernel
> > knowing it is dangerous. 
> 
> Yes. It is dangerous. This is not a good example. But, I think it is a
> useful feature. For example, the "setup page" you proposed later can be
> setup via amending /proc/kimgcore.
> 
> And, it is more useful for "invoking some code in physical mode". The
> procedure is something as follow:
> 
> 1. load some code executing in physical mode via kexec --load-preserve-context.
> 2. setup the parameters via amending /proc/kimgcore
> 3. execute the code in physical mode via kexec -e
> 4. get the result via reading /proc/kimgcore
> 5. setup another groups of parameters via amending /proc/kimgcore
> ...

This seems to be extended functionlity. If your focus is "Kexec based
hibernation" then I would think of initially keeping the implementation
simple and keeping patches small. Make kexec based hibernation work
and then extend functionality for other purposes.


> 
> > I would think that initially we can keep it simple and not suppot
> > resuming. Resuming is little tough as you need to make sure kernel's
> > complete env has been restored back. But in this case, we are playing
> > with resume env, for example, changing of vmcoreinfo etc.
> > 
> > Capability to resume the already booted kernel (Kernel B here), creates
> > a coupling with the kernel it has booted with in the past (Kernel A) and
> > that's why you end up modifying various things once you swith to B 
> > from C. I think we should boot B fresh all the time to save the hibernated
> > image of kernel A or C (Though it will take more time for hibernation).
> > It keeps things simple.
> > 
> > If you still want to support resuming the kernel, then I would think that in
> > resume path kernel should re-initialize/re-construct /proc/vmcore
> > (parse vmcoreinfo again) etc, instead of modifying the image of loaded kernel
> > through /proc/kimgcore interface.
> > 
> > We can probably think of a setup page where all the needed data is put and
> > this page is used as a communication medium between two kernels (Something
> > like the way, a page of data is passed between boot-loader and kernel to
> > communicate the info like command line etc).
> > 
> > So in the above example, kernel C will pass some data to kernel B (on
> > setup page). This data will also include the pointer to new ELF headers
> > (which also includes vmcoreinfo PT_NOTE). Resume path of kernel B
> > will parse this data and re-initialize vmcore accordingly.
> 
> Yes. A communication mechanism is needed between two kernels. The "setup
> page" and "PT_NOTE" earlier you proposed are two such mechanisms. It
> seems that the "setup page" is more powerful. I have defined a mechanism
> like "setup page" in my previous version. It is as follow:
> 
> 1. The second half of "control page" is used as the "setup page". That
> is, jump_back_entry + 0x800.
> 2. A magic number is provided in "setup page" to let krestore to
> distinguish between "normal executable" and "hibernated iamge" by check
> the magic number.
> 3. Other information such as vmcoreinfo can be added to "setup page".
> 4. Before one kernel jump to another kernel, the parameters are prepared
> by current kernel.
> 5. One kernel can check the parameters of another kernel by
> reading /proc/vmcore or /proc/kimgcore.
> 6. When memory image is saved in file. The parameters of hibernated
> kernel can be check by reading memory location jump_back_entry + 0x800.
> 
> You can check the details of this mechanism in my previous patch with
> title:
> 
> [PATCH 1/4 -mm] kexec based hibernation -v7 : kexec jump
> 
> The main issue of this mechanism is that: it is a kernel-to-kernel
> communication mechanism, while Eric Biederman thinks we should use only
> user-to-user communication mechanism. And he is not persuaded now.
> 
> Because kernel operations such as re-initialize/re-construct
> the /proc/vmcore, etc are needed for kexec jump or resuming. I think a
> "kernel-to-kernel" mechanism may be needed. But I don't know if Eric
> Biederman will agree with this.

Hmm... Personally I am more inclined to exchanging information between
two kernels on setup page, in a standard format (using ELF headers etc).
This information can be prepared by kexec-tools in user space and be). 
modified by purgatory (during transition to reflect the swapped pages.
Alternatively, one can modify this setup page info from user space through
some /proc/kimgcore like interface.  I prefer the first one...

Eric, what do you think abou the whole thing?

Thanks
Vivek

Re: [PATCH 0/3 -mm] kexec jump -v8

[Huang, Ying]

On Sun, 2008-01-06 at 15:49 -0500, Vivek Goyal wrote:
> > > Why should we exchange backup page map information between two kernels?
> > > What data has been swapped and how to restore it back should be known
> > > to the kernel who did it. I think the memory info and address of ELF
> > > headers we can still pass to second kernel the same way we do for
> > > /proc/vmcore. The only difference is that purgatory shall have to modify 
> > > the command line to reflect the new address of ELF headers just before
> > > jumping to new kernel (because of swapping).
> > > 
> > > We can also modify the actual ELF headers in purgatory to reflect the
> > > right data (because of swapping) and new kernel will not have to know
> > > anything about swapping.
> > >
[...]
>  
> > In addition to revising the ELF headers in purgatory, the /proc/kimgcore
> > interface can also be used to revise the ELF headers in kernel A. But
> > this needs to export backup pages map information to user space (another
> > sysfs/procfs file?).
> 
> This is kind of two step load process. Load image, then export backup
> page info to user space and then readjust the headers through
> /proc/kimgcore. I think purgatory doing it is cleaner.
[...] 
> > I think there may be no difference between two kinds of files. Nothing
> > prevents hibernated image produced through /proc/vmcore loaded with
> > normal kexec -l. I have just tested it. This means we need not use
> > krestore to resume the hibernated kernel after enhancing /sbin/kexec
> > (because it uses too much anonymous memory). And, the resuming process
> > will be more smooth.
> > 
> 
> Ok. So in a nutshell, any resumable image (be it is generated through
> /proc/vmcore or /proc/kimgcore) can be launched in the same way  (I think
> kexec --load-preserve-context). ?
> 
> In fact, if we are modifying the ELF headers in purgatory to reflect the
> swapped page action, then we don't require /proc/kimgcore interface
> at all?

But, how can purgatory get the "backup pages map" information? Which is
not available when purgatory is loaded. I think there are two methods:

Method1:

- A setup page is setup by original kernel, and the address of setup
page is passed to purgatory using a pre-defined protocol (such as in a
register). The "backup pages map" is in setup page.
- Purgatory uses the information in setup page to modify ELF headers
- kexeced kernel use ELF headers to build /proc/vmcore

Method2:

- A setup page is setup by /sbin/kexec
- The "backup pages map" information is written to purgatory
via /proc/kimgcore.
- Purgatory uses the information to modify ELF headers
- kexeced kernel use ELF headers to build /proc/vmcore


But, why not just exchanging the "backup pages map" information between
two kernels via the "setup page"? It is as follow:

- A setup page is setup by original kernel, and the address of setup
page is passed to kexeced kernel using a pre-defined protocol (such as
in a register). The "backup pages map" is in setup page.
- kexec kernel use "backup pages map" to build /proc/vmcore.

To keep it simple, another choice is not passing this "backup pages map"
information between two kernels now. The biggest issue is makedumpfile
can not exclude free pages without it, resulting in big hibernated
image. Another issue is that the information in /proc/vmcore is not
accurate, but that does not affect the hibernation implementation.

> In fact, if we are modifying the ELF headers in purgatory to reflect the
> swapped page action, then we don't require /proc/kimgcore interface
> at all?

/proc/kimgcore is mainly used to save the memory image of the kexeced
kernel. If this is not to be supported, /proc/kimgcore is not needed.
But, I think this is useful to accelerate the hibernating (no another
boot is needed for each hibernating).

> > For completion (although may be not a big issue now):
> > 
> > Whether using "krestore" or normal "kexec -l" depends on the memory used
> > by current kernel (/proc/iomem) and memory in image file (PT_LOAD
> > headers). If all memory in image file is outside memory used by current
> > kernel, "krestore" should be used. If all memory in image file is inside
> > memory used by current kernel, normal "kexec -l" should be used. If
> > there is intersection set between memory in image file and memory of
> > current kernel, the image file can not be loaded.
> > 
> 
> I think kexec should mask that difference. A user should be able to load
> a resumable image either by using "kexec -l" or "kexec
> --load-preserve-context" depending on whether user wants to come back to
> orignal kernel in future or not. Kexec-tools should recognize the image
> as resumable. Any page outside the current kernel can be written to
> final location through /dev/oldmem. And any pages which overlap with the
> current kernel, should be moved to destination when actual kexec 
> happens (existing functionality).
> 
> This will require merging krestore and kexec user space functionality
> so that resuming a hibernated image is effectively a "Kexec -l"
> operation.
> 
> So a user will not worry about whether he is kexecing a fresh kernel
> (bzImage or vmlinux) or resuming a already booted kernel. Kexec tools
> should determine that and setup the entry point accordingly (might
> require some purgatory changes to take care of transition while jumping
> to resume hibernated image).

Yes. This is a good idea. I will do it.

> > And, it is more useful for "invoking some code in physical mode". The
> > procedure is something as follow:
> > 
> > 1. load some code executing in physical mode via kexec --load-preserve-context.
> > 2. setup the parameters via amending /proc/kimgcore
> > 3. execute the code in physical mode via kexec -e
> > 4. get the result via reading /proc/kimgcore
> > 5. setup another groups of parameters via amending /proc/kimgcore
> > ...
> 
> This seems to be extended functionlity. If your focus is "Kexec based
> hibernation" then I would think of initially keeping the implementation
> simple and keeping patches small. Make kexec based hibernation work
> and then extend functionality for other purposes.

I think this is a important use case of "kexec jump" besides "kexec
based hibernation". But it can be separated from initial "kexec jump"
patchset if necessary.
 
[...]
> > The main issue of this mechanism is that: it is a kernel-to-kernel
> > communication mechanism, while Eric Biederman thinks we should use only
> > user-to-user communication mechanism. And he is not persuaded now.
> > 
> > Because kernel operations such as re-initialize/re-construct
> > the /proc/vmcore, etc are needed for kexec jump or resuming. I think a
> > "kernel-to-kernel" mechanism may be needed. But I don't know if Eric
> > Biederman will agree with this.
> 
> Hmm... Personally I am more inclined to exchanging information between
> two kernels on setup page, in a standard format (using ELF headers etc).
> This information can be prepared by kexec-tools in user space and be). 
> modified by purgatory (during transition to reflect the swapped pages.
> Alternatively, one can modify this setup page info from user space through
> some /proc/kimgcore like interface.  I prefer the first one...

Some information (such as "backup pages map") is not available
when /sbin/kexec is executed. So there should be a method to pass such
information to purgatory from original kernel. So why not change the
information between two kernel via "setup page" directly (need not
purgatory).

Best Regards,
Huang Ying