Re: [PATCH RFCv3 0/8] toolstack-based approach to pvhvm guest kexec

[Vitaly Kuznetsov <vkuznets@redhat.com>]

Vitaly Kuznetsov <vkuznets@redhat.com> writes:

> Changes from RFC/WIPv2:
>
> Here is a slightly different approach to memory reassignment. Instead of
> introducing new (and very doubtful) XENMEM_transfer operation introduce
> simple XEN_DOMCTL_set_recipient operation and do everything in free_domheap_pages()
> handler utilizing normal domain destroy path. This is better because:
> - The approach is general-enough
> - All memory pages are usually being freed when the domain is destroyed
> - No special grants handling required
> - Better supportability
>
> With regards to PV:
> Though XEN_DOMCTL_set_recipient works for both PV and HVM this patchset does not
> bring PV kexec/kdump support. xc_domain_soft_reset() is limited to work with HVM
> domains only. The main reason for that is: it is (in theory) possible to save p2m
> and rebuild them with the new domain but that would only allow us to resume execution
> from where we stopped. If we want to execute new kernel we need to build the same
> kernel/initrd/bootstrap_pagetables/... structure we build to boot PV domain initially.
> That however would destroy the original domain's memory thus making kdump impossible.
> To make everything work additional support from kexec userspace/linux kernel is
> required and I'm not sure it makes sense to implement all this stuff in the light of
> PVH.

I don't want to create noise but I didn't get many comments on this RFC
(the only one belongs to David). In case there are no comments on RFC
(design) level I can rebase and send first non-RFC.

I understand 4.5 release is the main focus atm though I would greatly
appreciate some feedback.

Thanks,

>
> Original description:
>
> When a PVHVM linux guest performs kexec there are lots of things which
> require taking care of:
> - shared info, vcpu_info
> - grants
> - event channels
> - ...
> Instead of taking care of all these things we can rebuild the domain
> performing kexec from scratch doing so-called soft-reboot.
>
> The idea was suggested by David Vrabel, Jan Beulich, and Konrad Rzeszutek Wilk.
>
> P.S. The patch series can be tested with PVHVM Linux guest with the following
> modifications:
>
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index c0cb11f..33c5cdd 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -33,6 +33,10 @@
>  #include <linux/memblock.h>
>  #include <linux/edd.h>
>
> +#ifdef CONFIG_KEXEC
> +#include <linux/kexec.h>
> +#endif
> +
>  #include <xen/xen.h>
>  #include <xen/events.h>
>  #include <xen/interface/xen.h>
> @@ -1810,6 +1814,22 @@ static struct notifier_block xen_hvm_cpu_notifier = {
>    .notifier_call   = xen_hvm_cpu_notify,
>  };
>
> +#ifdef CONFIG_KEXEC
> +static void xen_pvhvm_kexec_shutdown(void)
> +{
> +	native_machine_shutdown();
> +	if (kexec_in_progress) {
> +	   xen_reboot(SHUTDOWN_soft_reset);
> +	   }
> +}
> +
> +static void xen_pvhvm_crash_shutdown(struct pt_regs *regs)
> +{
> +	native_machine_crash_shutdown(regs);
> +	xen_reboot(SHUTDOWN_soft_reset);
> +}
> +#endif
> +
>  static void __init xen_hvm_guest_init(void)
>  {
> 	init_hvm_pv_info();
> @@ -1826,6 +1846,10 @@ static void __init xen_hvm_guest_init(void)
>    x86_init.irqs.intr_init = xen_init_IRQ;
>    xen_hvm_init_time_ops();
>    xen_hvm_init_mmu_ops();
> +#ifdef CONFIG_KEXEC
> +	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
> +	machine_ops.crash_shutdown = xen_pvhvm_crash_shutdown;
> +#endif
>  }
>
>  static bool xen_nopv = false;
> diff --git a/include/xen/interface/sched.h b/include/xen/interface/sched.h
> index 9ce0839..b5942a8 100644
> --- a/include/xen/interface/sched.h
> +++ b/include/xen/interface/sched.h
> @@ -107,5 +107,6 @@ struct sched_watchdog {
>  #define SHUTDOWN_suspend    2  /* Clean up, save suspend info, kill.         */
>  #define SHUTDOWN_crash      3  /* Tell controller we've crashed.             */
>  #define SHUTDOWN_watchdog   4  /* Restart because watchdog time expired.     */
> +#define SHUTDOWN_soft_reset 5  /* Soft-reset for kexec.                      */
>
>  #endif /* __XEN_PUBLIC_SCHED_H__ */
>
> Vitaly Kuznetsov (8):
>   xen: introduce SHUTDOWN_soft_reset shutdown reason
>   libxl: support SHUTDOWN_soft_reset shutdown reason
>   xen: introduce XEN_DOMCTL_set_recipient
>   libxc: support XEN_DOMCTL_set_recipient
>   libxl: add libxl__domain_soft_reset_destroy_old()
>   libxc: introduce soft reset for HVM domains
>   libxl: soft reset support
>   xsm: add XEN_DOMCTL_set_recipient support
>
>  tools/libxc/Makefile                |   1 +
>  tools/libxc/xc_domain.c             |  10 +
>  tools/libxc/xc_domain_soft_reset.c  | 394 ++++++++++++++++++++++++++++++++++++
>  tools/libxc/xenctrl.h               |  14 ++
>  tools/libxc/xenguest.h              |  20 ++
>  tools/libxl/libxl.c                 |  32 ++-
>  tools/libxl/libxl.h                 |   6 +
>  tools/libxl/libxl_create.c          | 103 +++++++++-
>  tools/libxl/libxl_internal.h        |   8 +
>  tools/libxl/libxl_types.idl         |   1 +
>  tools/libxl/xl_cmdimpl.c            |  24 ++-
>  tools/python/xen/lowlevel/xl/xl.c   |   1 +
>  xen/common/domain.c                 |   3 +
>  xen/common/domctl.c                 |  35 ++++
>  xen/common/page_alloc.c             |  26 ++-
>  xen/common/shutdown.c               |   7 +
>  xen/include/public/domctl.h         |  19 ++
>  xen/include/public/sched.h          |   3 +-
>  xen/include/xen/sched.h             |   1 +
>  xen/include/xsm/dummy.h             |   6 +
>  xen/include/xsm/xsm.h               |   6 +
>  xen/xsm/dummy.c                     |   1 +
>  xen/xsm/flask/hooks.c               |  17 ++
>  xen/xsm/flask/policy/access_vectors |  10 +
>  24 files changed, 726 insertions(+), 22 deletions(-)
>  create mode 100644 tools/libxc/xc_domain_soft_reset.c

-- 
  Vitaly