is kexec on Xen domU possible?

is kexec on Xen domU possible?

[Greg KH]

Hi all,

I was messing around with kexec and it seems to work just fine on KVM,
but for Xen domU images, it doesn't work at all.  Daniel, I saw some
patches from you back in September 2012 for adding this support for
Dom0, but they don't seem to have gone into the kernel (but other
patches went into kexec-tools at that time.)  You mention that domU
support is "easy" after your patches go in, is that because Dom0 needs
to support this, or is it something specific to only domU?

Also, what's the status of those patches for the kernel, I don't see
them reposted anywhere, did you drop them?

thanks,

greg k-h

Re: is kexec on Xen domU possible?

[Daniel Kiper]

On Thu, Jul 18, 2013 at 07:06:03PM -0700, Greg KH wrote:
> Hi all,
>
> I was messing around with kexec and it seems to work just fine on KVM,
> but for Xen domU images, it doesn't work at all.  Daniel, I saw some
> patches from you back in September 2012 for adding this support for
> Dom0, but they don't seem to have gone into the kernel (but other

At first I was going use existing in Xen kexec implementation for Dom0.
However, after some discussion on Xen-devel and LKML we stated that
this implementation is completely broken and should be rewriten. David
Vrabel from Citrix wrote new kexec implementation for Xen which does
not relay on Linux kernel. I hope that it will be included in Xen 4.4 release.

> patches went into kexec-tools at that time.)  You mention that domU

These are mostly fixes which were needed for planned Xen kexec support.
IIRC, they are also needed for systems using ancient Xen Linux Kernel 2.18.
However, most of this implementation will be replaced by new one written
by David Vrabel. It will contain support only for new Xen Dom0 kexec implentation.

> support is "easy" after your patches go in, is that because Dom0 needs
> to support this, or is it something specific to only domU?

In case of domU we should consider following cases:
  - PV guests: there is no support for kexec at this time;
    Once I wrote an implementatation for that type of guests
    for one company but according to our agreement I could not
    publish this code; However, I could use it as a base for
    publicly available kexec implementation; Currently, I do
    not have any plans to work on this due to some more important
    stuff to do; However, question about kexec support for PV
    guests is raised from time to time and maybe this issue
    will be much more important than others once,
  - HVM guests: kexec should work without any issue,
  - PVonHVM guests: IIRC, there were some issues with PV
    drivers but they were fixed some time ago by patches
    posted by Olaf Hering,
  - PVH guests: those type of guests are not available in Xen
    current releases yet; However, Konrad Wilk done some preliminary
    work on kexec support but there are still some issues to resolve.

I do not know what are you trying to do but if you would like
to get some crash dumps there is also another solution to that.
You could use xm/xl dump-core from Dom0 to get dumps of domU memory.

> Also, what's the status of those patches for the kernel, I don't see
> them reposted anywhere, did you drop them?

They were dropped. Please look above for details.

I hope that helps.

Daniel

Re: is kexec on Xen domU possible?

[Konrad Rzeszutek Wilk]

Hey Daniel,

Thank you for the nice summary.

>   - PVonHVM guests: IIRC, there were some issues with PV
>     drivers but they were fixed some time ago by patches
>     posted by Olaf Hering,

Which had to be reverted b/c they caused regressions with
32-bit guests during migration. But Greg if you are just looking
to do it on a guest without migration then cherrypicking
 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f
("xen PVonHVM: use E820_Reserved area for shared_info")
should do it for it.

Olaf, you wouldn't have any time to look at this some more now
that Xen 4.3 is out?

Re: is kexec on Xen domU possible?

[Brandon Philips]

On Fri, Jul 19, 2013 at 6:18 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
>   - PV guests: there is no support for kexec at this time;
>     Once I wrote an implementatation for that type of guests
>     for one company but according to our agreement I could not
>     publish this code; However, I could use it as a base for
>     publicly available kexec implementation; Currently, I do
>     not have any plans to work on this due to some more important
>     stuff to do; However, question about kexec support for PV
>     guests is raised from time to time and maybe this issue
>     will be much more important than others once,

To do kexec on PV guests is there a dependency on support from the dom0?

e.g. if we recreated your patches and merged them into 3.12 and booted
on a XenServer from a few years ago would it work?

What I want to do is use kexec on existing public cloud providers,
that use Xen, like AWS or Rackspace.

Thanks,

Brandon

Re: is kexec on Xen domU possible?

[Greg KH]

On Fri, Jul 19, 2013 at 03:18:19PM +0200, Daniel Kiper wrote:
> > support is "easy" after your patches go in, is that because Dom0 needs
> > to support this, or is it something specific to only domU?
> 
> In case of domU we should consider following cases:
>   - PV guests: there is no support for kexec at this time;
>     Once I wrote an implementatation for that type of guests
>     for one company but according to our agreement I could not
>     publish this code; However, I could use it as a base for
>     publicly available kexec implementation; Currently, I do
>     not have any plans to work on this due to some more important
>     stuff to do; However, question about kexec support for PV
>     guests is raised from time to time and maybe this issue
>     will be much more important than others once,
>   - HVM guests: kexec should work without any issue,
>   - PVonHVM guests: IIRC, there were some issues with PV
>     drivers but they were fixed some time ago by patches
>     posted by Olaf Hering,
>   - PVH guests: those type of guests are not available in Xen
>     current releases yet; However, Konrad Wilk done some preliminary
>     work on kexec support but there are still some issues to resolve.
> 
> I do not know what are you trying to do but if you would like
> to get some crash dumps there is also another solution to that.
> You could use xm/xl dump-core from Dom0 to get dumps of domU memory.

As Brandon said, we were trying to use kexec in a PV guest in domU to
run another kernel.  I had assumed this wouldn't need support from dom0.
As you have implemented this in the past, did you need to change dom0 in
order to achieve this, and if so, why?

The errors that the kexec tools seem to run into is finding the memory
to place the new kernel into, is that just an issue that PV guests
aren't given enough kernel memory in which to replicate themselves from
dom0?

thanks,

greg k-h

Re: is kexec on Xen domU possible?

[Daniel Kiper]

On Fri, Jul 19, 2013 at 07:58:40AM -0700, Brandon Philips wrote:
> On Fri, Jul 19, 2013 at 6:18 AM, Daniel Kiper <daniel.kiper@oracle.com> wrote:
> >   - PV guests: there is no support for kexec at this time;
> >     Once I wrote an implementatation for that type of guests
> >     for one company but according to our agreement I could not
> >     publish this code; However, I could use it as a base for
> >     publicly available kexec implementation; Currently, I do
> >     not have any plans to work on this due to some more important
> >     stuff to do; However, question about kexec support for PV
> >     guests is raised from time to time and maybe this issue
> >     will be much more important than others once,
>
> To do kexec on PV guests is there a dependency on support from the dom0?

No.

> e.g. if we recreated your patches and merged them into 3.12 and booted
> on a XenServer from a few years ago would it work?

No.

> What I want to do is use kexec on existing public cloud providers,
> that use Xen, like AWS or Rackspace.

Look for HVM or PVonHVM guests support in this providers.
Sadly, there is no support for kexec on PV guests.

Daniel

Re: is kexec on Xen domU possible?

[Daniel Kiper]

On Fri, Jul 19, 2013 at 08:12:43AM -0700, Greg KH wrote:
> On Fri, Jul 19, 2013 at 03:18:19PM +0200, Daniel Kiper wrote:
> > > support is "easy" after your patches go in, is that because Dom0 needs
> > > to support this, or is it something specific to only domU?
> >
> > In case of domU we should consider following cases:
> >   - PV guests: there is no support for kexec at this time;
> >     Once I wrote an implementatation for that type of guests
> >     for one company but according to our agreement I could not
> >     publish this code; However, I could use it as a base for
> >     publicly available kexec implementation; Currently, I do
> >     not have any plans to work on this due to some more important
> >     stuff to do; However, question about kexec support for PV
> >     guests is raised from time to time and maybe this issue
> >     will be much more important than others once,
> >   - HVM guests: kexec should work without any issue,
> >   - PVonHVM guests: IIRC, there were some issues with PV
> >     drivers but they were fixed some time ago by patches
> >     posted by Olaf Hering,
> >   - PVH guests: those type of guests are not available in Xen
> >     current releases yet; However, Konrad Wilk done some preliminary
> >     work on kexec support but there are still some issues to resolve.
> >
> > I do not know what are you trying to do but if you would like
> > to get some crash dumps there is also another solution to that.
> > You could use xm/xl dump-core from Dom0 to get dumps of domU memory.
>
> As Brandon said, we were trying to use kexec in a PV guest in domU to
> run another kernel.  I had assumed this wouldn't need support from dom0.

You are right.

> As you have implemented this in the past, did you need to change dom0 in
> order to achieve this, and if so, why?

It was strong requirement to not change anything in hypervisor or dom0.
I succeeded to do that but it requires changes in kernel and kexec-tools.

> The errors that the kexec tools seem to run into is finding the memory
> to place the new kernel into, is that just an issue that PV guests
> aren't given enough kernel memory in which to replicate themselves from
> dom0?

There are a lot of differences between baremetal machines and PV guests.
For example you are not able to do identity mapping per se in PV guests.
Arguments to new kernel are passed in completely different way. etc.

Daniel

Re: is kexec on Xen domU possible?

[Olaf Hering]

On Fri, Jul 19, Konrad Rzeszutek Wilk wrote:

> Olaf, you wouldn't have any time to look at this some more now
> that Xen 4.3 is out?

Currently no time to continue work on kexec for PVonHVM, sorry.

Olaf

Re: is kexec on Xen domU possible?

[Greg KH]

On Fri, Jul 19, 2013 at 08:32:35PM +0200, Daniel Kiper wrote:
> On Fri, Jul 19, 2013 at 08:12:43AM -0700, Greg KH wrote:
> > On Fri, Jul 19, 2013 at 03:18:19PM +0200, Daniel Kiper wrote:
> > > > support is "easy" after your patches go in, is that because Dom0 needs
> > > > to support this, or is it something specific to only domU?
> > >
> > > In case of domU we should consider following cases:
> > >   - PV guests: there is no support for kexec at this time;
> > >     Once I wrote an implementatation for that type of guests
> > >     for one company but according to our agreement I could not
> > >     publish this code; However, I could use it as a base for
> > >     publicly available kexec implementation; Currently, I do
> > >     not have any plans to work on this due to some more important
> > >     stuff to do; However, question about kexec support for PV
> > >     guests is raised from time to time and maybe this issue
> > >     will be much more important than others once,
> > >   - HVM guests: kexec should work without any issue,
> > >   - PVonHVM guests: IIRC, there were some issues with PV
> > >     drivers but they were fixed some time ago by patches
> > >     posted by Olaf Hering,
> > >   - PVH guests: those type of guests are not available in Xen
> > >     current releases yet; However, Konrad Wilk done some preliminary
> > >     work on kexec support but there are still some issues to resolve.
> > >
> > > I do not know what are you trying to do but if you would like
> > > to get some crash dumps there is also another solution to that.
> > > You could use xm/xl dump-core from Dom0 to get dumps of domU memory.
> >
> > As Brandon said, we were trying to use kexec in a PV guest in domU to
> > run another kernel.  I had assumed this wouldn't need support from dom0.
> 
> You are right.
> 
> > As you have implemented this in the past, did you need to change dom0 in
> > order to achieve this, and if so, why?
> 
> It was strong requirement to not change anything in hypervisor or dom0.
> I succeeded to do that but it requires changes in kernel and kexec-tools.
> 
> > The errors that the kexec tools seem to run into is finding the memory
> > to place the new kernel into, is that just an issue that PV guests
> > aren't given enough kernel memory in which to replicate themselves from
> > dom0?
> 
> There are a lot of differences between baremetal machines and PV guests.
> For example you are not able to do identity mapping per se in PV guests.
> Arguments to new kernel are passed in completely different way. etc.

Ok, thanks for confirming that it is possible, but doesn't currently
work for pv guests.

greg k-h

Re: is kexec on Xen domU possible?

[H. Peter Anvin]

On 07/19/2013 12:14 PM, Greg KH wrote:
>>
>>> The errors that the kexec tools seem to run into is finding the memory
>>> to place the new kernel into, is that just an issue that PV guests
>>> aren't given enough kernel memory in which to replicate themselves from
>>> dom0?
>>
>> There are a lot of differences between baremetal machines and PV guests.
>> For example you are not able to do identity mapping per se in PV guests.
>> Arguments to new kernel are passed in completely different way. etc.
> 
> Ok, thanks for confirming that it is possible, but doesn't currently
> work for pv guests.
> 

Also, in any virtualized environment the hypervisor can do a better job
for things like kdump, simply because it can provide two things that are
otherwise hard to do:

1. a known-good system state;
2. a known-clean kdump image.

As such, I do encourage the virtualization people to (also) develop
hypervisor-*aware* solutions for these kinds of things.

	-hpa

Re: is kexec on Xen domU possible?

[Daniel Kiper]

On Fri, Jul 19, 2013 at 01:58:05PM -0700, H. Peter Anvin wrote:
> On 07/19/2013 12:14 PM, Greg KH wrote:
> >>
> >>> The errors that the kexec tools seem to run into is finding the memory
> >>> to place the new kernel into, is that just an issue that PV guests
> >>> aren't given enough kernel memory in which to replicate themselves from
> >>> dom0?
> >>
> >> There are a lot of differences between baremetal machines and PV guests.
> >> For example you are not able to do identity mapping per se in PV guests.
> >> Arguments to new kernel are passed in completely different way. etc.
> >
> > Ok, thanks for confirming that it is possible, but doesn't currently
> > work for pv guests.
> >
>
> Also, in any virtualized environment the hypervisor can do a better job
> for things like kdump, simply because it can provide two things that are
> otherwise hard to do:
>
> 1. a known-good system state;
> 2. a known-clean kdump image.
>
> As such, I do encourage the virtualization people to (also) develop
> hypervisor-*aware* solutions for these kinds of things.

In general I agree but if you could not change hypervisor
and/or dom0 (e.g. you are using cloud providers which are
stick to old versions of Xen) then you have no choice.

Daniel

Re: is kexec on Xen domU possible?

[Eric W. Biederman]

Daniel Kiper <daniel.kiper@oracle.com> writes:

> On Fri, Jul 19, 2013 at 01:58:05PM -0700, H. Peter Anvin wrote:
>> On 07/19/2013 12:14 PM, Greg KH wrote:
>> >>
>> >>> The errors that the kexec tools seem to run into is finding the memory
>> >>> to place the new kernel into, is that just an issue that PV guests
>> >>> aren't given enough kernel memory in which to replicate themselves from
>> >>> dom0?
>> >>
>> >> There are a lot of differences between baremetal machines and PV guests.
>> >> For example you are not able to do identity mapping per se in PV guests.
>> >> Arguments to new kernel are passed in completely different way. etc.
>> >
>> > Ok, thanks for confirming that it is possible, but doesn't currently
>> > work for pv guests.
>> >
>>
>> Also, in any virtualized environment the hypervisor can do a better job
>> for things like kdump, simply because it can provide two things that are
>> otherwise hard to do:
>>
>> 1. a known-good system state;
>> 2. a known-clean kdump image.
>>
>> As such, I do encourage the virtualization people to (also) develop
>> hypervisor-*aware* solutions for these kinds of things.
>
> In general I agree but if you could not change hypervisor
> and/or dom0 (e.g. you are using cloud providers which are
> stick to old versions of Xen) then you have no choice.

Which tends to be where kexec on panic comes in most cases.  Getting
platform vendors to do something sane tends to be a multi-year political
effort of dubious worth while just solving the problem locally actually
gets the problem solved for those who care.

Eric

Re: is kexec on Xen domU possible?

[H. Peter Anvin]

On 07/22/2013 10:20 AM, Eric W. Biederman wrote:
>>>
>>> Also, in any virtualized environment the hypervisor can do a better job
>>> for things like kdump, simply because it can provide two things that are
>>> otherwise hard to do:
>>>
>>> 1. a known-good system state;
>>> 2. a known-clean kdump image.
>>>
>>> As such, I do encourage the virtualization people to (also) develop
>>> hypervisor-*aware* solutions for these kinds of things.
>>
>> In general I agree but if you could not change hypervisor
>> and/or dom0 (e.g. you are using cloud providers which are
>> stick to old versions of Xen) then you have no choice.
> 
> Which tends to be where kexec on panic comes in most cases.  Getting
> platform vendors to do something sane tends to be a multi-year political
> effort of dubious worth while just solving the problem locally actually
> gets the problem solved for those who care.
> 

It should not be a "one or the other" issue.

	-hpa

Re: is kexec on Xen domU possible?

[Greg KH]

On Mon, Jul 22, 2013 at 11:24:46AM -0700, H. Peter Anvin wrote:
> On 07/22/2013 10:20 AM, Eric W. Biederman wrote:
> >>>
> >>> Also, in any virtualized environment the hypervisor can do a better job
> >>> for things like kdump, simply because it can provide two things that are
> >>> otherwise hard to do:
> >>>
> >>> 1. a known-good system state;
> >>> 2. a known-clean kdump image.
> >>>
> >>> As such, I do encourage the virtualization people to (also) develop
> >>> hypervisor-*aware* solutions for these kinds of things.
> >>
> >> In general I agree but if you could not change hypervisor
> >> and/or dom0 (e.g. you are using cloud providers which are
> >> stick to old versions of Xen) then you have no choice.
> > 
> > Which tends to be where kexec on panic comes in most cases.  Getting
> > platform vendors to do something sane tends to be a multi-year political
> > effort of dubious worth while just solving the problem locally actually
> > gets the problem solved for those who care.
> > 
> 
> It should not be a "one or the other" issue.

I don't care about kdump, I care about kexec on domU for people who are
running on cloud providers with old versions of Xen so that they can
control what kernel they can boot, when they want to boot it.  If kdump
works as well, that's just a bonus, but it's down on the list of things
for me to be concerned about.

thanks,

greg k-h

Re: is kexec on Xen domU possible?

[H. Peter Anvin]

On 07/22/2013 11:33 AM, Greg KH wrote:
> 
> I don't care about kdump, I care about kexec on domU for people who are
> running on cloud providers with old versions of Xen so that they can
> control what kernel they can boot, when they want to boot it.  If kdump
> works as well, that's just a bonus, but it's down on the list of things
> for me to be concerned about.
> 

Another valid use case, to be sure.

	-hpa

Re: [Xen-devel] is kexec on Xen domU possible?

[Matt Wilson]

On Mon, Jul 22, 2013 at 11:33:15AM -0700, Greg KH wrote:
> On Mon, Jul 22, 2013 at 11:24:46AM -0700, H. Peter Anvin wrote:
> > On 07/22/2013 10:20 AM, Eric W. Biederman wrote:
> > >>>
> > >>> Also, in any virtualized environment the hypervisor can do a better job
> > >>> for things like kdump, simply because it can provide two things that are
> > >>> otherwise hard to do:
> > >>>
> > >>> 1. a known-good system state;
> > >>> 2. a known-clean kdump image.
> > >>>
> > >>> As such, I do encourage the virtualization people to (also) develop
> > >>> hypervisor-*aware* solutions for these kinds of things.
> > >>
> > >> In general I agree but if you could not change hypervisor
> > >> and/or dom0 (e.g. you are using cloud providers which are
> > >> stick to old versions of Xen) then you have no choice.
> > > 
> > > Which tends to be where kexec on panic comes in most cases.  Getting
> > > platform vendors to do something sane tends to be a multi-year political
> > > effort of dubious worth while just solving the problem locally actually
> > > gets the problem solved for those who care.
> > > 
> > 
> > It should not be a "one or the other" issue.
> 
> I don't care about kdump, I care about kexec on domU for people who are
> running on cloud providers with old versions of Xen so that they can
> control what kernel they can boot, when they want to boot it.  If kdump
> works as well, that's just a bonus, but it's down on the list of things
> for me to be concerned about.

Many Xen-based cloud providers provide a mechanism for users to boot
the kernels they want. For example you can use PV-GRUB on EC2
instances to boot a kernel that is stored within an AMI.

For more info:
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html

--msw

Re: [Xen-devel] is kexec on Xen domU possible?

[Greg KH]

On Tue, Jul 23, 2013 at 05:22:36PM -0700, Matt Wilson wrote:
> On Mon, Jul 22, 2013 at 11:33:15AM -0700, Greg KH wrote:
> > On Mon, Jul 22, 2013 at 11:24:46AM -0700, H. Peter Anvin wrote:
> > > On 07/22/2013 10:20 AM, Eric W. Biederman wrote:
> > > >>>
> > > >>> Also, in any virtualized environment the hypervisor can do a better job
> > > >>> for things like kdump, simply because it can provide two things that are
> > > >>> otherwise hard to do:
> > > >>>
> > > >>> 1. a known-good system state;
> > > >>> 2. a known-clean kdump image.
> > > >>>
> > > >>> As such, I do encourage the virtualization people to (also) develop
> > > >>> hypervisor-*aware* solutions for these kinds of things.
> > > >>
> > > >> In general I agree but if you could not change hypervisor
> > > >> and/or dom0 (e.g. you are using cloud providers which are
> > > >> stick to old versions of Xen) then you have no choice.
> > > > 
> > > > Which tends to be where kexec on panic comes in most cases.  Getting
> > > > platform vendors to do something sane tends to be a multi-year political
> > > > effort of dubious worth while just solving the problem locally actually
> > > > gets the problem solved for those who care.
> > > > 
> > > 
> > > It should not be a "one or the other" issue.
> > 
> > I don't care about kdump, I care about kexec on domU for people who are
> > running on cloud providers with old versions of Xen so that they can
> > control what kernel they can boot, when they want to boot it.  If kdump
> > works as well, that's just a bonus, but it's down on the list of things
> > for me to be concerned about.
> 
> Many Xen-based cloud providers provide a mechanism for users to boot
> the kernels they want. For example you can use PV-GRUB on EC2
> instances to boot a kernel that is stored within an AMI.
> 
> For more info:
>   http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html

Yes, that's quite true, but some don't, or they make it difficult to do
so.  Using kexec also allows you to "be the bootloader" and decide on
_which_ kernel you want to boot, independant of what cloud provider you
use, something that lots of people want in their quest to not dependant
on any one company.

thanks,

greg k-h

Re: [Xen-devel] is kexec on Xen domU possible?

[Eric W. Biederman]

Greg KH <gregkh@linuxfoundation.org> writes:

> On Tue, Jul 23, 2013 at 05:22:36PM -0700, Matt Wilson wrote:
>> On Mon, Jul 22, 2013 at 11:33:15AM -0700, Greg KH wrote:
>> > On Mon, Jul 22, 2013 at 11:24:46AM -0700, H. Peter Anvin wrote:
>> > > On 07/22/2013 10:20 AM, Eric W. Biederman wrote:
>> 
>> Many Xen-based cloud providers provide a mechanism for users to boot
>> the kernels they want. For example you can use PV-GRUB on EC2
>> instances to boot a kernel that is stored within an AMI.
>> 
>> For more info:
>>   http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html
>
> Yes, that's quite true, but some don't, or they make it difficult to do
> so.  Using kexec also allows you to "be the bootloader" and decide on
> _which_ kernel you want to boot, independant of what cloud provider you
> use, something that lots of people want in their quest to not dependant
> on any one company.

I would be more than happy to review and help get something merged that
sorts out kexec on Xen.

Eric