[PATCH RFC 0/4] arm: regarding live migration

[PATCH RFC 0/4] arm: regarding live migration

[Jaeyong Yoo]

Hi all,
I'm interested in developing live migration in xen arm and possibly 
the contribution to the community and I hope this patch series could be a start.

For this matter, I have following questions:

(1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh?

(2) After some overview of source code, I think the required parts
   for save/restore are the following:
      - xen-store info
      - shared info page
      - memory contents (no need for p2m table)
      - cpu/vcpu 
      - gic/vgic
      - drivers
   I think there are still important parts that I'm missing.
   I appreciate if you could give some advice :)

(3) Regarding split drivers, come to think of it, we have to store 
   both side (front/back) states, in-flight event channels, IRQs, etc.
   And those look like quite a work (although evtchn is migrated within vcpu)
   I appreciate  if you guys could share any hints from the experience of 
   migrating split drivers in x86.

Lastly I would like to note that the following patch series is just the 
concept work for reviewing my idea and they are quite preliminary.


Jaeyong Yoo (4):
  Create new directory for stroing hvm-related files in ARM.
  Implement arch_hvm_save and arch_hvm_load functions
  Implement save and restore for gic (template impl)
  Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl

 xen/arch/arm/Makefile                  |    2 +-
 xen/arch/arm/domctl.c                  |   58 +++++++++++++++-
 xen/arch/arm/hvm.c                     |   67 ------------------
 xen/arch/arm/hvm/Makefile              |    2 +
 xen/arch/arm/hvm/hvm.c                 |  118 ++++++++++++++++++++++++++++++++
 xen/arch/arm/hvm/save.c                |   69 +++++++++++++++++++
 xen/common/Makefile                    |    2 +
 xen/include/asm-arm/hvm/support.h      |   29 ++++++++
 xen/include/public/arch-arm/hvm/save.h |   36 ++++++++++
 9 files changed, 314 insertions(+), 69 deletions(-)
 delete mode 100644 xen/arch/arm/hvm.c
 create mode 100644 xen/arch/arm/hvm/Makefile
 create mode 100644 xen/arch/arm/hvm/hvm.c
 create mode 100644 xen/arch/arm/hvm/save.c
 create mode 100644 xen/include/asm-arm/hvm/support.h

-- 
1.7.9.5

Re: [PATCH RFC 0/4] arm: regarding live migration

[Ian Campbell]

On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:
> Hi all,
> I'm interested in developing live migration in xen arm and possibly 
> the contribution to the community and I hope this patch series could be a start.
> 
> For this matter, I have following questions:
> 
> (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh?

My personal preference is to avoid either of them, they are IMHO x86
specific terms. On ARM we (currently) have only "domains", there aren't
(yet) multiple types of domain

> (2) After some overview of source code, I think the required parts
>    for save/restore are the following:
>       - xen-store info

This should be recreated on the far side based on the guest
configuration, it doesn't need to be explicitly saved (this ties into
your #3 below)

>       - shared info page

This is either handled as part of the memory contents or is
reinitialised from scratch on the destination. I forget which but in any
case it doesn't need to be explicitly transferred, I don't think.
Perhaps its guest PFN location does need to be saved.

>       - memory contents (no need for p2m table)
>       - cpu/vcpu 
>       - gic/vgic

Yes

>       - drivers

Not sure which drivers you mean. Any emulated hardware would need its
state pickling, but PV drivers do not (see below).

>    I think there are still important parts that I'm missing.
>    I appreciate if you could give some advice :)
> 
> (3) Regarding split drivers, come to think of it, we have to store 
>    both side (front/back) states, in-flight event channels, IRQs, etc.
>    And those look like quite a work (although evtchn is migrated within vcpu)
>    I appreciate  if you guys could share any hints from the experience of 
>    migrating split drivers in x86.

There is no need to save any of this state, the devices are effectively
reconnected from scratch on the destination and the frontend devices are
responsible for replaying any inflight state on resume.

Ian.

> Lastly I would like to note that the following patch series is just the 
> concept work for reviewing my idea and they are quite preliminary.
> 
> 
> Jaeyong Yoo (4):
>   Create new directory for stroing hvm-related files in ARM.
>   Implement arch_hvm_save and arch_hvm_load functions
>   Implement save and restore for gic (template impl)
>   Implement XEN_DOMCTL_gethvmcontext part of arch_do_domctl
> 
>  xen/arch/arm/Makefile                  |    2 +-
>  xen/arch/arm/domctl.c                  |   58 +++++++++++++++-
>  xen/arch/arm/hvm.c                     |   67 ------------------
>  xen/arch/arm/hvm/Makefile              |    2 +
>  xen/arch/arm/hvm/hvm.c                 |  118 ++++++++++++++++++++++++++++++++
>  xen/arch/arm/hvm/save.c                |   69 +++++++++++++++++++
>  xen/common/Makefile                    |    2 +
>  xen/include/asm-arm/hvm/support.h      |   29 ++++++++
>  xen/include/public/arch-arm/hvm/save.h |   36 ++++++++++
>  9 files changed, 314 insertions(+), 69 deletions(-)
>  delete mode 100644 xen/arch/arm/hvm.c
>  create mode 100644 xen/arch/arm/hvm/Makefile
>  create mode 100644 xen/arch/arm/hvm/hvm.c
>  create mode 100644 xen/arch/arm/hvm/save.c
>  create mode 100644 xen/include/asm-arm/hvm/support.h
> 

Re: [PATCH RFC 0/4] arm: regarding live migration

[Ian Campbell]

On Wed, 2013-06-05 at 10:13 +0100, Ian Campbell wrote:
> On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:
> > Hi all,
> > I'm interested in developing live migration in xen arm and possibly 
> > the contribution to the community and I hope this patch series could be a start.
> > 
> > For this matter, I have following questions:
> > 
> > (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh?
> 
> My personal preference is to avoid either of them, they are IMHO x86
> specific terms. On ARM we (currently) have only "domains", there aren't
> (yet) multiple types of domain

BTW, I appreciate that the term HVM at least is part of the common
hypercall API which we share with x86 -- we just have to live with that
I think.

Re: [PATCH RFC 0/4] arm: regarding live migration

[Ian Campbell]

On Wed, 2013-06-05 at 14:46 +0900, Jaeyong Yoo wrote:
> Lastly I would like to note that the following patch series is just the 
> concept work for reviewing my idea and they are quite preliminary.

I took a look and made a few comments but as a preliminary framework it
looks good, thanks!

BTW, we are currently in feature freeze for the 4.3 release and
save/restore is clearly 4.4 material I think, but that shouldn't stop us
from working on the feature it just means it won't be committed right
away.

Ian.

Re: [PATCH RFC 0/4] arm: regarding live migration

[Jaeyong Yoo]

> > > For this matter, I have following questions:
> > > 
> > > (1) Is it OK to keep using the keyword "hvm"? Or, is it better to use pvh?
> > 
> > My personal preference is to avoid either of them, they are IMHO x86
> > specific terms. On ARM we (currently) have only "domains", there aren't
> > (yet) multiple types of domain
> 
> BTW, I appreciate that the term HVM at least is part of the common
> hypercall API which we share with x86 -- we just have to live with that
> I think.

Got it.

Jaeyong

Re: [PATCH RFC 0/4] arm: regarding live migration

[Jaeyong Yoo]

Hello Ian,
I got some more questions while implementing live migration in arm.

> > (3) Regarding split drivers, come to think of it, we have to store 
> >    both side (front/back) states, in-flight event channels, IRQs, etc.
> >    And those look like quite a work (although evtchn is migrated within vcpu)
> >    I appreciate  if you guys could share any hints from the experience of 
> >    migrating split drivers in x86.
> 
> There is no need to save any of this state, the devices are effectively
> reconnected from scratch on the destination and the frontend devices are
> responsible for replaying any inflight state on resume.
> 

I'm trying to find the suspending part of PV driver, but can not locate it.
(And, looks like there is no function like suspend in xen-blkfront.c)
One guess is by deleting vbd in xenstore, maybe hot-plug-out can suspend.
Could you give me some advice?

And, about grant table, if we effectively suspend and resume PV driver, I 
think we don't need to migrate grant table entires. Is this right?

Thanks,
Jaeyong

Re: [PATCH RFC 0/4] arm: regarding live migration

[Ian Campbell]

On Thu, 2013-06-13 at 01:15 +0000, Jaeyong Yoo wrote:
> Hello Ian,
> I got some more questions while implementing live migration in arm.
> 
> > > (3) Regarding split drivers, come to think of it, we have to store 
> > >    both side (front/back) states, in-flight event channels, IRQs, etc.
> > >    And those look like quite a work (although evtchn is migrated within vcpu)
> > >    I appreciate  if you guys could share any hints from the experience of 
> > >    migrating split drivers in x86.
> > 
> > There is no need to save any of this state, the devices are effectively
> > reconnected from scratch on the destination and the frontend devices are
> > responsible for replaying any inflight state on resume.
> > 
> 
> I'm trying to find the suspending part of PV driver, but can not locate it.
> (And, looks like there is no function like suspend in xen-blkfront.c)
> One guess is by deleting vbd in xenstore, maybe hot-plug-out can suspend.
> Could you give me some advice?

IIRC this is all handled by the block driver on resume and no work is
required on suspend. Essentially it simply replays the ring on resume. I
think it is done this way so that in checkpoint type saves the resume of
the original domain after checkpointing is cheap because nothing needs
to be done.

> And, about grant table, if we effectively suspend and resume PV driver, I 
> think we don't need to migrate grant table entires. Is this right?

Correct.

Ian.