From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bruno Alvisio Subject: Fwd: VM Live Migration with Local Storage Date: Sun, 11 Jun 2017 20:16:04 -0700 Message-ID: References: <20170222130030.r5tz32iupsosnjht@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6404915159779274323==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============6404915159779274323== Content-Type: multipart/alternative; boundary="089e08222430476b7f0551babdae" --089e08222430476b7f0551babdae Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I think it would be beneficial to add local disk migration feature for =E2=80=98blkback' backend since it is one of the mostly used backends. I wo= uld like to start a discussion about the design of the machinery needed to achieve this feature. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Objective Add a feature to migrate VMs that have local storage and use the blkback iface. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D User Interface Add a cmd line option in =E2=80=9Cxl migrate=E2=80=9D command to specify if= local disks need to be copied to the destination node. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Design 1. As part of the libxl_domain_suspend, the =E2=80=9Cdisk mirroring mach= inery=E2=80=9D starts an asynchronous job that copies the disks blocks from source to t= he destination. 2. The protocol to copy the disks should resemble the one used for memory copy: - Do first initial copy of the disk. - Check of sectors that have been written since copy started. For this, the blkback driver should be aware that migration of disk is happening a= nd in this case forward the write request to the =E2=80=9Cmigration machine= ry=E2=80=9D so that a record of dirty blocks are logged. - Migration machinery copies =E2=80=9Cdirty=E2=80=9D blocks until conver= gence. - Duplicate all the disk writes/reads to both disks in source and destinations node while VM is being suspended. Block Diagram +=E2=80=94------+ | VM | +-------+ | | I/O Write | V +----------+ +-----------+ +-------------+ | blkback | ----> | Source | sectors Stream | Destination | +----------+ | mirror |------------------>| mirror | | | machinery | I/O Writes | machinery | | +-----------+ +-------------+ | | | | | To I/O block layer | | | V V +----------+ +-------------+ | disk | | Mirrored | +----------+ | Disk | +-------------+ =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Initial Questions 1. Is it possible to leverage the current design of QEMU for drive mirroring for Xen? 2. What is the best place to implement this protocol? As part of Xen or the kernel? 3. Is it possible to use the same stream currently used for migrating the memory to also migrate the disk blocks? Any guidance/feedback for a more specific design is greatly appreciated. Thanks, Bruno On Wed, Feb 22, 2017 at 5:00 AM, Wei Liu wrote: > Hi Bruno > > Thanks for your interest. > > On Tue, Feb 21, 2017 at 10:34:45AM -0800, Bruno Alvisio wrote: > > Hello, > > > > I have been to doing some research and as far as I know XEN supports > > Live Migration > > of VMs that only have shared storage. (i.e. iSCSI) If the VM has been > > booted with local storage it cannot be live migrated. > > QEMU seems to support live migration with local storage (I have tested > using > > 'virsh migrate with the '--storage-copy-all' option) > > > > I am wondering if this still true in the latest XEN release. Are there > plans > > to add this functionality in future releases? I would be interested in > > contributing to the Xen Project by adding this functionality. > > > > No plan at the moment. > > Xen supports a wide variety of disk backends. QEMU is one of them. The > others are blktap (not upstreamed yet) and in-kernel blkback. The latter > two don't have the capability to copy local storage to the remote end. > > That said, I think it would be valuable to have such capability for QEMU > backed disks. We also need to design the machinery so that other > backends can be made to do the same thing in the future. > > If you want to undertake this project, I suggest you setup a Xen system, > read xl / libxl source code under tools directory and understand how > everything is put together. Reading source code could be daunting at > times, so don't hesitate to ask for pointers. After you have the big > picture in mind, we can then discuss how to implement the functionality > on xen-devel. > > Does this sound good to you? > > Wei. > > > Thanks, > > > > Bruno > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > https://lists.xen.org/xen-devel > > --089e08222430476b7f0551babdae Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,
<= br>
I= think it would be beneficial to add local disk migration feature for =E2= =80=98blkback' backend since it is one of the mostly used backends. I w= ould like to start a discussion about the design of the machinery needed to= achieve this feature.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D
Objective
Add a feature to migrate VMs that have local stora= ge and use the blkback iface.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
User Interface
Add a cmd line option in =E2=80= =9Cxl migrate=E2=80=9D command to specify if local disks need to be copied = to the destination node.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Design
  1. As part of the libxl_domain_suspend, th= e =E2=80=9Cdisk mirroring machinery=E2=80=9D starts an asynchronous job tha= t copies the disks blocks from source to the destination.
  2. The proto= col to copy the disks should resemble the one used for memory copy:
  • Do first initial= copy of the disk.
  • Check of sectors that have been written since co= py started. For this, the blkback driver should be aware that migration of = disk is happening and in this case forward the write request to the=C2=A0= =E2=80=9Cmigration machinery=E2=80=9D so that a record of dirty blocks are = logged.
  • Migration machinery copies =E2=80=9Cdirty=E2=80=9D blocks u= ntil convergence.
  • Duplicate all the disk writes/reads to both disks= in source and destinations node while VM is being suspended.

    • Block Diagram
       +=E2=80=94------+
       |  VM   |
       +-------+      |
          | I/O Write
          |
          V
    +----------+       =
+-----------+                   +-------------+
    |  blkback | ----> | =
 Source   |  sectors Stream   | Destination |
    +----------+       |  mirr=
or   |------------------>|   mirror    |
          |            | machine=
ry |   I/O Writes      |  machinery  |
          |            +-----------+ =
                  +-------------+
          |                               =
                   |
          |                                            =
      |
          | To I/O block layer                               |
      =
    |                                                  |
          V        =
                                          V
    +----------+                =
                       +-------------+
    |   disk   |                     =
                  |   Mirrored  |
    +----------+                          =
             |     Disk    |
                                               =
        +-------------+

    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
    Initial Questions
    1. Is it possible to leverage the current d= esign of QEMU for drive mirroring for Xen?
    2. What is the best place t= o implement this protocol? As part of Xen or the kernel?
    3. Is it poss= ible to use the same stream currently used for migrating the memory to also= migrate the disk blocks?

    Any guidance/feedbac= k for a more specific design is greatly appreciated.

    Thanks,

    Bruno

    On Wed, Feb 22, 2017 at 5:00 AM, Wei Liu <wei.liu2@citrix.com= > wrote:
    Hi Bruno

    Thanks for your interest.

    On Tue, Feb 21, 2017 at 10:34:45AM -0800, Bruno Alvisio wrote:
    > Hello,
    >
    > I have been to doing some research and as far as I know XEN supports > Live Migration
    > of VMs that only have shared storage. (i.e. iSCSI) If the VM has been<= br> > booted with local storage it cannot be live migrated.
    > QEMU seems to support live migration with local storage (I have tested= using
    > 'virsh migrate with the '--storage-copy-all' option)
    >
    > I am wondering if this still true in the latest XEN release. Are there= plans
    > to add this functionality in future releases? I would be interested in=
    > contributing to the Xen Project by adding this functionality.
    >

    No plan at the moment.

    Xen supports a wide variety of disk backends. QEMU is one of them. The
    others are blktap (not upstreamed yet) and in-kernel blkback. The latter two don't have the capability to copy local storage to the remote end.<= br>
    That said, I think it would be valuable to have such capability for QEMU backed disks. We also need to design the machinery so that other
    backends can be made to do the same thing in the future.

    If you want to undertake this project, I suggest you setup a Xen system, read xl / libxl source code under tools directory and understand how
    everything is put together. Reading source code could be daunting at
    times, so don't hesitate to ask for pointers. After you have the big picture in mind, we can then discuss how to implement the functionality
    on xen-devel.

    Does this sound good to you?

    Wei.

    > Thanks,
    >
    > Bruno

    > _______________________________________________
    > Xen-devel mailing list
    > Xen-devel= @lists.xen.org
    > https://lists.xen.org/xen-devel



    --089e08222430476b7f0551babdae-- --===============6404915159779274323== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============6404915159779274323==--