Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Bharata B Rao]

On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 
>   This is the 8th cut of my version of postcopy.
> 
> The userfaultfd linux kernel code is now in the upstream kernel
> tree, and so 4.3-rc3 can be used without modification.
> 
> This qemu series can be found at:
> https://github.com/orbitfp7/qemu.git
> on the wp3-postcopy-v8 tag
> 
> 
> Testing status:
>   * Tested heavily on x86
>   * Smoke tested on aarch64 (so it does work on different page sizes)
>   * Power is unhappy for me (but gets further than the htab problem
>     v7 used to have) (I get a kvm run failed)

Seems to be completing successfully on Power. But it takes 2min for the
migration status to transition from setup to active.

Host: 4.3.0-rc3+
Guest: 4.3.0-rc3+
QEMU: wp3-postcopy-v8 of your tree.

# ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -machine pseries -m 8G,slots=32,maxmem=32G -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -vga none -net nic,model=virtio -net user -redir tcp:2000::22 -smp 16,maxcpus=32 -serial pty

(qemu) migrate_set_capability x-postcopy-ram on
(qemu) migrate -d tcp:localhost:4444
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
Migration status: setup
total time: 0 milliseconds

same status for around 2min...

(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
Migration status: active
total time: 130089 milliseconds
expected downtime: 300 milliseconds
setup: 24 milliseconds
transferred ram: 79454 kbytes
throughput: 50.09 mbps
remaining ram: 7670688 kbytes
total ram: 8388864 kbytes
duplicate: 160684 pages
skipped: 0 pages
normal: 18860 pages
normal bytes: 75440 kbytes
dirty sync count: 1

(qemu) migrate_start_postcopy

(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
Migration status: postcopy-active
total time: 135627 milliseconds
expected downtime: 43 milliseconds
setup: 24 milliseconds
transferred ram: 338598 kbytes
throughput: 74.02 mbps
remaining ram: 1688384 kbytes
total ram: 8388864 kbytes
duplicate: 1600406 pages
skipped: 0 pages
normal: 75258 pages
normal bytes: 301032 kbytes
dirty sync count: 0
dirty pages rate: 98 pages

(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
Migration status: completed
total time: 136898 milliseconds
downtime: 685 milliseconds
setup: 24 milliseconds
transferred ram: 1194196 kbytes
throughput: 72.00 mbps
remaining ram: 0 kbytes
total ram: 8388864 kbytes
duplicate: 1810921 pages
skipped: 0 pages
normal: 286839 pages
normal bytes: 1147356 kbytes
dirty sync count: 2

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Dr. David Alan Gilbert]

* Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 
> >   This is the 8th cut of my version of postcopy.
> > 
> > The userfaultfd linux kernel code is now in the upstream kernel
> > tree, and so 4.3-rc3 can be used without modification.
> > 
> > This qemu series can be found at:
> > https://github.com/orbitfp7/qemu.git
> > on the wp3-postcopy-v8 tag
> > 
> > 
> > Testing status:
> >   * Tested heavily on x86
> >   * Smoke tested on aarch64 (so it does work on different page sizes)
> >   * Power is unhappy for me (but gets further than the htab problem
> >     v7 used to have) (I get a kvm run failed)
> 
> Seems to be completing successfully on Power. But it takes 2min for the
> migration status to transition from setup to active.
> 
> Host: 4.3.0-rc3+
> Guest: 4.3.0-rc3+
> QEMU: wp3-postcopy-v8 of your tree.
> 
> # ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -machine pseries -m 8G,slots=32,maxmem=32G -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -vga none -net nic,model=virtio -net user -redir tcp:2000::22 -smp 16,maxcpus=32 -serial pty
> 
> (qemu) migrate_set_capability x-postcopy-ram on
> (qemu) migrate -d tcp:localhost:4444
> (qemu) info migrate
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> Migration status: setup
> total time: 0 milliseconds
> 
> same status for around 2min...

That's interesting; I saw that behaviour on my aarch64 box, but not on
my power box or on x86.  Can you try using tcp:127.0.0.1:4444 to force
ipv4 (that fixed it for me on aarch64).  On the aarch box I found that
it still happened with head of tree qemu and so decided it wasn't my
postcopy world; I'm assuming what's happening is that it's trying
to connect to the IPv6 address, timing out and then trying IPv4.


> (qemu) info migrate
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> Migration status: active
> total time: 130089 milliseconds
> expected downtime: 300 milliseconds
> setup: 24 milliseconds
> transferred ram: 79454 kbytes
> throughput: 50.09 mbps
> remaining ram: 7670688 kbytes
> total ram: 8388864 kbytes
> duplicate: 160684 pages
> skipped: 0 pages
> normal: 18860 pages
> normal bytes: 75440 kbytes
> dirty sync count: 1
> 
> (qemu) migrate_start_postcopy
> 
> (qemu) info migrate
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> Migration status: postcopy-active
> total time: 135627 milliseconds
> expected downtime: 43 milliseconds
> setup: 24 milliseconds
> transferred ram: 338598 kbytes
> throughput: 74.02 mbps
> remaining ram: 1688384 kbytes
> total ram: 8388864 kbytes
> duplicate: 1600406 pages
> skipped: 0 pages
> normal: 75258 pages
> normal bytes: 301032 kbytes
> dirty sync count: 0
> dirty pages rate: 98 pages
> 
> (qemu) info migrate
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> Migration status: completed
> total time: 136898 milliseconds
> downtime: 685 milliseconds
> setup: 24 milliseconds
> transferred ram: 1194196 kbytes
> throughput: 72.00 mbps
> remaining ram: 0 kbytes
> total ram: 8388864 kbytes
> duplicate: 1810921 pages
> skipped: 0 pages
> normal: 286839 pages
> normal bytes: 1147356 kbytes
> dirty sync count: 2

Great; is the guest happy?

Dave

> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Bharata B Rao]

On Wed, Sep 30, 2015 at 11:53:04AM +0100, Dr. David Alan Gilbert wrote:
> * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> > On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > 
> > >   This is the 8th cut of my version of postcopy.
> > > 
> > > The userfaultfd linux kernel code is now in the upstream kernel
> > > tree, and so 4.3-rc3 can be used without modification.
> > > 
> > > This qemu series can be found at:
> > > https://github.com/orbitfp7/qemu.git
> > > on the wp3-postcopy-v8 tag
> > > 
> > > 
> > > Testing status:
> > >   * Tested heavily on x86
> > >   * Smoke tested on aarch64 (so it does work on different page sizes)
> > >   * Power is unhappy for me (but gets further than the htab problem
> > >     v7 used to have) (I get a kvm run failed)
> > 
> > Seems to be completing successfully on Power. But it takes 2min for the
> > migration status to transition from setup to active.
> > 
> > Host: 4.3.0-rc3+
> > Guest: 4.3.0-rc3+
> > QEMU: wp3-postcopy-v8 of your tree.
> > 
> > # ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -machine pseries -m 8G,slots=32,maxmem=32G -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -vga none -net nic,model=virtio -net user -redir tcp:2000::22 -smp 16,maxcpus=32 -serial pty
> > 
> > (qemu) migrate_set_capability x-postcopy-ram on
> > (qemu) migrate -d tcp:localhost:4444
> > (qemu) info migrate
> > capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> > Migration status: setup
> > total time: 0 milliseconds
> > 
> > same status for around 2min...
> 
> That's interesting; I saw that behaviour on my aarch64 box, but not on
> my power box or on x86.  Can you try using tcp:127.0.0.1:4444 to force
> ipv4 (that fixed it for me on aarch64).  On the aarch box I found that
> it still happened with head of tree qemu and so decided it wasn't my
> postcopy world; I'm assuming what's happening is that it's trying
> to connect to the IPv6 address, timing out and then trying IPv4.

Yes, s/localhost/127.0.0.1 helps.

> 
> Great; is the guest happy?

Yes, survived a reboot, postcopy migration back to the original host
and then a reboot.

Regards,
Bharata.

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Dr. David Alan Gilbert]

* Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> On Wed, Sep 30, 2015 at 11:53:04AM +0100, Dr. David Alan Gilbert wrote:
> > * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> > > On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> > > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > 
> > > >   This is the 8th cut of my version of postcopy.
> > > > 
> > > > The userfaultfd linux kernel code is now in the upstream kernel
> > > > tree, and so 4.3-rc3 can be used without modification.
> > > > 
> > > > This qemu series can be found at:
> > > > https://github.com/orbitfp7/qemu.git
> > > > on the wp3-postcopy-v8 tag
> > > > 
> > > > 
> > > > Testing status:
> > > >   * Tested heavily on x86
> > > >   * Smoke tested on aarch64 (so it does work on different page sizes)
> > > >   * Power is unhappy for me (but gets further than the htab problem
> > > >     v7 used to have) (I get a kvm run failed)
> > > 
> > > Seems to be completing successfully on Power. But it takes 2min for the
> > > migration status to transition from setup to active.
> > > 
> > > Host: 4.3.0-rc3+
> > > Guest: 4.3.0-rc3+
> > > QEMU: wp3-postcopy-v8 of your tree.
> > > 
> > > # ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -machine pseries -m 8G,slots=32,maxmem=32G -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -vga none -net nic,model=virtio -net user -redir tcp:2000::22 -smp 16,maxcpus=32 -serial pty
> > > 
> > > (qemu) migrate_set_capability x-postcopy-ram on
> > > (qemu) migrate -d tcp:localhost:4444
> > > (qemu) info migrate
> > > capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> > > Migration status: setup
> > > total time: 0 milliseconds
> > > 
> > > same status for around 2min...
> > 
> > That's interesting; I saw that behaviour on my aarch64 box, but not on
> > my power box or on x86.  Can you try using tcp:127.0.0.1:4444 to force
> > ipv4 (that fixed it for me on aarch64).  On the aarch box I found that
> > it still happened with head of tree qemu and so decided it wasn't my
> > postcopy world; I'm assuming what's happening is that it's trying
> > to connect to the IPv6 address, timing out and then trying IPv4.
> 
> Yes, s/localhost/127.0.0.1 helps.

OK; it sounds like there's something 'fun' going on with either the
bleeding edge kernel or the bleeding edge qemu with IPv6 then.

> > 
> > Great; is the guest happy?
> 
> Yes, survived a reboot, postcopy migration back to the original host
> and then a reboot.

OK, great.

Dave

> Regards,
> Bharata.
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Bharata B Rao]

On Wed, Sep 30, 2015 at 05:14:48PM +0100, Dr. David Alan Gilbert wrote:
> * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> > On Wed, Sep 30, 2015 at 11:53:04AM +0100, Dr. David Alan Gilbert wrote:
> > > * Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> > > > On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> > > > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > > 
> > > > >   This is the 8th cut of my version of postcopy.
> > > > > 
> > > > > The userfaultfd linux kernel code is now in the upstream kernel
> > > > > tree, and so 4.3-rc3 can be used without modification.
> > > > > 
> > > > > This qemu series can be found at:
> > > > > https://github.com/orbitfp7/qemu.git
> > > > > on the wp3-postcopy-v8 tag
> > > > > 
> > > > > 
> > > > > Testing status:
> > > > >   * Tested heavily on x86
> > > > >   * Smoke tested on aarch64 (so it does work on different page sizes)
> > > > >   * Power is unhappy for me (but gets further than the htab problem
> > > > >     v7 used to have) (I get a kvm run failed)
> > > > 
> > > > Seems to be completing successfully on Power. But it takes 2min for the
> > > > migration status to transition from setup to active.
> > > > 
> > > > Host: 4.3.0-rc3+
> > > > Guest: 4.3.0-rc3+
> > > > QEMU: wp3-postcopy-v8 of your tree.
> > > > 
> > > > # ./ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -machine pseries -m 8G,slots=32,maxmem=32G -device virtio-blk-pci,drive=rootdisk -drive file=/home/bharata/F20-snap1,if=none,cache=none,id=rootdisk,format=qcow2 -vga none -net nic,model=virtio -net user -redir tcp:2000::22 -smp 16,maxcpus=32 -serial pty
> > > > 
> > > > (qemu) migrate_set_capability x-postcopy-ram on
> > > > (qemu) migrate -d tcp:localhost:4444
> > > > (qemu) info migrate
> > > > capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off x-postcopy-ram: on 
> > > > Migration status: setup
> > > > total time: 0 milliseconds
> > > > 
> > > > same status for around 2min...
> > > 
> > > That's interesting; I saw that behaviour on my aarch64 box, but not on
> > > my power box or on x86.  Can you try using tcp:127.0.0.1:4444 to force
> > > ipv4 (that fixed it for me on aarch64).  On the aarch box I found that
> > > it still happened with head of tree qemu and so decided it wasn't my
> > > postcopy world; I'm assuming what's happening is that it's trying
> > > to connect to the IPv6 address, timing out and then trying IPv4.
> > 
> > Yes, s/localhost/127.0.0.1 helps.
> 
> OK; it sounds like there's something 'fun' going on with either the
> bleeding edge kernel or the bleeding edge qemu with IPv6 then.

It's the host kernel. Until 4.3.0-rc2 it works, the problem starts
from 4.3.0-rc3. Accessing monitor on localhost using telnet is also affected.

Regards,
Bharata.

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Dr. David Alan Gilbert]

* Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:

<snip>

> > > > That's interesting; I saw that behaviour on my aarch64 box, but not on
> > > > my power box or on x86.  Can you try using tcp:127.0.0.1:4444 to force
> > > > ipv4 (that fixed it for me on aarch64).  On the aarch box I found that
> > > > it still happened with head of tree qemu and so decided it wasn't my
> > > > postcopy world; I'm assuming what's happening is that it's trying
> > > > to connect to the IPv6 address, timing out and then trying IPv4.
> > > 
> > > Yes, s/localhost/127.0.0.1 helps.
> > 
> > OK; it sounds like there's something 'fun' going on with either the
> > bleeding edge kernel or the bleeding edge qemu with IPv6 then.
> 
> It's the host kernel. Until 4.3.0-rc2 it works, the problem starts
> from 4.3.0-rc3. Accessing monitor on localhost using telnet is also affected.

Hmm that is fun; I'll see if I can find a friendly kernel network person
to chat to.

Dave

> 
> Regards,
> Bharata.
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Bharata B Rao]

On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 
>   This is the 8th cut of my version of postcopy.
> 
> The userfaultfd linux kernel code is now in the upstream kernel
> tree, and so 4.3-rc3 can be used without modification.
> 
> This qemu series can be found at:
> https://github.com/orbitfp7/qemu.git
> on the wp3-postcopy-v8 tag
> 
> 
> Testing status:
>   * Tested heavily on x86
>   * Smoke tested on aarch64 (so it does work on different page sizes)
>   * Power is unhappy for me (but gets further than the htab problem
>     v7 used to have) (I get a kvm run failed)

As I said earlier, postcopy migration works on Power, but memory hotplug
seems to have some problem.

qemu-system-ppc64 ... -object memory-backend-ram,id=ram0,size=2G -device pc-dimm,memdev=ram0

qemu/exec.c:1278: find_ram_offset: Assertion `size != 0' failed.

Does this happen on x86 too ?

Regards,
Bharata.

Re: [Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Dr. David Alan Gilbert]

* Bharata B Rao (bharata@linux.vnet.ibm.com) wrote:
> On Mon, Sep 28, 2015 at 05:51:39PM +0100, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 
> >   This is the 8th cut of my version of postcopy.
> > 
> > The userfaultfd linux kernel code is now in the upstream kernel
> > tree, and so 4.3-rc3 can be used without modification.
> > 
> > This qemu series can be found at:
> > https://github.com/orbitfp7/qemu.git
> > on the wp3-postcopy-v8 tag
> > 
> > 
> > Testing status:
> >   * Tested heavily on x86
> >   * Smoke tested on aarch64 (so it does work on different page sizes)
> >   * Power is unhappy for me (but gets further than the htab problem
> >     v7 used to have) (I get a kvm run failed)
> 
> As I said earlier, postcopy migration works on Power, but memory hotplug
> seems to have some problem.
> 
> qemu-system-ppc64 ... -object memory-backend-ram,id=ram0,size=2G -device pc-dimm,memdev=ram0
> 
> qemu/exec.c:1278: find_ram_offset: Assertion `size != 0' failed.
> 
> Does this happen on x86 too ?

Hmm, yes it does - I hadn't tried that.
The problem is that I added a HOST_PAGE_ALIGN call during RAMBlock creation,
and -object gets parsed pretty early on, before the internal host page masks
have been setup.

The patch below passes a smoke test; I'll look to clean it up.

Thanks,

Dave

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index a3719b7..b4c4b6e 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -81,7 +81,6 @@ void cpu_gen_init(void);
 int cpu_gen_code(CPUArchState *env, struct TranslationBlock *tb,
                  int *gen_code_size_ptr);
 bool cpu_restore_state(CPUState *cpu, uintptr_t searched_pc);
-void page_size_init(void);
 
 void QEMU_NORETURN cpu_resume_from_signal(CPUState *cpu, void *puc);
 void QEMU_NORETURN cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index 01d29dd..ae3530c 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -491,4 +491,6 @@ int parse_debug_env(const char *name, int max, int initial);
 
 const char *qemu_ether_ntoa(const MACAddr *mac);
 
+void page_size_init(void);
+
 #endif
diff --git a/kvm-all.c b/kvm-all.c
index de1924c..62b71fe 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1461,7 +1461,6 @@ static int kvm_init(MachineState *ms)
      * page size for the system though.
      */
     assert(TARGET_PAGE_SIZE <= getpagesize());
-    page_size_init();
 
     s->sigmask_len = 8;
 
diff --git a/vl.c b/vl.c
index e211f6a..818075c 100644
--- a/vl.c
+++ b/vl.c
@@ -4249,6 +4249,7 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+    page_size_init();
     socket_init();
 
     if (qemu_opts_foreach(qemu_find_opts("object"),

> 
> Regards,
> Bharata.
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v8 01/54] Add postcopy documentation

[Juan Quintela]

Rsend to proper list.

Removed cc'd

Juan Quintela <quintela@redhat.com> wrote:
> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Reviewed-by: Amit Shah <amit.shah@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 04/54] Move configuration section writing

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:

Post proper list

remove cc'd

> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> The vmstate_configuration is currently written
>> in 'qemu_savevm_state_begin', move it to
>> 'qemu_savevm_state_header' since it's got a hard
>> requirement that it must be the 1st thing after
>> the header.
>> (In postcopy some 'command' sections get sent
>> early before the saving of the main sections
>> and hence before qemu_savevm_state_begin).
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 06/54] Rename mis->file to from_src_file

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:
> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:

Post proper list

Remove cc'd

>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> 'file' becomes confusing when you have flows in each direction;
>> rename to make it clear.
>> This leaves just the main forward direction ms->file, which is used
>> in a lot of places and is probably not worth renaming given the churn.
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 09/54] Add QEMU_MADV_NOHUGEPAGE

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:

Post proper list

Remove cc'd


> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> Add QEMU_MADV_NOHUGEPAGE as an OS-independent version of
>> MADV_NOHUGEPAGE.
>>
>> We include sys/mman.h before making the test to ensure
>> that we pick up the system defines.
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 10/54] migration/ram.c: Use RAMBlock rather than MemoryRegion

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:

Post proper list

Remove cc'd

> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> RAM migration mainly works on RAMBlocks but in a few places
>> uses data from MemoryRegions to access the same information that's
>> already held in RAMBlocks; clean it up just to avoid the
>> MemoryRegion use.
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

This was a leftover of when I tried to convert migration to use Memory
regions, yes, it didn't went too well

Re: [Qemu-devel] [PATCH v8 13/54] Move dirty page search state into separate structure

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:

Post proper list

Remove cc'd


> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> Pull the search state for one iteration of the dirty page
>> search into a structure.
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Reviewed-by: Amit Shah <amit.shah@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 17/54] Return path: socket_writev_buffer: Block even on non-blocking fd's

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:

Post proper list

Remove cc'd


> "Dr. David Alan Gilbert (git)" <dgilbert@redhat.com> wrote:
>> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>
>> The destination sets the fd to non-blocking on incoming migrations;
>> this also affects the return path from the destination, and thus we
>> need to make sure we can safely write to the return path.
>>
>> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Reviewed-by: Amit Shah <amit.shah@redhat.com>

 Reviewed-by: Juan Quintela <quintela@redhat.com>

 I think that my review by got lost on previous series O:-)

[Qemu-devel] [PATCH v8 00/54] Postcopy implementation

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

  This is the 8th cut of my version of postcopy.

The userfaultfd linux kernel code is now in the upstream kernel
tree, and so 4.3-rc3 can be used without modification.

This qemu series can be found at:
https://github.com/orbitfp7/qemu.git
on the wp3-postcopy-v8 tag


Testing status:
  * Tested heavily on x86
  * Smoke tested on aarch64 (so it does work on different page sizes)
  * Power is unhappy for me (but gets further than the htab problem
    v7 used to have) (I get a kvm run failed)

Note that patches:
   3 Init page size in qtest
   10 Use RAMBlock rather than Memory Region
   14,15 (splitting ram find and save block)
   36 Split out end of migration code

   have also been posted seperately during the last month and
can be taken separately from this series.

This work has been partially funded by the EU Orbit project:
  see http://www.orbitproject.eu/about/

v8
  Huge page changes
     The precopy phase is now allowed to keep transparent-huge-pages
     enabled, although these maybe split in the discard phase.  A change
     to the discard process now causes discards for unsent (as well as
     redirtied) pages; the combination of these changes means that the
     destination starts running with many of the precopy pages as huge
     pages, resulting in a significant performance benefit.  This change
     adds one more state ('discard') which is entered by the destination
     on reception of the 1st discard command.

  Add global_state_store for postcopy
  Moved postcopy_state back out of MigrationIncomingState
     During the end of the main migration thread reading state the
     postcopy state is read to see who should cleanup; in failure
     conditions there was a race between the state being read and the
     freeing of the MIS

  Stop calling the iterate method for non-postcopiable devices during postcopy
     Helps fix Power (thanks for Bharata for helping debug this)

  Review comment fixes
    rework of migration command parsing
    rework of postcopy_chunk_hostpages
    rework of discard code & protocol into start/length rather than start/end
    rename qemu_get_buffer_less_copy -> qemu_get_buffer_in_place
    split 'Postcopy end in migration thread' patch into two
    split of MIG_RP_MSG_REQ_PAGES into subtype with name
    Added comments documenting postcopy_state's use
    lots of other minor fixups

  Notes:
    I kept the mlock support (users are saying they wanted migration/postcopy
      with mlock)

    I'm keeping the x-  for now, until the libvirt interface gets finalised.

    There are two checkpatch errors, that I don't think are right to change:
     a) a 'typedef enum' it wants split - that's the way we do all our enums
        and would force a dummy name for the enum.
     b) A complaint about postcopy_ram_discard_version = 0 that's a global
        static; I could get rid of it by making my version 1, but it doesn't
        seem right to omit the '= 0 ' for a version constant.

Dave

Dr. David Alan Gilbert (54):
  Add postcopy documentation
  Provide runtime Target page information
  Init page sizes in qtest
  Move configuration section writing
  qemu_ram_block_from_host
  Rename mis->file to from_src_file
  Add qemu_get_buffer_in_place to avoid copies some of the time
  Add wrapper for setting blocking status on a QEMUFile
  Add QEMU_MADV_NOHUGEPAGE
  migration/ram.c: Use RAMBlock rather than MemoryRegion
  ram_debug_dump_bitmap: Dump a migration bitmap as text
  migrate_init: Call from savevm
  Move dirty page search state into separate structure
  ram_find_and_save_block: Split out the finding
  Rename save_live_complete to save_live_complete_precopy
  Return path: Open a return path on QEMUFile for sockets
  Return path: socket_writev_buffer: Block even on non-blocking fd's
  Migration commands
  Return path: Control commands
  Return path: Send responses from destination to source
  Return path: Source handling of return path
  Rework loadvm path for subloops
  Add migration-capability boolean for postcopy-ram.
  Add wrappers and handlers for sending/receiving the postcopy-ram
    migration messages.
  MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
  Modify save_live_pending for postcopy
  postcopy: OS support test
  migrate_start_postcopy: Command to trigger transition to postcopy
  MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
  Avoid sending vmdescription during postcopy
  Add qemu_savevm_state_complete_postcopy
  Postcopy: Maintain sentmap and calculate discard
  postcopy: Incoming initialisation
  postcopy: ram_enable_notify to switch on userfault
  Postcopy: Postcopy startup in migration thread
  Split out end of migration code from migration_thread
  Postcopy: End of iteration
  Page request:  Add MIG_RP_MSG_REQ_PAGES reverse command
  Page request: Process incoming page request
  Page request: Consume pages off the post-copy queue
  postcopy_ram.c: place_page and helpers
  Postcopy: Use helpers to map pages during migration
  Don't sync dirty bitmaps in postcopy
  Don't iterate on precopy-only devices during postcopy
  Host page!=target page: Cleanup bitmaps
  postcopy: Check order of received target pages
  Round up RAMBlock sizes to host page sizes
  Postcopy; Handle userfault requests
  Start up a postcopy/listener thread ready for incoming page data
  postcopy: Wire up loadvm_postcopy_handle_ commands
  Postcopy: Mark nohugepage before discard
  End of migration for postcopy
  Disable mlock around incoming postcopy
  Inhibit ballooning during postcopy

 balloon.c                        |  11 +
 docs/migration.txt               | 191 ++++++++
 exec.c                           |  72 ++-
 hmp-commands.hx                  |  15 +
 hmp.c                            |   7 +
 hmp.h                            |   1 +
 hw/ppc/spapr.c                   |   2 +-
 hw/virtio/virtio-balloon.c       |   4 +-
 include/exec/cpu-common.h        |   3 +
 include/exec/ram_addr.h          |   2 -
 include/migration/migration.h    | 128 ++++-
 include/migration/postcopy-ram.h |  99 ++++
 include/migration/qemu-file.h    |  10 +
 include/migration/vmstate.h      |   8 +-
 include/qemu/osdep.h             |   9 +
 include/qemu/typedefs.h          |   3 +
 include/sysemu/balloon.h         |   2 +
 include/sysemu/sysemu.h          |  46 +-
 migration/Makefile.objs          |   2 +-
 migration/block.c                |   9 +-
 migration/migration.c            | 753 +++++++++++++++++++++++++++--
 migration/postcopy-ram.c         | 763 ++++++++++++++++++++++++++++++
 migration/qemu-file-unix.c       | 111 ++++-
 migration/qemu-file.c            |  74 +++
 migration/ram.c                  | 995 +++++++++++++++++++++++++++++++++++----
 migration/savevm.c               | 825 ++++++++++++++++++++++++++++----
 qapi-schema.json                 |  18 +-
 qmp-commands.hx                  |  19 +
 qtest.c                          |   1 +
 trace-events                     |  81 +++-
 30 files changed, 3996 insertions(+), 268 deletions(-)
 create mode 100644 include/migration/postcopy-ram.h
 create mode 100644 migration/postcopy-ram.c

-- 
2.5.0