qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v7 00/11] XBRLE delta for live migration of large memory app
@ 2012-01-26 14:24 Orit Wasserman
  2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 01/11] Add cache handling functions Orit Wasserman
                   ` (11 more replies)
  0 siblings, 12 replies; 14+ messages in thread
From: Orit Wasserman @ 2012-01-26 14:24 UTC (permalink / raw)
  To: qemu-devel
  Cc: quintela, Petter Svard, stefanha, blauwirbel, Orit Wasserman,
	Benoit Hudzia, avi, Aidan Shribman

Changes from v6:
 1) add assert checks to ULEB encoding/decoding
 2) no need to send last zero run
	
Changes from v5:
1) Add migration capabilities
2) Use ULEB to encode run length
3) Do not send unmodified (dirty) page
3) Fix other patch comments

Using GCache or GHashTable requires allocating new buffer on every content change, 
so I decided to keep the simple cache implementation.

Todo :
     Use SSE for encoding
     Automatic activation/deactivation of the feature.

Changes from v4:
1) Rebase
2) divide patch into 9 patches
3) move memory allocation into cache_insert

By using XBZRLE (Xor Based Zero Run Length Encoding) we can reduce VM downtime
and total live-migration time of VMs running memory write intensive workloads
typical of large enterprise applications such as SAP ERP Systems, and generally
speaking for any application with a sparse memory update pattern.

The compression format uses the fact that we will have many zero (zero represents
an unchanged value). 
We repesent the page data delta by zero and non zero runs.
We represent a zero run with it's length (in bytes). 
We represent a non zero run with it's length (in bytes) and the data.
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)

page = zrun
       | zrun nzrun
       | zrun nzrun page

zrun = length

nzrun = length byte...

length = uleb128 encoded integer

On the sender side XBZRLE is used as a compact delta encoding of page updates,
retrieving the old page content from an LRU cache (default size of 512 MB). The
receiving side uses the existing page content and XBZRLE to decode the new page
content.

This is a more compact way to store the delta than the previous version.

This work was originally based on research results published VEE 2011: Evaluation of
Delta Compression Techniques for Efficient Live Migration of Large Virtual
Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta encoder
XBRLE was improved further using XBZRLE instead.

XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
ideal for in-line, real-time encoding such as is needed for live-migration.

A typical usage scenario:
    {qemu} migrate_set_cachesize 256m
    {qemu} migrate -d -u tcp:destination.host:4444
    {qemu} info migrate
    ...
    transferred ram-duplicate: A kbytes
    transferred ram-duplicate: B pages
    transferred ram-normal: C kbytes
    transferred ram-normal: D pages
    transferred ram-xbrle: E kbytes
    transferred ram-xbrle: F pages
    overflow ram-xbrle: G pages
    cache-hit ram-xbrle: H pages
    cache-lookup ram-xbrle: J pages

Testing: live migration with XBZRLE completed in 110 seconds, without live
migration was not able to complete.

A simple synthetic memory r/w load generator:
..    include <stdlib.h>
..    include <stdio.h>
..    int main()
..    {
..        char *buf = (char *) calloc(4096, 4096);
..        while (1) {
..            int i;
..            for (i = 0; i < 4096 * 4; i++) {
..                buf[i * 4096 / 4]++;
..            }
..            printf(".");
..        }
..    }

Signed-off-by: Benoit Hudzia <benoit.hudzia@sap.com>
Signed-off-by: Petter Svard <petters@cs.umu.se>
Signed-off-by: Aidan Shribman <aidan.shribman@sap.com>

Orit Wasserman (11):
  Add cache handling functions
  Add uleb encoding/decoding functions
  Add save_block_hdr function
  Add host_from_stream_offset_versioned function
  Add XBZRLE to ram_save_block and ram_save_live
  Add MigrationParams structure
  Add XBZRLE parameters to MigrationState
  Add migration capabilties
  Add set_cachesize command
  Add XBZRLE option to migrate command
  Add XBZRLE statstics information

 arch_init.c       |  459 +++++++++++++++++++++++++++++++++++++++++++++++++---
 block-migration.c |    8 +-
 hmp-commands.hx   |   36 ++++-
 hmp.c             |   31 ++++
 hmp.h             |    2 +
 migration.c       |   66 +++++++-
 migration.h       |   31 ++++-
 monitor.c         |    7 +
 qapi-schema.json  |   57 +++++++-
 qemu-common.h     |    1 +
 qmp-commands.hx   |   64 +++++++-
 savevm.c          |  135 +++++++++++++++-
 sysemu.h          |    5 +-
 vmstate.h         |    2 +-
 14 files changed, 838 insertions(+), 66 deletions(-)

-- 
1.7.6.5

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2012-02-24  9:14 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-26 14:24 [Qemu-devel] [PATCH v7 00/11] XBRLE delta for live migration of large memory app Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 01/11] Add cache handling functions Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 02/11] Add uleb encoding/decoding functions Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 03/11] Add save_block_hdr function Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 04/11] Add host_from_stream_offset_versioned function Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 05/11] Add XBZRLE to ram_save_block and ram_save_live Orit Wasserman
2012-01-29 10:52   ` Avi Kivity
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 06/11] Add MigrationParams structure Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 07/11] Add XBZRLE parameters to MigrationState Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 08/11] Add migration capabilties Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 09/11] Add set_cachesize command Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 10/11] Add XBZRLE option to migrate command Orit Wasserman
2012-01-26 14:24 ` [Qemu-devel] [PATCH v7 11/11] Add XBZRLE statstics information Orit Wasserman
2012-02-24  8:33 ` [Qemu-devel] [PATCH v7 00/11] XBRLE delta for live migration of large memory app Stefan Hajnoczi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).