From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54130) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YkaBh-0002AF-KP for qemu-devel@nongnu.org; Tue, 21 Apr 2015 11:34:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YkaBe-00020W-AK for qemu-devel@nongnu.org; Tue, 21 Apr 2015 11:34:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42970) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YkaBe-00020B-39 for qemu-devel@nongnu.org; Tue, 21 Apr 2015 11:34:10 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t3LFY8VT028829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 21 Apr 2015 11:34:09 -0400 Message-ID: <55366DEF.40708@redhat.com> Date: Tue, 21 Apr 2015 11:34:07 -0400 From: John Snow MIME-Version: 1.0 References: <20150421123547.GE5761@tesla.redhat.com> In-Reply-To: <20150421123547.GE5761@tesla.redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . . List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kashyap Chamarthy Cc: Kevin Wolf , qemu-devel , Stefan Hajnoczi Kashyap Chamarthy sent me a question asking for some clarifications on=20 testing my incremental backup series. I responded off-list but he has=20 asked me to reproduce my answer for the archives :) On 04/21/2015 08:35 AM, Kashyap Chamarthy wrote: > Hi John, > > I would have asked this question on #qemu on OFTC, but there are three > to four parallel conversations ongoing at the moment. . . > > So, I'm testing on a branch with your "v2.5" tree from > > https://github.com/jnsnow/qemu/commits/incremental-transactions > > $ git checkout -b v2.5-and-v6-inc-backup jsnow/incremental-transac= tions > $ git describe > v2.3.0-rc3-37-g6932a32 > > Test > ---- > > (1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way: > > /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system= -x86_64 \ > . . . > -drive file=3D/export/full_backup.img,if=3Dnone,id=3Ddrive-ide0-0-0,= format=3Dqcow2 \ > -qmp tcp:localhost:4444,server > > So, it's waiting on the server: > > $ ./2-min-qemu-invoc-qmp.sh > char device redirected to /dev/pts/31 (label charserial0) > QEMU waiting for connection on: disconnected:tcp:localhost:4444,serv= er > > > (2) Then, when I execute the `drive-backup` QMP command: > ----------------------------------- > #!/bin/bash > set -x > # Test incremental backup with `drive-backup` > > exec 3<>/dev/tcp/localhost/4444 > echo -e "{ 'execute': 'qmp_capabilities' }" >&3 > read response <&3 > echo $response > echo -e "{ 'execute': 'drive-backup', 'arguments': > { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'dirt= y-bitmap', 'target': > '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/test= s/incremental.0.img', 'mode': 'existing', 'format': 'qcow2' } }" >&3 > read response <&3 > echo $response > ----------------------------------- > > (3) Invoke it: > > $ ./invoke-drive-backup.sh > {"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2}, "pa= ckage": ""}, "capabilities": []}} > {"return": {}} > > > Now, I don't see any feedback notification on the QMP server as a resul= t > of the `drive-bacup` execution from step (3) unlike from a slightly old > test of regular `drive-backup` command[1]. > > NOTE: I tested with and without pre-creating the 'incremental.0.img' > file, to no avail. > > Am I missing anything obvious? > > > [1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt > (1) It doesn't appear as though you've created a bitmap in this example? (2) Incremental backup is designed to work basically exclusively with=20 'mode': 'existing' because it relies on the user (or the management utility/api) to keep track of what the "last backup" was. (I had actually attempted to add that convenience feature into QEMU, but=20 the sense was that the complexities of that were best left to libvirt,=20 and I agreed.) Here's the workflow (if the VM has not begun guest execution) (A) Create a bitmap ("bitmap0" is my usual choice) (B) Create an "anchor" or "tail" backup: This is a full backup of the drive, performed as usual. (C) Create the image to be used, using e.g. tail.qcow2 as the backing file. (D) Ensure that is clear by issuing a clear command. Look at bitmaps.md for an example. (E) You may optionally start the VM to allow the disk to change at this point. (F) You may now issue an incremental backup command: { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'dirty-bitmap', 'target': .../tail.qcow2, 'mode': 'existing', 'format': 'qcow2' } } The workflow is different if the VM has already started, for data safety reasons. (We need extra precautions to ensure the coherency of the incremental backup.) (A) Create a bitmap. ("bitmap0") (B) Using a single transaction, you must accomplish the following: (i) Create a full backup of 'drive-ide0-0-0' (ii) Clear bitmap0. This ensures that the bitmap tracks only writes that have occurred after the full backup was made. This is why the transactions are important. It's OK to let QEMU create the tail.qcow2 file here, so we don't need mode:existing. { 'execute': 'transaction', 'arguments': { 'actions': [ { 'type': 'block-dirty-bitmap-clear', 'data': { "node": 'drive-ide0-0-0', "name": 'bitmap0' } }, { 'type': 'drive-backup', 'data': { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'full', 'target': .../tail.qcow2, 'format': 'qcow2' } } ] } } (C) We need to create the incremental backup image outside of QEMU; # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2 (D) Now that we have a full backup and a synced bitmap, we can issue a standard incremental backup command at any time. { 'execute': 'drive-backup', 'arguments': { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'dirty-bitmap', 'target': .../tail.qcow2, 'mode': 'existing', 'format': 'qcow2' } } (3) You are probably not seeing any return from this script because it=20 looks as if you only read from the QMP stream once, which will just give=20 you the reply from accepting the QMP command, but the event that tells=20 you if it completed successfully or not occurs later, so you need one=20 more read. If you use bash to send/receive QMP commands, you need to=20 poll the fd to get event responses that occur later. --=20 =97js