Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . .

Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . .

[John Snow]

Kashyap Chamarthy sent me a question asking for some clarifications on 
testing my incremental backup series. I responded off-list but he has 
asked me to reproduce my answer for the archives :)

On 04/21/2015 08:35 AM, Kashyap Chamarthy wrote:
> Hi John,
>
> I would have asked this question on #qemu on OFTC, but there are three
> to four parallel conversations ongoing at the moment. . .
>
> So, I'm testing on a branch with your "v2.5" tree from
>
>      https://github.com/jnsnow/qemu/commits/incremental-transactions
>
>      $ git checkout -b v2.5-and-v6-inc-backup jsnow/incremental-transactions
>      $ git describe
>      v2.3.0-rc3-37-g6932a32
>
> Test
> ----
>
> (1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way:
>
>    /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system-x86_64 \
>    . . .
>    -drive file=/export/full_backup.img,if=none,id=drive-ide0-0-0,format=qcow2 \
>    -qmp tcp:localhost:4444,server
>
> So, it's waiting on the server:
>
>    $ ./2-min-qemu-invoc-qmp.sh
>    char device redirected to /dev/pts/31 (label charserial0)
>    QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
>
>
> (2) Then, when I execute the `drive-backup` QMP command:
> -----------------------------------
> #!/bin/bash
> set -x
> # Test incremental backup with `drive-backup`
>
> exec 3<>/dev/tcp/localhost/4444
> echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> read response <&3
> echo $response
> echo -e "{ 'execute': 'drive-backup', 'arguments':
>        { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'dirty-bitmap', 'target':
>        '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/tests/incremental.0.img', 'mode': 'existing', 'format': 'qcow2' } }" >&3
> read response <&3
> echo $response
> -----------------------------------
>
> (3) Invoke it:
>
> $ ./invoke-drive-backup.sh
> {"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2}, "package": ""}, "capabilities": []}}
> {"return": {}}
>
>
> Now, I don't see any feedback notification on the QMP server as a result
> of the `drive-bacup` execution from step (3) unlike from a slightly old
> test of regular `drive-backup` command[1].
>
> NOTE: I tested with and without pre-creating the 'incremental.0.img'
> file, to no avail.
>
> Am I missing anything obvious?
>
>
> [1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
>

(1) It doesn't appear as though you've created a bitmap in this example?

(2) Incremental backup is designed to work basically exclusively with 
'mode': 'existing' because it relies on the user (or the management
utility/api) to keep track of what the "last backup" was.

(I had actually attempted to add that convenience feature into QEMU, but 
the sense was that the complexities of that were best left to libvirt, 
and I agreed.)

Here's the workflow (if the VM has not begun guest execution)
   (A) Create a bitmap ("bitmap0" is my usual choice)
   (B) Create an "anchor" or "tail" backup:
       This is a full backup of the drive, performed as usual.
   (C) Create the image to be used, using e.g. tail.qcow2 as
       the backing file.
   (D) Ensure that <bitmap0> is clear by issuing a clear command. Look
       at bitmaps.md for an example.
   (E) You may optionally start the VM to allow the disk to change
       at this point.
   (F) You may now issue an incremental backup command:
       { 'execute': 'drive-backup',
         'arguments': {
           'device': 'drive-ide0-0-0',
           'bitmap': 'bitmap0',
           'sync': 'dirty-bitmap',
           'target': .../tail.qcow2,
           'mode': 'existing',
           'format': 'qcow2'
         }
       }

The workflow is different if the VM has already started,
for data safety reasons. (We need extra precautions to ensure
the coherency of the incremental backup.)

   (A) Create a bitmap. ("bitmap0")
   (B) Using a single transaction, you must accomplish the following:
     (i) Create a full backup of 'drive-ide0-0-0'
     (ii) Clear bitmap0.
          This ensures that the bitmap tracks only writes that have
          occurred after the full backup was made. This is why the
          transactions are important.

          It's OK to let QEMU create the tail.qcow2 file here,
          so we don't need mode:existing.

          { 'execute': 'transaction',
            'arguments': {
              'actions': [
                { 'type': 'block-dirty-bitmap-clear',
                  'data': {
                    "node": 'drive-ide0-0-0',
                    "name": 'bitmap0'
                  }
                },
                { 'type': 'drive-backup',
                  'data': {
                    'device': 'drive-ide0-0-0',
                    'bitmap': 'bitmap0',
                    'sync': 'full',
                    'target': .../tail.qcow2,
                    'format': 'qcow2'
                  }
                }
              ]
            }
          }

   (C) We need to create the incremental backup image outside of QEMU;
       # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2
   (D) Now that we have a full backup and a synced bitmap, we can
       issue a standard incremental backup command at any time.
       { 'execute': 'drive-backup',
         'arguments': {
           'device': 'drive-ide0-0-0',
           'bitmap': 'bitmap0',
           'sync': 'dirty-bitmap',
           'target': .../tail.qcow2,
           'mode': 'existing',
           'format': 'qcow2'
         }
       }

(3) You are probably not seeing any return from this script because it 
looks as if you only read from the QMP stream once, which will just give 
you the reply from accepting the QMP command, but the event that tells 
you if it completed successfully or not occurs later, so you need one 
more read. If you use bash to send/receive QMP commands, you need to 
poll the fd to get event responses that occur later.

-- 
—js

Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . .

[Kashyap Chamarthy]

On Tue, Apr 21, 2015 at 11:34:07AM -0400, John Snow wrote:
> Kashyap Chamarthy sent me a question asking for some clarifications on
> testing my incremental backup series. I responded off-list but he has asked
> me to reproduce my answer for the archives :)

Thanks for the detailed response on both online/offline cases further
below.

Just as a side note, I wrote a short version of the same email to the
list to your v 2.5 thread. I think you sent this response before
noticing them.  Sorry for the double emails. :-)

    http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02384.html
    http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02406.html

So for the record, your details below should give clues to answer those.

[. . .]

> >Test
> >----
> >
> >(1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way:
> >
> >   /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system-x86_64 \
> >   . . .
> >   -drive file=/export/full_backup.img,if=none,id=drive-ide0-0-0,format=qcow2 \
> >   -qmp tcp:localhost:4444,server
> >
> >So, it's waiting on the server:
> >
> >   $ ./2-min-qemu-invoc-qmp.sh
> >   char device redirected to /dev/pts/31 (label charserial0)
> >   QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
> >
> >
> >(2) Then, when I execute the `drive-backup` QMP command:
> >-----------------------------------
> >#!/bin/bash
> >set -x
> ># Test incremental backup with `drive-backup`
> >
> >exec 3<>/dev/tcp/localhost/4444
> >echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> >read response <&3
> >echo $response
> >echo -e "{ 'execute': 'drive-backup', 'arguments':
> >       { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 'dirty-bitmap', 'target':
> >       '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/tests/incremental.0.img', 'mode': 'existing', 'format': 'qcow2' } }" >&3
> >read response <&3
> >echo $response
> >-----------------------------------
> >
> >(3) Invoke it:
> >
> >$ ./invoke-drive-backup.sh
> >{"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2}, "package": ""}, "capabilities": []}}
> >{"return": {}}
> >
> >
> >Now, I don't see any feedback notification on the QMP server as a result
> >of the `drive-bacup` execution from step (3) unlike from a slightly old
> >test of regular `drive-backup` command[1].
> >
> >NOTE: I tested with and without pre-creating the 'incremental.0.img'
> >file, to no avail.
> >
> >Am I missing anything obvious?
> >
> >
> >[1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
> >
> 
> (1) It doesn't appear as though you've created a bitmap in this example?

You're right! Shame on me for missing the important bit. I should have:

    { 'execute': 'block-dirty-bitmap-add', 'arguments': { 'node': 'drive-ide0-0-0', 'name': 'bitmap0' } }


> (2) Incremental backup is designed to work basically exclusively with
> 'mode': 'existing' because it relies on the user (or the management
> utility/api) to keep track of what the "last backup" was.
> 
> (I had actually attempted to add that convenience feature into QEMU, but the
> sense was that the complexities of that were best left to libvirt, and I
> agreed.)
> 
> Here's the workflow (if the VM has not begun guest execution)
>   (A) Create a bitmap ("bitmap0" is my usual choice)
>   (B) Create an "anchor" or "tail" backup:
>       This is a full backup of the drive, performed as usual.
>   (C) Create the image to be used, using e.g. tail.qcow2 as
>       the backing file.
>   (D) Ensure that <bitmap0> is clear by issuing a clear command. Look
>       at bitmaps.md for an example.
>   (E) You may optionally start the VM to allow the disk to change
>       at this point.
>   (F) You may now issue an incremental backup command:
>       { 'execute': 'drive-backup',
>         'arguments': {
>           'device': 'drive-ide0-0-0',
>           'bitmap': 'bitmap0',
>           'sync': 'dirty-bitmap',
>           'target': .../tail.qcow2,
>           'mode': 'existing',
>           'format': 'qcow2'
>         }
>       }
> 
> The workflow is different if the VM has already started,
> for data safety reasons. (We need extra precautions to ensure
> the coherency of the incremental backup.)
> 
>   (A) Create a bitmap. ("bitmap0")
>   (B) Using a single transaction, you must accomplish the following:
>     (i) Create a full backup of 'drive-ide0-0-0'
>     (ii) Clear bitmap0.
>          This ensures that the bitmap tracks only writes that have
>          occurred after the full backup was made. This is why the
>          transactions are important.
> 
>          It's OK to let QEMU create the tail.qcow2 file here,
>          so we don't need mode:existing.
> 
>          { 'execute': 'transaction',
>            'arguments': {
>              'actions': [
>                { 'type': 'block-dirty-bitmap-clear',
>                  'data': {
>                    "node": 'drive-ide0-0-0',
>                    "name": 'bitmap0'
>                  }
>                },
>                { 'type': 'drive-backup',
>                  'data': {
>                    'device': 'drive-ide0-0-0',
>                    'bitmap': 'bitmap0',
>                    'sync': 'full',
>                    'target': .../tail.qcow2,
>                    'format': 'qcow2'
>                  }
>                }
>              ]
>            }
>          }
> 
>   (C) We need to create the incremental backup image outside of QEMU;
>       # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2
>   (D) Now that we have a full backup and a synced bitmap, we can
>       issue a standard incremental backup command at any time.
>       { 'execute': 'drive-backup',
>         'arguments': {
>           'device': 'drive-ide0-0-0',
>           'bitmap': 'bitmap0',
>           'sync': 'dirty-bitmap',
>           'target': .../tail.qcow2,
>           'mode': 'existing',
>           'format': 'qcow2'
>         }
>       }
> 
> (3) You are probably not seeing any return from this script because it looks
> as if you only read from the QMP stream once, which will just give you the
> reply from accepting the QMP command, but the event that tells you if it
> completed successfully or not occurs later, so you need one more read. If
> you use bash to send/receive QMP commands, you need to poll the fd to get
> event responses that occur later.

Thanks for these details! And, also didn't know of the "poll the fd"
bit you mention above below. As this shows, I just began poking at this,
so I'll obviously have to test more with different cases.


-- 
/kashyap