Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Gary R Hook]

On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
> On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
>
> [. . .]
>
> In my case, the block device is a QCOW2 disk image file. If I boot
> without using the disk image file which has the operating system, the
> domain will fail to boot, no?
>
> I see you're playing with NBD disks. I'll admit, I haven't played much
> with QEMU NBD, will have to experiment post holidays.

Back from the holidays, and back on this issue. I've learned a lot.

I've learned how to use the blockcopy command to create a local copy in 
a simple disk file:

virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait 
--verbose  --finish
virsh define my_domain.xml

and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It 
appears to be a perfect copy, as I expect it to be.

But while I see (per Kashyap's article, etc) that it can be useful in 
certain scenarios, it's not interesting to me. I would like to my copy 
to be off-system, and was hoping to use the NBD interface to accomplish 
that. So I tried this (a variant of the above), working on the same 
system because it's easier:

qemu-img create -f qcow2 /tmp/dsk.test.qcow2
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
nbd-client localhost 11112 /dev/nbd2
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain --wait --verbose --finish
virsh define my_domain.xml
nbd-client -d /dev/nbd2

and the qemu-nbd process exits, as I wish. I presume at this point that 
the new file has integrity.

I can take the qcow2 file that belongs to the domain and serve it up via 
NBD:

qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
nbd-client localhost 11112 /dev/nbd2
mount /dev/nbd2 -oloop /mnt/foo

and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.

If, however, I try to use my generated-via-NBD file, I get this:

# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
[1] 7672
# qemu-nbd: Could not find partition 1: Invalid argument

[1]+  Exit 1                  qemu-nbd --partition=1 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
[1] 7686
# qemu-nbd: Invalid partition 0
^C
[1]+  Exit 1                  qemu-nbd --partition=0 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
[1] 7699
# qemu-nbd: Could not find partition 2: Invalid argument
^C
[1]+  Exit 1                  qemu-nbd --partition=2 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
[1] 7830
# qemu-nbd: Could not find partition 3: Invalid argument

[1]+  Exit 1                  qemu-nbd --partition=3 -p11112 
$PWD/dsk.test.qcow2

I don't know what has been created, but it's not a copy of the original 
guest's disk. There's no partition there, it seems.

So yes, blockcopy works fine under certain conditions. But the NBD layer 
seems to really muck things up.

Or, more likely, I'm doing things wrong. I'm hoping someone can point 
out something obvious.

There's a recent thread about "Block Replication for Continuous 
Checkpointing" that is heading towards using NBD. I fail to understand 
how this is ever going to work, based on my explorations.


-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Kashyap Chamarthy]

On Thu, Jan 08, 2015 at 01:44:58PM -0600, Gary R Hook wrote:
> On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
> >On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
> >
> >[. . .]
> >
> >In my case, the block device is a QCOW2 disk image file. If I boot
> >without using the disk image file which has the operating system, the
> >domain will fail to boot, no?
> >
> >I see you're playing with NBD disks. I'll admit, I haven't played much
> >with QEMU NBD, will have to experiment post holidays.
> 
> Back from the holidays, and back on this issue. I've learned a lot.
> 
> I've learned how to use the blockcopy command to create a local copy in a
> simple disk file:
> 
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose
> --finish
> virsh define my_domain.xml

Yes, that's correct.

> and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It appears to
> be a perfect copy, as I expect it to be.

Good, this time you find it bootable, compared to your previous test.

> But while I see (per Kashyap's article, etc) that it can be useful in
> certain scenarios, it's not interesting to me. I would like to my copy to be
> off-system, and was hoping to use the NBD interface to accomplish that. So I
> tried this (a variant of the above), working on the same system because it's
> easier:
> 
> qemu-img create -f qcow2 /tmp/dsk.test.qcow2

A typo? You also need to provide a size here:

    $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G


For the rest, I'm afraid I still didn't manage time to test
the NBD scenario to give a meaningful response here. I'll let the others
who deal with NBD more often respond to it.

> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> nbd-client localhost 11112 /dev/nbd2
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain --wait --verbose --finish
> virsh define my_domain.xml
> nbd-client -d /dev/nbd2
> 
> and the qemu-nbd process exits, as I wish. I presume at this point that the
> new file has integrity.
> 
> I can take the qcow2 file that belongs to the domain and serve it up via
> NBD:
> 
> qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
> nbd-client localhost 11112 /dev/nbd2
> mount /dev/nbd2 -oloop /mnt/foo
> 
> and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.
> 
> If, however, I try to use my generated-via-NBD file, I get this:
> 
> # qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7672
> # qemu-nbd: Could not find partition 1: Invalid argument
> 
> [1]+  Exit 1                  qemu-nbd --partition=1 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7686
> # qemu-nbd: Invalid partition 0
> ^C
> [1]+  Exit 1                  qemu-nbd --partition=0 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7699
> # qemu-nbd: Could not find partition 2: Invalid argument
> ^C
> [1]+  Exit 1                  qemu-nbd --partition=2 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7830
> # qemu-nbd: Could not find partition 3: Invalid argument
> 
> [1]+  Exit 1                  qemu-nbd --partition=3 -p11112
> $PWD/dsk.test.qcow2
> 
> I don't know what has been created, but it's not a copy of the original
> guest's disk. There's no partition there, it seems.
> 
> So yes, blockcopy works fine under certain conditions. But the NBD layer
> seems to really muck things up.
> 
> Or, more likely, I'm doing things wrong. I'm hoping someone can point out
> something obvious.
> 
> There's a recent thread about "Block Replication for Continuous
> Checkpointing" that is heading towards using NBD. I fail to understand how
> this is ever going to work, based on my explorations.

-- 
/kashyap

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3663 bytes --]

On 01/08/2015 12:44 PM, Gary R Hook wrote:
> I've learned how to use the blockcopy command to create a local copy in
> a simple disk file:
> 
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait
> --verbose  --finish
> virsh define my_domain.xml
> 
> and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It
> appears to be a perfect copy, as I expect it to be.
> 
> But while I see (per Kashyap's article, etc) that it can be useful in
> certain scenarios, it's not interesting to me. I would like to my copy
> to be off-system,

Once you have the local dsk.copy.qcow2, you can then move it off-system
and serve it up however you'd like, including by NBD.

> and was hoping to use the NBD interface to accomplish
> that. So I tried this (a variant of the above), working on the same
> system because it's easier:
> 
> qemu-img create -f qcow2 /tmp/dsk.test.qcow2
> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> nbd-client localhost 11112 /dev/nbd2
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain --wait --verbose --finish

Missing a destination?  I'm guessing you meant /dev/nbd2 as the
pre-existing destination?

> virsh define my_domain.xml
> nbd-client -d /dev/nbd2
> 
> and the qemu-nbd process exits, as I wish. I presume at this point that
> the new file has integrity.

Where are you specifying the format?  I have not personally played with
NBD much.  But here's my guess: Even though /tmp/dsk.test.qcow2 is a
qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2.
 Thus, if you are trying to treat /dev/nbd2 as the destination of your
copy, you MUST tell qemu that the file format of the copy is to be raw
(regardless of the file format of the original that is being copied
from).  If you omit the --raw (also spelled --format=raw in newer
libvirt) parameter to the virsh blockcopy command, then libvirt has to
guess at the destination format; if the source was qcow2, then libvirt
will guess that the destination should be qcow2 as well.  But writing
qcow2 data to a raw NBD disk means you have created a nested file in
/tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file
whose contents are the raw data (not typical usage, and a bit weird to
wrap your head around).

> 
> If, however, I try to use my generated-via-NBD file, I get this:
> 
> # qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7672
> # qemu-nbd: Could not find partition 1: Invalid argument

It goes back to your format nesting mistake above - you are serving up a
nested qcow2 image rather than raw data, while the guest is trying to
treat the served up image as data.  Since the nested qcow2 data in
sector 0 of the served-up data is not a partition table, it is no wonder
that you can't find a partition.

> 
> I don't know what has been created, but it's not a copy of the original
> guest's disk. There's no partition there, it seems.

Or there is, but only if you peel off yet another layer of qcow2 :)

Try doing 'virsh blockcopy --raw /dev/nbd2 ...', to see if I was right
and that makes the difference.  Remember, blockcopy CAN change things so
that the destination (as seen by qemu) is a different format than the
source, and in the case of doing a blockcopy from a qcow2 source to a
raw-image NBD server, this is a case where you need to be explicit about
the difference in format.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Paolo Bonzini]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 08/01/2015 22:04, Eric Blake wrote:
> Where are you specifying the format?  I have not personally played
>  with NBD much.  But here's my guess: Even though 
> /tmp/dsk.test.qcow2 is a qcow2 file, the NBD server is serving up a
> RAW image through /dev/nbd2. Thus, if you are trying to treat 
> /dev/nbd2 as the destination of your copy, you MUST tell qemu that
>  the file format of the copy is to be raw (regardless of the file 
> format of the original that is being copied from).  If you omit the
> --raw (also spelled --format=raw in newer libvirt) parameter to the
> virsh blockcopy command, then libvirt has to guess at the 
> destination format; if the source was qcow2, then libvirt will 
> guess that the destination should be qcow2 as well.  But writing 
> qcow2 data to a raw NBD disk means you have created a nested file 
> in /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a 
> qcow2 file whose contents are the raw data (not typical usage, and
>  a bit weird to wrap your head around).

I think you're right.

Paolo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBAgAGBQJUrvL9AAoJEL/70l94x66DkusIAJX9/MtI65eOZeW57upOBxuX
NOEZA34X8f/JY3WUI6Fhyfeoi/EZet4lEtPDnqG5isrYYhm7TMmJFCLt0fu4pr1Z
KSOgMlGN/Ze5m+ty/XdoYn21Ujw/nmSplaMVAc0iqfSXUXnd15LWzICemjXNSnHS
zOp0y7uLEM64EygW4vHrPAZZo5JCFj2JbQiRUiWjDyT14JeJ7EWRJ3vYSrZEAclB
RVDlm/Q6p9wirFSNbjfZgo1Fn5cHvVvsGvmNvWwLOnijBGeQwOD5LaqSsuAD3ZLN
QAQw77eLnVxwES+lR96vQcVWOKfR2pECD9pDEAwTVTqn/xbw0dqWuEwMSIZbfXU=
=Asfg
-----END PGP SIGNATURE-----

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Gary R Hook]

On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:

>> qemu-img create -f qcow2 /tmp/dsk.test.qcow2
>
> A typo? You also need to provide a size here:
>
>      $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G

Yes, my mistake. The size is set to the potential size of the source 
disk, which in this case is 20G.

> For the rest, I'm afraid I still didn't manage time to test
> the NBD scenario to give a meaningful response here. I'll let the others
> who deal with NBD more often respond to it.

Well, that's the trick, right? No one that may have any experience has 
managed to pitch in to this conversation.

I can add this now: qemu-nbd seems to function like nbd-server, and 
concern itself with serving a filesystem. What's need, however, is a 
simple _file_. The challenge is getting that NBD-served thing to be 
viewed the same as a local disk file.

It would appear that the virsh option --raw is required. As far as I can 
tell, both virsh and qemu-nbd munge data with the intent of making it 
suitable for a qcow2 destination; the problem is that we don't need 
double-munging. So telling the blockcopy operation to "knock it off" 
seems to make it treat the NBD device the same as it does a simple disk 
file.

Yea!

So I find that the following:

>> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
>> nbd-client localhost 11112 /dev/nbd2
>> virsh dumpxml my_domain > my_domain.xml
>> virsh undefine my_domain
>> virsh blockcopy --domain my_domain --wait --verbose --finish

only requires the addition of "--raw" to the above command. Or, rather, 
what I really need, which is

virsh blockcopy --domain my_domain --raw

which I can then control with subsequent commands.

I'm kinda surprised no one else has tried to do this and lived to write 
about it.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Gary R Hook]

On 1/8/15 3:04 PM, Eric Blake wrote:
>
> Where are you specifying the format?  I have not personally played with
> NBD much.

This appears to be the pervasive situation. There's not much out there 
in google-land about this.

>  But here's my guess: Even though /tmp/dsk.test.qcow2 is a
> qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2.
>   Thus, if you are trying to treat /dev/nbd2 as the destination of your
> copy, you MUST tell qemu that the file format of the copy is to be raw
> (regardless of the file format of the original that is being copied
> from).  If you omit the --raw (also spelled --format=raw in newer
> libvirt) parameter to the virsh blockcopy command, then libvirt has to
> guess at the destination format; if the source was qcow2, then libvirt
> will guess that the destination should be qcow2 as well.  But writing
> qcow2 data to a raw NBD disk means you have created a nested file in
> /tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file
> whose contents are the raw data (not typical usage, and a bit weird to
> wrap your head around).

Yes, I _finally_ figured all of that out this afternoon after observing 
that --raw seemed to get everything working. Thank you for confirming 
the conclusions I arrived at independently.

I should turn this experience into a guest blog post, I suppose.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

Re: [Qemu-devel] [libvirt-users] Using virsh blockcopy -- what's it supposed to accomplish?

[Kashyap Chamarthy]

On Thu, Jan 08, 2015 at 06:04:20PM -0600, Gary R Hook wrote:
> On 1/8/15 2:21 PM, Kashyap Chamarthy wrote:
> 
> >>qemu-img create -f qcow2 /tmp/dsk.test.qcow2
> >
> >A typo? You also need to provide a size here:
> >
> >     $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G
> 
> Yes, my mistake. The size is set to the potential size of the source disk,
> which in this case is 20G.
> 
> >For the rest, I'm afraid I still didn't manage time to test
> >the NBD scenario to give a meaningful response here. I'll let the others
> >who deal with NBD more often respond to it.
> 
> Well, that's the trick, right? No one that may have any experience has
> managed to pitch in to this conversation.
>
> 
> I can add this now: qemu-nbd seems to function like nbd-server, and concern
> itself with serving a filesystem. What's need, however, is a simple _file_.
> The challenge is getting that NBD-served thing to be viewed the same as a
> local disk file.
> 
> It would appear that the virsh option --raw is required. As far as I can
> tell, both virsh and qemu-nbd munge data with the intent of making it
> suitable for a qcow2 destination; the problem is that we don't need
> double-munging. So telling the blockcopy operation to "knock it off" seems
> to make it treat the NBD device the same as it does a simple disk file.
> 
> Yea!
> 
> So I find that the following:
> 
> >>qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> >>nbd-client localhost 11112 /dev/nbd2
> >>virsh dumpxml my_domain > my_domain.xml
> >>virsh undefine my_domain
> >>virsh blockcopy --domain my_domain --wait --verbose --finish
> 
> only requires the addition of "--raw" to the above command. Or, rather, what
> I really need, which is
> 
> virsh blockcopy --domain my_domain --raw
> 
> which I can then control with subsequent commands.

Nice, I see that Eric and Paolo also concur with your observations.

> 
> I'm kinda surprised no one else has tried to do this and lived to write
> about it.

Well, the above is slightly an exaggerated statement, as you can guess
from other responses in this thread, not everyone will invest the time
to write everything they test explaining it to the world. Anyhow, the
issue seems to be resolved, and as you suggested in your other response
to write about this, that'd be useful. :-)

-- 
/kashyap