[Qemu-devel] [qemu-img] support for XVA

[Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

I'm thinking if support for XVA files could be added to qemu-img
The file-format is well known (it's just a tar archive) and there are scripts
that are able to convert an XVA file to a RAW image. (ie:
https://gist.github.com/miebach/0433947bcf053de23159)

Running these script on their own is very time consuming, as you have to
extract the XVA, convert any disk image from "single chunks" to a raw image
and then use qemu-img to convert from raw to qcow.

Maybe a native support will be able to skip some steps. (like the
conversion from
"single chunks" to raw and raw to qcow2)

Re: [Qemu-devel] [qemu-img] support for XVA

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

[adding libguestfs]

On 11/15/2017 09:52 AM, Gandalf Corvotempesta wrote:
> I'm thinking if support for XVA files could be added to qemu-img
> The file-format is well known (it's just a tar archive) and there are scripts
> that are able to convert an XVA file to a RAW image. (ie:
> https://gist.github.com/miebach/0433947bcf053de23159)
> 
> Running these script on their own is very time consuming, as you have to
> extract the XVA, convert any disk image from "single chunks" to a raw image
> and then use qemu-img to convert from raw to qcow.
> 
> Maybe a native support will be able to skip some steps. (like the
> conversion from
> "single chunks" to raw and raw to qcow2)

Another possibility might be writing an nbdkit plugin that can directly
read XVA, then you can connect qemu to the NBD server provided by
nbdkit, without having to teach qemu proper how to read the file.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

XVA is a tar archive.
I don't think would be possible to directly use the image stored inside
without extracting and merging each chunks

Any random reads would be impossible to do, only a huge sequential dump to
build the raw image

Il 15 nov 2017 6:33 PM, "Eric Blake" <eblake@redhat.com> ha scritto:

[adding libguestfs]

On 11/15/2017 09:52 AM, Gandalf Corvotempesta wrote:
> I'm thinking if support for XVA files could be added to qemu-img
> The file-format is well known (it's just a tar archive) and there are
scripts
> that are able to convert an XVA file to a RAW image. (ie:
> https://gist.github.com/miebach/0433947bcf053de23159)
>
> Running these script on their own is very time consuming, as you have to
> extract the XVA, convert any disk image from "single chunks" to a raw
image
> and then use qemu-img to convert from raw to qcow.
>
> Maybe a native support will be able to skip some steps. (like the
> conversion from
> "single chunks" to raw and raw to qcow2)

Another possibility might be writing an nbdkit plugin that can directly
read XVA, then you can connect qemu to the NBD server provided by
nbdkit, without having to teach qemu proper how to read the file.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 04:52:56PM +0100, Gandalf Corvotempesta wrote:
> I'm thinking if support for XVA files could be added to qemu-img
> The file-format is well known (it's just a tar archive) and there are scripts
> that are able to convert an XVA file to a RAW image. (ie:
> https://gist.github.com/miebach/0433947bcf053de23159)
> 
> Running these script on their own is very time consuming, as you have to
> extract the XVA, convert any disk image from "single chunks" to a raw image
> and then use qemu-img to convert from raw to qcow.
> 
> Maybe a native support will be able to skip some steps. (like the
> conversion from
> "single chunks" to raw and raw to qcow2)

Why not use the offset/size support in the raw driver?

We use that to access files inside OVA files (also a tar file with a
funny extension) in virt-v2v:

https://github.com/libguestfs/libguestfs/blob/994ca8d87361a4eb52a05bb4496b074fc9cb705a/v2v/input_ova.ml#L300

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: [Qemu-devel] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On 2017-11-15 18:44, Gandalf Corvotempesta wrote:
> XVA is a tar archive.
> I don't think would be possible to directly use the image stored inside
> without extracting and merging each chunks
> 
> Any random reads would be impossible to do, only a huge sequential dump to
> build the raw image

Well, you can't just add support to qemu-img alone either.  Every image
format supported by qemu-img is one supported by qemu as a whole and
thus needs a proper block driver that needs to support random accesses
as well.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
> Well, you can't just add support to qemu-img alone either.  Every image
> format supported by qemu-img is one supported by qemu as a whole and
> thus needs a proper block driver that needs to support random accesses
> as well.

I was talking about qemu-img convert, just to convert an XVA image to
something different, in a single pass, without having to extract the
tar.

Re: [Qemu-devel] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
> 2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>> Well, you can't just add support to qemu-img alone either.  Every image
>> format supported by qemu-img is one supported by qemu as a whole and
>> thus needs a proper block driver that needs to support random accesses
>> as well.
> 
> I was talking about qemu-img convert, just to convert an XVA image to
> something different, in a single pass, without having to extract the
> tar.

I know, but that doesn't work.  qemu-img convert uses the normal
general-purpose block drivers for that.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 09:07:12PM +0100, Max Reitz wrote:
> On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
> > 2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
> >> Well, you can't just add support to qemu-img alone either.  Every image
> >> format supported by qemu-img is one supported by qemu as a whole and
> >> thus needs a proper block driver that needs to support random accesses
> >> as well.
> > 
> > I was talking about qemu-img convert, just to convert an XVA image to
> > something different, in a single pass, without having to extract the
> > tar.
> 
> I know, but that doesn't work.  qemu-img convert uses the normal
> general-purpose block drivers for that.

In any case there's no need as qemu/qemu-img can already access files
inside an uncompressed tarball using the offset/size support added to
the raw driver exactly for this purpose:

https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03945.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On 2017-11-15 21:24, Richard W.M. Jones wrote:
> On Wed, Nov 15, 2017 at 09:07:12PM +0100, Max Reitz wrote:
>> On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
>>> 2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>>>> Well, you can't just add support to qemu-img alone either.  Every image
>>>> format supported by qemu-img is one supported by qemu as a whole and
>>>> thus needs a proper block driver that needs to support random accesses
>>>> as well.
>>>
>>> I was talking about qemu-img convert, just to convert an XVA image to
>>> something different, in a single pass, without having to extract the
>>> tar.
>>
>> I know, but that doesn't work.  qemu-img convert uses the normal
>> general-purpose block drivers for that.
> 
> In any case there's no need as qemu/qemu-img can already access files
> inside an uncompressed tarball using the offset/size support added to
> the raw driver exactly for this purpose:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03945.html

If that works, yes.  To me it doesn't look like XVA is just a single raw
image inside of a tarball, but I may very well be wrong, of course.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 09:27:14PM +0100, Max Reitz wrote:
> On 2017-11-15 21:24, Richard W.M. Jones wrote:
> > On Wed, Nov 15, 2017 at 09:07:12PM +0100, Max Reitz wrote:
> >> On 2017-11-15 21:06, Gandalf Corvotempesta wrote:
> >>> 2017-11-15 20:59 GMT+01:00 Max Reitz <mreitz@redhat.com>:
> >>>> Well, you can't just add support to qemu-img alone either.  Every image
> >>>> format supported by qemu-img is one supported by qemu as a whole and
> >>>> thus needs a proper block driver that needs to support random accesses
> >>>> as well.
> >>>
> >>> I was talking about qemu-img convert, just to convert an XVA image to
> >>> something different, in a single pass, without having to extract the
> >>> tar.
> >>
> >> I know, but that doesn't work.  qemu-img convert uses the normal
> >> general-purpose block drivers for that.
> > 
> > In any case there's no need as qemu/qemu-img can already access files
> > inside an uncompressed tarball using the offset/size support added to
> > the raw driver exactly for this purpose:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03945.html
> 
> If that works, yes.  To me it doesn't look like XVA is just a single raw
> image inside of a tarball, but I may very well be wrong, of course.

Gandalf, is there an XVA file publically available (pref. not
too big) that we can look at?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> Gandalf, is there an XVA file publically available (pref. not
> too big) that we can look at?

I can try to provide one, but it's simple:

# tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
---------- 0/0           42353 1970-01-01 01:00 ova.xml
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum


You can ignore the ova.xml and just use the "Ref:175" directory.
Inside the XVA you'll fine one "Ref" directory for each virtual disk
(ref number is different for each disk)
Inside each Ref directory, you'll get tons of 1MB file, corrisponding
to the RAW image.
You have to merge these files in a single raw file with just an
exception: numbers are not sequential.
as you can see above, we have: 00000000, 00000001, 00000003

The 00000002 is missing, because it's totally blank. XenServer doesn't
export any empty block, thus it will skip the corrisponding 1MB file.
When building the raw image, you have to replace empty blocks with 1MB
full of zeros.

This is (in addition to the tar extract) the most time-consuming part.
Currently I'm rebuilding a 250GB image, with tons of files to be
merge.
If qemu-img can be patched to automatically convert this kind of
format, I can save about 3 hours (30 minutes for extracting the
tarball, and about 2 hours to merge 250-300GB image)

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 09:41:20PM +0100, Gandalf Corvotempesta wrote:
> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> > Gandalf, is there an XVA file publically available (pref. not
> > too big) that we can look at?
> 
> I can try to provide one, but it's simple:
> 
> # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> ---------- 0/0           42353 1970-01-01 01:00 ova.xml
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum
> 
> 
> You can ignore the ova.xml and just use the "Ref:175" directory.
> Inside the XVA you'll fine one "Ref" directory for each virtual disk
> (ref number is different for each disk)
> Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> to the RAW image.
> You have to merge these files in a single raw file with just an
> exception: numbers are not sequential.
> as you can see above, we have: 00000000, 00000001, 00000003
> 
> The 00000002 is missing, because it's totally blank. XenServer doesn't
> export any empty block, thus it will skip the corrisponding 1MB file.
> When building the raw image, you have to replace empty blocks with 1MB
> full of zeros.
> 
> This is (in addition to the tar extract) the most time-consuming part.
> Currently I'm rebuilding a 250GB image, with tons of files to be
> merge.
> If qemu-img can be patched to automatically convert this kind of
> format, I can save about 3 hours (30 minutes for extracting the
> tarball, and about 2 hours to merge 250-300GB image)

I guess the nbdkit approach would be better given the multiple and
missing files within the tar file.

You'll have to use ‘tar tRvf file.xva’ to extract the offset and size
of each file.  (See the function ‘find_file_in_tar’ in virt-v2v source
for exactly how).

This will give you one offset/size/filename tuple for each file.  The
plugin will then simply need to calculate which file to access to
resolve each virtual file range (or substitute zeroes for missing
files).

Note that nbdkit lets you write plugins in high-level languages like
Perl or Python which would be ideally suited to this kind of task.

You can then use "captive nbdkit" (see the manual) like this:

  nbdkit -U - \
      perl script=/path/to/xva-parser.pl file=/path/to/file.xva \
      --run 'qemu-img convert $nbd -O qcow2 output.qcow2'

At no point does the tar file need to be fully unpacked and it should
all be reasonably efficient.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 09:05:00PM +0000, Richard W.M. Jones wrote:
> This will give you one offset/size/filename tuple for each file.  The
> plugin will then simply need to calculate which file to access to
> resolve each virtual file range (or substitute zeroes for missing
> files).

By which I mean, of course, that it accesses the offset/size directly
by opening the tar file.  The file doesn't have to be extracted.

I'm still interested if you have a small XVA example you can share.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 3301 bytes --]

On 2017-11-15 21:41, Gandalf Corvotempesta wrote:
> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
>> Gandalf, is there an XVA file publically available (pref. not
>> too big) that we can look at?
> 
> I can try to provide one, but it's simple:
> 
> # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> ---------- 0/0           42353 1970-01-01 01:00 ova.xml
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum
> 
> 
> You can ignore the ova.xml and just use the "Ref:175" directory.
> Inside the XVA you'll fine one "Ref" directory for each virtual disk
> (ref number is different for each disk)
> Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> to the RAW image.
> You have to merge these files in a single raw file with just an
> exception: numbers are not sequential.
> as you can see above, we have: 00000000, 00000001, 00000003
> 
> The 00000002 is missing, because it's totally blank. XenServer doesn't
> export any empty block, thus it will skip the corrisponding 1MB file.
> When building the raw image, you have to replace empty blocks with 1MB
> full of zeros.
> 
> This is (in addition to the tar extract) the most time-consuming part.
> Currently I'm rebuilding a 250GB image, with tons of files to be
> merge.
> If qemu-img can be patched to automatically convert this kind of
> format, I can save about 3 hours (30 minutes for extracting the
> tarball, and about 2 hours to merge 250-300GB image)

Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
and a size.  And you can use the null-co driver to emulate holes.  And
qemu-img convert supports concatenation.

Taking all of that together, it should be possible to create an input
"file" per 1 MB block and concatenate all of those with qemu-img
convert.  The only issue is that it'll get you a *really* long command line.

(And you'll still have to gunzip the archive before, of course.)

I'll try to write a script, but of course it's hart without an example...

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

I'm thinking on how to prove you a sample XVA
I have to create (and populate) a VM because an empty image will result in
an empty XVA
And a VM is 300-400Mb as minimum

Il 15 nov 2017 10:30 PM, "Max Reitz" <mreitz@redhat.com> ha scritto:

> On 2017-11-15 21:41, Gandalf Corvotempesta wrote:
> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> >> Gandalf, is there an XVA file publically available (pref. not
> >> too big) that we can look at?
> >
> > I can try to provide one, but it's simple:
> >
> > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> > ---------- 0/0           42353 1970-01-01 01:00 ova.xml
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum
> >
> >
> > You can ignore the ova.xml and just use the "Ref:175" directory.
> > Inside the XVA you'll fine one "Ref" directory for each virtual disk
> > (ref number is different for each disk)
> > Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> > to the RAW image.
> > You have to merge these files in a single raw file with just an
> > exception: numbers are not sequential.
> > as you can see above, we have: 00000000, 00000001, 00000003
> >
> > The 00000002 is missing, because it's totally blank. XenServer doesn't
> > export any empty block, thus it will skip the corrisponding 1MB file.
> > When building the raw image, you have to replace empty blocks with 1MB
> > full of zeros.
> >
> > This is (in addition to the tar extract) the most time-consuming part.
> > Currently I'm rebuilding a 250GB image, with tons of files to be
> > merge.
> > If qemu-img can be patched to automatically convert this kind of
> > format, I can save about 3 hours (30 minutes for extracting the
> > tarball, and about 2 hours to merge 250-300GB image)
>
> Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
> and a size.  And you can use the null-co driver to emulate holes.  And
> qemu-img convert supports concatenation.
>
> Taking all of that together, it should be possible to create an input
> "file" per 1 MB block and concatenate all of those with qemu-img
> convert.  The only issue is that it'll get you a *really* long command
> line.
>
> (And you'll still have to gunzip the archive before, of course.)
>
> I'll try to write a script, but of course it's hart without an example...
>
> Max
>
>

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 10:42:42PM +0100, Gandalf Corvotempesta wrote:
> I'm thinking on how to prove you a sample XVA
> I have to create (and populate) a VM because an empty image will result in
> an empty XVA
> And a VM is 300-400Mb as minimum

That's fine to download.  Also you could compress it (for transfer)
which should make it smaller.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86

Some XVAs

Il 15 nov 2017 10:42 PM, "Gandalf Corvotempesta" <
gandalf.corvotempesta@gmail.com> ha scritto:

> I'm thinking on how to prove you a sample XVA
> I have to create (and populate) a VM because an empty image will result in
> an empty XVA
> And a VM is 300-400Mb as minimum
>
> Il 15 nov 2017 10:30 PM, "Max Reitz" <mreitz@redhat.com> ha scritto:
>
>> On 2017-11-15 21:41, Gandalf Corvotempesta wrote:
>> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
>> >> Gandalf, is there an XVA file publically available (pref. not
>> >> too big) that we can look at?
>> >
>> > I can try to provide one, but it's simple:
>> >
>> > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
>> > ---------- 0/0           42353 1970-01-01 01:00 ova.xml
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000000.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000001.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000003.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000004.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000005.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000006.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000007.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000009.checksum
>> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
>> > ---------- 0/0              40 1970-01-01 01:00
>> Ref:175/00000010.checksum
>> >
>> >
>> > You can ignore the ova.xml and just use the "Ref:175" directory.
>> > Inside the XVA you'll fine one "Ref" directory for each virtual disk
>> > (ref number is different for each disk)
>> > Inside each Ref directory, you'll get tons of 1MB file, corrisponding
>> > to the RAW image.
>> > You have to merge these files in a single raw file with just an
>> > exception: numbers are not sequential.
>> > as you can see above, we have: 00000000, 00000001, 00000003
>> >
>> > The 00000002 is missing, because it's totally blank. XenServer doesn't
>> > export any empty block, thus it will skip the corrisponding 1MB file.
>> > When building the raw image, you have to replace empty blocks with 1MB
>> > full of zeros.
>> >
>> > This is (in addition to the tar extract) the most time-consuming part.
>> > Currently I'm rebuilding a 250GB image, with tons of files to be
>> > merge.
>> > If qemu-img can be patched to automatically convert this kind of
>> > format, I can save about 3 hours (30 minutes for extracting the
>> > tarball, and about 2 hours to merge 250-300GB image)
>>
>> Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
>> and a size.  And you can use the null-co driver to emulate holes.  And
>> qemu-img convert supports concatenation.
>>
>> Taking all of that together, it should be possible to create an input
>> "file" per 1 MB block and concatenate all of those with qemu-img
>> convert.  The only issue is that it'll get you a *really* long command
>> line.
>>
>> (And you'll still have to gunzip the archive before, of course.)
>>
>> I'll try to write a script, but of course it's hart without an example...
>>
>> Max
>>
>>

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 165 bytes --]

On 2017-11-15 22:50, Gandalf Corvotempesta wrote:
> https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86
> 
> Some XVAs

Thanks!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Wed, Nov 15, 2017 at 10:30:31PM +0100, Max Reitz wrote:
> Hmmm...  OK, so as Rich said, you can use the raw driver with an offset
> and a size.  And you can use the null-co driver to emulate holes.  And
> qemu-img convert supports concatenation.

Nice, I didn't know qemu-img convert could concatenate :-)

So that's two distinct ways to do this, neither needing any qemu
modifications.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

On 2017-11-15 22:50, Gandalf Corvotempesta wrote:
> https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86

Hmmm, that gives me a timeout when downloading.  And another image I got
gives me an "argument list too long" -- which I had feared already...
It has a size of 3 MB and Linux "only" allows 2...

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

Il 15 nov 2017 11:28 PM, "Max Reitz" <mreitz@redhat.com> ha scritto:

Hmmm, that gives me a timeout when downloading.  And another image I got
gives me an "argument list too long" -- which I had feared already...
It has a size of 3 MB and Linux "only" allows 2...


https://cloud.vates.fr/index.php/s/jynIvCjhso47nR8/download

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

On 2017-11-15 23:28, Max Reitz wrote:
> On 2017-11-15 22:50, Gandalf Corvotempesta wrote:
>> https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86
> 
> Hmmm, that gives me a timeout when downloading.  And another image I got
> gives me an "argument list too long" -- which I had feared already...
> It has a size of 3 MB and Linux "only" allows 2...

OK, being a bit clever helped a lot.

https://xanclic.moe/convert-xva.rb -- does this work?
(It seems to works on the two example images I found...)

An example is in the code, you use it like this:

$ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva

Ref:73
$ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73 \
                   -O qcow2 -p out.qcow2
    (100.00/100%)
$ ./qemu-img info out.qcow2
image: out.qcow2
file format: qcow2
virtual size: 80G (85916123136 bytes)
disk size: 4.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On 2017-11-15 23:55, Max Reitz wrote:
> On 2017-11-15 23:28, Max Reitz wrote:
>> On 2017-11-15 22:50, Gandalf Corvotempesta wrote:
>>> https://stacklet.com/downloads/XenServer-XVA-Template-Debian-7.8-Lightweight-x86
>>
>> Hmmm, that gives me a timeout when downloading.  And another image I got
>> gives me an "argument list too long" -- which I had feared already...
>> It has a size of 3 MB and Linux "only" allows 2...
> 
> OK, being a bit clever helped a lot.
> 
> https://xanclic.moe/convert-xva.rb -- does this work?
> (It seems to works on the two example images I found...)

Note that I have no idea how to get the total image size from the
ova.xml (or wherever else), so you may have to call qemu-img resize on
the output to get it to the right size.  (It only creates an image that
covers all of the input data blocks.)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

[-- Attachment #1: Type: text/plain, Size: 2832 bytes --]

Here's my solution, as a nbdkit plugin written in Perl.

As with Max's solution I don't bother to parse the virtual size out of
the XML file, so you need to specify that on the command line
otherwise the disk will be truncated to the largest extent stored in
the file.  Also the ‘.xva’ file must not be compressed.

  $ nbdkit perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296

  $ guestfish --ro --format=raw -a nbd://localhost -i
  
  Welcome to guestfish, the guest filesystem shell for
  editing virtual machine filesystems and disk images.
  
  Type: 'help' for help on commands
        'man' to read the manual
        'quit' to quit the shell
  
  Operating system: 8.2
  /dev/sda1 mounted on /
  
  ><fs> ll /
  total 100
  drwxr-xr-x 22 root root  4096 Jan  8  2016 .
  drwxr-xr-x 19 root root  4096 Nov 16 09:50 ..
  drwxrwxr-x  2 root root  4096 Jan  8  2016 bin
  drwxr-xr-x  3 root root  4096 Jan  8  2016 boot
  drwxr-xr-x  4 root root  4096 Jan  8  2016 dev
  drwxr-xr-x 87 root root  4096 Jan  8  2016 etc
  drwxr-xr-x  3 root root  4096 Jan  8  2016 home
  lrwxrwxrwx  1 root root    31 Jan  8  2016 initrd.img -> /boot/initrd.img-3.16.0-4-amd64
  drwxr-xr-x 14 root root  4096 Jan  8  2016 lib
  drwxr-xr-x  2 root root  4096 Jan  8  2016 lib64
  drwx------  2 root root 16384 Jan  8  2016 lost+found
  drwxr-xr-x  3 root root  4096 Jan  8  2016 media
  drwxr-xr-x  2 root root  4096 Jan  8  2016 mnt
  drwxr-xr-x  2 root root  4096 Jan  8  2016 opt
  drwxr-xr-x  2 root root  4096 May  4  2015 proc
  drwx------  2 root root  4096 Jan  8  2016 root
  drwxr-xr-x  2 root root  4096 Jan  8  2016 run
  drwxr-xr-x  2 root root  4096 Jan  8  2016 sbin
  drwxr-xr-x  2 root root  4096 Jan  8  2016 srv
  drwxr-xr-x  2 root root  4096 Apr  6  2015 sys
  drwxrwxrwt  7 root root  4096 Jan  8  2016 tmp
  drwxr-xr-x 10 root root  4096 Jan  8  2016 usr
  drwxr-xr-x 11 root root  4096 Jan  8  2016 var
  lrwxrwxrwx  1 root root    27 Jan  8  2016 vmlinuz -> boot/vmlinuz-3.16.0-4-amd64

I even managed to boot the Debian 8 guest from the sample .xva file:

  $ qemu-system-x86_64 -cpu host -machine accel=kvm:tcg -m 2048 -drive file=nbd:localhost:10809,format=raw,if=virtio,snapshot=on

although it was pretty slow ...

As mentioned before you can use this to do a qemu-img convert using
captive nbdkit:

  $ nbdkit -U - \
      perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
      --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2 -p'

HTH,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[-- Attachment #2: xva-reader.pl --]
[-- Type: application/x-perl, Size: 3227 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-16 11:01 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> As mentioned before you can use this to do a qemu-img convert using
> captive nbdkit:
>
>   $ nbdkit -U - \
>       perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
>       --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2 -p'


What if XVA is hosting 2 or more partitions ? Which partition are you
converting with this command ?

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

2017-11-15 23:55 GMT+01:00 Max Reitz <mreitz@redhat.com>:
> https://xanclic.moe/convert-xva.rb -- does this work?
> (It seems to works on the two example images I found...)
>
> An example is in the code, you use it like this:
>
> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73


It doesn't work on huge images:

# time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
/usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
qemu-img (Errno::E2BIG)
from /usr/bin/convert-xva.rb:119:in `<main>'

real 9m41.414s
user 0m19.280s
sys 0m3.260s

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

On Thu, Nov 16, 2017 at 11:07:10AM +0100, Gandalf Corvotempesta wrote:
> 2017-11-16 11:01 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> > As mentioned before you can use this to do a qemu-img convert using
> > captive nbdkit:
> >
> >   $ nbdkit -U - \
> >       perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \
> >       --run 'qemu-img convert -f raw $nbd -O qcow2 /var/tmp/output.qcow2 -p'
> 
> 
> What if XVA is hosting 2 or more partitions ? Which partition are you
> converting with this command ?

See "XXX" comments in the code, left as an exercise to the reader :-)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Tomáš Golembiovský]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]

On Wed, 15 Nov 2017 21:41:20 +0100
Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:

> 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:
> > Gandalf, is there an XVA file publically available (pref. not
> > too big) that we can look at?
> 
> I can try to provide one, but it's simple:
> 
> # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> ---------- 0/0           42353 1970-01-01 01:00 ova.xml
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
> ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
> ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum
> 
> 
> You can ignore the ova.xml and just use the "Ref:175" directory.
> Inside the XVA you'll fine one "Ref" directory for each virtual disk
> (ref number is different for each disk)
> Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> to the RAW image.
> You have to merge these files in a single raw file with just an
> exception: numbers are not sequential.
> as you can see above, we have: 00000000, 00000001, 00000003
> 
> The 00000002 is missing, because it's totally blank. XenServer doesn't
> export any empty block, thus it will skip the corrisponding 1MB file.
> When building the raw image, you have to replace empty blocks with 1MB
> full of zeros.
> 
> This is (in addition to the tar extract) the most time-consuming part.
> Currently I'm rebuilding a 250GB image, with tons of files to be
> merge.
> If qemu-img can be patched to automatically convert this kind of
> format, I can save about 3 hours (30 minutes for extracting the
> tarball, and about 2 hours to merge 250-300GB image)
> 

Hi,

I like what Richard and Max came up with. Pretty nifty solutions.
While there's nothing wrong with them I decided to take my own shot at
it. Since the blocks in tar file are pieces of raw image there is no
conversion happening. All in all it's just moving bytes from one place
to another. That means there shouldn't be a need for any heavy
machinery, right? :)

Attached is a shell script that uses dd to do the byte-shuffling.
I'm pretty sure this could even be safely parallelized by running
multiple instances of dd at the same time (e.g. with xargs). But I did
not try that.


    Tomas

-- 
Tomáš Golembiovský <tgolembi@redhat.com>

[-- Attachment #2: xva2raw.sh --]
[-- Type: application/x-shellscript, Size: 816 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Tomáš Golembiovský]

On Thu, 16 Nov 2017 13:56:16 +0100
Tomáš Golembiovský <tgolembi@redhat.com> wrote:

> On Wed, 15 Nov 2017 21:41:20 +0100
> Gandalf Corvotempesta <gandalf.corvotempesta@gmail.com> wrote:
> 
> > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones <rjones@redhat.com>:  
> > > Gandalf, is there an XVA file publically available (pref. not
> > > too big) that we can look at?  
> > 
> > I can try to provide one, but it's simple:
> > 
> > # tar tvf 20160630_124823_aa72_.xva.gz | head -n 50
> > ---------- 0/0           42353 1970-01-01 01:00 ova.xml
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000000
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000000.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000001
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000001.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000003
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000003.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000004
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000004.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000005
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000005.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000006
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000006.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000007
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000007.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000009
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000009.checksum
> > ---------- 0/0         1048576 1970-01-01 01:00 Ref:175/00000010
> > ---------- 0/0              40 1970-01-01 01:00 Ref:175/00000010.checksum
> > 
> > 
> > You can ignore the ova.xml and just use the "Ref:175" directory.
> > Inside the XVA you'll fine one "Ref" directory for each virtual disk
> > (ref number is different for each disk)
> > Inside each Ref directory, you'll get tons of 1MB file, corrisponding
> > to the RAW image.
> > You have to merge these files in a single raw file with just an
> > exception: numbers are not sequential.
> > as you can see above, we have: 00000000, 00000001, 00000003
> > 
> > The 00000002 is missing, because it's totally blank. XenServer doesn't
> > export any empty block, thus it will skip the corrisponding 1MB file.
> > When building the raw image, you have to replace empty blocks with 1MB
> > full of zeros.
> > 
> > This is (in addition to the tar extract) the most time-consuming part.
> > Currently I'm rebuilding a 250GB image, with tons of files to be
> > merge.
> > If qemu-img can be patched to automatically convert this kind of
> > format, I can save about 3 hours (30 minutes for extracting the
> > tarball, and about 2 hours to merge 250-300GB image)
> >   
> 
> Hi,
> 
> I like what Richard and Max came up with. Pretty nifty solutions.
> While there's nothing wrong with them I decided to take my own shot at
> it. Since the blocks in tar file are pieces of raw image there is no
> conversion happening. All in all it's just moving bytes from one place
> to another. That means there shouldn't be a need for any heavy
> machinery, right? :)
> 
> Attached is a shell script that uses dd to do the byte-shuffling.
> I'm pretty sure this could even be safely parallelized by running
> multiple instances of dd at the same time (e.g. with xargs). But I did
> not try that.

Oh yes, and one more thing: as with the other solutions I didn't bother
reading the XML for the target size. So resize may be necessary
afterwards.

    Tomas

-- 
Tomáš Golembiovský <tgolembi@redhat.com>

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Richard W.M. Jones]

I think it's even possible to modify that shell script to pipe into
nbdkit and from there to ‘qemu-img convert’.  I'm too lazy to actually
do that right now, but the basic idea is here:

  https://rwmj.wordpress.com/2014/10/14/streaming-nbd-server/

You'll probably have to add ‘-m 1’ to the qemu-img convert command line.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1708 bytes --]

On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
> 2017-11-15 23:55 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>> https://xanclic.moe/convert-xva.rb -- does this work?
>> (It seems to works on the two example images I found...)
>>
>> An example is in the code, you use it like this:
>>
>> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73
> 
> 
> It doesn't work on huge images:
> 
> # time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
> /usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
> qemu-img (Errno::E2BIG)
> from /usr/bin/convert-xva.rb:119:in `<main>'
> 
> real 9m41.414s
> user 0m19.280s
> sys 0m3.260s

Well, there is this:

https://github.com/XanClic/qemu.rb/blob/master/examples/convert-xva-online.rb

That should get around that restriction, but currently it is stupidly
slow (because it just issues all 1 MB copy operations simultaneously),
and it doesn't use qemu-img convert.  Instead, it uses a real qemu with
block jobs.  Ideally, you'd use it like this:

$ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva
Ref:73      81936 M
$ qemu-img create -f qcow2 [whatever options you need] disk1.qcow2 \
    81936M
Formatting 'disk1.qcow2', fmt=qcow2 size=85916123136 encryption=off
cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva \
    Ref:73 \
    '{"driver":"qcow2",
      "detect-zeroes":"unmap",
      "discard":"unmap",
      "file":{"driver":"file","filename":"disk1.qcow2"}}'
Adding block devices...
Starting block jobs...
100.00 % of jobs completed

(Maybe I can get it faster.)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

[Gandalf Corvotempesta]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

I did something different, that will build a raw image directly from a
xenserver export, on the fly.
Compared the resulting file (via MD5) with xenmygrate.py and there is a match.

Currently, this is the faster way to convert a XenServer image to a
raw file. Don't need to wait
for export, tar extract and conversion. It does all of that, at the
same time during the VM export in a single pass.

If someone interested in testing it, it would be apreciated.

2017-11-16 15:02 GMT+01:00 Max Reitz <mreitz@redhat.com>:
> On 2017-11-16 11:08, Gandalf Corvotempesta wrote:
>> 2017-11-15 23:55 GMT+01:00 Max Reitz <mreitz@redhat.com>:
>>> https://xanclic.moe/convert-xva.rb -- does this work?
>>> (It seems to works on the two example images I found...)
>>>
>>> An example is in the code, you use it like this:
>>>
>>> $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva Ref:73
>>
>>
>> It doesn't work on huge images:
>>
>> # time convert-xva.rb 20171115_193814_efff_.xva Ref:10 -O qcow2 disk1.qcow2
>> /usr/bin/convert-xva.rb:119:in `exec': Argument list too long -
>> qemu-img (Errno::E2BIG)
>> from /usr/bin/convert-xva.rb:119:in `<main>'
>>
>> real 9m41.414s
>> user 0m19.280s
>> sys 0m3.260s
>
> Well, there is this:
>
> https://github.com/XanClic/qemu.rb/blob/master/examples/convert-xva-online.rb
>
> That should get around that restriction, but currently it is stupidly
> slow (because it just issues all 1 MB copy operations simultaneously),
> and it doesn't use qemu-img convert.  Instead, it uses a real qemu with
> block jobs.  Ideally, you'd use it like this:
>
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva
> Ref:73      81936 M
> $ qemu-img create -f qcow2 [whatever options you need] disk1.qcow2 \
>     81936M
> Formatting 'disk1.qcow2', fmt=qcow2 size=85916123136 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> $ ./convert-xva-online.rb ~/Downloads/stats-appliance-2.36.020502.xva \
>     Ref:73 \
>     '{"driver":"qcow2",
>       "detect-zeroes":"unmap",
>       "discard":"unmap",
>       "file":{"driver":"file","filename":"disk1.qcow2"}}'
> Adding block devices...
> Starting block jobs...
> 100.00 % of jobs completed
>
> (Maybe I can get it faster.)
>
> Max
>

[-- Attachment #2: xva_conv.sh --]
[-- Type: application/x-sh, Size: 1146 bytes --]