From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46458) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biKVv-000652-IF for qemu-devel@nongnu.org; Fri, 09 Sep 2016 08:02:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1biKVp-0001sU-Kt for qemu-devel@nongnu.org; Fri, 09 Sep 2016 08:02:35 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55264) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biKVp-0001rR-Dc for qemu-devel@nongnu.org; Fri, 09 Sep 2016 08:02:29 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 76DD883F43 for ; Fri, 9 Sep 2016 12:02:28 +0000 (UTC) Date: Fri, 9 Sep 2016 13:02:26 +0100 From: "Richard W.M. Jones" Message-ID: <20160909120226.GZ4121@redhat.com> References: <20160909130349.2e675b30@fiorina> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20160909130349.2e675b30@fiorina> Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [Libguestfs] Extracting files from OVA is bad List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?utf-8?B?VG9tw6HFoSBHb2xlbWJpb3Zza8O9?= Cc: libguestfs , mskrivan@redhat.com, qemu-devel@nongnu.org, dgilbert@redhat.com On Fri, Sep 09, 2016 at 01:03:49PM +0200, Tom=C3=A1=C5=A1 Golembiovsk=C3=BD= wrote: > Hi, >=20 > recently we (oVirt) have started discussing whether the way virt-v2v > handles import from OVA files is good. And I would be interested in > ideas how it can be improved. It is likely somebody already gave some > thought to this problem. >=20 > TL;DR: Extracting the OVA before import is a problem for large VMs (in > sizes of TBs). Can we change something to prevent the extraction and > work directly over OVA? Specifically virt-v2v needs to do: qemu-img create -b -f qcow2 overlay.qcow= 2 qemu-img convert overlay.qcow2 output > What we consider a huge shortcoming is the fact that whole OVA is > extracted prior to the import into a temporary directory and processed > afterwards. Under normal situation user can have up to three copies of > the VM on his drive at the end of import: >=20 > * original OVA, > * temporary extracted files (will be deleted when virt-v2v terminates= , > * converted VM. >=20 >=20 > This is not a good idea for large VMs that have hunderds of GBs or even > TBs in size. The requirements on the necessary storage space can be > lessened with proper partitioning. I.e. source OVA and converted VM > don't end up on the same drive and TMPDIR is set to put even temporary > files somewhere else. But this is not a general solution. And sometimes > the necessary space may not be available at all. >=20 >=20 > The question is how to change the import path so that virt-v2v doesn't > have to extract the OVA. I can see the following solutions: >=20 > 1) Solve it virt-v2v: create a layer for directly accessing the files > in the archive. > > 2) Solve it in QEMU: create backing method that would allow creating > qemu disk backed by the archive.=20 As long as the tar file is not compressed, accessing a file within it should be trivial. I asked Kevin if there is a way to get qemu to access a disk image at an offset within another file, but there is no such feature at the moment. It's possible with `losetup', but that requires root :-( (At this point I would normally grumble about how easy this would be with a microkernel, but I won't do that now.) David Gilbert suggested looking at qemu-nbd which has an --offset option, allowing a particular offset with another file to be accessed. If we wanted to do it entirely within virt-v2v, I think this would be the way to go - the complex logic could be hidden inside v2v/input_ova.ml The second problem is to work out the right offset to use. I suspect this is something that http://www.libarchive.org/ can do, and that package is also in RHEL. We could even imagine a qemu block backend based on libarchive. > 3) Solve it on oVirt side: use some FUSE-based tool to provide > access to the archive and pass the OVA to virt-v2v not as a file bu= t > as directory. http://www.cybernoia.de/software/archivemount/ is one such tool which can do this. It's not in RHEL, but it seems to be based on libarchive. Rich. --=20 Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rj= ones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org