Converting images to stdout

Converting images to stdout

[Alberto Garcia]

Hi,

I haven't written here in a while :) but I have something small that I
would like to discuss.

Using qemu-img to convert an image and writing the result directly to
stdout is a question that has already been raised in the past (see
[1] for an example) and it's clear that it's generally not possible
because the images need to be seekable.

While I think that there's almost certainly no generic way to do
something like that for every combination of input and output formats,
I do think that it should be relatively easy to produce a qcow2 file
directly to stdout as long as the input file is on disk.

I'm interested in this use case, and I think that the method would be
as simple as this:

1. Decide a cluster size for the output qcow2 file.
2. Read the input file once to determine which clusters need to be
   allocated in the output file and which ones don't.
3. That infomation is enough to determine the number and contents of
   the refcount table, refcount blocks, and L1/L2 tables.
4. Write the qcow2 header + metadata + allocated data to stdout.

Since this would be qcow2-specific I would probably implement this as
a separate tool instead of adding it directly to qemu-img. This could
go to the scripts/ directory because I can imagine that such a tool
could be useful for other people.

Because this would be an external tool it would only support a qcow2
file with the default options. Other features like compression would
be out of scope.

For the same reason the input would be a raw file for simplicity,
other input files could be presented in raw format using
qemu-storage-daemon.

And that's the general idea, comments and questions are welcome.

Thanks!

Berto

[1] https://lists.nongnu.org/archive/html/qemu-discuss/2020-01/msg00014.html

Re: Converting images to stdout

[Eric Blake]

On Thu, Nov 16, 2023 at 06:48:09PM +0100, Alberto Garcia wrote:
> Hi,
> 
> I haven't written here in a while :) but I have something small that I
> would like to discuss.
> 
> Using qemu-img to convert an image and writing the result directly to
> stdout is a question that has already been raised in the past (see
> [1] for an example) and it's clear that it's generally not possible
> because the images need to be seekable.
> 
> While I think that there's almost certainly no generic way to do
> something like that for every combination of input and output formats,
> I do think that it should be relatively easy to produce a qcow2 file
> directly to stdout as long as the input file is on disk.

Indeed, it does seem like this should be easy enough to cobble together.

> 
> I'm interested in this use case, and I think that the method would be
> as simple as this:
> 
> 1. Decide a cluster size for the output qcow2 file.
> 2. Read the input file once to determine which clusters need to be
>    allocated in the output file and which ones don't.
> 3. That infomation is enough to determine the number and contents of
>    the refcount table, refcount blocks, and L1/L2 tables.
> 4. Write the qcow2 header + metadata + allocated data to stdout.

It may also be possible to write a qcow2 file that uses the external
data bit, so that you are only writing the qcow2 header + metadata,
and reusing the existing input file as the external data.

> 
> Since this would be qcow2-specific I would probably implement this as
> a separate tool instead of adding it directly to qemu-img. This could
> go to the scripts/ directory because I can imagine that such a tool
> could be useful for other people.

Also sounds reasonable.

> 
> Because this would be an external tool it would only support a qcow2
> file with the default options. Other features like compression would
> be out of scope.

Why is compression not viable?  Are you worried that the qcow2
metadata (such as refcounts) becomes too complex?

I've also wondered how easy or hard it would be to write a tool that
can take an existing qcow2 file and defragment and/or compress it
in-place (rather than having to copy it to a second qcow2 file).

> 
> For the same reason the input would be a raw file for simplicity,
> other input files could be presented in raw format using
> qemu-storage-daemon.

Yes, getting seekable input from any other format is easy enough; it's
the streaming output that is the trickier task.

> 
> And that's the general idea, comments and questions are welcome.
> 
> Thanks!
> 
> Berto
> 
> [1] https://lists.nongnu.org/archive/html/qemu-discuss/2020-01/msg00014.html
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: Converting images to stdout

[Alberto Garcia]

On Mon, Nov 20, 2023 at 05:23:27PM -0600, Eric Blake wrote:
> > I'm interested in this use case, and I think that the method would be
> > as simple as this:
> > 
> > 1. Decide a cluster size for the output qcow2 file.
> > 2. Read the input file once to determine which clusters need to be
> >    allocated in the output file and which ones don't.
> > 3. That infomation is enough to determine the number and contents of
> >    the refcount table, refcount blocks, and L1/L2 tables.
> > 4. Write the qcow2 header + metadata + allocated data to stdout.
> 
> It may also be possible to write a qcow2 file that uses the external
> data bit, so that you are only writing the qcow2 header + metadata,
> and reusing the existing input file as the external data.

Sure, although I'm not so certain about the use case here... also, the
input file might not be raw.

> > Because this would be an external tool it would only support
> > a qcow2 file with the default options. Other features like
> > compression would be out of scope.
> 
> Why is compression not viable?  Are you worried that the qcow2
> metadata (such as refcounts) becomes too complex?

Yeah, mostly... also, since the output file would be streamable it's
trivial to pipe it through gzip or whatever.

> I've also wondered how easy or hard it would be to write a tool that
> can take an existing qcow2 file and defragment and/or compress it
> in-place (rather than having to copy it to a second qcow2 file).

That sounds a bit more complex, but I guess it's doable. But not
something that I need atm :)

Berto