qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
To: Max Reitz <mreitz@redhat.com>
Cc: "Kevin Wolf" <kwolf@redhat.com>,
	ehabkost@redhat.com, qemu-block@nongnu.org,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Richard W.M. Jones" <rjones@redhat.com>,
	qemu-devel@nongnu.org, stefanha@redhat.com,
	"Michal Suchánek" <msuchanek@suse.de>
Subject: Re: [Qemu-devel] storing machine data in qcow images?
Date: Wed, 6 Jun 2018 15:41:41 +0100	[thread overview]
Message-ID: <20180606144140.GH2660@work-vm> (raw)
In-Reply-To: <66366ae1-1db8-7121-bfcc-f2096d0c954f@redhat.com>

* Max Reitz (mreitz@redhat.com) wrote:
> On 2018-06-06 16:02, Dr. David Alan Gilbert wrote:
> > * Max Reitz (mreitz@redhat.com) wrote:
> >> On 2018-06-06 14:16, Dr. David Alan Gilbert wrote:
> >>> * Max Reitz (mreitz@redhat.com) wrote:
> >>>> On 2018-06-06 13:37, Dr. David Alan Gilbert wrote:
> >>>>> * Max Reitz (mreitz@redhat.com) wrote:
> >>>>>> On 2018-06-06 13:19, Michal Suchánek wrote:
> >>>>>>> On Wed, 6 Jun 2018 13:02:53 +0200
> >>>>>>> Max Reitz <mreitz@redhat.com> wrote:
> >>>>>>>
> >>>>>>>> On 2018-06-06 12:32, Michal Suchánek wrote:
> >>>>>>>>> On Tue, 29 May 2018 12:14:15 +0200
> >>>>>>>>> Max Reitz <mreitz@redhat.com> wrote:
> >>>>
> >>>> [...]
> >>>>
> >>>>>>>>>> Unless I have got something terribly wrong (which is indeed a
> >>>>>>>>>> possibility!), to me this proposal means basically to turn qcow2
> >>>>>>>>>> into (1) a VM description format for qemu, and (2) to turn it into
> >>>>>>>>>> an archive format on the way.  
> >>>>>>>>>
> >>>>>>>>> And if you go all the way you can store multiple disks along with
> >>>>>>>>> the VM definition so you can have the whole appliance in one file.
> >>>>>>>>> It conveniently solves the problem of synchronizing snapshots across
> >>>>>>>>> multiple disk images and the question where to store the machine
> >>>>>>>>> state if you want to suspend it.   
> >>>>>>>>
> >>>>>>>> Yeah, but why make qcow2 that format?  That's what I completely fail
> >>>>>>>> to understand.
> >>>>>>>>
> >>>>>>>> If you want to have a single VM description file that contains the VM
> >>>>>>>> configuration and some qcow2/raw/whatever files along with it for the
> >>>>>>>> guest disk data, sure, go ahead.  But why does the format of the whole
> >>>>>>>> thing need to be qcow2?
> >>>>>>>
> >>>>>>> Because then qemu can access the disk data from the image directly
> >>>>>>> without any need for extraction, copying to different file, etc.
> >>>>>>
> >>>>>> This does not explain why it needs to be qcow2.  There is absolutely no
> >>>>>> reason why you couldn't use qcow2 files in-place inside of another file.
> >>>>>
> >>>>> Because then we'd have to change the whole stack to take advantage of
> >>>>> that.  Adding a feature into qcow2 means nothing else changes.
> >>>>
> >>>> Because it's a hack, right.  Storing binary data in a qcow2 file,
> >>>> completely ignoring it in qemu (and being completely unusable to any
> >>>> potential other users of the qcow2 format[1]) and only interpreting it
> >>>> somewhere up the stack is a hack.
> >>>
> >>> It's not a hack!
> >>> Seriously it's not.
> >>> There's nothing wrong with it being aimed higher up the stack than qemu,
> >>
> >> Not really, but storing that information in a disk image file is, from
> >> my perspective.  So far, qcow2 was always just for qemu.  (Hmm...  Maybe
> >> backing links weren't, but at least they were intended for qemu originally.)
> >>
> >> So this would mix information for different layers inside qcow2 which to
> >> me sounds weird.  Maybe I just have to get used to it.
> > 
> > The important point is it's explicitly for a different layer; we're not
> > mixing it - the guest can never get to this information.
> 
> Neither can it get to bitmaps, but bitmaps are still for qemu.
> 
> >                                                           It also saves
> > the higher level management layers ever having to look at the data the
> > guest can get to, which is a security advantage.
> 
> Er, well, yes, but guessing configuration options from the guest disk
> contents is definitely a bad idea, I agree on that anyway.
> 
> > From my point of view, it really is the sticky label on the disc rather
> > than the contents of it.
> 
> Sure, which is why I wouldn't put it in qcow2.  Content and meta-content
> is what qcow2 currently stores, but not how to use it.
> 
> >>> the problem we started off with was what happens when a user downloads
> >>> a VM image and tries to import it into their VM system;
> >>
> >> Well, the VM system should choke without a config file. O:-)
> >>
> >>>                                                         weve already
> >>> got 2+ layers of management stuff in there - I want the information to
> >>> guide those layers, not form a complete set of configuration.
> >>
> >> But I do.
> >>
> >> If we store some information, I don't see why we don't store all of it.
> > 
> > Hmm, now that generally I don't like:
> 
> Me neither.
> 
> >   a) That really would make it hypervisor specific
> 
> Yes.
> 
> >   b) It would be a lot of data
> 
> Yes.
> 
> >   c) Generally, the supplier of an image doesn't know how the end-user
> >      wants it configured - although for some appliances they might.
> 
> Well, yes.  But just storing very basic information limits the use case
> to a very basic case anyway, doesn't it?  So this wouldn't be worse.
> 
> Everything beyond a very basic use case can expect the user to take 30
> seconds to download and pass a config file.
> 
> >   d) Remember the only problem we had when we got here was how to stop
> >      the user shooting themselves in the foot by connecting the wrong
> >      image to the wrong VM type.
> 
> Hm.  How exactly is that shooting yourself in the foot?  Won't it just
> not work?
> 
> >                                   So I'm expecting to use this to
> >      contain requirements, nothing more.
> 
> I assumed you'd want to relieve users of having to specify config
> options in basic use cases.  This is why I believed it would be natural
> to expand that scope.
> 
> So why is it so dangerous to connect a disk you just downloaded to e.g.
> the wrong machine type?  I assumed it just wouldn't work and you'd try
> again, until you realized that maybe you should read the download
> description and do as it says ("download this config file, pass it").

That's bad!  Stuff should just-work; it currently just works, things
should get better and easier for our users.  And anyway, not working for
EFI for exmaple can be just a blank screen.  Seriously - keep it easy
for the user!

And with 'pc' type VMs being all that's around it does just-work.

> >>>> That is not necessarily a negative point, hacks can work wonderfully
> >>>> well, and they usually are simple, that is correct.  But the thing is
> >>>> that I feel like people have grand visions of what to get out of this.
> >>>> Imagine, a single file that can configure all and any VM!
> >>>>
> >>>> But hacks usually only solve a single issue.  Once you try to extend a
> >>>> hack, it breaks down and becomes insufficient.
> >>>>
> >>>> If we want a grand vision where a single file stores the whole VM, why
> >>>> not invest the work and make it right from the start?
> >>>
> >>> Because we won't get it right; however much we bikeshed about it
> >>> we'll just end up with a mess.
> >>
> >> Sure, but the same thing applies to putting it into qcow2.  The
> >> difference is, for something outside of qcow2, throwing it away and
> >> starting over is simple.
> >>
> >> When putting it into qcow2, we can only do that if we really just put a
> >> binary blob there that isn't described in the specification.
> > 
> > Well, it's why I'm going for defined key/values that are stored in the
> > blob and only a few of them.  We've got a reasonable chance of being
> > able to define what we want from 3-4 key/values, it should be a lot
> > easier than trying to define a grand scheme.
> 
> Yes, as long as we can agree on how we can justify to future generations
> why we really only want these very few very specific values.
> 
> >>>                                  The right thing is to put in something
> >>> to hold configuration and then review the items of configuration we
> >>> add properly as we define them.
> >>
> >> OK, but review them on what terms?  Whether they are simple enough?
> > 
> > Well, I'll take simple, make sense - whatever - feel free to be the
> > maintainer for that list!
> 
> OK, none. :-P
> 
> It would need to be a very strict requirement, in any case.  Just
> "simple" or "dense" do not suffice, because those can be stretched.
> 
> >> As I said, I would want a whole configuration if we allow some
> >> configuration.
> > 
> > Well then you could go for libvirt XML as the contents; but I think
> > that's crossing layers even more.
> > (I would have veered more to it being exactly the same as an OVA
> > description except for rjones dislike of it)
> 
> Well, the format doesn't really matter for now, I think it's most
> important to first talk about what kind of scope we want.
> 
> >> (More below)
> >>
> >>>> [1] Yes, I concede that there are probably no other users of qcow2.  But
> >>>> please forgive me for assuming that qcow2 was in a sense designed to be
> >>>> a rather general image format that not only qemu could use.
> >>>
> >>> What makes it QEMU specific?  It's basically just the same key/value
> >>> setup as OVA, except putting them inside the qcow2.
> >>
> >> Well, not necessarily qemu-specific, but
> >> ${management_software}-specific, which comes down to the same.  Or,
> >> well, I'd think that, but hold on.
> >>
> >>> We could use the same keys/value definitions as OVA in the blob,
> >>> although their definitions aren't very portable either.
> >>
> >> So you're proposing that we only add options that seem portable for any VM?
> > 
> > Hmm.  We should probably split them, so there should be general options
> > (e.g. minimum-ram) but also hypervisor specifics
> > (qemu.machine-class=q35); but that doesn't mean you can't add keys
> > for multiple hypervisors into the one blob.  I mean
> > something like:
> >     minimum-ram = 1G
> >     qemu.machine-class = q35
> >     anothervm.machine-class = ....
> 
> Well, and that's my issue.  Once you have application-specific info, you
> can go wild.  And I would go wild, without a reasonable and strict
> requirement that the information we want to store has to fulfill.
> 
> For the record, I would've liked it if you'd said "only portable
> options".  But I would have replied that I would fear we'd still end up
> with someone saying "I'd like to store X and Y, let's just put them into
> the specification, then they are portable [even if only this stack
> supports them]" and we wouldn't really have won anything.

I couldn't second guess every other hypervisor on the planet to know
whether specifying a machine class would work for them.

Dave

> Max
> 



--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

  reply	other threads:[~2018-06-06 14:42 UTC|newest]

Thread overview: 157+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-05-18 15:30 [Qemu-devel] storing machine data in qcow images? Michael S. Tsirkin
2018-05-18 16:49 ` Eduardo Habkost
2018-05-18 17:09 ` Daniel P. Berrangé
2018-05-18 17:41   ` Eduardo Habkost
2018-05-19  6:05     ` Markus Armbruster
2018-05-21 18:29       ` Eduardo Habkost
2018-05-21 18:44         ` Daniel P. Berrangé
2018-05-21 19:01           ` Eduardo Habkost
2018-05-23 11:19             ` Markus Armbruster
2018-05-23 12:13               ` Eduardo Habkost
2018-05-23 16:35                 ` Markus Armbruster
2018-05-29 14:06                   ` Dr. David Alan Gilbert
2018-06-05 21:58                   ` Michal Suchánek
2018-05-21 20:18     ` Daniel P. Berrangé
2018-05-21 20:33       ` Eduardo Habkost
2018-05-24  9:58         ` Kashyap Chamarthy
2018-05-22  7:35   ` Gerd Hoffmann
2018-05-22 10:53     ` Eduardo Habkost
2018-05-22 14:19     ` Michael S. Tsirkin
2018-05-22 15:02       ` Kevin Wolf
2018-05-22 15:14         ` Eduardo Habkost
2018-05-23  2:12         ` Fam Zheng
2018-05-23  9:16           ` Kevin Wolf
2018-05-23 14:46             ` Michael S. Tsirkin
2018-05-24 11:17   ` Richard W.M. Jones
2018-05-29 14:03     ` Dr. David Alan Gilbert
2018-05-29 14:14       ` Eduardo Habkost
2018-05-29 14:51         ` Richard W.M. Jones
2018-05-29 15:31         ` Dr. David Alan Gilbert
2018-05-22  8:50 ` Philipp Hahn
2018-05-24 11:32 ` Richard W.M. Jones
2018-05-24 14:56   ` Michael S. Tsirkin
2018-05-24 15:08     ` Kevin Wolf
2018-05-24 15:19       ` Michael S. Tsirkin
2018-05-24 15:20       ` Richard W.M. Jones
2018-05-24 16:25         ` Markus Armbruster
2018-05-28 18:10   ` Max Reitz
2018-05-28 18:30     ` Richard W.M. Jones
2018-05-28 18:38       ` Kevin Wolf
2018-05-28 18:44         ` Max Reitz
2018-05-28 19:09           ` Kevin Wolf
2018-05-29  9:23             ` Max Reitz
2018-05-29 10:14               ` Kevin Wolf
2018-05-29 13:16                 ` Eduardo Habkost
2018-05-28 21:20         ` Richard W.M. Jones
2018-05-28 21:25           ` Richard W.M. Jones
2018-05-29  6:44             ` Kevin Wolf
2018-05-29 10:14               ` Max Reitz
2018-06-05  9:21                 ` Dr. David Alan Gilbert
2018-06-05 19:03                   ` Eduardo Habkost
2018-06-05 19:47                     ` Michael S. Tsirkin
2018-06-05 19:54                       ` [Qemu-devel] [Qemu-block] " Eric Blake
2018-06-05 19:58                         ` Richard W.M. Jones
2018-06-05 20:09                           ` Eric Blake
2018-06-05 20:28                             ` Michael S. Tsirkin
2018-06-05 20:46                               ` Eric Blake
2018-06-05 21:26                                 ` Michael S. Tsirkin
2018-06-06  8:07                               ` Dr. David Alan Gilbert
2018-06-06  6:23                           ` Gerd Hoffmann
2018-06-05 20:06                         ` Michael S. Tsirkin
2018-06-06  6:26                     ` [Qemu-devel] " Gerd Hoffmann
2018-06-06  9:44                     ` Dr. David Alan Gilbert
2018-06-06 13:35                       ` Eduardo Habkost
2018-06-06 11:02                   ` Max Reitz
2018-06-06 11:14                     ` Dr. David Alan Gilbert
2018-06-06 11:26                       ` Max Reitz
2018-06-06 12:00                         ` Dr. David Alan Gilbert
2018-06-06 12:59                           ` Max Reitz
2018-06-06 14:31                             ` Dr. David Alan Gilbert
2018-06-06 14:37                               ` Daniel P. Berrangé
2018-06-06 14:42                                 ` Dr. David Alan Gilbert
2018-06-06 14:51                               ` Max Reitz
2018-06-06 15:05                                 ` Dr. David Alan Gilbert
2018-06-06 15:36                                   ` Eric Blake
2018-06-06 16:11                                     ` Michal Suchánek
2018-06-06 16:37                                       ` Eric Blake
2018-06-06 16:32                                     ` Daniel P. Berrangé
2018-06-06 16:36                                       ` Dr. David Alan Gilbert
2018-06-07 10:02                                       ` Andrea Bolognani
2018-06-07 10:22                                         ` Daniel P. Berrangé
2018-06-07 11:17                                           ` Andrea Bolognani
2018-06-07 12:38                                             ` Daniel P. Berrangé
2018-06-07 13:49                                               ` Dr. David Alan Gilbert
2018-06-07 14:06                                                 ` Andrea Bolognani
2018-06-07 14:45                                                   ` Dr. David Alan Gilbert
2018-06-07 14:56                                                     ` Andrea Bolognani
2018-06-07 15:25                                                       ` Dr. David Alan Gilbert
2018-06-07 20:38                                                         ` Gerd Hoffmann
2018-06-07 10:32                                         ` Richard W.M. Jones
2018-06-07 10:35                                           ` Dr. David Alan Gilbert
2018-06-07 10:36                                           ` Daniel P. Berrangé
2018-06-07 10:54                                             ` Andrea Bolognani
2018-06-07 19:24                                               ` Laszlo Ersek
2018-06-08  8:21                                                 ` Dr. David Alan Gilbert
2018-06-08  8:41                                                   ` Daniel P. Berrangé
2018-06-08  8:53                                                     ` Dr. David Alan Gilbert
2018-06-07 21:19                                               ` Michael S. Tsirkin
2018-06-07 21:18                                             ` Michael S. Tsirkin
2018-06-07 10:51                                           ` Andrea Bolognani
2018-06-07 19:38                                             ` Laszlo Ersek
2018-06-06 17:49                                   ` Max Reitz
2018-06-06 15:09                                 ` Michael S. Tsirkin
2018-06-06 17:06                                   ` Max Reitz
2018-06-07 21:43                                     ` Michael S. Tsirkin
2018-06-09 21:34                                       ` Max Reitz
2018-06-11  2:06                                         ` Michael S. Tsirkin
2018-06-11  8:16                                           ` Michal Suchánek
2018-06-06 11:42                       ` Richard W.M. Jones
2018-06-06 11:48                         ` Daniel P. Berrangé
2018-06-06 11:53                           ` Max Reitz
2018-06-06 12:03                           ` Dr. David Alan Gilbert
2018-06-06 13:15                             ` Max Reitz
2018-06-06 12:29                           ` Richard W.M. Jones
2018-06-06 11:22                     ` [Qemu-devel] [Qemu-block] " Peter Krempa
2018-06-06 10:32                 ` [Qemu-devel] " Michal Suchánek
2018-06-06 11:02                   ` Max Reitz
2018-06-06 11:19                     ` Michal Suchánek
2018-06-06 11:32                       ` Max Reitz
2018-06-06 11:37                         ` Dr. David Alan Gilbert
2018-06-06 11:44                           ` Max Reitz
2018-06-06 12:16                             ` Dr. David Alan Gilbert
2018-06-06 13:22                               ` Max Reitz
2018-06-06 14:02                                 ` Dr. David Alan Gilbert
2018-06-06 14:33                                   ` Max Reitz
2018-06-06 14:41                                     ` Dr. David Alan Gilbert [this message]
2018-06-06 14:55                                       ` Max Reitz
2018-06-06 15:25                                         ` Michal Suchánek
2018-06-06 18:02                                           ` Max Reitz
2018-06-06 18:33                                             ` Michal Suchánek
2018-06-06 18:36                                               ` Eduardo Habkost
2018-06-07 18:27                                                 ` [Qemu-devel] [Qemu-block] " Kashyap Chamarthy
2018-06-06 13:42                             ` [Qemu-devel] " Eduardo Habkost
2018-06-06 14:55                               ` Michael S. Tsirkin
2018-06-06 14:57                                 ` Max Reitz
2018-06-11 14:10                                 ` Kevin Wolf
2018-06-06 14:46                             ` Michael S. Tsirkin
2018-06-06 15:04                               ` Max Reitz
2018-06-06 11:43                         ` Michal Suchánek
2018-06-06 11:52                           ` Max Reitz
2018-06-06 12:13                             ` Michal Suchánek
2018-06-06 13:14                               ` Max Reitz
2018-06-06 13:45                                 ` Michal Suchánek
2018-06-06 13:50                                   ` Daniel P. Berrangé
2018-06-06 14:14                                     ` Eduardo Habkost
2018-06-06 14:21                                       ` Max Reitz
2018-06-06 14:24                                       ` Daniel P. Berrangé
2018-06-06 14:17                                   ` Max Reitz
2018-06-06 16:10                                     ` Eduardo Habkost
2018-06-06 18:09                                       ` Max Reitz
2018-06-11  8:44                         ` Richard W.M. Jones
2018-06-06 11:40                     ` Richard W.M. Jones
2018-06-06 14:31                       ` Michael S. Tsirkin
2018-06-06 14:43                     ` Michael S. Tsirkin
2018-06-06 14:57                       ` Eric Blake
2018-06-06 20:39                         ` Eric Blake
2018-06-06 21:01                           ` Gerd Hoffmann
2018-06-06 15:02                       ` Max Reitz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180606144140.GH2660@work-vm \
    --to=dgilbert@redhat.com \
    --cc=ehabkost@redhat.com \
    --cc=kwolf@redhat.com \
    --cc=mreitz@redhat.com \
    --cc=mst@redhat.com \
    --cc=msuchanek@suse.de \
    --cc=qemu-block@nongnu.org \
    --cc=qemu-devel@nongnu.org \
    --cc=rjones@redhat.com \
    --cc=stefanha@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).