Re: Maximum QMP reply size

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Thu, 15 Sept 2022 at 16:21, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > On Tue, 6 Sept 2022 at 20:41, John Snow <jsnow@redhat.com> wrote:
> > > > Hi, I suspect I have asked this before, but I didn't write it down in
> > > > a comment, so I forget my justification...
> > > >
> > > > In the QMP lib, we need to set a buffering limit for how big a QMP
> > > > message can be -- In practice, I found that the largest possible
> > > > response was the QAPI schema reply, and I set the code to this:
> > > >
> > > >     # Maximum allowable size of read buffer
> > > >     _limit = (64 * 1024)
> > > >
> > > > However, I didn't document if this was a reasonable limit or just a
> > > > "worksforme" one. I assume that there's no hard limit for the protocol
> > > > or the implementation thereof in QEMU. Is there any kind of value here
> > > > that would be more sensible than another?
> > > >
> > > > I'm worried that if replies get bigger in the future (possibly in some
> > > > degenerate case I am presently unaware of) that the library default
> > > > will become nonsensical.
> > >
> > > There are some QMP commands which return lists of things
> > > where we put no inherent limit on how many things there
> > > are in the list, like qom-list-types. We'd have to be getting
> > > a bit enthusiastic about defining types for that to get
> > > up towards 64K's worth of response, but it's not inherently
> > > impossible. I think using human-monitor-command to send
> > > an 'xp' HMP command is also a way to get back an arbitrarily
> > > large string (just ask for a lot of memory to be dumped).
> >
> > We could put size limits on xp; most Humans will only dump a few kB
> > maximum like that, any larger and you can dump to file.
> 
> Sure, we could, but why? It's not clear to me why a consumer
> of QMP needs to impose a maximum message size limit on it:

That was just a suggestion if someone did want to limit it; if 'xp'
sizing is the only thing holding us back.

> I thought it was just JSON. Fixed buffer sizes are very 1980s :-)

But generic JSON is awful for this, given there are no constraints on
field ordering, a parser has to read the whole JSON message before
being able to determine if anything in it makes sense.  You can't
generally 'stream' parse it, so do end up reading into buffers.

> If this is a common requirement then should we define something
> in the protocol where the client says "I can support messages
> up to this big" and then the server has to split things up?

I fear you'll have to do that for every command type that outputs
a lot of data; doing it generally could be tricky.

Dave

> -- PMM
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK