RFC: towards systemd socket activation in q-s-d

RFC: towards systemd socket activation in q-s-d

[Eric Blake]

In https://bugzilla.redhat.com/show_bug.cgi?id=2055229, the question
was raised on how to make qemu-storage-daemon sufficiently powerful to
be a full-blown replacement to qemu-nbd.  One of the features still
lacking is the ability to do systemd socket activation (qemu-nbd does
this, qemu-storage-daemon needs a way to do it).

But that bug further noted that systemd supports LISTEN_FDNAMES to
supply names to a passed-in fd (right now, qemu-nbd does not use
names, but merely expects one fd in LISTEN_FDS).  Dan had the idea
that it would be nice to write a systemd file that passes in a socket
name for a QMP socket, as in:

 [Socket]
 ListenStream=/var/run/myapp/qsd.qmp
 FileDescriptorName=qmp
 Service=myapp-qsd.service

and further notes that QAPI SocketAddressType supports @fd which is a
name in QMP (a previously-added fd passed through the older 'getfd'
command, rather than the newer 'add-fd' command), but an integer on
the command line.  With LISTEN_FDNAMES, we could mix systemd socket
activation with named fds for any command line usage that already
supports SocketAddressType (not limited to just q-s-d usage).

I'm at a point where I can take a shot at implementing this, but want
some feedback on whether it is better to try to shoehorn a generic
solution into the existing @fd member of the SocketAddressType union,
or whether it would be better to add yet another union member
@systemd-fd or some similar name to make it explicit when a command
line parameter wants to refer to an fd being passed through systemd
socket activation LISTEN_FDS and friends.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: RFC: towards systemd socket activation in q-s-d

[Richard W.M. Jones]

On Fri, Jan 27, 2023 at 03:26:15PM -0600, Eric Blake wrote:
> In https://bugzilla.redhat.com/show_bug.cgi?id=2055229, the question
> was raised on how to make qemu-storage-daemon sufficiently powerful to
> be a full-blown replacement to qemu-nbd.  One of the features still
> lacking is the ability to do systemd socket activation (qemu-nbd does
> this, qemu-storage-daemon needs a way to do it).
> 
> But that bug further noted that systemd supports LISTEN_FDNAMES to
> supply names to a passed-in fd (right now, qemu-nbd does not use
> names, but merely expects one fd in LISTEN_FDS).  Dan had the idea
> that it would be nice to write a systemd file that passes in a socket
> name for a QMP socket, as in:
> 
>  [Socket]
>  ListenStream=/var/run/myapp/qsd.qmp
>  FileDescriptorName=qmp
>  Service=myapp-qsd.service
> 
> and further notes that QAPI SocketAddressType supports @fd which is a
> name in QMP (a previously-added fd passed through the older 'getfd'
> command, rather than the newer 'add-fd' command), but an integer on
> the command line.  With LISTEN_FDNAMES, we could mix systemd socket
> activation with named fds for any command line usage that already
> supports SocketAddressType (not limited to just q-s-d usage).

I couldn't find a good description of LISTEN_FDNAMES anywhere, and we
don't use it in libnbd / nbdkit / qemu-nbd right now.  So here's my
interpretation of what this environment variable means ...

Currently systemd socket activation in qemu-nbd or nbdkit uses only
LISTEN_PID and LISTEN_FDS, usually with a single fd being passed.
(I'll ignore LISTEN_PID.)  So:

  LISTEN_FDS=1
    fd 3 --> NBD socket

This works because there's only one NBD protocol, it doesn't need to
be named.

However if we imagine that a superserver wants to run a webserver, it
might need to pass two sockets carrying distinct protocols (HTTP and
HTTPS).  In this case it would use:

  LISTEN_FDS=2
    fd 3 --> HTTP socket
    fd 4 --> HTTPS socket
  LISTEN_FDNAMES=http:https

LISTEN_FDNAMES is telling the webserver that the first socket is HTTP
and the second is HTTPS, so it knows which protocol to negotiate on
each socket.

I believe what you're saying above is that you'd like to have the
ability to pass multiple sockets to q-s-d carrying distinct protocols,
with the obvious ones being NBD + QMP (although other combinations
could be possible, eg. NBD + vhost + QMP?):

  LISTEN_FDS=2
    fd 3 --> NBD socket
    fd 4 --> QMP socket
  LISTEN_FDNAMES=nbd:qmp

If my understanding is correct, then this makes sense.  We might also
modify libnbd to pass LISTEN_FDNAMES=nbd in:

  https://gitlab.com/nbdkit/libnbd/-/blob/master/generator/states-connect-socket-activation.c

(Current servers would ignore the new environment variable.)

> I'm at a point where I can take a shot at implementing this, but want
> some feedback on whether it is better to try to shoehorn a generic
> solution into the existing @fd member of the SocketAddressType union,
> or whether it would be better to add yet another union member
> @systemd-fd or some similar name to make it explicit when a command
> line parameter wants to refer to an fd being passed through systemd
> socket activation LISTEN_FDS and friends.

It sounds like the q-s-d command lines will be even more convoluted
and inelegant than before.

That's fine, but please keep qemu-nbd around as a sane alternative.
It might in the end just be a wrapper that translates the command line
to q-s-d.  I don't believe it's ever going to be possible or desirable
to deprecate or remove qemu-nbd.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: RFC: towards systemd socket activation in q-s-d

[Daniel P. Berrangé]

On Fri, Jan 27, 2023 at 03:26:15PM -0600, Eric Blake wrote:
> In https://bugzilla.redhat.com/show_bug.cgi?id=2055229, the question
> was raised on how to make qemu-storage-daemon sufficiently powerful to
> be a full-blown replacement to qemu-nbd.  One of the features still
> lacking is the ability to do systemd socket activation (qemu-nbd does
> this, qemu-storage-daemon needs a way to do it).
> 
> But that bug further noted that systemd supports LISTEN_FDNAMES to
> supply names to a passed-in fd (right now, qemu-nbd does not use
> names, but merely expects one fd in LISTEN_FDS).  Dan had the idea
> that it would be nice to write a systemd file that passes in a socket
> name for a QMP socket, as in:
> 
>  [Socket]
>  ListenStream=/var/run/myapp/qsd.qmp
>  FileDescriptorName=qmp
>  Service=myapp-qsd.service
> 
> and further notes that QAPI SocketAddressType supports @fd which is a
> name in QMP (a previously-added fd passed through the older 'getfd'
> command, rather than the newer 'add-fd' command), but an integer on
> the command line.  With LISTEN_FDNAMES, we could mix systemd socket
> activation with named fds for any command line usage that already
> supports SocketAddressType (not limited to just q-s-d usage).

Yes, I think should really frame the problem more generally than
merely q-s-d and/or single sockets, so we don't tie ourselves into
something overly limited in scope.

I'm particularly thinking about our long term desire to be able to
fully manage QEMU system emulators via QMP with zero command line
guest config aspects. q-s-d is simply special in that it is fairly
close in operation to what we want to eventially achieve with the
QEMU system emulators with near 100% QMP configurability.


Currently with runtime configuration we can use getfd to pass a
FD over a UNIX socket, associating it with a name, that you can
reference in SocketAddress structs (and potentially elsewhere).

If we assume that QEMU could be started with  -config /some/guest.json
which contains a sequence of QMP commands, then the 'getfd' command is
not usable because QEMU would just be loading the commands from the
JSON file and have no UNIX socket over which to receive the FDs. 

Obviously at startup QEMU can trivially inherit the FDs from whatever
spawned it. The only task is to identify the FDs that are passed into,
and systemd defined a mechanism for this using LISTEN_FDNAMES. IOW the
socket activation can fully replace 'getfd' for purpose of initial
startup. This will get rid of the annoying difference that SocketAddress
only allows numeric FDs at startup and named FDs at runtime, by making
named FDs the consistent standard. We could thus deprecate the use of
non-named numeric FDs in SocketAddress to improve our sanity.

The question is how to define semantics for the LISTEN_FDNAMES while
also still remaining back compat with the existing QEMU utilities
that allow socket activation. Some kind of naming scheme would need
to be decided upon, as well as handling the use of activation without
LISTEN_FDNAMES BEING SET. 

> I'm at a point where I can take a shot at implementing this, but want
> some feedback on whether it is better to try to shoehorn a generic
> solution into the existing @fd member of the SocketAddressType union,
> or whether it would be better to add yet another union member
> @systemd-fd or some similar name to make it explicit when a command
> line parameter wants to refer to an fd being passed through systemd
> socket activation LISTEN_FDS and friends.

I don't think we should change SocketAddress at all, just use the
@fd member that already exists, by fixing its semantics to always
take an alphanumeric FD name and deprecate the use of pure numeric
FD numbers.

Currently the socket code has an annoying structural connection
to the monitor code, because the monitor is tracking the passed
in FDs.

It would be nice to separate this by introducing some dedicated
APIs for handling named FDs. The monitor can feed named FDs into
this from getfd, and the CLI can feed named FDS into this from
activation.   The socket code can then use these named FD handling
APIs and thus isolate/detach itself from the monitor.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: RFC: towards systemd socket activation in q-s-d

[Richard W.M. Jones]

On Mon, Jan 30, 2023 at 02:58:01PM +0000, Daniel P. Berrangé wrote:
> Obviously at startup QEMU can trivially inherit the FDs from whatever
> spawned it. The only task is to identify the FDs that are passed into,
> and systemd defined a mechanism for this using LISTEN_FDNAMES. IOW the
> socket activation can fully replace 'getfd' for purpose of initial
> startup. This will get rid of the annoying difference that SocketAddress
> only allows numeric FDs at startup and named FDs at runtime, by making
> named FDs the consistent standard. We could thus deprecate the use of
> non-named numeric FDs in SocketAddress to improve our sanity.
> 
> The question is how to define semantics for the LISTEN_FDNAMES while
> also still remaining back compat with the existing QEMU utilities
> that allow socket activation. Some kind of naming scheme would need
> to be decided upon, as well as handling the use of activation without
> LISTEN_FDNAMES being set. 

If I understand LISTEN_FDNAMES correctly, it's the names of the
protocols to be used (rather clumsily expressed through IANA
registered names from /etc/services).  It would be valid to use
LISTEN_FDNAMES=http:http for example, for a service that must use HTTP
on both sockets.

In other words it's not just names of file descriptors that you can
make up.

However as there is zero documentation for this environment variable,
who knows what it's really supposed to be ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

Re: RFC: towards systemd socket activation in q-s-d

[Daniel P. Berrangé]

On Mon, Jan 30, 2023 at 03:44:48PM +0000, Richard W.M. Jones wrote:
> On Mon, Jan 30, 2023 at 02:58:01PM +0000, Daniel P. Berrangé wrote:
> > Obviously at startup QEMU can trivially inherit the FDs from whatever
> > spawned it. The only task is to identify the FDs that are passed into,
> > and systemd defined a mechanism for this using LISTEN_FDNAMES. IOW the
> > socket activation can fully replace 'getfd' for purpose of initial
> > startup. This will get rid of the annoying difference that SocketAddress
> > only allows numeric FDs at startup and named FDs at runtime, by making
> > named FDs the consistent standard. We could thus deprecate the use of
> > non-named numeric FDs in SocketAddress to improve our sanity.
> > 
> > The question is how to define semantics for the LISTEN_FDNAMES while
> > also still remaining back compat with the existing QEMU utilities
> > that allow socket activation. Some kind of naming scheme would need
> > to be decided upon, as well as handling the use of activation without
> > LISTEN_FDNAMES being set. 
> 
> If I understand LISTEN_FDNAMES correctly, it's the names of the
> protocols to be used (rather clumsily expressed through IANA
> registered names from /etc/services).  It would be valid to use
> LISTEN_FDNAMES=http:http for example, for a service that must use HTTP
> on both sockets.

That's not accurate, it is not related to IANA service names.

> In other words it's not just names of file descriptors that you can
> make up.

By default the FD name matches the name of the systemd .socket unit
file, but it can be set to an arbitrary string using FileDescriptorName=.
It is upto the application to decide if it wants to require specific
file naming schemes, or is happy to receive the default system .socket
based names.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: RFC: towards systemd socket activation in q-s-d

[Daniel P. Berrangé]

On Mon, Jan 30, 2023 at 02:58:01PM +0000, Daniel P. Berrangé wrote:
> > I'm at a point where I can take a shot at implementing this, but want
> > some feedback on whether it is better to try to shoehorn a generic
> > solution into the existing @fd member of the SocketAddressType union,
> > or whether it would be better to add yet another union member
> > @systemd-fd or some similar name to make it explicit when a command
> > line parameter wants to refer to an fd being passed through systemd
> > socket activation LISTEN_FDS and friends.
> 
> I don't think we should change SocketAddress at all, just use the
> @fd member that already exists, by fixing its semantics to always
> take an alphanumeric FD name and deprecate the use of pure numeric
> FD numbers.
> 
> Currently the socket code has an annoying structural connection
> to the monitor code, because the monitor is tracking the passed
> in FDs.
> 
> It would be nice to separate this by introducing some dedicated
> APIs for handling named FDs. The monitor can feed named FDs into
> this from getfd, and the CLI can feed named FDS into this from
> activation.   The socket code can then use these named FD handling
> APIs and thus isolate/detach itself from the monitor.

Some worked examples of what I would like to see being possible.

First the bare minimum, allowing a zero-conf setup (ok not possible
today, but lets assume QEMU can configure machine hardware entirely
via QMP):

  $ cat /etc/system/system/myvm.service
  [Unit]
  Description=My VM
  [Service]
  Exec=/usr/bin/qemu-system-x86_64 

  $ cat /etc/system/system/myvm-qmp.socket
  [Unit]
  Description=My VM QMP
  [Socket]
  ListenStream=/var/run/myvm-qmp.socket
  FileDescriptorName=qmp

So in this example, /usr/bin/qemu-system-x86_64 would be started
with no args, and LISTEN_FDS=1 and LISTEN_FDNAMES=qmp. QEMU will
have declared that 'qmp' is a reserved activation name that results
in creation of a QMP chardev.

This would be functionally equivalent to the following more explicit
syntax, which avoids the reserved name:

  $ cat /etc/system/system/myvm.service
  [Unit]
  Description=My VM
  [Service]
  Exec=/usr/bin/qemu-system-x86_64 -chardev socket,fd=control,id=qmp0 -qmp chardev=qmp0

  $ cat /etc/system/system/myvm-qmp.socket
  [Unit]
  Description=My VM QMP
  [Socket]
  ListenStream=/var/run/myvm-qmp.socket
  FileDescriptorName=control

which is LISTEN_FDS=1, LISTEN_FDNAMES=control

A more advanced usage would be to pass many FDs for different
tasks:

  $ cat /etc/system/system/myvm.service
  [Unit]
  Description=My VM
  [Service]
  Exec=/usr/bin/qemu-system-x86_64 \
     -chardev socket,fd=control,id=qmp0 -qmp chardev=qmp0 \
     -vnc fd=vnc-listen

  $ cat /etc/system/system/myvm-qmp.socket
  [Unit]
  Description=My VM QMP
  [Socket]
  ListenStream=/var/run/myvm-qmp.socket
  FileDescriptorName=control

  $ cat /etc/system/system/myvm-vnc.socket
  [Unit]
  Description=My VM VNC
  [Socket]
  ListenStream=0.0.0.0:5901
  FileDescriptorName=vnc

which is LISTEN_FDS=2, LISTEN_FDNAMES=control,vnc


This can apply to q-s-d too with the -export flag taking a FD name.
eg here we use the implicit QMP socket, and the explicit NBD exports:

  $ cat /etc/system/system/mydisks.service
  [Unit]
  Description=My DISKS
  [Service]
  Exec=/usr/bin/qemu-storage-daemon \
     -blockdev id=pubdisk,.... \
     -blockdev id=privdisk,.... \
     -export type=nbd,id=nbdpub,node-name=pubdisk,addr.type=fd,addr.fd=nbd-public
     -export type=nbd,id=nbdpub,node-name=privdisk,addr.type=fd,addr.fd=nbd-private

  $ cat /etc/system/system/mydisks-qmp.socket
  [Unit]
  Description=My disks QMP
  [Socket]
  ListenStream=/var/run/mydisks-qmp.socket
  FileDescriptorName=qmp

  $ cat /etc/system/system/mydisks-nbd-public.socket
  [Unit]
  Description=My disks Public
  [Socket]
  ListenStream=0.0.0.0:9000
  FileDescriptorName=nbd-public

  $ cat /etc/system/system/mydisks-nbd-private.socket
  [Unit]
  Description=My disks private
  [Socket]
  ListenStream=/var/run/mydisks-nbd-private.sock
  FileDescriptorName=nbd-private


which is LISTEN_FDS=3, LISTEN_FDNAMES=qmp,nbd-public,nbd-private, giving
you a QMP server on a UNIX socket, a private disk on a NBD UNIX socket
and a public disk on a NBD TCP socket.

The above system units are overly simplified for illustration, real
world ones would have more such as SocketMode for unit sockets and
dependencies, etc

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: RFC: towards systemd socket activation in q-s-d

[Richard W.M. Jones]

On Mon, Jan 30, 2023 at 04:45:08PM +0000, Daniel P. Berrangé wrote:
> which is LISTEN_FDS=2, LISTEN_FDNAMES=control,vnc

Colon for separating the elements not comma.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

Re: RFC: towards systemd socket activation in q-s-d

[Kevin Wolf]

Am 30.01.2023 um 15:58 hat Daniel P. Berrangé geschrieben:
> > I'm at a point where I can take a shot at implementing this, but want
> > some feedback on whether it is better to try to shoehorn a generic
> > solution into the existing @fd member of the SocketAddressType union,
> > or whether it would be better to add yet another union member
> > @systemd-fd or some similar name to make it explicit when a command
> > line parameter wants to refer to an fd being passed through systemd
> > socket activation LISTEN_FDS and friends.
> 
> I don't think we should change SocketAddress at all, just use the
> @fd member that already exists, by fixing its semantics to always
> take an alphanumeric FD name and deprecate the use of pure numeric
> FD numbers.

So you're suggesting that socket activation FDs automagically show up as
named FDs in QEMU?

It's just a gut feeling, but it feels a bit like one of those convenient
shortcuts that we later regretted. So far named FDs are those that the
user explicitly created using QMP. Implicitly adding in additional ones
from another source might be trying to treat two different things the
same - you could even consider them being from two different namespaces.

This is why I would have considered a new SocketAddress variant the safe
default.

> Currently the socket code has an annoying structural connection
> to the monitor code, because the monitor is tracking the passed
> in FDs.

Do we intend socket activation FDs do work with the same operations as
named FDs created with 'getfd'? In particular, I suppose 'closefd' would
still make sense, and referring to the FD should work in the same way,
too. This might be an argument in favour of treating it as a named FD
like you suggest, because then the only different thing about it is that
it magically shows up with a name coming from systemd rather than QMP.

> It would be nice to separate this by introducing some dedicated
> APIs for handling named FDs. The monitor can feed named FDs into
> this from getfd, and the CLI can feed named FDS into this from
> activation.   The socket code can then use these named FD handling
> APIs and thus isolate/detach itself from the monitor.

That structure would make sense to me, even without adding socket
activation.

Do we have places where we would potentially use the file descriptor in
other ways than referring to it in a SocketAddress(Legacy)? If we do,
then covering socket activation at the file descriptor level is pretty
much a requirement.

Kevin

Re: RFC: towards systemd socket activation in q-s-d

[Daniel P. Berrangé]

On Tue, Jan 31, 2023 at 12:17:07PM +0100, Kevin Wolf wrote:
> Am 30.01.2023 um 15:58 hat Daniel P. Berrangé geschrieben:
> > > I'm at a point where I can take a shot at implementing this, but want
> > > some feedback on whether it is better to try to shoehorn a generic
> > > solution into the existing @fd member of the SocketAddressType union,
> > > or whether it would be better to add yet another union member
> > > @systemd-fd or some similar name to make it explicit when a command
> > > line parameter wants to refer to an fd being passed through systemd
> > > socket activation LISTEN_FDS and friends.
> > 
> > I don't think we should change SocketAddress at all, just use the
> > @fd member that already exists, by fixing its semantics to always
> > take an alphanumeric FD name and deprecate the use of pure numeric
> > FD numbers.
> 
> So you're suggesting that socket activation FDs automagically show up as
> named FDs in QEMU?

Yep.

> It's just a gut feeling, but it feels a bit like one of those convenient
> shortcuts that we later regretted. So far named FDs are those that the
> user explicitly created using QMP. Implicitly adding in additional ones
> from another source might be trying to treat two different things the
> same - you could even consider them being from two different namespaces.
> 
> This is why I would have considered a new SocketAddress variant the safe
> default.

I would say that both 'getfd' and 'LISTEN_FDS' are explicitly controlled
by the user who owns the QEMU process. The former case the user has to
talk QMP, the latter case the user has to configure systemd units (or
an compatible impl). In both cases the user gets a free choice overing
the names associated with the FDs, though systemd gives some default
naming if the user doesn't specify a choice themselves.

> > Currently the socket code has an annoying structural connection
> > to the monitor code, because the monitor is tracking the passed
> > in FDs.
> 
> Do we intend socket activation FDs do work with the same operations as
> named FDs created with 'getfd'? In particular, I suppose 'closefd' would
> still make sense, and referring to the FD should work in the same way,
> too. This might be an argument in favour of treating it as a named FD
> like you suggest, because then the only different thing about it is that
> it magically shows up with a name coming from systemd rather than QMP.

Yeah, closefd and anything else that consumes thue result of 'getfd'
ought to continue to work.

> > It would be nice to separate this by introducing some dedicated
> > APIs for handling named FDs. The monitor can feed named FDs into
> > this from getfd, and the CLI can feed named FDS into this from
> > activation.   The socket code can then use these named FD handling
> > APIs and thus isolate/detach itself from the monitor.
> 
> That structure would make sense to me, even without adding socket
> activation.
> 
> Do we have places where we would potentially use the file descriptor in
> other ways than referring to it in a SocketAddress(Legacy)? If we do,
> then covering socket activation at the file descriptor level is pretty
> much a requirement.

getfd works in combination with socket FDs and thus SocketAddress.

We also have the add-fd QMP command which passes in file/block/char
based FDs that populate the /dev/fdset/NN  namespace, and thus used
by qemu_open().  Socket activation wouldn't overlap with this, since
as its name suggests the passed FDs are only targetting stream based
FDs, aka sockets or named FIFOs.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|