Re: Best practices to handle shared objects through qemu upgrades?

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, Nov 01, 2019 at 08:14:08AM +0100, Christian Ehrhardt wrote:
> Hi everyone,
> we've got a bug report recently - on handling qemu .so's through
> upgrades - that got me wondering how to best handle it.
> After checking with Paolo yesterday that there is no obvious solution
> that I missed we agreed this should be brought up on the list for
> wider discussion.
> Maybe there already is a good best practise out there, or if it
> doesn't exist we might want to agree upon one going forward.
> Let me outline the case and the ideas brought up so far.
> 
> Case
> - You have qemu representing a Guest
> - Due to other constraints e.g. PT you can't live migrate (which would
> be preferred)
> - You haven't used a specific shared object yet - lets say RBD storage
> driver as example
> - Qemu gets an update, packaging replaces the .so files on disk
> - The Qemu process and the .so files on disk now have a mismatch in $buildid
> - If you hotplug an RBD device it will fail to load the (now new) .so

What happens when it fails to load ?  Does the user get a graceful
error message or does QEMU abort ? I'd hope the former.

> 
> On almost any other service than "qemu representing a VM" the answer
> is "restart it", some even re-exec in place to keep things up and
> running.
> 
> Ideas so far:
> a) Modules are checked by build-id, so keep them in a per build-id dir on disk
>   - qemu could be made looking preferred in -$buildid dir first
>   - do not remove the packages with .so's on upgrades
>   - needs a not-too-complex way to detect which buildids running qemu processes
>     have for packaging to be able to "autoclean later"
>   - Needs some dependency juggling for Distro packaging but IMHO can be made
>     to work if above simple "probing buildid of running qemu" would exist

So this needs a bunch of special QEMU hacks in package mgmt tools
to prevent the package upgrade & cleanup later. This does not look
like a viable strategy to me.

> 
> b) Preload the modules before upgrade
>   - One could load the .so files before upgrade
>   - The open file reference will keep the content around even with the
> on disk file gone
>   - lacking a 'load-module' command that would require fake hotplugs
> which seems wrong
>   - Required additional upgrade pre-planning
>   - kills most benefits of modular code without an actual need for it
> being loaded

Well there's two benefits to modular approach

 - Allow a single build to be selectively installed on a host or container
   image, such that the install disk footprint is reduced
 - Allow a faster startup such that huge RBD libraries dont slow down
   startup of VMs not using RBD disks.

Preloading the modules before upgrade doesn't have to the second benefit.
We just have to make sure the pre loading doesn't impact the VM startup
performance.

IOW, register a SIGUSR2 handler which preloads all modules it finds on
disk. Have a pre-uninstall option on the .so package that sends SIGUSR2
to all QEMU processes. The challenge of course is that signals are
async. You might suggest a QMP command, but only 1 process can have the
QMP monitor open at any time and that's libvirt. Adding a second QMP
monitor instance is possible but kind of gross for this purpose.

Another option would be to pre-load the modules during startup, but
do it asynchronously, so that its not blocking overall VM startup.
eg just before starting the mainloop, spawn a background thread to
load all remaining modules.

This will potentially degrade performance of the guest CPUs a bit,
but avoids the latency spike from being synchronous in the startup
path.


> c) go back to non modular build
>   - No :-)
> 
> d) anything else out there?

e) Don't do upgrades on a host with running VMs :-)

   Upgrades can break the running VM even ignoring this particular
   QEMU module scenario. 

f) Simply document that if you upgrade with running VMs that some
   features like hotplug of RBD will become unavialable. Users can
   then avoid upgrades if that matters to them.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|