From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97D2EC3F2CD for ; Wed, 4 Mar 2020 09:39:25 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5FE0D2073D for ; Wed, 4 Mar 2020 09:39:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5FE0D2073D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=canonical.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j9QUu-0006He-G3 for qemu-devel@archiver.kernel.org; Wed, 04 Mar 2020 04:39:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:32971) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j9QUC-0005pf-46 for qemu-devel@nongnu.org; Wed, 04 Mar 2020 04:38:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j9QU9-00081l-T0 for qemu-devel@nongnu.org; Wed, 04 Mar 2020 04:38:40 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:35311) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j9QU9-0007z7-Jb for qemu-devel@nongnu.org; Wed, 04 Mar 2020 04:38:37 -0500 Received: from mail-ot1-f69.google.com ([209.85.210.69]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1j9QU7-0007In-Dr for qemu-devel@nongnu.org; Wed, 04 Mar 2020 09:38:35 +0000 Received: by mail-ot1-f69.google.com with SMTP id x21so721043otp.6 for ; Wed, 04 Mar 2020 01:38:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=k9G8OPFnChlEiNPznUCFrYXWPPkqrFzK/jYCnbaEJks=; b=i2DhJYVE89xPJDR4om/J4CfBE2hTvTMM4YhHSJN2ZAI08zkJCJS86YIvYmPIstHHoN pfSx3zHgCK4N4DWshENuxpaVj8n+OYZ2Ex0qYw98Rg/721T6GDayUNCsPq/mP5tJNGUx BiS2NuTm9pNyCZuF3LeSu2JW9veqqkb4QE9zepbxLBLqyuwGyMOwMxrJ+JLgtZz9O7C/ 15i8Rg7MQaOWR6v/w+QxwvLk5ifTqSFDSw2+hMLb9ZONAJzySf5mEmFSWz22f8+B/e6c ZTBJhCJ7XKQDNxB0FrPFhbZxDZ58COfuW7a3Nf9Q/vaBmHSREwWCkg7NqXdTF/j8/4Jq Sj2w== X-Gm-Message-State: ANhLgQ3ZyxT1bhbqO10ijZX0/2waUQdgjNgmJbCVh8lHULNpFERUqy93 RF9GoSPzqE1PYKyhmC8Y/hVdjuhkzPc8O4bwN7g1yszWCkWKt+NFVqqwnoVf/q7RGopFu7CDwSm co0JirVzR1WZp4S9DZXYceChu0Z7WRV4tjOcTIKI4Ty04rqMM X-Received: by 2002:aca:dc56:: with SMTP id t83mr1155693oig.105.1583314714340; Wed, 04 Mar 2020 01:38:34 -0800 (PST) X-Google-Smtp-Source: ADFU+vvu2rJ/OEvP43yoaAxWflHxU01/XAxhsp9hGWtPbMc4TjYZXDCTG5MOYEyT78L9xwx8q1FVMMext3Dmzf39+k8= X-Received: by 2002:aca:dc56:: with SMTP id t83mr1155686oig.105.1583314714119; Wed, 04 Mar 2020 01:38:34 -0800 (PST) MIME-Version: 1.0 References: <20191101093403.GE11296@redhat.com> In-Reply-To: <20191101093403.GE11296@redhat.com> From: Christian Ehrhardt Date: Wed, 4 Mar 2020 10:37:44 +0100 Message-ID: Subject: Re: Best practices to handle shared objects through qemu upgrades? To: =?UTF-8?Q?Daniel_P=2E_Berrang=C3=A9?= Content-Type: multipart/alternative; boundary="0000000000001a63ed05a0042fa0" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 91.189.89.112 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , qemu-devel Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --0000000000001a63ed05a0042fa0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Nov 1, 2019 at 10:34 AM Daniel P. Berrang=C3=A9 wrote: > On Fri, Nov 01, 2019 at 08:14:08AM +0100, Christian Ehrhardt wrote: > > Hi everyone, > > we've got a bug report recently - on handling qemu .so's through > > upgrades - that got me wondering how to best handle it. > > After checking with Paolo yesterday that there is no obvious solution > > that I missed we agreed this should be brought up on the list for > > wider discussion. > > Maybe there already is a good best practise out there, or if it > > doesn't exist we might want to agree upon one going forward. > > Let me outline the case and the ideas brought up so far. > > > > Case > > - You have qemu representing a Guest > > - Due to other constraints e.g. PT you can't live migrate (which would > > be preferred) > > - You haven't used a specific shared object yet - lets say RBD storage > > driver as example > > - Qemu gets an update, packaging replaces the .so files on disk > > - The Qemu process and the .so files on disk now have a mismatch in > $buildid > > - If you hotplug an RBD device it will fail to load the (now new) .so > > What happens when it fails to load ? Does the user get a graceful > error message or does QEMU abort ? I'd hope the former. > > > > > On almost any other service than "qemu representing a VM" the answer > > is "restart it", some even re-exec in place to keep things up and > > running. > > > > Ideas so far: > > a) Modules are checked by build-id, so keep them in a per build-id dir > on disk > > - qemu could be made looking preferred in -$buildid dir first > > - do not remove the packages with .so's on upgrades > > - needs a not-too-complex way to detect which buildids running qemu > processes > > have for packaging to be able to "autoclean later" > > - Needs some dependency juggling for Distro packaging but IMHO can be > made > > to work if above simple "probing buildid of running qemu" would exi= st > > So this needs a bunch of special QEMU hacks in package mgmt tools > to prevent the package upgrade & cleanup later. This does not look > like a viable strategy to me. > > > > > b) Preload the modules before upgrade > > - One could load the .so files before upgrade > > - The open file reference will keep the content around even with the > > on disk file gone > > - lacking a 'load-module' command that would require fake hotplugs > > which seems wrong > > - Required additional upgrade pre-planning > > - kills most benefits of modular code without an actual need for it > > being loaded > > Well there's two benefits to modular approach > > - Allow a single build to be selectively installed on a host or containe= r > image, such that the install disk footprint is reduced > - Allow a faster startup such that huge RBD libraries dont slow down > startup of VMs not using RBD disks. > > Preloading the modules before upgrade doesn't have to the second benefit. > We just have to make sure the pre loading doesn't impact the VM startup > performance. > > IOW, register a SIGUSR2 handler which preloads all modules it finds on > disk. Have a pre-uninstall option on the .so package that sends SIGUSR2 > to all QEMU processes. The challenge of course is that signals are > async. You might suggest a QMP command, but only 1 process can have the > QMP monitor open at any time and that's libvirt. Adding a second QMP > monitor instance is possible but kind of gross for this purpose. > > Another option would be to pre-load the modules during startup, but > do it asynchronously, so that its not blocking overall VM startup. > eg just before starting the mainloop, spawn a background thread to > load all remaining modules. > > This will potentially degrade performance of the guest CPUs a bit, > but avoids the latency spike from being synchronous in the startup > path. > > > > c) go back to non modular build > > - No :-) > > > > d) anything else out there? > > e) Don't do upgrades on a host with running VMs :-) > > Upgrades can break the running VM even ignoring this particular > QEMU module scenario. > > f) Simply document that if you upgrade with running VMs that some > features like hotplug of RBD will become unavialable. Users can > then avoid upgrades if that matters to them. > Hi, I've come back to this after a while and now think all the pre-load or load-command Ideas we had were in vain. They would be overly complex and need a lot of integration into different places to trigger them. All of that would not be well integrated in the trigger of the issue itself which usually is a package upgrade. But qemu already does try to load modules from different places and with a slight extension there I think we could provide something that packaging (the actual place knowing about upgrades) can use to avoid this issue. I'll reply to this thread with a patch for your consideration in a few minutes. There is already a Ubuntu 20.04 test build with the qemu and packaging changes in [1]. The related Debian/Ubuntu packaging changes themselves can be seen in [2]. I hope that helps to illustrate how it would work overall [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3961 [2]: https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/log/?h=3Dbug-1847361= -miss-old-so-on-upgrade-UBUNTU > Regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > > --=20 Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd --0000000000001a63ed05a0042fa0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Nov 1, 2019 at 10:34 AM Danie= l P. Berrang=C3=A9 <berrange@redh= at.com> wrote:
On Fri, Nov 01, 2019 at 08:14:08AM +0100, Christian Ehrhardt wrote: > Hi everyone,
> we've got a bug report recently - on handling qemu .so's throu= gh
> upgrades - that got me wondering how to best handle it.
> After checking with Paolo yesterday that there is no obvious solution<= br> > that I missed we agreed this should be brought up on the list for
> wider discussion.
> Maybe there already is a good best practise out there, or if it
> doesn't exist we might want to agree upon one going forward.
> Let me outline the case and the ideas brought up so far.
>
> Case
> - You have qemu representing a Guest
> - Due to other constraints e.g. PT you can't live migrate (which w= ould
> be preferred)
> - You haven't used a specific shared object yet - lets say RBD sto= rage
> driver as example
> - Qemu gets an update, packaging replaces the .so files on disk
> - The Qemu process and the .so files on disk now have a mismatch in $b= uildid
> - If you hotplug an RBD device it will fail to load the (now new) .so<= br>
What happens when it fails to load ?=C2=A0 Does the user get a graceful
error message or does QEMU abort ? I'd hope the former.

>
> On almost any other service than "qemu representing a VM" th= e answer
> is "restart it", some even re-exec in place to keep things u= p and
> running.
>
> Ideas so far:
> a) Modules are checked by build-id, so keep them in a per build-id dir= on disk
>=C2=A0 =C2=A0- qemu could be made looking preferred in -$buildid dir fi= rst
>=C2=A0 =C2=A0- do not remove the packages with .so's on upgrades >=C2=A0 =C2=A0- needs a not-too-complex way to detect which buildids run= ning qemu processes
>=C2=A0 =C2=A0 =C2=A0have for packaging to be able to "autoclean la= ter"
>=C2=A0 =C2=A0- Needs some dependency juggling for Distro packaging but = IMHO can be made
>=C2=A0 =C2=A0 =C2=A0to work if above simple "probing buildid of ru= nning qemu" would exist

So this needs a bunch of special QEMU hacks in package mgmt tools
to prevent the package upgrade & cleanup later. This does not look
like a viable strategy to me.

>
> b) Preload the modules before upgrade
>=C2=A0 =C2=A0- One could load the .so files before upgrade
>=C2=A0 =C2=A0- The open file reference will keep the content around eve= n with the
> on disk file gone
>=C2=A0 =C2=A0- lacking a 'load-module' command that would requi= re fake hotplugs
> which seems wrong
>=C2=A0 =C2=A0- Required additional upgrade pre-planning
>=C2=A0 =C2=A0- kills most benefits of modular code without an actual ne= ed for it
> being loaded

Well there's two benefits to modular approach

=C2=A0- Allow a single build to be selectively installed on a host or conta= iner
=C2=A0 =C2=A0image, such that the install disk footprint is reduced
=C2=A0- Allow a faster startup such that huge RBD libraries dont slow down<= br> =C2=A0 =C2=A0startup of VMs not using RBD disks.

Preloading the modules before upgrade doesn't have to the second benefi= t.
We just have to make sure the pre loading doesn't impact the VM startup=
performance.

IOW, register a SIGUSR2 handler which preloads all modules it finds on
disk. Have a pre-uninstall option on the .so package that sends SIGUSR2
to all QEMU processes. The challenge of course is that signals are
async. You might suggest a QMP command, but only 1 process can have the
QMP monitor open at any time and that's libvirt. Adding a second QMP monitor instance is possible but kind of gross for this purpose.

Another option would be to pre-load the modules during startup, but
do it asynchronously, so that its not blocking overall VM startup.
eg just before starting the mainloop, spawn a background thread to
load all remaining modules.

This will potentially degrade performance of the guest CPUs a bit,
but avoids the latency spike from being synchronous in the startup
path.


> c) go back to non modular build
>=C2=A0 =C2=A0- No :-)
>
> d) anything else out there?

e) Don't do upgrades on a host with running VMs :-)

=C2=A0 =C2=A0Upgrades can break the running VM even ignoring this particula= r
=C2=A0 =C2=A0QEMU module scenario.

f) Simply document that if you upgrade with running VMs that some
=C2=A0 =C2=A0features like hotplug of RBD will become unavialable. Users ca= n
=C2=A0 =C2=A0then avoid upgrades if that matters to them.
<= div>
Hi,
I've come back to this after a while a= nd now think all the pre-load or load-command Ideas we had were in vain.
They would be overly complex and need a lot of integration into dif= ferent places to trigger them.
All of that would not be well inte= grated in the trigger of the issue itself which usually=C2=A0is a package u= pgrade.

But qemu already does try to load modules = from different places and with a slight extension there I think we could
provide something that packaging (the actual place knowing about=C2= =A0upgrades) can use to avoid this issue.

I'll= reply to this thread with a patch for your consideration in a few minutes.=

There is already a Ubuntu 20.04 test build with t= he qemu and packaging changes in [1].
The related Debian/Ubuntu p= ackaging changes themselves can be seen in [2].
I hope that helps= to illustrate how it would work overall



--
Christian Ehrhardt
Staff Engineer, Ubuntu Ser= ver
Canonical Ltd
--0000000000001a63ed05a0042fa0--