From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC959C64E8A for ; Wed, 2 Dec 2020 17:25:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0DEBE21D42 for ; Wed, 2 Dec 2020 17:25:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DEBE21D42 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:41588 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kkVsX-0006MO-QY for qemu-devel@archiver.kernel.org; Wed, 02 Dec 2020 12:25:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kkVr4-0005hS-4c for qemu-devel@nongnu.org; Wed, 02 Dec 2020 12:23:50 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38912) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kkVr1-0007qv-FN for qemu-devel@nongnu.org; Wed, 02 Dec 2020 12:23:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606929825; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LptDdb8uLb4u0JEYB3nQBkJWXVpJYjKfq0rkay5yXXo=; b=eCeQN5p2/D+xRttSSHy26g4gMZVCzv4WIHp7jsd+qAxEP70tGjtgGo2AsbwogCc3CR0Df9 G6rzgPYyLKdBie9kG4oRxpovNKmZRjFk8SeLVKrBtcNCAIwu8I3ZxFJJoP77MA+bJibSae Ui+niHc0w0NOTEkfGOdF0OAbeHJVrsc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-63-XYed1ZZxPGewbsZfGSLdqg-1; Wed, 02 Dec 2020 12:23:39 -0500 X-MC-Unique: XYed1ZZxPGewbsZfGSLdqg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AA8EC57000; Wed, 2 Dec 2020 17:23:37 +0000 (UTC) Received: from redhat.com (ovpn-115-57.ams2.redhat.com [10.36.115.57]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E52F960855; Wed, 2 Dec 2020 17:23:32 +0000 (UTC) Date: Wed, 2 Dec 2020 17:23:29 +0000 From: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= To: priyankar jain Subject: Re: [RFC] dbus-vmstate: Connect to the dbus only during the migration phase Message-ID: <20201202172329.GM2360260@redhat.com> References: <1605810535-51254-1-git-send-email-priyankar.jain@nutanix.com> <20201119184724.GO579364@redhat.com> <057612c5-a9b8-c7be-c710-1b635aa361be@nutanix.com> <20201202161659.GL2360260@redhat.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.14.6 (2020-07-11) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=berrange@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=216.205.24.124; envelope-from=berrange@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -35 X-Spam_score: -3.6 X-Spam_bar: --- X-Spam_report: (-3.6 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.495, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= Cc: Peter Turschmid , qemu-devel@nongnu.org, =?utf-8?Q?Marc-Andr=C3=A9?= Lureau , Raphael Norwitz Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Wed, Dec 02, 2020 at 10:33:01PM +0530, priyankar jain wrote: > On 02/12/20 9:46 pm, Daniel P. Berrangé wrote: > > On Wed, Dec 02, 2020 at 09:25:27PM +0530, priyankar jain wrote: > > > On 20/11/20 12:17 am, Daniel P. Berrangé wrote: > > > > On Thu, Nov 19, 2020 at 06:28:55PM +0000, Priyankar Jain wrote: > > > > > Today, dbus-vmstate maintains a constant connection to the dbus. This is > > > > > problematic for a number of reasons: > > > > > 1. If dbus-vmstate is attached during power-on, then the device holds > > > > > the unused connection for a long period of time until migration > > > > > is triggered, thus unnecessarily occupying dbus. > > > > > 2. Similarly, if the dbus is restarted in the time period between VM > > > > > power-on (dbus-vmstate initialisation) and migration, then the > > > > > migration will fail. The only way to recover would be by > > > > > re-initialising the dbus-vmstate object. > > > > > 3. If dbus is not available during VM power-on, then currently dbus-vmstate > > > > > initialisation fails, causing power-on to fail. > > > > > 4. For a system with large number of VMs, having multiple QEMUs connected to > > > > > the same dbus can lead to a DoS for new connections. > > > > > > > > The expectation is that there is a *separate* dbus daemon created for > > > > each QEMU instance. There should never be multiple QEMUs connected to > > > > the same dbus instance, nor should it ever connect to the common dbus > > > > instances provided by most Linux distros. > > > > > > > > None of these 4 issues should apply when each QEMU has its own dedicated > > > > dbus instance AFAICT. > > > > > > > > > > > > Regards, > > > > Daniel > > > > > > > > > > How does having a separate dbus daemon resolve issue (2)? If any daemon > > > restarts between VM power-on and migration, the dbus-vmstate object for that > > > VM would have to be reinitialized, no? > > > > The private dbus damon for QEMU is expected to exist for the lifetime of > > that QEMU process. > > Totally agree on the expectation. But any external stimuli (maybe > unintended) can easily break this condition, and this would indeed result > into a situation where the VM is basically non migratable until the VM is > powered cycled or the dbus-vmstate is removed by manual intervention. > Secondly, having dbus-vmstate backend connect to dbus at migration time > eases the complexity for any management plane to recover in these failure > situations by monitoring dbus and restarting it with the same params if dbus > gets killed without affecting QEMU. The vmstate support is just one use case for the dbus daemon. The intention when specifying dbus was that this serves as a general purpose framework on which other functionality will be built for mgmt of helper processes associated with QEMU VM. So ultimately it is thought that the dbus service will be relevant throuh the lifetime of the VM. Obviously it doesn't appear that way right now, but I'm wary about optimizing to dynamically connect/disconnect around migration since we're expecting it to be in use any arbitrary times for other things long term. > > > > Secondly, on a setup with large number of VMs, having separate dbus-daemons > > > leads to high cummulative memory usage by dbus daemons, is it a feasible > > > approach to spawn a new dbus-daemon for every QEMU, given the fact that > > > majority of the security aspect lies with the dbus peers, apart from the > > > SELinux checks provided by dbus. > > > > The memory usage of a dbus daemon shouldn't be that high. A large portion > > of the memory footprint should be readony pages shared between all dbus > > procsses. The private usage should be a functional of number of clients > > and the message traffic. Do you have any measured figures you're concerned > > with ? > > One of our setup had a long running private dbus-daemon (nearly 4-5 days) in > the destination hypervisor after performing migration, it was showing the > following memory usage (figures in kB): > Virt - 90980 > Rss - 19576 > Total - 110556 > > Extrapolating these figures for 100s of daemons results in considerable Rss > usage. These figures were taken using `top` linux utility. But I had not > considered the readonly shared pages aspect at the time of capture. Yep, you can't just multiply 'Rss' by the number of instances, as that'll way over-estimate the overhead. Need to look at the private Rss only. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|