From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF884C433F5 for ; Tue, 8 Feb 2022 16:58:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 543174B120; Tue, 8 Feb 2022 11:58:35 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0RFClE+b4RpW; Tue, 8 Feb 2022 11:58:33 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id E4B1F4A105; Tue, 8 Feb 2022 11:58:33 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BB4F94A105 for ; Tue, 8 Feb 2022 11:58:32 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JIyAzFDzHk31 for ; Tue, 8 Feb 2022 11:58:31 -0500 (EST) Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 52A2B49F5F for ; Tue, 8 Feb 2022 11:58:31 -0500 (EST) Received: by mail-pj1-f44.google.com with SMTP id t14-20020a17090a3e4e00b001b8f6032d96so2241593pjm.2 for ; Tue, 08 Feb 2022 08:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=gtD1WqxflWJB+W3B+2IPTB0tczwLG7fEAfX6erBc0MU5nljMZXr+E8SiEnTM4N+BFg RjxyhEm/cmjmv0ervzymU069H+VhD045lnQ39rcRknECCnyaucw8K93VEVf8UoKDZSew jdP2LEemYZcR5i2XXtiiLw/uLGKW8q/gojZST44XnLhBy+AmIOEsB1Iw9vJRMUTEWeLR 04asKhrQVyx/py9Gao6ECyqb62nkk/SpW8Cm42Mt4FPy1xSjZMUrEsAZmNS1hS0HcFd5 G+KT9JLD/hjAa7U929ByWGFWRp/jjxzZ8/OA5yppdq3NtjrTwzncuBmrOEXYNjpnmH68 H4Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=v+7tUSnM04K9x+TxU6Y85RVhxnDG71hbzwSA2iyr7D77/HyP7g1usKAHOX1u6iGFeX aQnf86JmK7bSQ9pVUTeVF2AglyHspV6Y+LfIfDCfDeELNiBvFVFKzYtY8N+Yw+NiSawV cWCk+NdD9fvXLub02iYUC29KrALBzwVZACyJzHd1Mstb1LpYDYV9xwacpkHYFc18uM+c ILNJI0SFCUxcxXyp2zz1M4AUaCX+LPDyR6IGtiFK5HWP1oVKi5aX8GkQjR4d5XZfnDdB 9P9uyl4rLc5rJ10F6JT4yr6iEckbMKZ6ZPJuIQR7Hga8LWLmG3yAGOMEGkcXTd3bzWjB zzNw== X-Gm-Message-State: AOAM530UpowNL7aN8952zRdzWm59WqdsRxj5+wq73/qsCAVAWc2QRwAb no2LRD5UhCSzY75TJWiVXfnDdw== X-Google-Smtp-Source: ABdhPJycqIE2rS7QGrW4hrzwOVH1TbvshmoHJyww4j5tVSIRNytldRpHF1r23ExSbwF4NMrAS2Xnog== X-Received: by 2002:a17:902:8f96:: with SMTP id z22mr5455290plo.2.1644339510071; Tue, 08 Feb 2022 08:58:30 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m20sm16957314pfk.215.2022.02.08.08.58.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Feb 2022 08:58:29 -0800 (PST) Date: Tue, 8 Feb 2022 16:58:26 +0000 From: Sean Christopherson To: Oliver Upton Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87ilyitt6e.wl-maz@kernel.org> <87lf3drmvp.wl-maz@kernel.org> <875yq88app.wl-maz@kernel.org> <878ruld72v.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: kvm@vger.kernel.org, Marc Zyngier , pshier@google.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Tue, Feb 08, 2022, Oliver Upton wrote: > Hi Marc, > > On Tue, Feb 8, 2022 at 1:46 AM Marc Zyngier wrote: > > > > KVM currently restricts the vcpu features to be unified across vcpus, > > > > but that's only an implementation choice. > > > > > > But that implementation choice has become ABI, no? How could support > > > for asymmetry be added without requiring userspace opt-in or breaking > > > existing VMMs that depend on feature unification? > > > > Of course, you'd need some sort of advertising of this new behaviour. > > > > One thing I would like to add to the current state of thing is an > > indication of whether the effects of a sysreg being written from > > userspace are global or local to a vcpu. You'd need a new capability, > > and an extra flag added to the encoding of each register. > > Ah. I think that is a much more reasonable fit then. VMMs unaware of > this can continue to migrate new bits (albeit at the cost of > potentially higher lock contention for the per-VM stuff), and those > that do can reap the benefits of writing such attributes exactly once. But the "proper" usage is no different than adding support for VM-scoped variants of KVM_{G,S}ET_ONE_REG and friends, and a VM-scoped variant is conceptually a lot cleaner IMO. And making them truly VM-scoped means KVM can do things like support sysregs that are immutable after vCPUs are created. So long as KVM defaults to '0' for all such registers, lack of migration support in userspace that isn't aware of the new API, i.e. doesn't do KVM_GET_REG_LIST at a VM-scope, is a nop because said userspace also won't modify the registers in the first place. The only "unsolvable" problem that is avoided by usurping the per-vCPU ioctls is rollback to a userspace VMM that isn't aware of the per-VM ioctls, but it doesn't seem too onerous to tell userspace "don't use these unless your entire fleet has upgraded", especially since that requirement/advisement is true for the KVM side with respect to new registers regardless of how those registers are accessed. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 45D5BC433F5 for ; Tue, 8 Feb 2022 16:59:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YpymrQke+eJ8KbkGE9HKrNFDflhDsw4zSvoNUNc0lhs=; b=fLh71K74ObjmRy FD5SYXDJMasyuI5pFAQEc/C/zcV160t3u7AWbP51odbHxuIr+9puJzDdKUEUxdALI+BE0WBTZVJwF ylRDfdHXZe9Ne3cO3IlPQgBOBcLDPH9gxRM5v3sOxFzdhH75e22JZ0dpaxUii9GnwL5/Ves5lWnlC H2ogpMQ/7GLuey+X2ektiOEkFOpAHuGr3AmImyvcXvL/+lQ4Uuh9QYJfcKJaWTztQoGlBDk7l0ECU mY0JeYnNJygdbBarnQRSTQNUiPG1YyQ8SKwVy8Ivu26BqoVeL+TPIJBkn3OmHT1F/eb8wzUKD++En EzihI8JDLNHdPAnWuW3g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHTp4-00Exhc-VM; Tue, 08 Feb 2022 16:58:35 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHTp0-00Exh4-Pk for linux-arm-kernel@lists.infradead.org; Tue, 08 Feb 2022 16:58:32 +0000 Received: by mail-pl1-x631.google.com with SMTP id w20so5076604plq.12 for ; Tue, 08 Feb 2022 08:58:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=gtD1WqxflWJB+W3B+2IPTB0tczwLG7fEAfX6erBc0MU5nljMZXr+E8SiEnTM4N+BFg RjxyhEm/cmjmv0ervzymU069H+VhD045lnQ39rcRknECCnyaucw8K93VEVf8UoKDZSew jdP2LEemYZcR5i2XXtiiLw/uLGKW8q/gojZST44XnLhBy+AmIOEsB1Iw9vJRMUTEWeLR 04asKhrQVyx/py9Gao6ECyqb62nkk/SpW8Cm42Mt4FPy1xSjZMUrEsAZmNS1hS0HcFd5 G+KT9JLD/hjAa7U929ByWGFWRp/jjxzZ8/OA5yppdq3NtjrTwzncuBmrOEXYNjpnmH68 H4Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=M2yj3XNq7czOl8AJhaSXyuZofWi2Fnsn1rYp+tCP/EJYSCy73oeW1MGmovj3kgd4Uw fIDHf1v9wQHGWbWn3yVMkCe17kPdHLXkExpPmyhX8QgT9mLj9wgb7w/2NFbtGxR6Hnv2 FqTnkNlEJitHBOj5FNCDX3NeHCGDEVdBqdUldbJiDsc7Z9rF4c7/lwIzwlPY64jtygVU kVb4B2NFEK3ZIqKL/ZxLNZhXlld8pz0Mg4UmDD3F7PXMPQPbXrDUiQaFa/UBJfgtFjWX udQ8l2Kpe2Dr2FfePvfwReIzQBu5bcaV9n1J6h8hCeTTCnIo0cd/B8jn0yyvttAbSYPa S9aA== X-Gm-Message-State: AOAM531PgmFK0P2zwZLGDg5RXKYDDbvRZcsQ1gvKewyUl+yNCFVquVBn 2+aCFHRe4Yy6rKgx+WNfOhgjwnbgyv8Tww== X-Google-Smtp-Source: ABdhPJycqIE2rS7QGrW4hrzwOVH1TbvshmoHJyww4j5tVSIRNytldRpHF1r23ExSbwF4NMrAS2Xnog== X-Received: by 2002:a17:902:8f96:: with SMTP id z22mr5455290plo.2.1644339510071; Tue, 08 Feb 2022 08:58:30 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m20sm16957314pfk.215.2022.02.08.08.58.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Feb 2022 08:58:29 -0800 (PST) Date: Tue, 8 Feb 2022 16:58:26 +0000 From: Sean Christopherson To: Oliver Upton Cc: Marc Zyngier , Raghavendra Rao Ananta , Andrew Jones , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87ilyitt6e.wl-maz@kernel.org> <87lf3drmvp.wl-maz@kernel.org> <875yq88app.wl-maz@kernel.org> <878ruld72v.wl-maz@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220208_085830_869267_722D1AC8 X-CRM114-Status: GOOD ( 21.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Feb 08, 2022, Oliver Upton wrote: > Hi Marc, > > On Tue, Feb 8, 2022 at 1:46 AM Marc Zyngier wrote: > > > > KVM currently restricts the vcpu features to be unified across vcpus, > > > > but that's only an implementation choice. > > > > > > But that implementation choice has become ABI, no? How could support > > > for asymmetry be added without requiring userspace opt-in or breaking > > > existing VMMs that depend on feature unification? > > > > Of course, you'd need some sort of advertising of this new behaviour. > > > > One thing I would like to add to the current state of thing is an > > indication of whether the effects of a sysreg being written from > > userspace are global or local to a vcpu. You'd need a new capability, > > and an extra flag added to the encoding of each register. > > Ah. I think that is a much more reasonable fit then. VMMs unaware of > this can continue to migrate new bits (albeit at the cost of > potentially higher lock contention for the per-VM stuff), and those > that do can reap the benefits of writing such attributes exactly once. But the "proper" usage is no different than adding support for VM-scoped variants of KVM_{G,S}ET_ONE_REG and friends, and a VM-scoped variant is conceptually a lot cleaner IMO. And making them truly VM-scoped means KVM can do things like support sysregs that are immutable after vCPUs are created. So long as KVM defaults to '0' for all such registers, lack of migration support in userspace that isn't aware of the new API, i.e. doesn't do KVM_GET_REG_LIST at a VM-scope, is a nop because said userspace also won't modify the registers in the first place. The only "unsolvable" problem that is avoided by usurping the per-vCPU ioctls is rollback to a userspace VMM that isn't aware of the per-VM ioctls, but it doesn't seem too onerous to tell userspace "don't use these unless your entire fleet has upgraded", especially since that requirement/advisement is true for the KVM side with respect to new registers regardless of how those registers are accessed. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AAC2C433F5 for ; Tue, 8 Feb 2022 16:58:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345008AbiBHQ6c (ORCPT ); Tue, 8 Feb 2022 11:58:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230391AbiBHQ6b (ORCPT ); Tue, 8 Feb 2022 11:58:31 -0500 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1E30C061578 for ; Tue, 8 Feb 2022 08:58:30 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id w1so4769553plb.6 for ; Tue, 08 Feb 2022 08:58:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=gtD1WqxflWJB+W3B+2IPTB0tczwLG7fEAfX6erBc0MU5nljMZXr+E8SiEnTM4N+BFg RjxyhEm/cmjmv0ervzymU069H+VhD045lnQ39rcRknECCnyaucw8K93VEVf8UoKDZSew jdP2LEemYZcR5i2XXtiiLw/uLGKW8q/gojZST44XnLhBy+AmIOEsB1Iw9vJRMUTEWeLR 04asKhrQVyx/py9Gao6ECyqb62nkk/SpW8Cm42Mt4FPy1xSjZMUrEsAZmNS1hS0HcFd5 G+KT9JLD/hjAa7U929ByWGFWRp/jjxzZ8/OA5yppdq3NtjrTwzncuBmrOEXYNjpnmH68 H4Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=BehLNzAs/zQ1lF/lRFmUiYEG0MqDcTlN+Zi4GMNO4po=; b=6ZlAfO9r4Gt9mMh0+yBulTLwvVD/X8Fof7KL9giBQgU27GyRHK9zaDZKxqZLUgBcBJ clYE6rCI5KhPWbU6+pnSmVZOUNthaFSOzls3RezyDtWmGG7Aac/UnPNs5KAwJGOLDrJm x/aJRQw/v2nKjeAyezM1IpEdWiJf7f642T1y8aAK5pXMRJ0YvGdLzV8O+isAmW94gfTV trcUGtAw4Q/qshm5npjHLrOA7O9PwQW9SyYbLHj3BEIs0nJExwE4L0lIKyVgcv9rQ98p cIMqecNQweooQl9A3vTtq2AGsSr44HGDwPRmvZW4pPPYIdwQPp1foNn/+mmb6unOlwCF +Zmw== X-Gm-Message-State: AOAM530vq7CgtVu7LlEq/HqIP3TfQCZAx5bc7bEbDkL1BBVpGj8mhgOP cZONscx9y0T9uGu5kx6LsRJNpg== X-Google-Smtp-Source: ABdhPJycqIE2rS7QGrW4hrzwOVH1TbvshmoHJyww4j5tVSIRNytldRpHF1r23ExSbwF4NMrAS2Xnog== X-Received: by 2002:a17:902:8f96:: with SMTP id z22mr5455290plo.2.1644339510071; Tue, 08 Feb 2022 08:58:30 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m20sm16957314pfk.215.2022.02.08.08.58.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Feb 2022 08:58:29 -0800 (PST) Date: Tue, 8 Feb 2022 16:58:26 +0000 From: Sean Christopherson To: Oliver Upton Cc: Marc Zyngier , Raghavendra Rao Ananta , Andrew Jones , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe Message-ID: References: <87ilyitt6e.wl-maz@kernel.org> <87lf3drmvp.wl-maz@kernel.org> <875yq88app.wl-maz@kernel.org> <878ruld72v.wl-maz@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, Feb 08, 2022, Oliver Upton wrote: > Hi Marc, > > On Tue, Feb 8, 2022 at 1:46 AM Marc Zyngier wrote: > > > > KVM currently restricts the vcpu features to be unified across vcpus, > > > > but that's only an implementation choice. > > > > > > But that implementation choice has become ABI, no? How could support > > > for asymmetry be added without requiring userspace opt-in or breaking > > > existing VMMs that depend on feature unification? > > > > Of course, you'd need some sort of advertising of this new behaviour. > > > > One thing I would like to add to the current state of thing is an > > indication of whether the effects of a sysreg being written from > > userspace are global or local to a vcpu. You'd need a new capability, > > and an extra flag added to the encoding of each register. > > Ah. I think that is a much more reasonable fit then. VMMs unaware of > this can continue to migrate new bits (albeit at the cost of > potentially higher lock contention for the per-VM stuff), and those > that do can reap the benefits of writing such attributes exactly once. But the "proper" usage is no different than adding support for VM-scoped variants of KVM_{G,S}ET_ONE_REG and friends, and a VM-scoped variant is conceptually a lot cleaner IMO. And making them truly VM-scoped means KVM can do things like support sysregs that are immutable after vCPUs are created. So long as KVM defaults to '0' for all such registers, lack of migration support in userspace that isn't aware of the new API, i.e. doesn't do KVM_GET_REG_LIST at a VM-scope, is a nop because said userspace also won't modify the registers in the first place. The only "unsolvable" problem that is avoided by usurping the per-vCPU ioctls is rollback to a userspace VMM that isn't aware of the per-VM ioctls, but it doesn't seem too onerous to tell userspace "don't use these unless your entire fleet has upgraded", especially since that requirement/advisement is true for the KVM side with respect to new registers regardless of how those registers are accessed.