From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBC69C432BE for ; Wed, 25 Aug 2021 18:17:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8B8D3610C9 for ; Wed, 25 Aug 2021 18:17:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8B8D3610C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nJEwQcwA9i88Q52hR3OxPLS0TSlIJYv0PRKZR3VzAp0=; b=BnGxb1NU0OQYza 7s4utXityFpDjJBXFM0vEipUlljBZZ6KRTQNE85mVJvsLpd4eWDNuSm9NuEdIST9hcqPAuF0TNHDT N5Lu7zqG2Jirdt4L6rRKfstTYOMwJsUpf8cMCD2AzFvFZ546sXbdcGx5ZPRmvR2bylQJHpbco+foV LJJ8P3XM+MlgFIQVRGcnWHF8avMP7vlj/W0pY6vP7iz1Jatx/h/jAxpJomEcqbHLLxJtleW/YqPWq epIHJKsl6Hr6x1pLcoQmP2PMBlr/kq3ACiDp97axuk7K8K+UqVs7CfqleGaySzCzK5C1BS+ckdCIc k8f+mm3L1VPsZIfA01bg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIxQj-008ELs-D5; Wed, 25 Aug 2021 18:15:17 +0000 Received: from mail-lf1-x12e.google.com ([2a00:1450:4864:20::12e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIxQf-008EL3-Ry for linux-arm-kernel@lists.infradead.org; Wed, 25 Aug 2021 18:15:15 +0000 Received: by mail-lf1-x12e.google.com with SMTP id y34so773923lfa.8 for ; Wed, 25 Aug 2021 11:15:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1QhcI1IjXPKa7HYJu5yUeiCjoFqDIsZs1IlqeQNR1lY=; b=VHozpR4Wlvty/jole+2KSgwfwJqt1fe7dBwxqGumFynobcOVl5gcedIGXUmMYUx70b BSwZtOGFxM6+2T93t1wxtbje9ffq9cL3PTMdmNWLBt5c6/X6/wymTulSsvuHenl/KbBC takhBP3OYv0/cqFf8tAIQYw3un/xxNhpWn4wbhe8FS/rL3RGHyvoIlzchlkZ2C8nn9Mp ZhDhUkeqg6MdFFyyDwIyP1qQeFJG1A5vYAIvXvFN7ey1iId7EZua+bLUNqc2S+tLa1FJ 0PYrwKklLGw8EuFjPmUDPWiyUregqV+qYd8Gcl3y/zrMTY6ESjjUJhruCvw7zDemrVpg G1tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1QhcI1IjXPKa7HYJu5yUeiCjoFqDIsZs1IlqeQNR1lY=; b=ruXUdKutlH5GIlrBO/lW95e301Na16v54gmt+xeDa0e3Csp6cPNIwKWTkg9/N9fLqx WKMyP/pZibhA5YmNsd/AL0iQLfPzFJR9qGLwldooZDCaJPdFb5dxJO4aW8BX8JK2QYrK brNhmY0cZzPJRhb0DOChEMNY6AYw7HB6fTQ/bbn2VPkZZUG6odrZOhxEKUMqGhrAtmzz tXBzEXSnpcSMZjGcbUvgAO6R74BaetT3JAycn22sVgnSgoDvSsJ3tpBV5kQlJhyuh1uW 7/ErOUE5twrC0FJaK0OC3GU04eg9ZsE3zCRrS0Or2h9zPtvz9yaI7EC9jVaquwlD5CWG sTQw== X-Gm-Message-State: AOAM531Yn3orWeAfFR1vGrAqMj1o1nQwCSj+jZBCDuZlX3kaXBCAkzG8 QXvTA0TjwuoQmY34M0CIoYNPKvaJAcGlnQRoSgwWLA== X-Google-Smtp-Source: ABdhPJzjaChjPX6dvqV/GQwrjrZ4/dwEB+a0xuKcQ+jo5ni1J1XiNEgEDTiTxiLtHMNW8vL3J4U8VZqo1i55vhk+UGM= X-Received: by 2002:a05:6512:3ba4:: with SMTP id g36mr34272950lfv.80.1629915311065; Wed, 25 Aug 2021 11:15:11 -0700 (PDT) MIME-Version: 1.0 References: <87mtp5q3gx.wl-maz@kernel.org> <87fsuxq049.wl-maz@kernel.org> <20210825150713.5rpwzm4grfn7akcw@gator.home> In-Reply-To: <20210825150713.5rpwzm4grfn7akcw@gator.home> From: Oliver Upton Date: Wed, 25 Aug 2021 11:14:59 -0700 Message-ID: Subject: Re: KVM/arm64: Guest ABI changes do not appear rollback-safe To: Andrew Jones Cc: Marc Zyngier , kvmarm@lists.cs.columbia.edu, pshier@google.com, ricarkol@google.com, rananta@google.com, reijiw@google.com, jingzhangos@google.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Alexandru.Elisei@arm.com, suzuki.poulose@arm.com, Peter Maydell X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210825_111513_978654_0E0C8A37 X-CRM114-Status: GOOD ( 58.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 25, 2021 at 8:07 AM Andrew Jones wrote: > > On Wed, Aug 25, 2021 at 11:39:34AM +0100, Marc Zyngier wrote: > > On Wed, 25 Aug 2021 11:02:28 +0100, > > Oliver Upton wrote: > > > > > > On Wed, Aug 25, 2021 at 2:27 AM Marc Zyngier wrote: > > > > > Exposing new hypercalls to guests in this manner seems very unsafe to > > > > > me. Suppose an operator is trying to upgrade from kernel N to kernel > > > > > N+1, which brings in the new 'widget' hypercall. Guests are live > > > > > migrated onto the N+1 kernel, but the operator finds a defect that > > > > > warrants a kernel rollback. VMs are then migrated from kernel N+1 -> N. > > > > > Any guests that discovered the 'widget' hypercall are likely going to > > > > > get fussy _very_ quickly on the old kernel. > > > > > > > > This goes against what we decided to support for the *only* publicly > > > > available VMM that cares about save/restore, which is that we only > > > > move forward and don't rollback. > > > > > > Ah, I was definitely missing this context. Current behavior makes much > > > more sense then. > > > > > > > Hypercalls are the least of your > > > > worries, and there is a whole range of other architectural features > > > > that will have also appeared/disappeared (your own CNTPOFF series is a > > > > glaring example of this). > > > > > > Isn't that a tad bit different though? I'll admit, I'm just as guilty > > > with my own series forgetting to add a KVM_CAP (oops), but it is in my > > > queue to kick out with the fix for nVHE/ptimer. Nonetheless, if a user > > > takes up a new KVM UAPI, it is up to the user to run on a new kernel. > > > > The two are linked. Exposing a new register to userspace and/or guest > > result in the same thing: you can't rollback. That's specially true in > > the QEMU case, which *learns* from the kernel what registers are > > available, and doesn't maintain a fixed list. > > > > > My concerns are explicitly with the 'under the nose' changes, where > > > KVM modifies the guest feature set without userspace opting in. Based > > > on your comment, though, it would appear that other parts of KVM are > > > affected too. > > > > Any new system register that is exposed by a new kernel feature breaks > > rollback. And so far, we only consider it a bug if the set of exposed > > registers reduces. Anything can be added safely (as checked by one of > > the selftests added by Drew). > > > > < It doesn't have to be rollback safety, either. There may > > > simply be a hypercall which an operator doesn't want to give its > > > guests, and it needs a way to tell KVM to hide it. > > > > Fair enough. But this has to be done in a scalable way, which > > individual capability cannot provide. > > > > > > > Have I missed something blatantly obvious, or do others see this as an > > > > > issue as well? I'll reply with an example of adding opt-out for PTP. > > > > > I'm sure other hypercalls could be handled similarly. > > > > > > > > Why do we need this? For future hypercalls, we could have some buy-in > > > > capabilities. For existing ones, it is too late, and negative features > > > > are just too horrible. > > > > > > Oh, agreed on the nastiness. Lazy hack to realize the intended > > > functional change.. > > > > Well, you definitely achieved your goal of attracting my attention :). > > > > > > For KVM-specific hypercalls, we could get the VMM to save/restore the > > > > bitmap of supported functions. That would be "less horrible". This > > > > could be implemented using extra "firmware pseudo-registers" such as > > > > the ones described in Documentation/virt/kvm/arm/psci.rst. > > > > > > This seems more reasonable, especially since we do this for migrating > > > the guest's PSCI version. > > > > > > Alternatively, I had thought about using a VM attribute, given the > > > fact that it is non-architectural information and we avoid ABI issues > > > in KVM_GET_REG_LIST without buy-in through a KVM_CAP. > > > > The whole point is that these settings get exposed by > > KVM_GET_REG_LIST, as this is QEMU's way to dump a VM state. Given that > > we already have this for things like the spectre management state, we > > can just as well expose the bitmaps that deal with the KVM-specific > > hypercalls. After all, this falls into the realm of "KVM as VM > > firmware". > > > > For ARM-architected hypercalls (TRNG, stolen time), we may need a > > similar extension. > > > > Thanks for including me Marc. I think you've mentioned all the examples > of why we don't generally expect N+1 -> N migrations to work that I > can think of. While some of the examples like get-reg-list could > eventually be eliminated if we had CPU models to tighten our machine type > state, I think N+1 -> N migrations will always be best effort at most. > > I agree with giving userspace control over the exposer of the hypercalls > though. Using pseudo-registers for that purpose rather than a pile of > CAPs also seems reasonable to me. > > And, while I don't think this patch is going to proceed, I thought I'd > point out that the opt-out approach doesn't help much with expanding > our migration support unless we require the VMM to be upgraded first. > > And, even then, the (N_kern, N+1_vmm) -> (N+1_kern, N_vmm) case won't > work as expected, since the source enforce opt-out, but the destination > won't. Right, there's going to need to be a fence in both kernel and VMM versions. Before the fence, you can't rollback with either component. Once on the other side of the fence, the user may freely migrate between kernel + VMM combinations. > Also, since the VMM doesn't key off the kernel version, for the > most part N+1 VMMs won't know when they're supposed to opt-out or not, > leaving it to the user to ensure they consider everything. opt-in > usually only needs the user to consider what machine type they want to > launch. Going the register route will implicitly require opt-out for all old hypercalls. We exposed them unconditionally to the guest before, and we must uphold that behavior. The default value for the bitmap will have those features set. Any hypercalls added after that register interface will then require explicit opt-in from userspace. With regards to the pseudoregister interface, how would a VMM discover new bits? From my perspective, you need to have two bitmaps that the VMM can get at: the set of supported feature bits and the active bitmap of features for a running guest. > Thanks, > drew > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel