From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 750D6C12002 for ; Wed, 21 Jul 2021 15:46:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 46A3B61003 for ; Wed, 21 Jul 2021 15:46:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 46A3B61003 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n01tBo7yz8nWxNVcT2+TLv4PE4n0aeADQd6v0BufheE=; b=B6WTwOsrG0NKiA qvPWpRjZsL5Wu6ChL/Bw/3RA+6lLz1Rilmytaxlq/uRZqdo7sl5LQhEBDIFGxGMD/QY3Sq3OkEB7f qRZridj7GHZgddp3Xp/UsmUV3LpEL7wwVe400QaidgV22uiWxG5WAybN3Wd37czGoTj2ZwdxKRPny BRVXyRgTAWqUlIGBKXAtdJsRbD0ILTRu40tBc5QORn/1+kI4wVuNJLucQGKD0cqENAZaFCLsGGZMZ VIh0IlHoi7KsAGlaEePo0O/Qn1vjifDPVXhDqL6UWopcj46oYaCsElDV1D7y6Qm4KXf9yLnms+hsq j9oGAyoZqM5iKkWUJWcg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m6EOS-00GOt2-6j; Wed, 21 Jul 2021 15:44:20 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m6E8t-00GJ6Q-Se for linux-arm-kernel@lists.infradead.org; Wed, 21 Jul 2021 15:28:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626881294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=FdvVljLxaRoKu02p/XKHt8pLC1+aQhmZqpHxlBYeaQA=; b=PYY/5ucUTwznAd325Yj1dNAuKntmjki+r9Jkb0mmdQkavf9GBNNwuekAiOZAm5xIWeqUf+ AfaO+TFMKFb8dIWJUJQ8wtHyOIAjMZFM5KsFkOfhklsHDyvz74kU8/VeRQvLLg8zr1ZHXZ dZzZFOW/8Pjmv2wTBFGSfILx2nol+4c= Received: from mail-il1-f199.google.com (mail-il1-f199.google.com [209.85.166.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-508-yXzVPRpGMQChhXIowjqW4A-1; Wed, 21 Jul 2021 11:28:11 -0400 X-MC-Unique: yXzVPRpGMQChhXIowjqW4A-1 Received: by mail-il1-f199.google.com with SMTP id w8-20020a056e021c88b02902095727d18dso1767036ill.17 for ; Wed, 21 Jul 2021 08:28:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=FdvVljLxaRoKu02p/XKHt8pLC1+aQhmZqpHxlBYeaQA=; b=gSZXob2WBonXDmqF3R2m8hCUB6B/vo+lslyKQeCSPqblSwXqTNour5SOUo3QlQ5etz GTY9yHJilUq7INjSqf3FhxSnYj+nWbVK09FSwJv0MGdZ9+ehYPCsAjmcftVPFErtGldm ESwMyCbKmLZToKkWeFidrpRDaT9xjy0U8SOQdn8+t3gps2fuM9ElJ1PsSPNFdXrNG3dj sc2Q23nvaHBM0ZvMXdkEVz7oi9PLkdoM1X2h78v9p/P28a7haczm/e536w+nKaBCdyK7 PYDH8zcZHA7vPTedc2vHBCOHqJOJvzJh9/sLqsrpE4+41sDwqXpVZSSw+Z8LRwzhMNfz hi0g== X-Gm-Message-State: AOAM531JigfmKkMKQrmRxOCgHyTZg7a6Mmr7l2F/g6XHsqt9XEyMzjal 3zagBHMwY0fI7uFnbpW3oX4UQtUEGjZBKGgioreKDf9LTZsI7/Ry9cLMEmH1ZZPvbIPJkH2YdCj mMlP/sZyRfCCwmzYHGf2cPEc7RdiLzZ9PjcQ= X-Received: by 2002:a02:cf31:: with SMTP id s17mr31776938jar.46.1626881291060; Wed, 21 Jul 2021 08:28:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyXsUlUSfaaSu/VgVW9B0i+z/O7lclfXTzWxt6V521RZmWaIu4pVkdUyLATZI2zGAntaYUAnw== X-Received: by 2002:a02:cf31:: with SMTP id s17mr31776920jar.46.1626881290789; Wed, 21 Jul 2021 08:28:10 -0700 (PDT) Received: from gator ([140.82.166.162]) by smtp.gmail.com with ESMTPSA id k4sm1848796ilu.67.2021.07.21.08.28.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 08:28:10 -0700 (PDT) Date: Wed, 21 Jul 2021 17:28:08 +0200 From: Andrew Jones To: Oliver Upton Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, Marc Zyngier , Raghavendra Rao Anata , Peter Shier , Sean Christopherson , David Matlack , Paolo Bonzini , linux-arm-kernel@lists.infradead.org, Jim Mattson Subject: Re: [PATCH v2 00/12] KVM: Add idempotent controls for migrating system counter state Message-ID: <20210721152808.lsnphkl3urz6bu3v@gator> References: <20210716212629.2232756-1-oupton@google.com> MIME-Version: 1.0 In-Reply-To: <20210716212629.2232756-1-oupton@google.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=drjones@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210721_082816_074734_D3CA9F1E X-CRM114-Status: GOOD ( 27.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Jul 16, 2021 at 09:26:17PM +0000, Oliver Upton wrote: > KVM's current means of saving/restoring system counters is plagued with > temporal issues. At least on ARM64 and x86, we migrate the guest's > system counter by-value through the respective guest system register > values (cntvct_el0, ia32_tsc). Restoring system counters by-value is > brittle as the state is not idempotent: the host system counter is still > oscillating between the attempted save and restore. Furthermore, VMMs > may wish to transparently live migrate guest VMs, meaning that they > include the elapsed time due to live migration blackout in the guest > system counter view. The VMM thread could be preempted for any number of > reasons (scheduler, L0 hypervisor under nested) between the time that > it calculates the desired guest counter value and when KVM actually sets > this counter state. > > Despite the value-based interface that we present to userspace, KVM > actually has idempotent guest controls by way of system counter offsets. > We can avoid all of the issues associated with a value-based interface > by abstracting these offset controls in new ioctls. This series > introduces new vCPU device attributes to provide userspace access to the > vCPU's system counter offset. > > Patch 1 adopts Paolo's suggestion, augmenting the KVM_{GET,SET}_CLOCK > ioctls to provide userspace with a (host_tsc, realtime) instant. This is > essential for a VMM to perform precise migration of the guest's system > counters. > > Patches 2-3 add support for x86 by shoehorning the new controls into the > pre-existing synchronization heuristics. > > Patches 4-5 implement a test for the new additions to > KVM_{GET,SET}_CLOCK. > > Patches 6-7 implement at test for the tsc offset attribute introduced in > patch 3. > > Patch 8 adds a device attribute for the arm64 virtual counter-timer > offset. > > Patch 9 extends the test from patch 7 to cover the arm64 virtual > counter-timer offset. > > Patch 10 adds a device attribute for the arm64 physical counter-timer > offset. Currently, this is implemented as a synthetic register, forcing > the guest to trap to the host and emulating the offset in the fast exit > path. Later down the line we will have hardware with FEAT_ECV, which > allows the hypervisor to perform physical counter-timer offsetting in > hardware (CNTPOFF_EL2). > > Patch 11 extends the test from patch 7 to cover the arm64 physical > counter-timer offset. > > Patch 12 introduces a benchmark to measure the overhead of emulation in > patch 10. > > Physical counter benchmark > -------------------------- > > The following data was collected by running 10000 iterations of the > benchmark test from Patch 6 on an Ampere Mt. Jade reference server, A 2S > machine with 2 80-core Ampere Altra SoCs. Measurements were collected > for both VHE and nVHE operation using the `kvm-arm.mode=` command-line > parameter. > > nVHE > ---- > > +--------------------+--------+---------+ > | Metric | Native | Trapped | > +--------------------+--------+---------+ > | Average | 54ns | 148ns | > | Standard Deviation | 124ns | 122ns | > | 95th Percentile | 258ns | 348ns | > +--------------------+--------+---------+ > > VHE > --- > > +--------------------+--------+---------+ > | Metric | Native | Trapped | > +--------------------+--------+---------+ > | Average | 53ns | 152ns | > | Standard Deviation | 92ns | 94ns | > | 95th Percentile | 204ns | 307ns | > +--------------------+--------+---------+ > > This series applies cleanly to the following commit: > > 1889228d80fe ("KVM: selftests: smm_test: Test SMM enter from L2") > > v1 -> v2: > - Reimplemented as vCPU device attributes instead of a distinct ioctl. > - Added the (realtime, host_tsc) instant support to > KVM_{GET,SET}_CLOCK > - Changed the arm64 implementation to broadcast counter offset values > to all vCPUs in a guest. This upholds the architectural expectations > of a consistent counter-timer across CPUs. > - Fixed a bug with traps in VHE mode. We now configure traps on every > transition into a guest to handle differing VMs (trapped, emulated). > Oops, I see there's a v3 of this series. I'll switch to reviewing that. I think my comments / r-b's apply to that version as well though. Thanks, drew _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel