From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Thu, 18 Nov 2021 19:46:56 +0000 Subject: [PATCH v3 08/12] KVM: Propagate vcpu explicitly to mark_page_dirty_in_slot() In-Reply-To: <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> References: <20211117174003.297096-1-dwmw2@infradead.org> <20211117174003.297096-9-dwmw2@infradead.org> <85d9fec17f32c3eb9e100e56b91af050.squirrel@twosheds.infradead.org> <4c48546b-eb4a-dff7-cc38-5df54f73f5d4@redhat.com> <20b5952e76c54a3a5dfe5a898e3b835404ac6fb1.camel@infradead.org> <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Thu, Nov 18, 2021, David Woodhouse wrote: > > > On 18 November 2021 18:50:55 GMT, Sean Christopherson wrote: > >On Thu, Nov 18, 2021, Sean Christopherson wrote: > >> On Thu, Nov 18, 2021, David Woodhouse wrote: > >> > That leaves the one in TDP MMU handle_changed_spte_dirty_log() which > >> > AFAICT can trigger the same crash seen by butt3rflyh4ck ? can't that > >> > happen from a thread where kvm_get_running_vcpu() is NULL too? For that > >> > one I'm not sure. > >> > >> I think could be trigger in the TDP MMU via kvm_mmu_notifier_release() > >> -> kvm_mmu_zap_all(), e.g. if the userspace VMM exits while dirty logging is > >> enabled. That should be easy to (dis)prove via a selftest. > > > >Scratch that, the dirty log update is guarded by the new_spte being present, so > >zapping of any kind won't trigger it. > > > >Currently, I believe the only path that would create a present SPTE without an > >active vCPU is mmu_notifer.change_pte, but that squeaks by because its required > >to be wrapped with invalidate_range_{start,end}(MMU_NOTIFY_CLEAR), and KVM zaps > >in that situation. > > Is it sufficient to have *an* active vCPU? What if a VMM has threads for > active vCPUs but is doing mmap/munmap on a *different* thread? Does that not > suffer the same crash? It is sufficient for the current physical CPU to have an active vCPU, which is generally guaranteed in the MMU code because, with a few exceptions, populating SPTEs is done in vCPU context. mmap() will never directly trigger SPTE creation, KVM first requires a vCPU to fault on the new address. munmap() is a pure zap flow, i.e. won't create a present SPTE and trigger the writeback of the dirty bit. That's also why I dislike using kvm_get_running_vcpu(); when it's needed, there's a valid vCPU from the caller, but it deliberately gets dropped and indirectly picked back up. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F4BDC433EF for ; Thu, 18 Nov 2021 19:47:07 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id BC4D1619BB for ; Thu, 18 Nov 2021 19:47:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org BC4D1619BB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 586304B231; Thu, 18 Nov 2021 14:47:06 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EDFFtWmFf2zX; Thu, 18 Nov 2021 14:47:05 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 152BA4B1F3; Thu, 18 Nov 2021 14:47:05 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id CB1444B1BA for ; Thu, 18 Nov 2021 14:47:03 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vaNpKaIDmEdL for ; Thu, 18 Nov 2021 14:47:02 -0500 (EST) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 7981A4B177 for ; Thu, 18 Nov 2021 14:47:02 -0500 (EST) Received: by mail-pl1-f179.google.com with SMTP id m24so6165302pls.10 for ; Thu, 18 Nov 2021 11:47:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=TJltHTqmUDTIgL1P24a20gD79MoRsYyT+oWNI7Zk1KCB0kxMW9nay123E+ZoFctpYI aiNnMLJVER0OaOWjGQFAAodEb7XssAPkbTv3HZ/7Xvc56U/KwoqRQGvrj5PKP0bcKR4k 2vvhP28esD9If7TAFeNjBviDlcYadmbYqHQK4EKbmvqKoyyDe2mAxq3N83OoFp8cHZvs tW/44+HbAaU7BnQeov5HYKclsNsjLoArhh5d8PMaS08emGsC4hW0iBqIihQVo007XCgE f+/05PLBrEHbV7h1EwSp8E42irdKiOdyzv/mpEc9ne+Zn3+nJLJHbZQwUjsMNM99uGwK D+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=aW/dPcQM3UcP0gz2U76t7PYuEVIxiJHkTA514wTHOdfoUvzFqMe1mC/w1Wf53McTJB T07VDQOx613YwNHKyE6HrbVuFUBJe+eOtl4AAdKByUJxo9zQEf0BCJRKtm97KB0awbr7 DJCg85SndVakwf5DBJo3w6ZYXIToYjpsPB5kAM9nIcZwG98qQrlWCh5MRassbBZ39Bv2 Z2soEIE+fkApayk/kRcSFIFhlJ/4Yr2mzGusqN0H1Ezkj2mE//9FzkSQ4DeEBvvPkMJp nyuKmPCRF2PhNbN5SGdfvvURwx5YSw7NhJWTPzPZHfosffl5ZDzniSM3EijrX7JoZSU2 QGXg== X-Gm-Message-State: AOAM5328PDhoy8bGjQ37MuE2Do6FOFKIIlHaswTXI70aiQEEbHztOukA EUFAtM6sCCINow01eH824TFiLw== X-Google-Smtp-Source: ABdhPJw2jmwgTmZEgqtXJQ8BT+0frIfcOwv6QQsFYGUkgZtQBdhdiIC61pVtTpnGW9Ux3cY+6emP4Q== X-Received: by 2002:a17:90b:4d09:: with SMTP id mw9mr13604040pjb.238.1637264821320; Thu, 18 Nov 2021 11:47:01 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id t40sm435895pfg.107.2021.11.18.11.46.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 11:46:59 -0800 (PST) Date: Thu, 18 Nov 2021 19:46:56 +0000 From: Sean Christopherson To: David Woodhouse Subject: Re: [PATCH v3 08/12] KVM: Propagate vcpu explicitly to mark_page_dirty_in_slot() Message-ID: References: <20211117174003.297096-1-dwmw2@infradead.org> <20211117174003.297096-9-dwmw2@infradead.org> <85d9fec17f32c3eb9e100e56b91af050.squirrel@twosheds.infradead.org> <4c48546b-eb4a-dff7-cc38-5df54f73f5d4@redhat.com> <20b5952e76c54a3a5dfe5a898e3b835404ac6fb1.camel@infradead.org> <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> Cc: Anup Patel , "wanpengli @ tencent . com" , kvm , Benjamin Herrenschmidt , Joao Martins , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-s390@vger.kernel.org, Michael Ellerman , "joro @ 8bytes . org" , Huacai Chen , Christian Borntraeger , Aleksandar Markovic , karahmed@amazon.com, Catalin Marinas , Boris Ostrovsky , linux-arm-kernel , "jmattson @ google . com" , "mtosatti @ redhat . com" , linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , "vkuznets @ redhat . com" , linuxppc-dev@lists.ozlabs.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu T24gVGh1LCBOb3YgMTgsIDIwMjEsIERhdmlkIFdvb2Rob3VzZSB3cm90ZToKPiAKPiAKPiBPbiAx OCBOb3ZlbWJlciAyMDIxIDE4OjUwOjU1IEdNVCwgU2VhbiBDaHJpc3RvcGhlcnNvbiA8c2Vhbmpj QGdvb2dsZS5jb20+IHdyb3RlOgo+ID5PbiBUaHUsIE5vdiAxOCwgMjAyMSwgU2VhbiBDaHJpc3Rv cGhlcnNvbiB3cm90ZToKPiA+PiBPbiBUaHUsIE5vdiAxOCwgMjAyMSwgRGF2aWQgV29vZGhvdXNl IHdyb3RlOgo+ID4+ID4gVGhhdCBsZWF2ZXMgdGhlIG9uZSBpbiBURFAgTU1VIGhhbmRsZV9jaGFu Z2VkX3NwdGVfZGlydHlfbG9nKCkgd2hpY2gKPiA+PiA+IEFGQUlDVCBjYW4gdHJpZ2dlciB0aGUg c2FtZSBjcmFzaCBzZWVuIGJ5IGJ1dHQzcmZseWg0Y2sg4oCUIGNhbid0IHRoYXQKPiA+PiA+IGhh cHBlbiBmcm9tIGEgdGhyZWFkIHdoZXJlIGt2bV9nZXRfcnVubmluZ192Y3B1KCkgaXMgTlVMTCB0 b28/IEZvciB0aGF0Cj4gPj4gPiBvbmUgSSdtIG5vdCBzdXJlLgo+ID4+IAo+ID4+IEkgdGhpbmsg Y291bGQgYmUgdHJpZ2dlciBpbiB0aGUgVERQIE1NVSB2aWEga3ZtX21tdV9ub3RpZmllcl9yZWxl YXNlKCkKPiA+PiAtPiBrdm1fbW11X3phcF9hbGwoKSwgZS5nLiBpZiB0aGUgdXNlcnNwYWNlIFZN TSBleGl0cyB3aGlsZSBkaXJ0eSBsb2dnaW5nIGlzCj4gPj4gZW5hYmxlZC4gIFRoYXQgc2hvdWxk IGJlIGVhc3kgdG8gKGRpcylwcm92ZSB2aWEgYSBzZWxmdGVzdC4KPiA+Cj4gPlNjcmF0Y2ggdGhh dCwgdGhlIGRpcnR5IGxvZyB1cGRhdGUgaXMgZ3VhcmRlZCBieSB0aGUgbmV3X3NwdGUgYmVpbmcg cHJlc2VudCwgc28KPiA+emFwcGluZyBvZiBhbnkga2luZCB3b24ndCB0cmlnZ2VyIGl0Lgo+ID4K PiA+Q3VycmVudGx5LCBJIGJlbGlldmUgdGhlIG9ubHkgcGF0aCB0aGF0IHdvdWxkIGNyZWF0ZSBh IHByZXNlbnQgU1BURSB3aXRob3V0IGFuCj4gPmFjdGl2ZSB2Q1BVIGlzIG1tdV9ub3RpZmVyLmNo YW5nZV9wdGUsIGJ1dCB0aGF0IHNxdWVha3MgYnkgYmVjYXVzZSBpdHMgcmVxdWlyZWQKPiA+dG8g YmUgd3JhcHBlZCB3aXRoIGludmFsaWRhdGVfcmFuZ2Vfe3N0YXJ0LGVuZH0oTU1VX05PVElGWV9D TEVBUiksIGFuZCBLVk0gemFwcwo+ID5pbiB0aGF0IHNpdHVhdGlvbi4KPiAKPiBJcyBpdCBzdWZm aWNpZW50IHRvIGhhdmUgKmFuKiBhY3RpdmUgdkNQVT8gIFdoYXQgaWYgYSBWTU0gaGFzIHRocmVh ZHMgZm9yCj4gYWN0aXZlIHZDUFVzIGJ1dCBpcyBkb2luZyBtbWFwL211bm1hcCBvbiBhICpkaWZm ZXJlbnQqIHRocmVhZD8gRG9lcyB0aGF0IG5vdAo+IHN1ZmZlciB0aGUgc2FtZSBjcmFzaD8KCkl0 IGlzIHN1ZmZpY2llbnQgZm9yIHRoZSBjdXJyZW50IHBoeXNpY2FsIENQVSB0byBoYXZlIGFuIGFj dGl2ZSB2Q1BVLCB3aGljaCBpcwpnZW5lcmFsbHkgZ3VhcmFudGVlZCBpbiB0aGUgTU1VIGNvZGUg YmVjYXVzZSwgd2l0aCBhIGZldyBleGNlcHRpb25zLCBwb3B1bGF0aW5nClNQVEVzIGlzIGRvbmUg aW4gdkNQVSBjb250ZXh0LgoKbW1hcCgpIHdpbGwgbmV2ZXIgZGlyZWN0bHkgdHJpZ2dlciBTUFRF IGNyZWF0aW9uLCBLVk0gZmlyc3QgcmVxdWlyZXMgYSB2Q1BVIHRvCmZhdWx0IG9uIHRoZSBuZXcg YWRkcmVzcy4gIG11bm1hcCgpIGlzIGEgcHVyZSB6YXAgZmxvdywgaS5lLiB3b24ndCBjcmVhdGUg YQpwcmVzZW50IFNQVEUgYW5kIHRyaWdnZXIgdGhlIHdyaXRlYmFjayBvZiB0aGUgZGlydHkgYml0 LgoKVGhhdCdzIGFsc28gd2h5IEkgZGlzbGlrZSB1c2luZyBrdm1fZ2V0X3J1bm5pbmdfdmNwdSgp OyB3aGVuIGl0J3MgbmVlZGVkLCB0aGVyZSdzCmEgdmFsaWQgdkNQVSBmcm9tIHRoZSBjYWxsZXIs IGJ1dCBpdCBkZWxpYmVyYXRlbHkgZ2V0cyBkcm9wcGVkIGFuZCBpbmRpcmVjdGx5CnBpY2tlZCBi YWNrIHVwLgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpr dm1hcm0gbWFpbGluZyBsaXN0Cmt2bWFybUBsaXN0cy5jcy5jb2x1bWJpYS5lZHUKaHR0cHM6Ly9s aXN0cy5jcy5jb2x1bWJpYS5lZHUvbWFpbG1hbi9saXN0aW5mby9rdm1hcm0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B60C433F5 for ; Thu, 18 Nov 2021 19:47:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 287D161A56 for ; Thu, 18 Nov 2021 19:47:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229914AbhKRTuC (ORCPT ); Thu, 18 Nov 2021 14:50:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229656AbhKRTuC (ORCPT ); Thu, 18 Nov 2021 14:50:02 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA17AC061574 for ; Thu, 18 Nov 2021 11:47:01 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id nh10-20020a17090b364a00b001a69adad5ebso6699987pjb.2 for ; Thu, 18 Nov 2021 11:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=TJltHTqmUDTIgL1P24a20gD79MoRsYyT+oWNI7Zk1KCB0kxMW9nay123E+ZoFctpYI aiNnMLJVER0OaOWjGQFAAodEb7XssAPkbTv3HZ/7Xvc56U/KwoqRQGvrj5PKP0bcKR4k 2vvhP28esD9If7TAFeNjBviDlcYadmbYqHQK4EKbmvqKoyyDe2mAxq3N83OoFp8cHZvs tW/44+HbAaU7BnQeov5HYKclsNsjLoArhh5d8PMaS08emGsC4hW0iBqIihQVo007XCgE f+/05PLBrEHbV7h1EwSp8E42irdKiOdyzv/mpEc9ne+Zn3+nJLJHbZQwUjsMNM99uGwK D+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=R5Dflp6lLA4S/HU7Mt2lxTaj37Hk3ZDgUYUO9Eu6J1iVDUzX+hGe2HQ7O2cSxgNVmJ Qr2u2CMzhw9RYdpjr4HiTaBl9PCX62E80hPdFrXW2+kw5wZhpqynGlLQdFpVO6tAzGaS xgaw1/OGPGfgr0NUFnj9Tnm0K2BBa/679H1EuD+ZmGTHUOWJm8Mpqd86cP4XnUIrmlo5 Zqm7dDNlx6O+bcWXlG3PHYodMp7RkXWO9aN+Ue+r89LhLOXkQK2ooTzr2xf/6w0/VLKD hGoO1sJFTbiUksoUl/SJuIN/OlHOoXzwxTQyJyR+2zGhW4qV0o8kN7JDITEZ6lYsU6+q pfgw== X-Gm-Message-State: AOAM530HYiIYE103RmuYvpoyVYMhOSniAheimk2WAVpiXDwTMjGmxtkN 248h3stt4t24fxz9EFkbHKm81g== X-Google-Smtp-Source: ABdhPJw2jmwgTmZEgqtXJQ8BT+0frIfcOwv6QQsFYGUkgZtQBdhdiIC61pVtTpnGW9Ux3cY+6emP4Q== X-Received: by 2002:a17:90b:4d09:: with SMTP id mw9mr13604040pjb.238.1637264821320; Thu, 18 Nov 2021 11:47:01 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id t40sm435895pfg.107.2021.11.18.11.46.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 11:46:59 -0800 (PST) Date: Thu, 18 Nov 2021 19:46:56 +0000 From: Sean Christopherson To: David Woodhouse Cc: Paolo Bonzini , kvm , Boris Ostrovsky , Joao Martins , "jmattson @ google . com" , "wanpengli @ tencent . com" , "vkuznets @ redhat . com" , "mtosatti @ redhat . com" , "joro @ 8bytes . org" , karahmed@amazon.com, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Will Deacon , Huacai Chen , Aleksandar Markovic , Michael Ellerman , Benjamin Herrenschmidt , Anup Patel , Christian Borntraeger , kvmarm@lists.cs.columbia.edu, linux-arm-kernel , linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-s390@vger.kernel.org Subject: Re: [PATCH v3 08/12] KVM: Propagate vcpu explicitly to mark_page_dirty_in_slot() Message-ID: References: <20211117174003.297096-1-dwmw2@infradead.org> <20211117174003.297096-9-dwmw2@infradead.org> <85d9fec17f32c3eb9e100e56b91af050.squirrel@twosheds.infradead.org> <4c48546b-eb4a-dff7-cc38-5df54f73f5d4@redhat.com> <20b5952e76c54a3a5dfe5a898e3b835404ac6fb1.camel@infradead.org> <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org On Thu, Nov 18, 2021, David Woodhouse wrote: > > > On 18 November 2021 18:50:55 GMT, Sean Christopherson wrote: > >On Thu, Nov 18, 2021, Sean Christopherson wrote: > >> On Thu, Nov 18, 2021, David Woodhouse wrote: > >> > That leaves the one in TDP MMU handle_changed_spte_dirty_log() which > >> > AFAICT can trigger the same crash seen by butt3rflyh4ck — can't that > >> > happen from a thread where kvm_get_running_vcpu() is NULL too? For that > >> > one I'm not sure. > >> > >> I think could be trigger in the TDP MMU via kvm_mmu_notifier_release() > >> -> kvm_mmu_zap_all(), e.g. if the userspace VMM exits while dirty logging is > >> enabled. That should be easy to (dis)prove via a selftest. > > > >Scratch that, the dirty log update is guarded by the new_spte being present, so > >zapping of any kind won't trigger it. > > > >Currently, I believe the only path that would create a present SPTE without an > >active vCPU is mmu_notifer.change_pte, but that squeaks by because its required > >to be wrapped with invalidate_range_{start,end}(MMU_NOTIFY_CLEAR), and KVM zaps > >in that situation. > > Is it sufficient to have *an* active vCPU? What if a VMM has threads for > active vCPUs but is doing mmap/munmap on a *different* thread? Does that not > suffer the same crash? It is sufficient for the current physical CPU to have an active vCPU, which is generally guaranteed in the MMU code because, with a few exceptions, populating SPTEs is done in vCPU context. mmap() will never directly trigger SPTE creation, KVM first requires a vCPU to fault on the new address. munmap() is a pure zap flow, i.e. won't create a present SPTE and trigger the writeback of the dirty bit. That's also why I dislike using kvm_get_running_vcpu(); when it's needed, there's a valid vCPU from the caller, but it deliberately gets dropped and indirectly picked back up. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76D2FC433F5 for ; Thu, 18 Nov 2021 19:47:51 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 92CAA61A52 for ; Thu, 18 Nov 2021 19:47:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 92CAA61A52 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Hw9Lr72pBz3c7v for ; Fri, 19 Nov 2021 06:47:48 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=TJltHTqm; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=seanjc@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=TJltHTqm; dkim-atps=neutral Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Hw9L25sp5z2yPs for ; Fri, 19 Nov 2021 06:47:04 +1100 (AEDT) Received: by mail-pj1-x102e.google.com with SMTP id np6-20020a17090b4c4600b001a90b011e06so6665270pjb.5 for ; Thu, 18 Nov 2021 11:47:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=TJltHTqmUDTIgL1P24a20gD79MoRsYyT+oWNI7Zk1KCB0kxMW9nay123E+ZoFctpYI aiNnMLJVER0OaOWjGQFAAodEb7XssAPkbTv3HZ/7Xvc56U/KwoqRQGvrj5PKP0bcKR4k 2vvhP28esD9If7TAFeNjBviDlcYadmbYqHQK4EKbmvqKoyyDe2mAxq3N83OoFp8cHZvs tW/44+HbAaU7BnQeov5HYKclsNsjLoArhh5d8PMaS08emGsC4hW0iBqIihQVo007XCgE f+/05PLBrEHbV7h1EwSp8E42irdKiOdyzv/mpEc9ne+Zn3+nJLJHbZQwUjsMNM99uGwK D+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=278LAycss92ZlaA1H2kECAp7ckK9ulO2yW89hftz+302cynN6CHu+L5UfZIosB7mvq 8cpoQXAOpMssoultsqDNBym134qiH2M3cHn8Uhs2qkJohnr/LYdaZhKdTklXFq1TAeTo JebdzNDIVmnuMubEtzPqU8IfhyDJyqBa2XQqIkaSBK/IUad7lrcmSg7JNQGHXyQAThH3 VSWiyePlDEAFp62/R822Cc/i4y4H8QjY6uh5zd4JimwRVfTQ1VSuYZhAvTRDzK/NdTU1 AwSrcacIGetJeuWHnQaJZ+2Px7c0tLMA6hVyhgNQyoN15/Jrp3C4M2SJ+wGzR5uirxlP XTVg== X-Gm-Message-State: AOAM530UZGorHZ74oNVWN7g/rmbOC4sbad6cju5rPuoyH+eUuRw3gDLI QcTKdIA4fBWUGV8fAvjknVxwrA== X-Google-Smtp-Source: ABdhPJw2jmwgTmZEgqtXJQ8BT+0frIfcOwv6QQsFYGUkgZtQBdhdiIC61pVtTpnGW9Ux3cY+6emP4Q== X-Received: by 2002:a17:90b:4d09:: with SMTP id mw9mr13604040pjb.238.1637264821320; Thu, 18 Nov 2021 11:47:01 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id t40sm435895pfg.107.2021.11.18.11.46.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 11:46:59 -0800 (PST) Date: Thu, 18 Nov 2021 19:46:56 +0000 From: Sean Christopherson To: David Woodhouse Subject: Re: [PATCH v3 08/12] KVM: Propagate vcpu explicitly to mark_page_dirty_in_slot() Message-ID: References: <20211117174003.297096-1-dwmw2@infradead.org> <20211117174003.297096-9-dwmw2@infradead.org> <85d9fec17f32c3eb9e100e56b91af050.squirrel@twosheds.infradead.org> <4c48546b-eb4a-dff7-cc38-5df54f73f5d4@redhat.com> <20b5952e76c54a3a5dfe5a898e3b835404ac6fb1.camel@infradead.org> <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , "wanpengli @ tencent . com" , kvm , Joao Martins , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-s390@vger.kernel.org, "joro @ 8bytes . org" , Huacai Chen , Christian Borntraeger , Aleksandar Markovic , karahmed@amazon.com, Catalin Marinas , Suzuki K Poulose , Boris Ostrovsky , Alexandru Elisei , linux-arm-kernel , "jmattson @ google . com" , "mtosatti @ redhat . com" , linux-mips@vger.kernel.org, James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , "vkuznets @ redhat . com" , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Nov 18, 2021, David Woodhouse wrote: > > > On 18 November 2021 18:50:55 GMT, Sean Christopherson wrote: > >On Thu, Nov 18, 2021, Sean Christopherson wrote: > >> On Thu, Nov 18, 2021, David Woodhouse wrote: > >> > That leaves the one in TDP MMU handle_changed_spte_dirty_log() which > >> > AFAICT can trigger the same crash seen by butt3rflyh4ck — can't that > >> > happen from a thread where kvm_get_running_vcpu() is NULL too? For that > >> > one I'm not sure. > >> > >> I think could be trigger in the TDP MMU via kvm_mmu_notifier_release() > >> -> kvm_mmu_zap_all(), e.g. if the userspace VMM exits while dirty logging is > >> enabled. That should be easy to (dis)prove via a selftest. > > > >Scratch that, the dirty log update is guarded by the new_spte being present, so > >zapping of any kind won't trigger it. > > > >Currently, I believe the only path that would create a present SPTE without an > >active vCPU is mmu_notifer.change_pte, but that squeaks by because its required > >to be wrapped with invalidate_range_{start,end}(MMU_NOTIFY_CLEAR), and KVM zaps > >in that situation. > > Is it sufficient to have *an* active vCPU? What if a VMM has threads for > active vCPUs but is doing mmap/munmap on a *different* thread? Does that not > suffer the same crash? It is sufficient for the current physical CPU to have an active vCPU, which is generally guaranteed in the MMU code because, with a few exceptions, populating SPTEs is done in vCPU context. mmap() will never directly trigger SPTE creation, KVM first requires a vCPU to fault on the new address. munmap() is a pure zap flow, i.e. won't create a present SPTE and trigger the writeback of the dirty bit. That's also why I dislike using kvm_get_running_vcpu(); when it's needed, there's a valid vCPU from the caller, but it deliberately gets dropped and indirectly picked back up. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFC95C433F5 for ; Thu, 18 Nov 2021 19:48:37 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 97B1A61A52 for ; Thu, 18 Nov 2021 19:48:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 97B1A61A52 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1CzSmCESvyYBQX13GbkZW97QlS31K5eczyMc0dtPOsg=; b=VOBA0EcjwSQZmV Bvrrbh2asyX3Uf4n0mj7qYDU+wl6WnB8WKq3E091QLe7nj+9wVtZLXJBF2LmnzXGtJOJjlL0YLXdD HYaJ4QiNDhOUcPQH5tLQbEB4aMJS47kmB9B2R7/sXOPDOPCSVH+/1h8ABbtrCill5wr417cuNG4hG jz6RKaPdIXfNbgR8zcRUa+2iJBL89p5kI3wwxDJLg+PVB+ck5mcN/SUJGa+Rb3bo3Y/OM1SpU5S5p 71+RUIeka5h8gnsYd/tvzhvB0w+JjZVtaeEBG12cWWX9VavoIlZW3BqUcCtlcdKAF5np/IpEH5tYQ MMrOcgo6p6n6GMbzWs4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnnNC-008m7k-8Y; Thu, 18 Nov 2021 19:47:06 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnnN8-008m6d-9V for linux-arm-kernel@lists.infradead.org; Thu, 18 Nov 2021 19:47:03 +0000 Received: by mail-pl1-x62c.google.com with SMTP id v19so6175536plo.7 for ; Thu, 18 Nov 2021 11:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=TJltHTqmUDTIgL1P24a20gD79MoRsYyT+oWNI7Zk1KCB0kxMW9nay123E+ZoFctpYI aiNnMLJVER0OaOWjGQFAAodEb7XssAPkbTv3HZ/7Xvc56U/KwoqRQGvrj5PKP0bcKR4k 2vvhP28esD9If7TAFeNjBviDlcYadmbYqHQK4EKbmvqKoyyDe2mAxq3N83OoFp8cHZvs tW/44+HbAaU7BnQeov5HYKclsNsjLoArhh5d8PMaS08emGsC4hW0iBqIihQVo007XCgE f+/05PLBrEHbV7h1EwSp8E42irdKiOdyzv/mpEc9ne+Zn3+nJLJHbZQwUjsMNM99uGwK D+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=etzXRtUGTr8lCEC3N+zBz9sC9KLaK/1vTaoNub+XSrA=; b=sgul5xVUsSdWq0OGLI27q+QjJplfsb15B2h6QAUDk2J18vFJRpi9jGwcUMUCXiGz9f TMXoqbr6YLjoDKe9y53vsqpXR3CSjEXdhRw8+XrrWpgVZOAf/tPy5jn6FPZzI34GquJs O28mlwKfCVhDtZP0vLbyxuGXjSAiS0v1whopoZ5BFu5PmluN3oQswWpiqTnAVf475J09 8xziIdfmnwyajDeMErub2TLG+Ux/DfuH5heH6Cjfbj+oG2GGpTTkRLbFCy6TE8kXZVLJ W9laR3X+r1O9v2MBZsgVcL958pv9+cfPsU07JYqtM1MIG2Twgj45Cg83eCzZFyE+xRXm CmKA== X-Gm-Message-State: AOAM531VWTCzVWIx2AhMlT/gCh0s7udqCAKCpqZvvs5yJml0SAGNaIsb Q3wk4Nmgi3DQ3ja4RuTv2NWDjg== X-Google-Smtp-Source: ABdhPJw2jmwgTmZEgqtXJQ8BT+0frIfcOwv6QQsFYGUkgZtQBdhdiIC61pVtTpnGW9Ux3cY+6emP4Q== X-Received: by 2002:a17:90b:4d09:: with SMTP id mw9mr13604040pjb.238.1637264821320; Thu, 18 Nov 2021 11:47:01 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id t40sm435895pfg.107.2021.11.18.11.46.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 11:46:59 -0800 (PST) Date: Thu, 18 Nov 2021 19:46:56 +0000 From: Sean Christopherson To: David Woodhouse Cc: Paolo Bonzini , kvm , Boris Ostrovsky , Joao Martins , "jmattson @ google . com" , "wanpengli @ tencent . com" , "vkuznets @ redhat . com" , "mtosatti @ redhat . com" , "joro @ 8bytes . org" , karahmed@amazon.com, Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , Catalin Marinas , Will Deacon , Huacai Chen , Aleksandar Markovic , Michael Ellerman , Benjamin Herrenschmidt , Anup Patel , Christian Borntraeger , kvmarm@lists.cs.columbia.edu, linux-arm-kernel , linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-s390@vger.kernel.org Subject: Re: [PATCH v3 08/12] KVM: Propagate vcpu explicitly to mark_page_dirty_in_slot() Message-ID: References: <20211117174003.297096-1-dwmw2@infradead.org> <20211117174003.297096-9-dwmw2@infradead.org> <85d9fec17f32c3eb9e100e56b91af050.squirrel@twosheds.infradead.org> <4c48546b-eb4a-dff7-cc38-5df54f73f5d4@redhat.com> <20b5952e76c54a3a5dfe5a898e3b835404ac6fb1.camel@infradead.org> <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <35AEC3FD-B46A-451D-B7D5-4B1BDD5407BD@infradead.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211118_114702_344580_E0346DD3 X-CRM114-Status: GOOD ( 20.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1LCBOb3YgMTgsIDIwMjEsIERhdmlkIFdvb2Rob3VzZSB3cm90ZToKPiAKPiAKPiBPbiAx OCBOb3ZlbWJlciAyMDIxIDE4OjUwOjU1IEdNVCwgU2VhbiBDaHJpc3RvcGhlcnNvbiA8c2Vhbmpj QGdvb2dsZS5jb20+IHdyb3RlOgo+ID5PbiBUaHUsIE5vdiAxOCwgMjAyMSwgU2VhbiBDaHJpc3Rv cGhlcnNvbiB3cm90ZToKPiA+PiBPbiBUaHUsIE5vdiAxOCwgMjAyMSwgRGF2aWQgV29vZGhvdXNl IHdyb3RlOgo+ID4+ID4gVGhhdCBsZWF2ZXMgdGhlIG9uZSBpbiBURFAgTU1VIGhhbmRsZV9jaGFu Z2VkX3NwdGVfZGlydHlfbG9nKCkgd2hpY2gKPiA+PiA+IEFGQUlDVCBjYW4gdHJpZ2dlciB0aGUg c2FtZSBjcmFzaCBzZWVuIGJ5IGJ1dHQzcmZseWg0Y2sg4oCUIGNhbid0IHRoYXQKPiA+PiA+IGhh cHBlbiBmcm9tIGEgdGhyZWFkIHdoZXJlIGt2bV9nZXRfcnVubmluZ192Y3B1KCkgaXMgTlVMTCB0 b28/IEZvciB0aGF0Cj4gPj4gPiBvbmUgSSdtIG5vdCBzdXJlLgo+ID4+IAo+ID4+IEkgdGhpbmsg Y291bGQgYmUgdHJpZ2dlciBpbiB0aGUgVERQIE1NVSB2aWEga3ZtX21tdV9ub3RpZmllcl9yZWxl YXNlKCkKPiA+PiAtPiBrdm1fbW11X3phcF9hbGwoKSwgZS5nLiBpZiB0aGUgdXNlcnNwYWNlIFZN TSBleGl0cyB3aGlsZSBkaXJ0eSBsb2dnaW5nIGlzCj4gPj4gZW5hYmxlZC4gIFRoYXQgc2hvdWxk IGJlIGVhc3kgdG8gKGRpcylwcm92ZSB2aWEgYSBzZWxmdGVzdC4KPiA+Cj4gPlNjcmF0Y2ggdGhh dCwgdGhlIGRpcnR5IGxvZyB1cGRhdGUgaXMgZ3VhcmRlZCBieSB0aGUgbmV3X3NwdGUgYmVpbmcg cHJlc2VudCwgc28KPiA+emFwcGluZyBvZiBhbnkga2luZCB3b24ndCB0cmlnZ2VyIGl0Lgo+ID4K PiA+Q3VycmVudGx5LCBJIGJlbGlldmUgdGhlIG9ubHkgcGF0aCB0aGF0IHdvdWxkIGNyZWF0ZSBh IHByZXNlbnQgU1BURSB3aXRob3V0IGFuCj4gPmFjdGl2ZSB2Q1BVIGlzIG1tdV9ub3RpZmVyLmNo YW5nZV9wdGUsIGJ1dCB0aGF0IHNxdWVha3MgYnkgYmVjYXVzZSBpdHMgcmVxdWlyZWQKPiA+dG8g YmUgd3JhcHBlZCB3aXRoIGludmFsaWRhdGVfcmFuZ2Vfe3N0YXJ0LGVuZH0oTU1VX05PVElGWV9D TEVBUiksIGFuZCBLVk0gemFwcwo+ID5pbiB0aGF0IHNpdHVhdGlvbi4KPiAKPiBJcyBpdCBzdWZm aWNpZW50IHRvIGhhdmUgKmFuKiBhY3RpdmUgdkNQVT8gIFdoYXQgaWYgYSBWTU0gaGFzIHRocmVh ZHMgZm9yCj4gYWN0aXZlIHZDUFVzIGJ1dCBpcyBkb2luZyBtbWFwL211bm1hcCBvbiBhICpkaWZm ZXJlbnQqIHRocmVhZD8gRG9lcyB0aGF0IG5vdAo+IHN1ZmZlciB0aGUgc2FtZSBjcmFzaD8KCkl0 IGlzIHN1ZmZpY2llbnQgZm9yIHRoZSBjdXJyZW50IHBoeXNpY2FsIENQVSB0byBoYXZlIGFuIGFj dGl2ZSB2Q1BVLCB3aGljaCBpcwpnZW5lcmFsbHkgZ3VhcmFudGVlZCBpbiB0aGUgTU1VIGNvZGUg YmVjYXVzZSwgd2l0aCBhIGZldyBleGNlcHRpb25zLCBwb3B1bGF0aW5nClNQVEVzIGlzIGRvbmUg aW4gdkNQVSBjb250ZXh0LgoKbW1hcCgpIHdpbGwgbmV2ZXIgZGlyZWN0bHkgdHJpZ2dlciBTUFRF IGNyZWF0aW9uLCBLVk0gZmlyc3QgcmVxdWlyZXMgYSB2Q1BVIHRvCmZhdWx0IG9uIHRoZSBuZXcg YWRkcmVzcy4gIG11bm1hcCgpIGlzIGEgcHVyZSB6YXAgZmxvdywgaS5lLiB3b24ndCBjcmVhdGUg YQpwcmVzZW50IFNQVEUgYW5kIHRyaWdnZXIgdGhlIHdyaXRlYmFjayBvZiB0aGUgZGlydHkgYml0 LgoKVGhhdCdzIGFsc28gd2h5IEkgZGlzbGlrZSB1c2luZyBrdm1fZ2V0X3J1bm5pbmdfdmNwdSgp OyB3aGVuIGl0J3MgbmVlZGVkLCB0aGVyZSdzCmEgdmFsaWQgdkNQVSBmcm9tIHRoZSBjYWxsZXIs IGJ1dCBpdCBkZWxpYmVyYXRlbHkgZ2V0cyBkcm9wcGVkIGFuZCBpbmRpcmVjdGx5CnBpY2tlZCBi YWNrIHVwLgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K bGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZy YWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGlu dXgtYXJtLWtlcm5lbAo=