From mboxrd@z Thu Jan 1 00:00:00 1970 From: Samuel Holland Date: Fri, 13 Sep 2024 21:52:07 -0500 Subject: [PATCH v4 09/10] RISC-V: KVM: Allow Smnpm and Ssnpm extensions for guests In-Reply-To: References: <20240829010151.2813377-1-samuel.holland@sifive.com> <20240829010151.2813377-10-samuel.holland@sifive.com> <20ab0fa2-d5dd-446d-9fff-a3ef82e8db35@sifive.com> <4c010cb1-b57c-427e-a241-1dd3ab15f2ce@sifive.com> Message-ID: <8e474b14-e963-4d3e-8240-37f662e7bd8a@sifive.com> List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi Anup, On 2024-09-05 12:18 AM, Anup Patel wrote: > On Wed, Sep 4, 2024 at 9:25?PM Samuel Holland wrote: >> >> On 2024-09-04 10:20 AM, Anup Patel wrote: >>> On Wed, Sep 4, 2024 at 8:27?PM Samuel Holland wrote: >>>> >>>> Hi Anup, >>>> >>>> On 2024-09-04 9:45 AM, Anup Patel wrote: >>>>> On Wed, Sep 4, 2024 at 8:01?PM Samuel Holland wrote: >>>>>> On 2024-09-04 7:17 AM, Anup Patel wrote: >>>>>>> On Thu, Aug 29, 2024 at 6:32?AM Samuel Holland >>>>>>> wrote: >>>>>>>> >>>>>>>> The interface for controlling pointer masking in VS-mode is henvcfg.PMM, >>>>>>>> which is part of the Ssnpm extension, even though pointer masking in >>>>>>>> HS-mode is provided by the Smnpm extension. As a result, emulating Smnpm >>>>>>>> in the guest requires (only) Ssnpm on the host. >>>>>>>> >>>>>>>> Since the guest configures Smnpm through the SBI Firmware Features >>>>>>>> interface, the extension can be disabled by failing the SBI call. Ssnpm >>>>>>>> cannot be disabled without intercepting writes to the senvcfg CSR. >>>>>>>> >>>>>>>> Signed-off-by: Samuel Holland >>>>>>>> --- >>>>>>>> >>>>>>>> (no changes since v2) >>>>>>>> >>>>>>>> Changes in v2: >>>>>>>> - New patch for v2 >>>>>>>> >>>>>>>> arch/riscv/include/uapi/asm/kvm.h | 2 ++ >>>>>>>> arch/riscv/kvm/vcpu_onereg.c | 3 +++ >>>>>>>> 2 files changed, 5 insertions(+) >>>>>>>> >>>>>>>> diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> index e97db3296456..4f24201376b1 100644 >>>>>>>> --- a/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> +++ b/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> @@ -175,6 +175,8 @@ enum KVM_RISCV_ISA_EXT_ID { >>>>>>>> KVM_RISCV_ISA_EXT_ZCF, >>>>>>>> KVM_RISCV_ISA_EXT_ZCMOP, >>>>>>>> KVM_RISCV_ISA_EXT_ZAWRS, >>>>>>>> + KVM_RISCV_ISA_EXT_SMNPM, >>>>>>>> + KVM_RISCV_ISA_EXT_SSNPM, >>>>>>>> KVM_RISCV_ISA_EXT_MAX, >>>>>>>> }; >>>>>>>> >>>>>>>> diff --git a/arch/riscv/kvm/vcpu_onereg.c b/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> index b319c4c13c54..6f833ec2344a 100644 >>>>>>>> --- a/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> +++ b/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> @@ -34,9 +34,11 @@ static const unsigned long kvm_isa_ext_arr[] = { >>>>>>>> [KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m, >>>>>>>> [KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v, >>>>>>>> /* Multi letter extensions (alphabetically sorted) */ >>>>>>>> + [KVM_RISCV_ISA_EXT_SMNPM] = RISCV_ISA_EXT_SSNPM, >>>>>>> >>>>>>> Why not use KVM_ISA_EXT_ARR() macro here ? >>>>>> >>>>>> Because the extension name in the host does not match the extension name in the >>>>>> guest. Pointer masking for HS mode is provided by Smnpm. Pointer masking for VS >>>>>> mode is provided by Ssnpm at the hardware level, but this needs to appear to the >>>>>> guest as if Smnpm was implemented, since the guest thinks it is running on bare >>>>>> metal. >>>>> >>>>> Okay, makes sense. >>>>> >>>>>> >>>>>>>> KVM_ISA_EXT_ARR(SMSTATEEN), >>>>>>>> KVM_ISA_EXT_ARR(SSAIA), >>>>>>>> KVM_ISA_EXT_ARR(SSCOFPMF), >>>>>>>> + KVM_ISA_EXT_ARR(SSNPM), >>>>>>>> KVM_ISA_EXT_ARR(SSTC), >>>>>>>> KVM_ISA_EXT_ARR(SVINVAL), >>>>>>>> KVM_ISA_EXT_ARR(SVNAPOT), >>>>>>>> @@ -129,6 +131,7 @@ static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext) >>>>>>>> case KVM_RISCV_ISA_EXT_M: >>>>>>>> /* There is not architectural config bit to disable sscofpmf completely */ >>>>>>>> case KVM_RISCV_ISA_EXT_SSCOFPMF: >>>>>>>> + case KVM_RISCV_ISA_EXT_SSNPM: >>>>>>> >>>>>>> Why not add KVM_RISCV_ISA_EXT_SMNPM here ? >>>>>>> >>>>>>> Disabling Smnpm from KVM user space is very different from >>>>>>> disabling Smnpm from Guest using SBI FWFT extension. >>>>>> >>>>>> Until a successful SBI FWFT call to KVM to enable pointer masking for VS mode, >>>>>> the existence of Smnpm has no visible effect on the guest. So failing the SBI >>>>>> call is sufficient to pretend that the hardware does not support Smnpm. >>>>>> >>>>>>> The KVM user space should always add Smnpm in the >>>>>>> Guest ISA string whenever the Host ISA string has it. >>>>>> >>>>>> I disagree. Allowing userspace to disable extensions is useful for testing and >>>>>> to support migration to hosts which do not support those extensions. So I would >>>>>> only add extensions to this list if there is no possible way to disable them. >>>>> >>>>> I am not saying to disallow KVM user space disabling Smnpm. >>>> >>>> Then I'm confused. This is the "return false;" switch case inside >>>> kvm_riscv_vcpu_isa_disable_allowed(). If I add KVM_RISCV_ISA_EXT_SMNPM here, >>>> then (unless I am misreading the code) I am disallowing KVM userspace from >>>> disabling Smnpm in the guest (i.e. preventing KVM userspace from removing Smnpm >>>> from the guest ISA string). If that is not desired, then why do you suggest I >>>> add KVM_RISCV_ISA_EXT_SMNPM here? >>> >>> Yes, adding KVM_RISCV_ISA_EXT_SMNPM here means KVM >>> user space can't disable it using ONE_REG interface but KVM user >>> space can certainly not add it in the Guest ISA string. >> >> Is there a problem with allowing KVM userspace to disable the ISA extension with >> the ONE_REG interface? >> >> If KVM userspace removes Smnpm from the ISA string without the host kernel's >> knowledge, that doesn't actually prevent the guest from successfully calling >> sbi_fwft_set(POINTER_MASKING_PMLEN, ...), so it doesn't guarantee that the VM >> can be migrated to a host without pointer masking support. So the ONE_REG >> interface still has value. (And that's my answer to your original question "Why >> not add KVM_RISCV_ISA_EXT_SMNPM here ?") > > Currently, disabling KVM_RISCV_ISA_EXT_SMNPM via ONE_REG > will only clear the corresponding bit in VCPU isa bitmap. Basically, the > KVM user space disabling KVM_RISCV_ISA_EXT_SMNPM for Guest > changes nothing for the Guest/VM. > > On other hand, disabling KVM_RISCV_ISA_EXT_SVPBMT via > ONE_REG will not only clear it from VCPU isa bitmap but also > disable Svpmbt from henvcfg CSR for the Guest/VM. > > In other words, if disabling an ISA extension is allowed by the > kvm_riscv_vcpu_isa_disable_allowed() then the Guest/VM must > see a different behaviour when the ISA extension is disabled by > KVM user space. > >> >>>>> The presence of Smnpm in ISA only means that it is present in HW >>>>> but it needs to be explicitly configured/enabled using SBI FWFT. >>>>> >>>>> KVM user space can certainly disable extensions by not adding it to >>>>> ISA string based on the KVMTOOL/QEMU-KVM command line option. >>>>> Additionally, when SBI FWFT is added to KVM RISC-V. It will have its >>>>> own way to explicitly disable firmware features from KVM user space. >>>> >>>> I think we agree on this, but your explanation here appears to conflict with >>>> your suggested code change. Apologies if I'm missing something. >>> >>> I think the confusion is about what does it mean when Smnpm is present >>> in the ISA string. We have two approaches: >>> >>> 1) Presence of Smnpm in ISA string only means it is present in HW but >>> says nothing about its enable/disable state. To configure/enable >>> Smnpm, the supervisor must use SBI FWFT. >>> >>> 2) Presence of Smnpm in ISA string means it is present in HW and >>> enabled at boot-time. To re-configure/disable Smnpm, the supervisor >>> must use SBI FWFT. >>> >>> I am suggesting approach #1 but I am guessing you are leaning towards >>> approach #2 ? >>> >>> For approach #2, additional hencfg.PMM configuration is required in >>> this patch based on the state of KVM_RISCV_ISA_EXT_SMNPM. >> >> No, I am definitely suggesting only approach #1. My proposal for adding pointer >> masking to the SBI FWFT extension[1] specifies the feature as disabled by >> default, and this would apply both inside and ouside a VM. >> >> But I am also suggesting that the ONE_REG interface is a useful way to >> completely hide the extension from the guest, like we do for other extensions >> such as Svpbmt. The only difference between something like Svpbmt and Smnpm is >> that instead of clearing a bit in henvcfg to hide the extension from the guest, >> we reject calls to sbi_fwft_set(POINTER_MASKING_PMLEN, ...) when the ISA >> extension is hidden from the guest. > > I think we are converging towards the same thing. > > How about this ? > > For this series, lets add KVM_RISCV_ISA_EXT_SMNPM to > kvm_riscv_vcpu_isa_disable_allowed() so that for the time > being KVM user space can't disable Smnpm. > > In the future, a separate series which adds SBI FWFT to > KVM RISC-V will remove KVM_RISCV_ISA_EXT_SMNPM > from the kvm_riscv_vcpu_isa_disable_allowed() because > disabling Smnpm from KVM user space would mean that > the POINTER_MASKING_PMLEN firmware feature is > not available to the Guest/VM. > > This means in the future (after SBI FWFT is implemented in > KVM RISC-V), Guest with Smnpm disabled can be migrated > to a host without pointer masking. OK, that is a reasonable compromise. I'll do that for v5. Regards, Samuel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1BA7FA3758 for ; Sat, 14 Sep 2024 02:52:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZTl4JaFnbvWEHti3HXFrD/NVfdgrJjsMymJVw1Vd5Ac=; b=c1uIV8aUsBNW12 IvTkLjFWx/6cVGe+GgxrVnWmbxq1oawCMT/vr7GMwUgKHpY1cTuMcOb7EsyraJZKKX7/3OiLM6Lw+ Fcik96pRgpwHAxAZddbRo6kV8vWW5ouVeo9Ca9gTiVu3lzbT9jNxiBtHK8H9WnuboypZcsBJzWo/V fYJukW5+dTZH4w9K65FeFLynzw12Vc930MPEDg2VE3kEq6A5LJtqPWitirxWtLBptDkYCvzEpIj63 iQVdg0waO8BJ1bMhglDWtnJqRYpNpShYy8AH+GeOKzkwkvn2YuKAN+lnnq1Az3gBw/KSzxbzP207S Wwgdf+gKXlWiTdH/2edA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1spItS-0000000HXT9-3kBU; Sat, 14 Sep 2024 02:52:14 +0000 Received: from mail-io1-xd2c.google.com ([2607:f8b0:4864:20::d2c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1spItP-0000000HXSI-1xy9 for linux-riscv@lists.infradead.org; Sat, 14 Sep 2024 02:52:13 +0000 Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-824ee14f7bfso102466039f.1 for ; Fri, 13 Sep 2024 19:52:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1726282330; x=1726887130; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=PcgocvC07KwdgYkvDV/qMfScM5+LqmPejCCEAA1uMJI=; b=mWQNuzeJa5+AHUvbhiFDuJ4aYv9/dPMSdDnz6/pUJDVkDIOhk7LlC+v7qdQxVDAImB 1ALXCRfWzXTIzoIlsYTXuvC/eHBCghSV61OOGP4jb3Zrmcuxewa6/yYhek5NAKtBNIpA WCukHIuzU+hnHjetMZYFmOmWw+NjJmPWii7fCAlDL1ltHDFIk0/z+Zl96DV4wwXZ9n2G wOIEN9oG3OIwzzmzt29gUjvpJSMckgkLSLtHuMKVPqaysG3KGiIpcMnoMWPeh9GyQltt uTlGrPE3p2vpHKIWBhDJSLaXHpjQFB0B15SJKNwS+u65/nF4vg1vU8GhAzd5sFRoC1Zo Gn3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726282330; x=1726887130; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PcgocvC07KwdgYkvDV/qMfScM5+LqmPejCCEAA1uMJI=; b=m/qMgioClm5liXmIsbdGL3lZtpGDoHAdLCxlkK5YsOcszDRNXc+MkCVS8yv1n+Q4V/ 0ECXP3dOCM/Km6yn3bN33EjEhfdgvbjxsRtWR/ZWNw+X1cWpqGdVdsk84lgUklPatGHa 1afDSDHdmp8h8sE2HzMoeR8EAlZFag2A2gqY73Cy+aBzY7zjp5JWQkzif1YF8vhhvVCl DVS2YJYSS+J9QbxUpzvE9w0u4huFMZdLQwy1sNPZATxZMf5Kd7BnLRXvkXvTDT8U2bCf Z6FVJ7Zu03/I2ZizHa1uqfk+rP2UGxldgWiLU62n7PIE35cQ1v6DibrAhccSJCjrZVoI Ltpw== X-Forwarded-Encrypted: i=1; AJvYcCWBVnpYVlnKbwf7FhW0xZ93ZWzdh4+v5QJbwAfJccYT05xtKU+mvgXiC/Og7dzSHwYseKZexsfmIZ9jNA==@lists.infradead.org X-Gm-Message-State: AOJu0Yz9saotsQHhnhYqdVyWlHMY572GbhUq4jNaopoooUln0MizgEgD 745pC68eytOjra5lj48EZYkKIHVauYcQfjN4ptok6HBlzrht8Y5Tc0Jz4oKpn00= X-Google-Smtp-Source: AGHT+IEaD4v+a/RKnKmId+r+Iz4R86fNRYyVeVXynZnVMTjpM3fXydjExpGYdtf8Iye3kIi/wgX9bQ== X-Received: by 2002:a05:6e02:5ad:b0:3a0:8dae:8b06 with SMTP id e9e14a558f8ab-3a08dae8c99mr24837465ab.9.1726282330175; Fri, 13 Sep 2024 19:52:10 -0700 (PDT) Received: from [100.64.0.1] ([147.124.94.167]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a092dfe1d8sm1609685ab.13.2024.09.13.19.52.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 13 Sep 2024 19:52:09 -0700 (PDT) Message-ID: <8e474b14-e963-4d3e-8240-37f662e7bd8a@sifive.com> Date: Fri, 13 Sep 2024 21:52:07 -0500 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 09/10] RISC-V: KVM: Allow Smnpm and Ssnpm extensions for guests To: Anup Patel Cc: Anup Patel , Palmer Dabbelt , linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, Catalin Marinas , linux-kernel@vger.kernel.org, Conor Dooley , kasan-dev@googlegroups.com, Atish Patra , Evgenii Stepanov , Krzysztof Kozlowski , Rob Herring , "Kirill A . Shutemov" , kvm-riscv@lists.infradead.org References: <20240829010151.2813377-1-samuel.holland@sifive.com> <20240829010151.2813377-10-samuel.holland@sifive.com> <20ab0fa2-d5dd-446d-9fff-a3ef82e8db35@sifive.com> <4c010cb1-b57c-427e-a241-1dd3ab15f2ce@sifive.com> From: Samuel Holland Content-Language: en-US In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240913_195211_556622_EDC7095E X-CRM114-Status: GOOD ( 41.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org SGkgQW51cCwKCk9uIDIwMjQtMDktMDUgMTI6MTggQU0sIEFudXAgUGF0ZWwgd3JvdGU6Cj4gT24g V2VkLCBTZXAgNCwgMjAyNCBhdCA5OjI14oCvUE0gU2FtdWVsIEhvbGxhbmQgPHNhbXVlbC5ob2xs YW5kQHNpZml2ZS5jb20+IHdyb3RlOgo+Pgo+PiBPbiAyMDI0LTA5LTA0IDEwOjIwIEFNLCBBbnVw IFBhdGVsIHdyb3RlOgo+Pj4gT24gV2VkLCBTZXAgNCwgMjAyNCBhdCA4OjI34oCvUE0gU2FtdWVs IEhvbGxhbmQgPHNhbXVlbC5ob2xsYW5kQHNpZml2ZS5jb20+IHdyb3RlOgo+Pj4+Cj4+Pj4gSGkg QW51cCwKPj4+Pgo+Pj4+IE9uIDIwMjQtMDktMDQgOTo0NSBBTSwgQW51cCBQYXRlbCB3cm90ZToK Pj4+Pj4gT24gV2VkLCBTZXAgNCwgMjAyNCBhdCA4OjAx4oCvUE0gU2FtdWVsIEhvbGxhbmQgPHNh bXVlbC5ob2xsYW5kQHNpZml2ZS5jb20+IHdyb3RlOgo+Pj4+Pj4gT24gMjAyNC0wOS0wNCA3OjE3 IEFNLCBBbnVwIFBhdGVsIHdyb3RlOgo+Pj4+Pj4+IE9uIFRodSwgQXVnIDI5LCAyMDI0IGF0IDY6 MzLigK9BTSBTYW11ZWwgSG9sbGFuZAo+Pj4+Pj4+IDxzYW11ZWwuaG9sbGFuZEBzaWZpdmUuY29t PiB3cm90ZToKPj4+Pj4+Pj4KPj4+Pj4+Pj4gVGhlIGludGVyZmFjZSBmb3IgY29udHJvbGxpbmcg cG9pbnRlciBtYXNraW5nIGluIFZTLW1vZGUgaXMgaGVudmNmZy5QTU0sCj4+Pj4+Pj4+IHdoaWNo IGlzIHBhcnQgb2YgdGhlIFNzbnBtIGV4dGVuc2lvbiwgZXZlbiB0aG91Z2ggcG9pbnRlciBtYXNr aW5nIGluCj4+Pj4+Pj4+IEhTLW1vZGUgaXMgcHJvdmlkZWQgYnkgdGhlIFNtbnBtIGV4dGVuc2lv bi4gQXMgYSByZXN1bHQsIGVtdWxhdGluZyBTbW5wbQo+Pj4+Pj4+PiBpbiB0aGUgZ3Vlc3QgcmVx dWlyZXMgKG9ubHkpIFNzbnBtIG9uIHRoZSBob3N0Lgo+Pj4+Pj4+Pgo+Pj4+Pj4+PiBTaW5jZSB0 aGUgZ3Vlc3QgY29uZmlndXJlcyBTbW5wbSB0aHJvdWdoIHRoZSBTQkkgRmlybXdhcmUgRmVhdHVy ZXMKPj4+Pj4+Pj4gaW50ZXJmYWNlLCB0aGUgZXh0ZW5zaW9uIGNhbiBiZSBkaXNhYmxlZCBieSBm YWlsaW5nIHRoZSBTQkkgY2FsbC4gU3NucG0KPj4+Pj4+Pj4gY2Fubm90IGJlIGRpc2FibGVkIHdp dGhvdXQgaW50ZXJjZXB0aW5nIHdyaXRlcyB0byB0aGUgc2VudmNmZyBDU1IuCj4+Pj4+Pj4+Cj4+ Pj4+Pj4+IFNpZ25lZC1vZmYtYnk6IFNhbXVlbCBIb2xsYW5kIDxzYW11ZWwuaG9sbGFuZEBzaWZp dmUuY29tPgo+Pj4+Pj4+PiAtLS0KPj4+Pj4+Pj4KPj4+Pj4+Pj4gKG5vIGNoYW5nZXMgc2luY2Ug djIpCj4+Pj4+Pj4+Cj4+Pj4+Pj4+IENoYW5nZXMgaW4gdjI6Cj4+Pj4+Pj4+ICAtIE5ldyBwYXRj aCBmb3IgdjIKPj4+Pj4+Pj4KPj4+Pj4+Pj4gIGFyY2gvcmlzY3YvaW5jbHVkZS91YXBpL2FzbS9r dm0uaCB8IDIgKysKPj4+Pj4+Pj4gIGFyY2gvcmlzY3Yva3ZtL3ZjcHVfb25lcmVnLmMgICAgICB8 IDMgKysrCj4+Pj4+Pj4+ICAyIGZpbGVzIGNoYW5nZWQsIDUgaW5zZXJ0aW9ucygrKQo+Pj4+Pj4+ Pgo+Pj4+Pj4+PiBkaWZmIC0tZ2l0IGEvYXJjaC9yaXNjdi9pbmNsdWRlL3VhcGkvYXNtL2t2bS5o IGIvYXJjaC9yaXNjdi9pbmNsdWRlL3VhcGkvYXNtL2t2bS5oCj4+Pj4+Pj4+IGluZGV4IGU5N2Ri MzI5NjQ1Ni4uNGYyNDIwMTM3NmIxIDEwMDY0NAo+Pj4+Pj4+PiAtLS0gYS9hcmNoL3Jpc2N2L2lu Y2x1ZGUvdWFwaS9hc20va3ZtLmgKPj4+Pj4+Pj4gKysrIGIvYXJjaC9yaXNjdi9pbmNsdWRlL3Vh cGkvYXNtL2t2bS5oCj4+Pj4+Pj4+IEBAIC0xNzUsNiArMTc1LDggQEAgZW51bSBLVk1fUklTQ1Zf SVNBX0VYVF9JRCB7Cj4+Pj4+Pj4+ICAgICAgICAgS1ZNX1JJU0NWX0lTQV9FWFRfWkNGLAo+Pj4+ Pj4+PiAgICAgICAgIEtWTV9SSVNDVl9JU0FfRVhUX1pDTU9QLAo+Pj4+Pj4+PiAgICAgICAgIEtW TV9SSVNDVl9JU0FfRVhUX1pBV1JTLAo+Pj4+Pj4+PiArICAgICAgIEtWTV9SSVNDVl9JU0FfRVhU X1NNTlBNLAo+Pj4+Pj4+PiArICAgICAgIEtWTV9SSVNDVl9JU0FfRVhUX1NTTlBNLAo+Pj4+Pj4+ PiAgICAgICAgIEtWTV9SSVNDVl9JU0FfRVhUX01BWCwKPj4+Pj4+Pj4gIH07Cj4+Pj4+Pj4+Cj4+ Pj4+Pj4+IGRpZmYgLS1naXQgYS9hcmNoL3Jpc2N2L2t2bS92Y3B1X29uZXJlZy5jIGIvYXJjaC9y aXNjdi9rdm0vdmNwdV9vbmVyZWcuYwo+Pj4+Pj4+PiBpbmRleCBiMzE5YzRjMTNjNTQuLjZmODMz ZWMyMzQ0YSAxMDA2NDQKPj4+Pj4+Pj4gLS0tIGEvYXJjaC9yaXNjdi9rdm0vdmNwdV9vbmVyZWcu Ywo+Pj4+Pj4+PiArKysgYi9hcmNoL3Jpc2N2L2t2bS92Y3B1X29uZXJlZy5jCj4+Pj4+Pj4+IEBA IC0zNCw5ICszNCwxMSBAQCBzdGF0aWMgY29uc3QgdW5zaWduZWQgbG9uZyBrdm1faXNhX2V4dF9h cnJbXSA9IHsKPj4+Pj4+Pj4gICAgICAgICBbS1ZNX1JJU0NWX0lTQV9FWFRfTV0gPSBSSVNDVl9J U0FfRVhUX20sCj4+Pj4+Pj4+ICAgICAgICAgW0tWTV9SSVNDVl9JU0FfRVhUX1ZdID0gUklTQ1Zf SVNBX0VYVF92LAo+Pj4+Pj4+PiAgICAgICAgIC8qIE11bHRpIGxldHRlciBleHRlbnNpb25zIChh bHBoYWJldGljYWxseSBzb3J0ZWQpICovCj4+Pj4+Pj4+ICsgICAgICAgW0tWTV9SSVNDVl9JU0Ff RVhUX1NNTlBNXSA9IFJJU0NWX0lTQV9FWFRfU1NOUE0sCj4+Pj4+Pj4KPj4+Pj4+PiBXaHkgbm90 IHVzZSBLVk1fSVNBX0VYVF9BUlIoKSBtYWNybyBoZXJlID8KPj4+Pj4+Cj4+Pj4+PiBCZWNhdXNl IHRoZSBleHRlbnNpb24gbmFtZSBpbiB0aGUgaG9zdCBkb2VzIG5vdCBtYXRjaCB0aGUgZXh0ZW5z aW9uIG5hbWUgaW4gdGhlCj4+Pj4+PiBndWVzdC4gUG9pbnRlciBtYXNraW5nIGZvciBIUyBtb2Rl IGlzIHByb3ZpZGVkIGJ5IFNtbnBtLiBQb2ludGVyIG1hc2tpbmcgZm9yIFZTCj4+Pj4+PiBtb2Rl IGlzIHByb3ZpZGVkIGJ5IFNzbnBtIGF0IHRoZSBoYXJkd2FyZSBsZXZlbCwgYnV0IHRoaXMgbmVl ZHMgdG8gYXBwZWFyIHRvIHRoZQo+Pj4+Pj4gZ3Vlc3QgYXMgaWYgU21ucG0gd2FzIGltcGxlbWVu dGVkLCBzaW5jZSB0aGUgZ3Vlc3QgdGhpbmtzIGl0IGlzIHJ1bm5pbmcgb24gYmFyZQo+Pj4+Pj4g bWV0YWwuCj4+Pj4+Cj4+Pj4+IE9rYXksIG1ha2VzIHNlbnNlLgo+Pj4+Pgo+Pj4+Pj4KPj4+Pj4+ Pj4gICAgICAgICBLVk1fSVNBX0VYVF9BUlIoU01TVEFURUVOKSwKPj4+Pj4+Pj4gICAgICAgICBL Vk1fSVNBX0VYVF9BUlIoU1NBSUEpLAo+Pj4+Pj4+PiAgICAgICAgIEtWTV9JU0FfRVhUX0FSUihT U0NPRlBNRiksCj4+Pj4+Pj4+ICsgICAgICAgS1ZNX0lTQV9FWFRfQVJSKFNTTlBNKSwKPj4+Pj4+ Pj4gICAgICAgICBLVk1fSVNBX0VYVF9BUlIoU1NUQyksCj4+Pj4+Pj4+ICAgICAgICAgS1ZNX0lT QV9FWFRfQVJSKFNWSU5WQUwpLAo+Pj4+Pj4+PiAgICAgICAgIEtWTV9JU0FfRVhUX0FSUihTVk5B UE9UKSwKPj4+Pj4+Pj4gQEAgLTEyOSw2ICsxMzEsNyBAQCBzdGF0aWMgYm9vbCBrdm1fcmlzY3Zf dmNwdV9pc2FfZGlzYWJsZV9hbGxvd2VkKHVuc2lnbmVkIGxvbmcgZXh0KQo+Pj4+Pj4+PiAgICAg ICAgIGNhc2UgS1ZNX1JJU0NWX0lTQV9FWFRfTToKPj4+Pj4+Pj4gICAgICAgICAvKiBUaGVyZSBp cyBub3QgYXJjaGl0ZWN0dXJhbCBjb25maWcgYml0IHRvIGRpc2FibGUgc3Njb2ZwbWYgY29tcGxl dGVseSAqLwo+Pj4+Pj4+PiAgICAgICAgIGNhc2UgS1ZNX1JJU0NWX0lTQV9FWFRfU1NDT0ZQTUY6 Cj4+Pj4+Pj4+ICsgICAgICAgY2FzZSBLVk1fUklTQ1ZfSVNBX0VYVF9TU05QTToKPj4+Pj4+Pgo+ Pj4+Pj4+IFdoeSBub3QgYWRkIEtWTV9SSVNDVl9JU0FfRVhUX1NNTlBNIGhlcmUgPwo+Pj4+Pj4+ Cj4+Pj4+Pj4gRGlzYWJsaW5nIFNtbnBtIGZyb20gS1ZNIHVzZXIgc3BhY2UgaXMgdmVyeSBkaWZm ZXJlbnQgZnJvbQo+Pj4+Pj4+IGRpc2FibGluZyBTbW5wbSBmcm9tIEd1ZXN0IHVzaW5nIFNCSSBG V0ZUIGV4dGVuc2lvbi4KPj4+Pj4+Cj4+Pj4+PiBVbnRpbCBhIHN1Y2Nlc3NmdWwgU0JJIEZXRlQg Y2FsbCB0byBLVk0gdG8gZW5hYmxlIHBvaW50ZXIgbWFza2luZyBmb3IgVlMgbW9kZSwKPj4+Pj4+ IHRoZSBleGlzdGVuY2Ugb2YgU21ucG0gaGFzIG5vIHZpc2libGUgZWZmZWN0IG9uIHRoZSBndWVz dC4gU28gZmFpbGluZyB0aGUgU0JJCj4+Pj4+PiBjYWxsIGlzIHN1ZmZpY2llbnQgdG8gcHJldGVu ZCB0aGF0IHRoZSBoYXJkd2FyZSBkb2VzIG5vdCBzdXBwb3J0IFNtbnBtLgo+Pj4+Pj4KPj4+Pj4+ PiBUaGUgS1ZNIHVzZXIgc3BhY2Ugc2hvdWxkIGFsd2F5cyBhZGQgU21ucG0gaW4gdGhlCj4+Pj4+ Pj4gR3Vlc3QgSVNBIHN0cmluZyB3aGVuZXZlciB0aGUgSG9zdCBJU0Egc3RyaW5nIGhhcyBpdC4K Pj4+Pj4+Cj4+Pj4+PiBJIGRpc2FncmVlLiBBbGxvd2luZyB1c2Vyc3BhY2UgdG8gZGlzYWJsZSBl eHRlbnNpb25zIGlzIHVzZWZ1bCBmb3IgdGVzdGluZyBhbmQKPj4+Pj4+IHRvIHN1cHBvcnQgbWln cmF0aW9uIHRvIGhvc3RzIHdoaWNoIGRvIG5vdCBzdXBwb3J0IHRob3NlIGV4dGVuc2lvbnMuIFNv IEkgd291bGQKPj4+Pj4+IG9ubHkgYWRkIGV4dGVuc2lvbnMgdG8gdGhpcyBsaXN0IGlmIHRoZXJl IGlzIG5vIHBvc3NpYmxlIHdheSB0byBkaXNhYmxlIHRoZW0uCj4+Pj4+Cj4+Pj4+IEkgYW0gbm90 IHNheWluZyB0byBkaXNhbGxvdyBLVk0gdXNlciBzcGFjZSBkaXNhYmxpbmcgU21ucG0uCj4+Pj4K Pj4+PiBUaGVuIEknbSBjb25mdXNlZC4gVGhpcyBpcyB0aGUgInJldHVybiBmYWxzZTsiIHN3aXRj aCBjYXNlIGluc2lkZQo+Pj4+IGt2bV9yaXNjdl92Y3B1X2lzYV9kaXNhYmxlX2FsbG93ZWQoKS4g SWYgSSBhZGQgS1ZNX1JJU0NWX0lTQV9FWFRfU01OUE0gaGVyZSwKPj4+PiB0aGVuICh1bmxlc3Mg SSBhbSBtaXNyZWFkaW5nIHRoZSBjb2RlKSBJIGFtIGRpc2FsbG93aW5nIEtWTSB1c2Vyc3BhY2Ug ZnJvbQo+Pj4+IGRpc2FibGluZyBTbW5wbSBpbiB0aGUgZ3Vlc3QgKGkuZS4gcHJldmVudGluZyBL Vk0gdXNlcnNwYWNlIGZyb20gcmVtb3ZpbmcgU21ucG0KPj4+PiBmcm9tIHRoZSBndWVzdCBJU0Eg c3RyaW5nKS4gSWYgdGhhdCBpcyBub3QgZGVzaXJlZCwgdGhlbiB3aHkgZG8geW91IHN1Z2dlc3Qg SQo+Pj4+IGFkZCBLVk1fUklTQ1ZfSVNBX0VYVF9TTU5QTSBoZXJlPwo+Pj4KPj4+IFllcywgYWRk aW5nIEtWTV9SSVNDVl9JU0FfRVhUX1NNTlBNIGhlcmUgbWVhbnMgS1ZNCj4+PiB1c2VyIHNwYWNl IGNhbid0IGRpc2FibGUgaXQgdXNpbmcgT05FX1JFRyBpbnRlcmZhY2UgYnV0IEtWTSB1c2VyCj4+ PiBzcGFjZSBjYW4gY2VydGFpbmx5IG5vdCBhZGQgaXQgaW4gdGhlIEd1ZXN0IElTQSBzdHJpbmcu Cj4+Cj4+IElzIHRoZXJlIGEgcHJvYmxlbSB3aXRoIGFsbG93aW5nIEtWTSB1c2Vyc3BhY2UgdG8g ZGlzYWJsZSB0aGUgSVNBIGV4dGVuc2lvbiB3aXRoCj4+IHRoZSBPTkVfUkVHIGludGVyZmFjZT8K Pj4KPj4gSWYgS1ZNIHVzZXJzcGFjZSByZW1vdmVzIFNtbnBtIGZyb20gdGhlIElTQSBzdHJpbmcg d2l0aG91dCB0aGUgaG9zdCBrZXJuZWwncwo+PiBrbm93bGVkZ2UsIHRoYXQgZG9lc24ndCBhY3R1 YWxseSBwcmV2ZW50IHRoZSBndWVzdCBmcm9tIHN1Y2Nlc3NmdWxseSBjYWxsaW5nCj4+IHNiaV9m d2Z0X3NldChQT0lOVEVSX01BU0tJTkdfUE1MRU4sIC4uLiksIHNvIGl0IGRvZXNuJ3QgZ3VhcmFu dGVlIHRoYXQgdGhlIFZNCj4+IGNhbiBiZSBtaWdyYXRlZCB0byBhIGhvc3Qgd2l0aG91dCBwb2lu dGVyIG1hc2tpbmcgc3VwcG9ydC4gU28gdGhlIE9ORV9SRUcKPj4gaW50ZXJmYWNlIHN0aWxsIGhh cyB2YWx1ZS4gKEFuZCB0aGF0J3MgbXkgYW5zd2VyIHRvIHlvdXIgb3JpZ2luYWwgcXVlc3Rpb24g IldoeQo+PiBub3QgYWRkIEtWTV9SSVNDVl9JU0FfRVhUX1NNTlBNIGhlcmUgPyIpCj4gCj4gQ3Vy cmVudGx5LCBkaXNhYmxpbmcgS1ZNX1JJU0NWX0lTQV9FWFRfU01OUE0gdmlhIE9ORV9SRUcKPiB3 aWxsIG9ubHkgY2xlYXIgdGhlIGNvcnJlc3BvbmRpbmcgYml0IGluIFZDUFUgaXNhIGJpdG1hcC4g QmFzaWNhbGx5LCB0aGUKPiBLVk0gdXNlciBzcGFjZSBkaXNhYmxpbmcgS1ZNX1JJU0NWX0lTQV9F WFRfU01OUE0gZm9yIEd1ZXN0Cj4gY2hhbmdlcyBub3RoaW5nIGZvciB0aGUgR3Vlc3QvVk0uCj4g Cj4gT24gb3RoZXIgaGFuZCwgZGlzYWJsaW5nIEtWTV9SSVNDVl9JU0FfRVhUX1NWUEJNVCB2aWEK PiBPTkVfUkVHIHdpbGwgbm90IG9ubHkgY2xlYXIgaXQgZnJvbSBWQ1BVIGlzYSBiaXRtYXAgYnV0 IGFsc28KPiBkaXNhYmxlIFN2cG1idCBmcm9tIGhlbnZjZmcgQ1NSIGZvciB0aGUgR3Vlc3QvVk0u Cj4gCj4gSW4gb3RoZXIgd29yZHMsIGlmIGRpc2FibGluZyBhbiBJU0EgZXh0ZW5zaW9uIGlzIGFs bG93ZWQgYnkgdGhlCj4ga3ZtX3Jpc2N2X3ZjcHVfaXNhX2Rpc2FibGVfYWxsb3dlZCgpIHRoZW4g dGhlIEd1ZXN0L1ZNIG11c3QKPiBzZWUgYSBkaWZmZXJlbnQgYmVoYXZpb3VyIHdoZW4gdGhlIElT QSBleHRlbnNpb24gaXMgZGlzYWJsZWQgYnkKPiBLVk0gdXNlciBzcGFjZS4KPiAKPj4KPj4+Pj4g VGhlIHByZXNlbmNlIG9mIFNtbnBtIGluIElTQSBvbmx5IG1lYW5zIHRoYXQgaXQgaXMgcHJlc2Vu dCBpbiBIVwo+Pj4+PiBidXQgaXQgbmVlZHMgdG8gYmUgZXhwbGljaXRseSBjb25maWd1cmVkL2Vu YWJsZWQgdXNpbmcgU0JJIEZXRlQuCj4+Pj4+Cj4+Pj4+IEtWTSB1c2VyIHNwYWNlIGNhbiBjZXJ0 YWlubHkgZGlzYWJsZSBleHRlbnNpb25zIGJ5IG5vdCBhZGRpbmcgaXQgdG8KPj4+Pj4gSVNBIHN0 cmluZyBiYXNlZCBvbiB0aGUgS1ZNVE9PTC9RRU1VLUtWTSBjb21tYW5kIGxpbmUgb3B0aW9uLgo+ Pj4+PiBBZGRpdGlvbmFsbHksIHdoZW4gU0JJIEZXRlQgaXMgYWRkZWQgdG8gS1ZNIFJJU0MtVi4g SXQgd2lsbCBoYXZlIGl0cwo+Pj4+PiBvd24gd2F5IHRvIGV4cGxpY2l0bHkgZGlzYWJsZSBmaXJt d2FyZSBmZWF0dXJlcyBmcm9tIEtWTSB1c2VyIHNwYWNlLgo+Pj4+Cj4+Pj4gSSB0aGluayB3ZSBh Z3JlZSBvbiB0aGlzLCBidXQgeW91ciBleHBsYW5hdGlvbiBoZXJlIGFwcGVhcnMgdG8gY29uZmxp Y3Qgd2l0aAo+Pj4+IHlvdXIgc3VnZ2VzdGVkIGNvZGUgY2hhbmdlLiBBcG9sb2dpZXMgaWYgSSdt IG1pc3Npbmcgc29tZXRoaW5nLgo+Pj4KPj4+IEkgdGhpbmsgdGhlIGNvbmZ1c2lvbiBpcyBhYm91 dCB3aGF0IGRvZXMgaXQgbWVhbiB3aGVuIFNtbnBtIGlzIHByZXNlbnQKPj4+IGluIHRoZSBJU0Eg c3RyaW5nLiBXZSBoYXZlIHR3byBhcHByb2FjaGVzOgo+Pj4KPj4+IDEpIFByZXNlbmNlIG9mIFNt bnBtIGluIElTQSBzdHJpbmcgb25seSBtZWFucyBpdCBpcyBwcmVzZW50IGluIEhXIGJ1dAo+Pj4g ICAgIHNheXMgbm90aGluZyBhYm91dCBpdHMgZW5hYmxlL2Rpc2FibGUgc3RhdGUuIFRvIGNvbmZp Z3VyZS9lbmFibGUKPj4+ICAgICBTbW5wbSwgdGhlIHN1cGVydmlzb3IgbXVzdCB1c2UgU0JJIEZX RlQuCj4+Pgo+Pj4gMikgUHJlc2VuY2Ugb2YgU21ucG0gaW4gSVNBIHN0cmluZyBtZWFucyBpdCBp cyBwcmVzZW50IGluIEhXIGFuZAo+Pj4gICAgIGVuYWJsZWQgYXQgYm9vdC10aW1lLiBUbyByZS1j b25maWd1cmUvZGlzYWJsZSBTbW5wbSwgdGhlIHN1cGVydmlzb3IKPj4+ICAgICBtdXN0IHVzZSBT QkkgRldGVC4KPj4+Cj4+PiBJIGFtIHN1Z2dlc3RpbmcgYXBwcm9hY2ggIzEgYnV0IEkgYW0gZ3Vl c3NpbmcgeW91IGFyZSBsZWFuaW5nIHRvd2FyZHMKPj4+IGFwcHJvYWNoICMyID8KPj4+Cj4+PiBG b3IgYXBwcm9hY2ggIzIsIGFkZGl0aW9uYWwgaGVuY2ZnLlBNTSBjb25maWd1cmF0aW9uIGlzIHJl cXVpcmVkIGluCj4+PiB0aGlzIHBhdGNoIGJhc2VkIG9uIHRoZSBzdGF0ZSBvZiBLVk1fUklTQ1Zf SVNBX0VYVF9TTU5QTS4KPj4KPj4gTm8sIEkgYW0gZGVmaW5pdGVseSBzdWdnZXN0aW5nIG9ubHkg YXBwcm9hY2ggIzEuIE15IHByb3Bvc2FsIGZvciBhZGRpbmcgcG9pbnRlcgo+PiBtYXNraW5nIHRv IHRoZSBTQkkgRldGVCBleHRlbnNpb25bMV0gc3BlY2lmaWVzIHRoZSBmZWF0dXJlIGFzIGRpc2Fi bGVkIGJ5Cj4+IGRlZmF1bHQsIGFuZCB0aGlzIHdvdWxkIGFwcGx5IGJvdGggaW5zaWRlIGFuZCBv dXNpZGUgYSBWTS4KPj4KPj4gQnV0IEkgYW0gYWxzbyBzdWdnZXN0aW5nIHRoYXQgdGhlIE9ORV9S RUcgaW50ZXJmYWNlIGlzIGEgdXNlZnVsIHdheSB0bwo+PiBjb21wbGV0ZWx5IGhpZGUgdGhlIGV4 dGVuc2lvbiBmcm9tIHRoZSBndWVzdCwgbGlrZSB3ZSBkbyBmb3Igb3RoZXIgZXh0ZW5zaW9ucwo+ PiBzdWNoIGFzIFN2cGJtdC4gVGhlIG9ubHkgZGlmZmVyZW5jZSBiZXR3ZWVuIHNvbWV0aGluZyBs aWtlIFN2cGJtdCBhbmQgU21ucG0gaXMKPj4gdGhhdCBpbnN0ZWFkIG9mIGNsZWFyaW5nIGEgYml0 IGluIGhlbnZjZmcgdG8gaGlkZSB0aGUgZXh0ZW5zaW9uIGZyb20gdGhlIGd1ZXN0LAo+PiB3ZSBy ZWplY3QgY2FsbHMgdG8gc2JpX2Z3ZnRfc2V0KFBPSU5URVJfTUFTS0lOR19QTUxFTiwgLi4uKSB3 aGVuIHRoZSBJU0EKPj4gZXh0ZW5zaW9uIGlzIGhpZGRlbiBmcm9tIHRoZSBndWVzdC4KPiAKPiBJ IHRoaW5rIHdlIGFyZSBjb252ZXJnaW5nIHRvd2FyZHMgdGhlIHNhbWUgdGhpbmcuCj4gCj4gSG93 IGFib3V0IHRoaXMgPwo+IAo+IEZvciB0aGlzIHNlcmllcywgbGV0cyBhZGQgS1ZNX1JJU0NWX0lT QV9FWFRfU01OUE0gdG8KPiBrdm1fcmlzY3ZfdmNwdV9pc2FfZGlzYWJsZV9hbGxvd2VkKCkgc28g dGhhdCBmb3IgdGhlIHRpbWUKPiBiZWluZyBLVk0gdXNlciBzcGFjZSBjYW4ndCBkaXNhYmxlIFNt bnBtLgo+IAo+IEluIHRoZSBmdXR1cmUsIGEgc2VwYXJhdGUgc2VyaWVzIHdoaWNoIGFkZHMgU0JJ IEZXRlQgdG8KPiBLVk0gUklTQy1WIHdpbGwgcmVtb3ZlIEtWTV9SSVNDVl9JU0FfRVhUX1NNTlBN Cj4gZnJvbSB0aGUga3ZtX3Jpc2N2X3ZjcHVfaXNhX2Rpc2FibGVfYWxsb3dlZCgpIGJlY2F1c2UK PiBkaXNhYmxpbmcgU21ucG0gZnJvbSBLVk0gdXNlciBzcGFjZSB3b3VsZCBtZWFuIHRoYXQKPiB0 aGUgUE9JTlRFUl9NQVNLSU5HX1BNTEVOIGZpcm13YXJlIGZlYXR1cmUgaXMKPiBub3QgYXZhaWxh YmxlIHRvIHRoZSBHdWVzdC9WTS4KPiAKPiBUaGlzIG1lYW5zIGluIHRoZSBmdXR1cmUgKGFmdGVy IFNCSSBGV0ZUIGlzIGltcGxlbWVudGVkIGluCj4gS1ZNIFJJU0MtViksIEd1ZXN0IHdpdGggU21u cG0gZGlzYWJsZWQgY2FuIGJlIG1pZ3JhdGVkCj4gdG8gYSBob3N0IHdpdGhvdXQgcG9pbnRlciBt YXNraW5nLgoKT0ssIHRoYXQgaXMgYSByZWFzb25hYmxlIGNvbXByb21pc2UuIEknbGwgZG8gdGhh dCBmb3IgdjUuCgpSZWdhcmRzLApTYW11ZWwKCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNjdiBtYWlsaW5nIGxpc3QKbGludXgtcmlzY3ZA bGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xp c3RpbmZvL2xpbnV4LXJpc2N2Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FFB211185 for ; Sat, 14 Sep 2024 02:52:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726282333; cv=none; b=Oxsr0/GH7FSpwj9Zr4/b1AMUi7DhRJF6vzXAgbl/FWH5qMy+oih1XaHDGiroN1+Gh9EdfEUwXB8/s8nH/yeT9/yoHczLEG56sC9y7PVe4cIb0tfvgSl73DpkuAPdJbTDeiXPd4h76ZCdfKbHDWgrggyIgzN5NQNwheBm/sCZ7Co= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726282333; c=relaxed/simple; bh=1Gr5pJH9pMrNB4Z2ewdxf3KB031tjCazkwMcq3AHEks=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=VRkzwidtA+Hv4sVgqKkq9ZgCISVM4hC7qMFBe+/KwMKQaH22gJbpLf1IJ1H2zi5mJPu0JXOhvMQNYWEg4MqIzJPagm7P+2nrkkj1wTvtepKDg870mOifv1suU0EvkrejsMvczW7H4mLuBsbKxXybuoJWZS8/o0HCR1dbV9P7/YM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com; spf=pass smtp.mailfrom=sifive.com; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b=O7/l4jf6; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="O7/l4jf6" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-82ce1cd2026so98118339f.3 for ; Fri, 13 Sep 2024 19:52:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1726282330; x=1726887130; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=PcgocvC07KwdgYkvDV/qMfScM5+LqmPejCCEAA1uMJI=; b=O7/l4jf6WeHi3/qunfph2cyQj94ZPUkAtgNgcMFvigFIzUCCXx9Q8kuFw1Tz52eyiu a04cS3mJ/OeBbfxXrrPIUX7kyjTrTFROQOjL9Llx9Wta/I3kBvM0W4pksc/l+4T4N7nt 3yGauRwmLcCCWBNBkyJH0tWFKKylkcrNMFOVeJKzC2KY6xiSsMXMNLZ5SkKGHMDqkFT6 /wVMqYk09lis4/r83VD3lDgyWWnNUimTOGCHGnefVRETqy1K2lUDtZzOEzc5nBBZRDy2 jzVbx480MMm8rlllzyUMm9bcQvnCJlEhjwZ5AV9SzG2FbQCdxcdRIjcP5SyaypbVHhq/ VevQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726282330; x=1726887130; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PcgocvC07KwdgYkvDV/qMfScM5+LqmPejCCEAA1uMJI=; b=I2LSRuahYmA0N3EeAkzjisDHX+BKCnxKqp0788p6zLdkLsDYIrfRkiJ2YOiq0VWRRC fdD6wT2Ubi1+8WpCHUSdsousXqji0NIn0FQEZlc1NBMa095BoNs7sb7vB7SiiqMdOAg6 AJkM7BqGdu6e5UVXYBDuSXK3ZPC2IXzWi7DYOaBI91VQf+MHfOBJAL+coblKsTgPaz7j 2dnv7xncJduhUAUj4aotBUKKwU+PkaidrNWdZ5zVsaiQG1N0Z/KWzMnguCwhxal6oE/C 8nIkYmfbbkOsMqa0KdFd5zQYm+azSAigTHrKcqU2HEGwgQiO7rctSYKscBFsUSsv8zmr ce1A== X-Forwarded-Encrypted: i=1; AJvYcCW0XNWpEuMNVmirCa4lzJQ0AxfVTPXoMcsxNU6HxXjMcJczEY4PxXjNrDytPQ/9Jvbdw/kNiWLDnj34@vger.kernel.org X-Gm-Message-State: AOJu0Yyd8UvOU/a19MMUecOIOqHHXM2KJUKTEghHLnQ7Mh2NBmCK2Ro+ aLIJ1i+1MzX4pNyE1QZzDMtvkiLpKqOxb7VW5kNavO6srbS9kK0WWArfpBmUH/8= X-Google-Smtp-Source: AGHT+IEaD4v+a/RKnKmId+r+Iz4R86fNRYyVeVXynZnVMTjpM3fXydjExpGYdtf8Iye3kIi/wgX9bQ== X-Received: by 2002:a05:6e02:5ad:b0:3a0:8dae:8b06 with SMTP id e9e14a558f8ab-3a08dae8c99mr24837465ab.9.1726282330175; Fri, 13 Sep 2024 19:52:10 -0700 (PDT) Received: from [100.64.0.1] ([147.124.94.167]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a092dfe1d8sm1609685ab.13.2024.09.13.19.52.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 13 Sep 2024 19:52:09 -0700 (PDT) Message-ID: <8e474b14-e963-4d3e-8240-37f662e7bd8a@sifive.com> Date: Fri, 13 Sep 2024 21:52:07 -0500 Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 09/10] RISC-V: KVM: Allow Smnpm and Ssnpm extensions for guests To: Anup Patel Cc: Anup Patel , Palmer Dabbelt , linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, Catalin Marinas , linux-kernel@vger.kernel.org, Conor Dooley , kasan-dev@googlegroups.com, Atish Patra , Evgenii Stepanov , Krzysztof Kozlowski , Rob Herring , "Kirill A . Shutemov" , kvm-riscv@lists.infradead.org References: <20240829010151.2813377-1-samuel.holland@sifive.com> <20240829010151.2813377-10-samuel.holland@sifive.com> <20ab0fa2-d5dd-446d-9fff-a3ef82e8db35@sifive.com> <4c010cb1-b57c-427e-a241-1dd3ab15f2ce@sifive.com> From: Samuel Holland Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi Anup, On 2024-09-05 12:18 AM, Anup Patel wrote: > On Wed, Sep 4, 2024 at 9:25 PM Samuel Holland wrote: >> >> On 2024-09-04 10:20 AM, Anup Patel wrote: >>> On Wed, Sep 4, 2024 at 8:27 PM Samuel Holland wrote: >>>> >>>> Hi Anup, >>>> >>>> On 2024-09-04 9:45 AM, Anup Patel wrote: >>>>> On Wed, Sep 4, 2024 at 8:01 PM Samuel Holland wrote: >>>>>> On 2024-09-04 7:17 AM, Anup Patel wrote: >>>>>>> On Thu, Aug 29, 2024 at 6:32 AM Samuel Holland >>>>>>> wrote: >>>>>>>> >>>>>>>> The interface for controlling pointer masking in VS-mode is henvcfg.PMM, >>>>>>>> which is part of the Ssnpm extension, even though pointer masking in >>>>>>>> HS-mode is provided by the Smnpm extension. As a result, emulating Smnpm >>>>>>>> in the guest requires (only) Ssnpm on the host. >>>>>>>> >>>>>>>> Since the guest configures Smnpm through the SBI Firmware Features >>>>>>>> interface, the extension can be disabled by failing the SBI call. Ssnpm >>>>>>>> cannot be disabled without intercepting writes to the senvcfg CSR. >>>>>>>> >>>>>>>> Signed-off-by: Samuel Holland >>>>>>>> --- >>>>>>>> >>>>>>>> (no changes since v2) >>>>>>>> >>>>>>>> Changes in v2: >>>>>>>> - New patch for v2 >>>>>>>> >>>>>>>> arch/riscv/include/uapi/asm/kvm.h | 2 ++ >>>>>>>> arch/riscv/kvm/vcpu_onereg.c | 3 +++ >>>>>>>> 2 files changed, 5 insertions(+) >>>>>>>> >>>>>>>> diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> index e97db3296456..4f24201376b1 100644 >>>>>>>> --- a/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> +++ b/arch/riscv/include/uapi/asm/kvm.h >>>>>>>> @@ -175,6 +175,8 @@ enum KVM_RISCV_ISA_EXT_ID { >>>>>>>> KVM_RISCV_ISA_EXT_ZCF, >>>>>>>> KVM_RISCV_ISA_EXT_ZCMOP, >>>>>>>> KVM_RISCV_ISA_EXT_ZAWRS, >>>>>>>> + KVM_RISCV_ISA_EXT_SMNPM, >>>>>>>> + KVM_RISCV_ISA_EXT_SSNPM, >>>>>>>> KVM_RISCV_ISA_EXT_MAX, >>>>>>>> }; >>>>>>>> >>>>>>>> diff --git a/arch/riscv/kvm/vcpu_onereg.c b/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> index b319c4c13c54..6f833ec2344a 100644 >>>>>>>> --- a/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> +++ b/arch/riscv/kvm/vcpu_onereg.c >>>>>>>> @@ -34,9 +34,11 @@ static const unsigned long kvm_isa_ext_arr[] = { >>>>>>>> [KVM_RISCV_ISA_EXT_M] = RISCV_ISA_EXT_m, >>>>>>>> [KVM_RISCV_ISA_EXT_V] = RISCV_ISA_EXT_v, >>>>>>>> /* Multi letter extensions (alphabetically sorted) */ >>>>>>>> + [KVM_RISCV_ISA_EXT_SMNPM] = RISCV_ISA_EXT_SSNPM, >>>>>>> >>>>>>> Why not use KVM_ISA_EXT_ARR() macro here ? >>>>>> >>>>>> Because the extension name in the host does not match the extension name in the >>>>>> guest. Pointer masking for HS mode is provided by Smnpm. Pointer masking for VS >>>>>> mode is provided by Ssnpm at the hardware level, but this needs to appear to the >>>>>> guest as if Smnpm was implemented, since the guest thinks it is running on bare >>>>>> metal. >>>>> >>>>> Okay, makes sense. >>>>> >>>>>> >>>>>>>> KVM_ISA_EXT_ARR(SMSTATEEN), >>>>>>>> KVM_ISA_EXT_ARR(SSAIA), >>>>>>>> KVM_ISA_EXT_ARR(SSCOFPMF), >>>>>>>> + KVM_ISA_EXT_ARR(SSNPM), >>>>>>>> KVM_ISA_EXT_ARR(SSTC), >>>>>>>> KVM_ISA_EXT_ARR(SVINVAL), >>>>>>>> KVM_ISA_EXT_ARR(SVNAPOT), >>>>>>>> @@ -129,6 +131,7 @@ static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext) >>>>>>>> case KVM_RISCV_ISA_EXT_M: >>>>>>>> /* There is not architectural config bit to disable sscofpmf completely */ >>>>>>>> case KVM_RISCV_ISA_EXT_SSCOFPMF: >>>>>>>> + case KVM_RISCV_ISA_EXT_SSNPM: >>>>>>> >>>>>>> Why not add KVM_RISCV_ISA_EXT_SMNPM here ? >>>>>>> >>>>>>> Disabling Smnpm from KVM user space is very different from >>>>>>> disabling Smnpm from Guest using SBI FWFT extension. >>>>>> >>>>>> Until a successful SBI FWFT call to KVM to enable pointer masking for VS mode, >>>>>> the existence of Smnpm has no visible effect on the guest. So failing the SBI >>>>>> call is sufficient to pretend that the hardware does not support Smnpm. >>>>>> >>>>>>> The KVM user space should always add Smnpm in the >>>>>>> Guest ISA string whenever the Host ISA string has it. >>>>>> >>>>>> I disagree. Allowing userspace to disable extensions is useful for testing and >>>>>> to support migration to hosts which do not support those extensions. So I would >>>>>> only add extensions to this list if there is no possible way to disable them. >>>>> >>>>> I am not saying to disallow KVM user space disabling Smnpm. >>>> >>>> Then I'm confused. This is the "return false;" switch case inside >>>> kvm_riscv_vcpu_isa_disable_allowed(). If I add KVM_RISCV_ISA_EXT_SMNPM here, >>>> then (unless I am misreading the code) I am disallowing KVM userspace from >>>> disabling Smnpm in the guest (i.e. preventing KVM userspace from removing Smnpm >>>> from the guest ISA string). If that is not desired, then why do you suggest I >>>> add KVM_RISCV_ISA_EXT_SMNPM here? >>> >>> Yes, adding KVM_RISCV_ISA_EXT_SMNPM here means KVM >>> user space can't disable it using ONE_REG interface but KVM user >>> space can certainly not add it in the Guest ISA string. >> >> Is there a problem with allowing KVM userspace to disable the ISA extension with >> the ONE_REG interface? >> >> If KVM userspace removes Smnpm from the ISA string without the host kernel's >> knowledge, that doesn't actually prevent the guest from successfully calling >> sbi_fwft_set(POINTER_MASKING_PMLEN, ...), so it doesn't guarantee that the VM >> can be migrated to a host without pointer masking support. So the ONE_REG >> interface still has value. (And that's my answer to your original question "Why >> not add KVM_RISCV_ISA_EXT_SMNPM here ?") > > Currently, disabling KVM_RISCV_ISA_EXT_SMNPM via ONE_REG > will only clear the corresponding bit in VCPU isa bitmap. Basically, the > KVM user space disabling KVM_RISCV_ISA_EXT_SMNPM for Guest > changes nothing for the Guest/VM. > > On other hand, disabling KVM_RISCV_ISA_EXT_SVPBMT via > ONE_REG will not only clear it from VCPU isa bitmap but also > disable Svpmbt from henvcfg CSR for the Guest/VM. > > In other words, if disabling an ISA extension is allowed by the > kvm_riscv_vcpu_isa_disable_allowed() then the Guest/VM must > see a different behaviour when the ISA extension is disabled by > KVM user space. > >> >>>>> The presence of Smnpm in ISA only means that it is present in HW >>>>> but it needs to be explicitly configured/enabled using SBI FWFT. >>>>> >>>>> KVM user space can certainly disable extensions by not adding it to >>>>> ISA string based on the KVMTOOL/QEMU-KVM command line option. >>>>> Additionally, when SBI FWFT is added to KVM RISC-V. It will have its >>>>> own way to explicitly disable firmware features from KVM user space. >>>> >>>> I think we agree on this, but your explanation here appears to conflict with >>>> your suggested code change. Apologies if I'm missing something. >>> >>> I think the confusion is about what does it mean when Smnpm is present >>> in the ISA string. We have two approaches: >>> >>> 1) Presence of Smnpm in ISA string only means it is present in HW but >>> says nothing about its enable/disable state. To configure/enable >>> Smnpm, the supervisor must use SBI FWFT. >>> >>> 2) Presence of Smnpm in ISA string means it is present in HW and >>> enabled at boot-time. To re-configure/disable Smnpm, the supervisor >>> must use SBI FWFT. >>> >>> I am suggesting approach #1 but I am guessing you are leaning towards >>> approach #2 ? >>> >>> For approach #2, additional hencfg.PMM configuration is required in >>> this patch based on the state of KVM_RISCV_ISA_EXT_SMNPM. >> >> No, I am definitely suggesting only approach #1. My proposal for adding pointer >> masking to the SBI FWFT extension[1] specifies the feature as disabled by >> default, and this would apply both inside and ouside a VM. >> >> But I am also suggesting that the ONE_REG interface is a useful way to >> completely hide the extension from the guest, like we do for other extensions >> such as Svpbmt. The only difference between something like Svpbmt and Smnpm is >> that instead of clearing a bit in henvcfg to hide the extension from the guest, >> we reject calls to sbi_fwft_set(POINTER_MASKING_PMLEN, ...) when the ISA >> extension is hidden from the guest. > > I think we are converging towards the same thing. > > How about this ? > > For this series, lets add KVM_RISCV_ISA_EXT_SMNPM to > kvm_riscv_vcpu_isa_disable_allowed() so that for the time > being KVM user space can't disable Smnpm. > > In the future, a separate series which adds SBI FWFT to > KVM RISC-V will remove KVM_RISCV_ISA_EXT_SMNPM > from the kvm_riscv_vcpu_isa_disable_allowed() because > disabling Smnpm from KVM user space would mean that > the POINTER_MASKING_PMLEN firmware feature is > not available to the Guest/VM. > > This means in the future (after SBI FWFT is implemented in > KVM RISC-V), Guest with Smnpm disabled can be migrated > to a host without pointer masking. OK, that is a reasonable compromise. I'll do that for v5. Regards, Samuel