From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Thu, 5 Oct 2023 20:21:15 -0700 Subject: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes In-Reply-To: References: Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Thu, Oct 05, 2023, Fuad Tabba wrote: > Hi Sean, > > On Tue, Oct 3, 2023 at 9:51?PM Sean Christopherson wrote: > > > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/SHARED, > > > just a way of tracking in the host kernel of what is shared (as opposed to > > > the hypervisor, which already has the knowledge). The solution could simply > > > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its own > > > tracking of the status of the guest pages, and only selects KVM_PRIVATE_MEM. > > > > At the risk of overstepping my bounds, I think that effectively giving the guest > > full control over what is shared vs. private is a mistake. It more or less locks > > pKVM into a single model, and even within that model, dealing with errors and/or > > misbehaving guests becomes unnecessarily problematic. > > > > Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the userspace > > side of pKVM could simply "reflect" all conversion hypercalls, and terminate the > > VM on errors. But the cost is very minimal, e.g. a single extra ioctl() per > > converion, and the upside is that pKVM won't be stuck if a use case comes along > > that wants to go beyond "all conversion requests either immediately succeed or > > terminate the guest". > > Now that I understand the purpose of KVM_SET_MEMORY_ATTRIBUTES, I > agree. However, pKVM needs to track at the host kernel (i.e., EL1) > whether guest memory is shared or private. Why does EL1 need it's own view/opinion? E.g. is it to avoid a accessing data that is still private according to EL2 (on behalf of the guest)? Assuming that's the case, why can't EL1 wait until it gets confirmation from EL2 that the data is fully shared before doing whatever it is that needs to be done? Ah, is the problem that whether or not .mmap() is allowed keys off of the state of the memory attributes? If that's so, then yeah, an internal flag in attributes is probably the way to go. It doesn't need to be a "host kernel private" flag though, e.g. an IN_FLUX flag to capture that the attributes aren't fully realized might be more intuitive for readers, and might have utility for other attributes in the future too. > One approach would be to add another flag to the attributes that > tracks the host kernel view. The way KVM_SET_MEMORY_ATTRIBUTES is > implemented now, userspace can zero it, so in that case, that > operation would need to be masked to avoid that. > > Another approach would be to have a pKVM-specific xarray (or similar) > to do the tracking, but since there is a structure that's already > doing something similar (i.e.,the attributes array), it seems like it > would be unnecessary overhead. > > Do you have any ideas or preferences? > > Cheers, > /fuad From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E16D629 for ; Fri, 6 Oct 2023 03:21:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DhJkizQL" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d81486a0382so2310047276.0 for ; Thu, 05 Oct 2023 20:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696562477; x=1697167277; darn=lists.linux.dev; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=DhJkizQLWq69l2K4n1HKXzUs6uHDR7d7SmUeh4eoycjwlA10+sRVaPf2jwae9FB1Wc MLgTyVdSVZk8JULYEqABuHTsIm4vRTBIKdYYhyoXm9+yKnjGE5IX3vOwfvYnCsSNeF7Z 3O7DTICWJUajhOuQSVuT/RAbBkKJt5915sMIOxCK3aEETIohUo2eN4ouJPTB+RhX/sWH dZMpSNo8XtL3yRgXIp8R3VDNcE1Z+qoudSk5NoDGBVbCSmqT+KGc3VtuFsZ+n1fP0xYe Wsd9/6xPfj8PerlV864131T1fdZPnR/0vOn84M9uv/ABVR1VQMV8TvPCSaWjLy7zWgWa 2XLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696562477; x=1697167277; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=RtXxcEiOg21p7V1nfvPRWWKQB228HP9v+vlDQz3chCXPSxLaR4p6ScHgUxTC2eZcYw 7HchZTgUTEN5zVebuClNwobdY78dDBxc/ExRZslHB9MnGfZbgziEcQCY6bgeNdz8lGzi 4V9C4JOF81auXPZJDUlmBFbjPhQXS5NUU2UukZSd17qYEB1l0qixPuzr2tuVh3XT6DZx 1bXY2iTt6qDypqpi5hbo/8aAuFVsVgDQ9Ny6JlcrYpchdDa2/X9EPDKWgp4jM/auxVek bfCIR3GQoQBg1HUAlYzFOopRErKSeY++Idwg2thBj0cNzMho1lSHCqtWaUea/b8jtko2 uDiQ== X-Gm-Message-State: AOJu0Yygs0kRHIGyV/T6R2b0MHDDcEB/jiIeOMNVOKL9gN55SzAgMw9U t7ewpR2UQ46Qvww+fqXMA+zd3BEZt5Y= X-Google-Smtp-Source: AGHT+IHbgJX1dul2brLKqtxy3zNH4Wq8osuz6z+RCm0MrNAV7Qoo4mqOPY/NVTlIcosnwAkmq4EHKfjNEDc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:ae41:0:b0:d78:a78:6fc7 with SMTP id g1-20020a25ae41000000b00d780a786fc7mr101904ybe.6.1696562477545; Thu, 05 Oct 2023 20:21:17 -0700 (PDT) Date: Thu, 5 Oct 2023 20:21:15 -0700 In-Reply-To: Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , KVM , "moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)" , KVMARM , LinuxMIPS , linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, open list , Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Thu, Oct 05, 2023, Fuad Tabba wrote: > Hi Sean, >=20 > On Tue, Oct 3, 2023 at 9:51=E2=80=AFPM Sean Christopherson wrote: > > > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/S= HARED, > > > just a way of tracking in the host kernel of what is shared (as oppos= ed to > > > the hypervisor, which already has the knowledge). The solution could = simply > > > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its o= wn > > > tracking of the status of the guest pages, and only selects KVM_PRIVA= TE_MEM. > > > > At the risk of overstepping my bounds, I think that effectively giving = the guest > > full control over what is shared vs. private is a mistake. It more or = less locks > > pKVM into a single model, and even within that model, dealing with erro= rs and/or > > misbehaving guests becomes unnecessarily problematic. > > > > Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the= userspace > > side of pKVM could simply "reflect" all conversion hypercalls, and term= inate the > > VM on errors. But the cost is very minimal, e.g. a single extra ioctl(= ) per > > converion, and the upside is that pKVM won't be stuck if a use case com= es along > > that wants to go beyond "all conversion requests either immediately suc= ceed or > > terminate the guest". >=20 > Now that I understand the purpose of KVM_SET_MEMORY_ATTRIBUTES, I > agree. However, pKVM needs to track at the host kernel (i.e., EL1) > whether guest memory is shared or private. Why does EL1 need it's own view/opinion? E.g. is it to avoid a accessing d= ata that is still private according to EL2 (on behalf of the guest)? Assuming that's the case, why can't EL1 wait until it gets confirmation fro= m EL2 that the data is fully shared before doing whatever it is that needs to be = done? Ah, is the problem that whether or not .mmap() is allowed keys off of the s= tate of the memory attributes? If that's so, then yeah, an internal flag in att= ributes is probably the way to go. It doesn't need to be a "host kernel private" f= lag though, e.g. an IN_FLUX flag to capture that the attributes aren't fully re= alized might be more intuitive for readers, and might have utility for other attri= butes in the future too. > One approach would be to add another flag to the attributes that > tracks the host kernel view. The way KVM_SET_MEMORY_ATTRIBUTES is > implemented now, userspace can zero it, so in that case, that > operation would need to be masked to avoid that. >=20 > Another approach would be to have a pKVM-specific xarray (or similar) > to do the tracking, but since there is a structure that's already > doing something similar (i.e.,the attributes array), it seems like it > would be unnecessary overhead. >=20 > Do you have any ideas or preferences? >=20 > Cheers, > /fuad From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0A92E92FDD for ; Fri, 6 Oct 2023 03:21:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=18UzxoC3oImJWSR2sgOO4Eszi6mlAUXhoKGJ78DbYt0=; b=ABnYyDVx8MU54+AXujQssjMAoa Spz7SdXV1sbpSWZWjoluRyq+yAwm29N2dipHsukhBBeJ8NBnDkoybFHuD2eymZ6KSHZM2tsMELqaP d1vjVkmrZsRWfJLUU1OZSTGWxtb1XKWM71WuoWM30Lo1aUB9dEIBQ2dtscHX8yJnnAXbHt2R1mwPt Nvlxq9U5KOlstwADSSjVYm6oxokzWHKNL5gnrHFiafqIJmXKtbTyg98b4sJzG9APwBI+bBUhvdpKu r38S9G5V9W+7VZjtWESKGvpfqvZfwAvntSuvtSaRzq800eHEqmnENF3Bb2A0LfHtwzd1dFAtja5z9 7M0mrlkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qobP2-004qfN-2N; Fri, 06 Oct 2023 03:21:24 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qobOy-004qcR-1I for linux-riscv@lists.infradead.org; Fri, 06 Oct 2023 03:21:23 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59b59e1ac70so25289957b3.1 for ; Thu, 05 Oct 2023 20:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696562477; x=1697167277; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=j6dJ8gWiLBN0fbJWWKgxrbiJQQrcqNRfU9Kf7QJrDBF2P4juuFkTyecxJ3J3O3Y/NQ jUxCZ1YmB1+EG4XeC3NnB7EMJ9Ig/kXZ004yJPAdhMSOOzoGMw0moOF/srrAzo4gn0K9 p0nK74Qe7HHq/+THNlgRROvlkiET72a9tckmrGzSgvrSDfFu3MhZSrmJ3l5elP35WQ2g MYZ27Uox8kjcM+4r2PikdvLqcuN2u1/bTDP0bguUD6eDs5tSp7Svyo8daIIJzGkT6p4d yvL2uYjT0FZH+8IXK4X1+lm6X3waGsP8swA/AL/kFHq1gzXfFC2YH3v7TZpMlKUx/mrt HxSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696562477; x=1697167277; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=HKJrW7YNl4p2s8nPIoMmy3b7Y+MUv1UzXg9JayjTcUEF+w+YIGM0cCXLqwTzBbCSeX 8gzKboXTJFe98BoNBc60G0mvQ30tGei6A7KA8umXvvSvClyi+k40TzSWBVeCGtJ/Uj1Q sE6YwYO5/XRa7bPF8K6OLTfAIAGLzv7ueQCPnyMqGiYLMM/0sfB8x0TaBQwACx0pO1hF cQrO/sIhk7w9dN2YvpuV3a1O2YnUaj8IM9k0O7ATgVtronnnSqrErtlZd7z0hgxEnMb8 P0vVa8p75LBY4b8J09WagffwOlMwwz8iK36j1RidIWzyUv95LEsplNAlHar0pfVFQ/i8 zrJw== X-Gm-Message-State: AOJu0YzHhEN2MCAgnQ+NVYoYUGqUKHvt8mZbDTDiZmeVeah6ekRKJDqV nzCUm8BrpTNJ6RsiNZGX5lPwU6X3BIY= X-Google-Smtp-Source: AGHT+IHbgJX1dul2brLKqtxy3zNH4Wq8osuz6z+RCm0MrNAV7Qoo4mqOPY/NVTlIcosnwAkmq4EHKfjNEDc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:ae41:0:b0:d78:a78:6fc7 with SMTP id g1-20020a25ae41000000b00d780a786fc7mr101904ybe.6.1696562477545; Thu, 05 Oct 2023 20:21:17 -0700 (PDT) Date: Thu, 5 Oct 2023 20:21:15 -0700 In-Reply-To: Mime-Version: 1.0 References: Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , KVM , "moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)" , KVMARM , LinuxMIPS , linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, open list , Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231005_202120_441322_9C2CDAC2 X-CRM114-Status: GOOD ( 27.58 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVGh1LCBPY3QgMDUsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gSGkgU2VhbiwKPiAKPiBP biBUdWUsIE9jdCAzLCAyMDIzIGF0IDk6NTHigK9QTSBTZWFuIENocmlzdG9waGVyc29uIDxzZWFu amNAZ29vZ2xlLmNvbT4gd3JvdGU6Cj4gPiA+IExpa2UgSSBzYWlkLCBwS1ZNIGRvZXNuJ3QgbmVl ZCBhIHVzZXJzcGFjZSBBQkkgZm9yIG1hbmFnaW5nIFBSSVZBVEUvU0hBUkVELAo+ID4gPiBqdXN0 IGEgd2F5IG9mIHRyYWNraW5nIGluIHRoZSBob3N0IGtlcm5lbCBvZiB3aGF0IGlzIHNoYXJlZCAo YXMgb3Bwb3NlZCB0bwo+ID4gPiB0aGUgaHlwZXJ2aXNvciwgd2hpY2ggYWxyZWFkeSBoYXMgdGhl IGtub3dsZWRnZSkuIFRoZSBzb2x1dGlvbiBjb3VsZCBzaW1wbHkKPiA+ID4gYmUgdGhhdCBwS1ZN IGRvZXMgbm90IGVuYWJsZSBLVk1fR0VORVJJQ19NRU1PUllfQVRUUklCVVRFUywgaGFzIGl0cyBv d24KPiA+ID4gdHJhY2tpbmcgb2YgdGhlIHN0YXR1cyBvZiB0aGUgZ3Vlc3QgcGFnZXMsIGFuZCBv bmx5IHNlbGVjdHMgS1ZNX1BSSVZBVEVfTUVNLgo+ID4KPiA+IEF0IHRoZSByaXNrIG9mIG92ZXJz dGVwcGluZyBteSBib3VuZHMsIEkgdGhpbmsgdGhhdCBlZmZlY3RpdmVseSBnaXZpbmcgdGhlIGd1 ZXN0Cj4gPiBmdWxsIGNvbnRyb2wgb3ZlciB3aGF0IGlzIHNoYXJlZCB2cy4gcHJpdmF0ZSBpcyBh IG1pc3Rha2UuICBJdCBtb3JlIG9yIGxlc3MgbG9ja3MKPiA+IHBLVk0gaW50byBhIHNpbmdsZSBt b2RlbCwgYW5kIGV2ZW4gd2l0aGluIHRoYXQgbW9kZWwsIGRlYWxpbmcgd2l0aCBlcnJvcnMgYW5k L29yCj4gPiBtaXNiZWhhdmluZyBndWVzdHMgYmVjb21lcyB1bm5lY2Vzc2FyaWx5IHByb2JsZW1h dGljLgo+ID4KPiA+IFVzaW5nIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgbWF5IG5vdCBwcm92 aWRlIHZhbHVlICp0b2RheSosIGUuZy4gdGhlIHVzZXJzcGFjZQo+ID4gc2lkZSBvZiBwS1ZNIGNv dWxkIHNpbXBseSAicmVmbGVjdCIgYWxsIGNvbnZlcnNpb24gaHlwZXJjYWxscywgYW5kIHRlcm1p bmF0ZSB0aGUKPiA+IFZNIG9uIGVycm9ycy4gIEJ1dCB0aGUgY29zdCBpcyB2ZXJ5IG1pbmltYWws IGUuZy4gYSBzaW5nbGUgZXh0cmEgaW9jdGwoKSBwZXIKPiA+IGNvbnZlcmlvbiwgYW5kIHRoZSB1 cHNpZGUgaXMgdGhhdCBwS1ZNIHdvbid0IGJlIHN0dWNrIGlmIGEgdXNlIGNhc2UgY29tZXMgYWxv bmcKPiA+IHRoYXQgd2FudHMgdG8gZ28gYmV5b25kICJhbGwgY29udmVyc2lvbiByZXF1ZXN0cyBl aXRoZXIgaW1tZWRpYXRlbHkgc3VjY2VlZCBvcgo+ID4gdGVybWluYXRlIHRoZSBndWVzdCIuCj4g Cj4gTm93IHRoYXQgSSB1bmRlcnN0YW5kIHRoZSBwdXJwb3NlIG9mIEtWTV9TRVRfTUVNT1JZX0FU VFJJQlVURVMsIEkKPiBhZ3JlZS4gSG93ZXZlciwgcEtWTSBuZWVkcyB0byB0cmFjayBhdCB0aGUg aG9zdCBrZXJuZWwgKGkuZS4sIEVMMSkKPiB3aGV0aGVyIGd1ZXN0IG1lbW9yeSBpcyBzaGFyZWQg b3IgcHJpdmF0ZS4KCldoeSBkb2VzIEVMMSBuZWVkIGl0J3Mgb3duIHZpZXcvb3Bpbmlvbj8gIEUu Zy4gaXMgaXQgdG8gYXZvaWQgYSBhY2Nlc3NpbmcgZGF0YQp0aGF0IGlzIHN0aWxsIHByaXZhdGUg YWNjb3JkaW5nIHRvIEVMMiAob24gYmVoYWxmIG9mIHRoZSBndWVzdCk/CgpBc3N1bWluZyB0aGF0 J3MgdGhlIGNhc2UsIHdoeSBjYW4ndCBFTDEgd2FpdCB1bnRpbCBpdCBnZXRzIGNvbmZpcm1hdGlv biBmcm9tIEVMMgp0aGF0IHRoZSBkYXRhIGlzIGZ1bGx5IHNoYXJlZCBiZWZvcmUgZG9pbmcgd2hh dGV2ZXIgaXQgaXMgdGhhdCBuZWVkcyB0byBiZSBkb25lPwoKQWgsIGlzIHRoZSBwcm9ibGVtIHRo YXQgd2hldGhlciBvciBub3QgLm1tYXAoKSBpcyBhbGxvd2VkIGtleXMgb2ZmIG9mIHRoZSBzdGF0 ZQpvZiB0aGUgbWVtb3J5IGF0dHJpYnV0ZXM/ICBJZiB0aGF0J3Mgc28sIHRoZW4geWVhaCwgYW4g aW50ZXJuYWwgZmxhZyBpbiBhdHRyaWJ1dGVzCmlzIHByb2JhYmx5IHRoZSB3YXkgdG8gZ28uICBJ dCBkb2Vzbid0IG5lZWQgdG8gYmUgYSAiaG9zdCBrZXJuZWwgcHJpdmF0ZSIgZmxhZwp0aG91Z2gs IGUuZy4gYW4gSU5fRkxVWCBmbGFnIHRvIGNhcHR1cmUgdGhhdCB0aGUgYXR0cmlidXRlcyBhcmVu J3QgZnVsbHkgcmVhbGl6ZWQKbWlnaHQgYmUgbW9yZSBpbnR1aXRpdmUgZm9yIHJlYWRlcnMsIGFu ZCBtaWdodCBoYXZlIHV0aWxpdHkgZm9yIG90aGVyIGF0dHJpYnV0ZXMKaW4gdGhlIGZ1dHVyZSB0 b28uCgo+IE9uZSBhcHByb2FjaCB3b3VsZCBiZSB0byBhZGQgYW5vdGhlciBmbGFnIHRvIHRoZSBh dHRyaWJ1dGVzIHRoYXQKPiB0cmFja3MgdGhlIGhvc3Qga2VybmVsIHZpZXcuIFRoZSB3YXkgS1ZN X1NFVF9NRU1PUllfQVRUUklCVVRFUyBpcwo+IGltcGxlbWVudGVkIG5vdywgdXNlcnNwYWNlIGNh biB6ZXJvIGl0LCBzbyBpbiB0aGF0IGNhc2UsIHRoYXQKPiBvcGVyYXRpb24gd291bGQgbmVlZCB0 byBiZSBtYXNrZWQgdG8gYXZvaWQgdGhhdC4KPiAKPiBBbm90aGVyIGFwcHJvYWNoIHdvdWxkIGJl IHRvIGhhdmUgYSBwS1ZNLXNwZWNpZmljIHhhcnJheSAob3Igc2ltaWxhcikKPiB0byBkbyB0aGUg dHJhY2tpbmcsIGJ1dCBzaW5jZSB0aGVyZSBpcyBhIHN0cnVjdHVyZSB0aGF0J3MgYWxyZWFkeQo+ IGRvaW5nIHNvbWV0aGluZyBzaW1pbGFyIChpLmUuLHRoZSBhdHRyaWJ1dGVzIGFycmF5KSwgaXQg c2VlbXMgbGlrZSBpdAo+IHdvdWxkIGJlIHVubmVjZXNzYXJ5IG92ZXJoZWFkLgo+IAo+IERvIHlv dSBoYXZlIGFueSBpZGVhcyBvciBwcmVmZXJlbmNlcz8KPiAKPiBDaGVlcnMsCj4gL2Z1YWQKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2 IG1haWxpbmcgbGlzdApsaW51eC1yaXNjdkBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0 cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtcmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A73D4E92FDD for ; Fri, 6 Oct 2023 03:22:13 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=CR0iuFWP; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S1tyX2F0kz3dlT for ; Fri, 6 Oct 2023 14:22:12 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=CR0iuFWP; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--seanjc.bounces.google.com (client-ip=2607:f8b0:4864:20::1149; helo=mail-yw1-x1149.google.com; envelope-from=3lx0fzqykdnqi40d926ee6b4.2ecb8dknff2-34lb8iji.epb01i.eh6@flex--seanjc.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S1txZ0NP0z3c4R for ; Fri, 6 Oct 2023 14:21:20 +1100 (AEDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59f4f2a9ef0so25211827b3.2 for ; Thu, 05 Oct 2023 20:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696562477; x=1697167277; darn=lists.ozlabs.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=CR0iuFWPtUldTYzXYpsDwiZcEIwCFsXQAdu1rWCmlZMpPCsIq9qZXd3j5N9ursujEE j/WHbBAdDFKz0qNoU7SFliNniJWLr50ngRZ3wLIGjhukowgGY5UQH/sXt05mWf5EwifM 8hEu8JHS6dUWRc2xbfKClXiMKfF/DJzZhYcjn+8OwyFMOIJjuC/8pxc7Bhgp2cCqqWhQ g8Q6FJlp2Xy/KYE/SNEU/SiEix1xujgmBAE3PX38duFf8/1LjJt0WSNljd3Fu6I4tYTt Bjt0FqY4buYhKv829xKGdtCJ5xB7nej0PyuqxzV7x/DxLwJwkKgaslp6stGAOrSGBXUB 5lsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696562477; x=1697167277; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=XeXWO5F9VUHcm9UUL2yBNuR68E68ObBX2euL2rLqJDVvZgBKB6H+5QUnW+jQwmHl2X 7OJ7xqTnzTPjQxbupeB/ibwTZBO2w7xQ+9GsE0+xRTYDNeLD3wZGFRHuwxg5G1xRTw5a nMVUuHKMmWB63GSxeZI3gX+G2nJjRcfIPrPF7W3XuRkVpcg92bLSr8GIJeFtRynkMJmi PuUAWQHdvHnDGXiO7MYIXhfE/rTNThE0y3iN0gWGr09H+lRq+WQNBJhIBqdNSWj/hwkg tCQGnydyLYVNhJ14MzLUO1fDPthp8UXclubhUD93j0pFmDox1W+aX3NEBNigcGZ53EYX iyRg== X-Gm-Message-State: AOJu0YwejWLirUTNEVFYHJoygb4vdKypEdQbDebYC1rKqG0ilk4XK9Ja 0jEzSvFHwDM1QAl061cBdq3BHrNHqGc= X-Google-Smtp-Source: AGHT+IHbgJX1dul2brLKqtxy3zNH4Wq8osuz6z+RCm0MrNAV7Qoo4mqOPY/NVTlIcosnwAkmq4EHKfjNEDc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:ae41:0:b0:d78:a78:6fc7 with SMTP id g1-20020a25ae41000000b00d780a786fc7mr101904ybe.6.1696562477545; Thu, 05 Oct 2023 20:21:17 -0700 (PDT) Date: Thu, 5 Oct 2023 20:21:15 -0700 In-Reply-To: Mime-Version: 1.0 References: Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: KVM , David Hildenbrand , Yu Zhang , open list , linux-mm@kvack.org, Chao Peng , linux-riscv@lists.infradead.org, Isaku Yamahata , Paul Moore , Marc Zyngier , Huacai Chen , James Morris , "Matthew Wilcox \(Oracle\)" , Wang , Vlastimil Babka , Jarkko Sakkinen , "Serge E. Hallyn" , Maciej Szmigiero , Albert Ou , Michael Roth , Ackerley Tng , Paul Walmsley , KVMARM , "moderated list:ARM64 PORT \(AARCH64 ARCHITECTURE\)" , Isaku Yamahata , Quentin Perret , Liam Merwick , LinuxMIPS , Oliver Upton , linux-security-module@vger.kernel.org, Palmer Dabbelt , "Kirill A . Shutemov" , kvm-riscv@lists.infradead.org, Anup Patel , linux-fsdevel@vger.kernel.org, Paolo Bonzini , Andrew Morton , Vishal Annapurve , linuxppc-dev@lists.ozlabs.org, Xu Yilun , Anish Moorthy Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Oct 05, 2023, Fuad Tabba wrote: > Hi Sean, >=20 > On Tue, Oct 3, 2023 at 9:51=E2=80=AFPM Sean Christopherson wrote: > > > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/S= HARED, > > > just a way of tracking in the host kernel of what is shared (as oppos= ed to > > > the hypervisor, which already has the knowledge). The solution could = simply > > > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its o= wn > > > tracking of the status of the guest pages, and only selects KVM_PRIVA= TE_MEM. > > > > At the risk of overstepping my bounds, I think that effectively giving = the guest > > full control over what is shared vs. private is a mistake. It more or = less locks > > pKVM into a single model, and even within that model, dealing with erro= rs and/or > > misbehaving guests becomes unnecessarily problematic. > > > > Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the= userspace > > side of pKVM could simply "reflect" all conversion hypercalls, and term= inate the > > VM on errors. But the cost is very minimal, e.g. a single extra ioctl(= ) per > > converion, and the upside is that pKVM won't be stuck if a use case com= es along > > that wants to go beyond "all conversion requests either immediately suc= ceed or > > terminate the guest". >=20 > Now that I understand the purpose of KVM_SET_MEMORY_ATTRIBUTES, I > agree. However, pKVM needs to track at the host kernel (i.e., EL1) > whether guest memory is shared or private. Why does EL1 need it's own view/opinion? E.g. is it to avoid a accessing d= ata that is still private according to EL2 (on behalf of the guest)? Assuming that's the case, why can't EL1 wait until it gets confirmation fro= m EL2 that the data is fully shared before doing whatever it is that needs to be = done? Ah, is the problem that whether or not .mmap() is allowed keys off of the s= tate of the memory attributes? If that's so, then yeah, an internal flag in att= ributes is probably the way to go. It doesn't need to be a "host kernel private" f= lag though, e.g. an IN_FLUX flag to capture that the attributes aren't fully re= alized might be more intuitive for readers, and might have utility for other attri= butes in the future too. > One approach would be to add another flag to the attributes that > tracks the host kernel view. The way KVM_SET_MEMORY_ATTRIBUTES is > implemented now, userspace can zero it, so in that case, that > operation would need to be masked to avoid that. >=20 > Another approach would be to have a pKVM-specific xarray (or similar) > to do the tracking, but since there is a structure that's already > doing something similar (i.e.,the attributes array), it seems like it > would be unnecessary overhead. >=20 > Do you have any ideas or preferences? >=20 > Cheers, > /fuad From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA215E92FDB for ; Fri, 6 Oct 2023 03:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MeuEwuWZTtCgF/u0R6l6nq1qR1LIfxv4fBFkTwYi17w=; b=KeKmjw9pTMXPNvBJHHCXPoxFg+ xc+brSNX0WxgEd7yLcT5dcrwPqgG6IrTL0uKiF4tYYg9PbwJopXmyjW/i3dVibG2QkgIGg5mU29Ny 9/C2cg2Az7gCkepIWx08RUkPk9HXvO+AJFslHQzhn8i+kbXLoy0isARgEqvqqhIvdodiVFYXXkdcY YnIip5S9/MsKWi1fiP2vZvez1pena0eqaNfMVhCb4A/Nvjer97/Yi8I7b0pf+aEZhakWQVGQlH66q Nvc+4XBqCxNdwwiyWyDNVZyizXuWe6ltwOGPZfsLoxnTKdLJ3YkT//c+dbc++VMyd28UiHgHaE7SJ qFh01zSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qobP3-004qfZ-1I; Fri, 06 Oct 2023 03:21:25 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qobOy-004qcP-1J for linux-arm-kernel@lists.infradead.org; Fri, 06 Oct 2023 03:21:23 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d866d13c637so2279617276.3 for ; Thu, 05 Oct 2023 20:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696562477; x=1697167277; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=j6dJ8gWiLBN0fbJWWKgxrbiJQQrcqNRfU9Kf7QJrDBF2P4juuFkTyecxJ3J3O3Y/NQ jUxCZ1YmB1+EG4XeC3NnB7EMJ9Ig/kXZ004yJPAdhMSOOzoGMw0moOF/srrAzo4gn0K9 p0nK74Qe7HHq/+THNlgRROvlkiET72a9tckmrGzSgvrSDfFu3MhZSrmJ3l5elP35WQ2g MYZ27Uox8kjcM+4r2PikdvLqcuN2u1/bTDP0bguUD6eDs5tSp7Svyo8daIIJzGkT6p4d yvL2uYjT0FZH+8IXK4X1+lm6X3waGsP8swA/AL/kFHq1gzXfFC2YH3v7TZpMlKUx/mrt HxSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696562477; x=1697167277; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+GCfD3J/utVXt7fNFcXS7tndSxc76Lg25lhLquZNVGo=; b=Mz0nE1H0WOa51r+rsSqyz0wn0DQI5E3x0zkuBNEHU9uny70sp1uRyzKB4g4EBc8s9/ YiX0WvlORljbvKdCi04LDpr0hFyRz8Enw5Fr5tEAFrnnv/gZLFjsRTP7/0/7ZffnLQGC d24M9w2IDhpb3Nr1e41JSfKcjFR25irAmWVtcLG8bZgEKq1hSuqSEkN+HvOCx2BDWoWh /LA0UoT93BYvASzArN9orPNO9VrYx6PhESVl6xZChtcPHQhc1a6wHBIEMAKXdrBuoaHt mdFuxq5+sycIgP6PT2V0nOAwwzT5yPH0HGWPNt7tm+mKx3aYpeVAb9/NidGmPfQaBa2q 5N1w== X-Gm-Message-State: AOJu0YyrCjWUCoBaIN1G9z8Uh/uLYm6tQ+WGntwZHh5Xy7GUBbJ3ZxSQ g2MJdjNkk1pfNIfpFqm+11ip5QYGBkM= X-Google-Smtp-Source: AGHT+IHbgJX1dul2brLKqtxy3zNH4Wq8osuz6z+RCm0MrNAV7Qoo4mqOPY/NVTlIcosnwAkmq4EHKfjNEDc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:ae41:0:b0:d78:a78:6fc7 with SMTP id g1-20020a25ae41000000b00d780a786fc7mr101904ybe.6.1696562477545; Thu, 05 Oct 2023 20:21:17 -0700 (PDT) Date: Thu, 5 Oct 2023 20:21:15 -0700 In-Reply-To: Mime-Version: 1.0 References: Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , KVM , "moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)" , KVMARM , LinuxMIPS , linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, open list , Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231005_202120_444210_F6A87645 X-CRM114-Status: GOOD ( 29.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1LCBPY3QgMDUsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gSGkgU2VhbiwKPiAKPiBP biBUdWUsIE9jdCAzLCAyMDIzIGF0IDk6NTHigK9QTSBTZWFuIENocmlzdG9waGVyc29uIDxzZWFu amNAZ29vZ2xlLmNvbT4gd3JvdGU6Cj4gPiA+IExpa2UgSSBzYWlkLCBwS1ZNIGRvZXNuJ3QgbmVl ZCBhIHVzZXJzcGFjZSBBQkkgZm9yIG1hbmFnaW5nIFBSSVZBVEUvU0hBUkVELAo+ID4gPiBqdXN0 IGEgd2F5IG9mIHRyYWNraW5nIGluIHRoZSBob3N0IGtlcm5lbCBvZiB3aGF0IGlzIHNoYXJlZCAo YXMgb3Bwb3NlZCB0bwo+ID4gPiB0aGUgaHlwZXJ2aXNvciwgd2hpY2ggYWxyZWFkeSBoYXMgdGhl IGtub3dsZWRnZSkuIFRoZSBzb2x1dGlvbiBjb3VsZCBzaW1wbHkKPiA+ID4gYmUgdGhhdCBwS1ZN IGRvZXMgbm90IGVuYWJsZSBLVk1fR0VORVJJQ19NRU1PUllfQVRUUklCVVRFUywgaGFzIGl0cyBv d24KPiA+ID4gdHJhY2tpbmcgb2YgdGhlIHN0YXR1cyBvZiB0aGUgZ3Vlc3QgcGFnZXMsIGFuZCBv bmx5IHNlbGVjdHMgS1ZNX1BSSVZBVEVfTUVNLgo+ID4KPiA+IEF0IHRoZSByaXNrIG9mIG92ZXJz dGVwcGluZyBteSBib3VuZHMsIEkgdGhpbmsgdGhhdCBlZmZlY3RpdmVseSBnaXZpbmcgdGhlIGd1 ZXN0Cj4gPiBmdWxsIGNvbnRyb2wgb3ZlciB3aGF0IGlzIHNoYXJlZCB2cy4gcHJpdmF0ZSBpcyBh IG1pc3Rha2UuICBJdCBtb3JlIG9yIGxlc3MgbG9ja3MKPiA+IHBLVk0gaW50byBhIHNpbmdsZSBt b2RlbCwgYW5kIGV2ZW4gd2l0aGluIHRoYXQgbW9kZWwsIGRlYWxpbmcgd2l0aCBlcnJvcnMgYW5k L29yCj4gPiBtaXNiZWhhdmluZyBndWVzdHMgYmVjb21lcyB1bm5lY2Vzc2FyaWx5IHByb2JsZW1h dGljLgo+ID4KPiA+IFVzaW5nIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgbWF5IG5vdCBwcm92 aWRlIHZhbHVlICp0b2RheSosIGUuZy4gdGhlIHVzZXJzcGFjZQo+ID4gc2lkZSBvZiBwS1ZNIGNv dWxkIHNpbXBseSAicmVmbGVjdCIgYWxsIGNvbnZlcnNpb24gaHlwZXJjYWxscywgYW5kIHRlcm1p bmF0ZSB0aGUKPiA+IFZNIG9uIGVycm9ycy4gIEJ1dCB0aGUgY29zdCBpcyB2ZXJ5IG1pbmltYWws IGUuZy4gYSBzaW5nbGUgZXh0cmEgaW9jdGwoKSBwZXIKPiA+IGNvbnZlcmlvbiwgYW5kIHRoZSB1 cHNpZGUgaXMgdGhhdCBwS1ZNIHdvbid0IGJlIHN0dWNrIGlmIGEgdXNlIGNhc2UgY29tZXMgYWxv bmcKPiA+IHRoYXQgd2FudHMgdG8gZ28gYmV5b25kICJhbGwgY29udmVyc2lvbiByZXF1ZXN0cyBl aXRoZXIgaW1tZWRpYXRlbHkgc3VjY2VlZCBvcgo+ID4gdGVybWluYXRlIHRoZSBndWVzdCIuCj4g Cj4gTm93IHRoYXQgSSB1bmRlcnN0YW5kIHRoZSBwdXJwb3NlIG9mIEtWTV9TRVRfTUVNT1JZX0FU VFJJQlVURVMsIEkKPiBhZ3JlZS4gSG93ZXZlciwgcEtWTSBuZWVkcyB0byB0cmFjayBhdCB0aGUg aG9zdCBrZXJuZWwgKGkuZS4sIEVMMSkKPiB3aGV0aGVyIGd1ZXN0IG1lbW9yeSBpcyBzaGFyZWQg b3IgcHJpdmF0ZS4KCldoeSBkb2VzIEVMMSBuZWVkIGl0J3Mgb3duIHZpZXcvb3Bpbmlvbj8gIEUu Zy4gaXMgaXQgdG8gYXZvaWQgYSBhY2Nlc3NpbmcgZGF0YQp0aGF0IGlzIHN0aWxsIHByaXZhdGUg YWNjb3JkaW5nIHRvIEVMMiAob24gYmVoYWxmIG9mIHRoZSBndWVzdCk/CgpBc3N1bWluZyB0aGF0 J3MgdGhlIGNhc2UsIHdoeSBjYW4ndCBFTDEgd2FpdCB1bnRpbCBpdCBnZXRzIGNvbmZpcm1hdGlv biBmcm9tIEVMMgp0aGF0IHRoZSBkYXRhIGlzIGZ1bGx5IHNoYXJlZCBiZWZvcmUgZG9pbmcgd2hh dGV2ZXIgaXQgaXMgdGhhdCBuZWVkcyB0byBiZSBkb25lPwoKQWgsIGlzIHRoZSBwcm9ibGVtIHRo YXQgd2hldGhlciBvciBub3QgLm1tYXAoKSBpcyBhbGxvd2VkIGtleXMgb2ZmIG9mIHRoZSBzdGF0 ZQpvZiB0aGUgbWVtb3J5IGF0dHJpYnV0ZXM/ICBJZiB0aGF0J3Mgc28sIHRoZW4geWVhaCwgYW4g aW50ZXJuYWwgZmxhZyBpbiBhdHRyaWJ1dGVzCmlzIHByb2JhYmx5IHRoZSB3YXkgdG8gZ28uICBJ dCBkb2Vzbid0IG5lZWQgdG8gYmUgYSAiaG9zdCBrZXJuZWwgcHJpdmF0ZSIgZmxhZwp0aG91Z2gs IGUuZy4gYW4gSU5fRkxVWCBmbGFnIHRvIGNhcHR1cmUgdGhhdCB0aGUgYXR0cmlidXRlcyBhcmVu J3QgZnVsbHkgcmVhbGl6ZWQKbWlnaHQgYmUgbW9yZSBpbnR1aXRpdmUgZm9yIHJlYWRlcnMsIGFu ZCBtaWdodCBoYXZlIHV0aWxpdHkgZm9yIG90aGVyIGF0dHJpYnV0ZXMKaW4gdGhlIGZ1dHVyZSB0 b28uCgo+IE9uZSBhcHByb2FjaCB3b3VsZCBiZSB0byBhZGQgYW5vdGhlciBmbGFnIHRvIHRoZSBh dHRyaWJ1dGVzIHRoYXQKPiB0cmFja3MgdGhlIGhvc3Qga2VybmVsIHZpZXcuIFRoZSB3YXkgS1ZN X1NFVF9NRU1PUllfQVRUUklCVVRFUyBpcwo+IGltcGxlbWVudGVkIG5vdywgdXNlcnNwYWNlIGNh biB6ZXJvIGl0LCBzbyBpbiB0aGF0IGNhc2UsIHRoYXQKPiBvcGVyYXRpb24gd291bGQgbmVlZCB0 byBiZSBtYXNrZWQgdG8gYXZvaWQgdGhhdC4KPiAKPiBBbm90aGVyIGFwcHJvYWNoIHdvdWxkIGJl IHRvIGhhdmUgYSBwS1ZNLXNwZWNpZmljIHhhcnJheSAob3Igc2ltaWxhcikKPiB0byBkbyB0aGUg dHJhY2tpbmcsIGJ1dCBzaW5jZSB0aGVyZSBpcyBhIHN0cnVjdHVyZSB0aGF0J3MgYWxyZWFkeQo+ IGRvaW5nIHNvbWV0aGluZyBzaW1pbGFyIChpLmUuLHRoZSBhdHRyaWJ1dGVzIGFycmF5KSwgaXQg c2VlbXMgbGlrZSBpdAo+IHdvdWxkIGJlIHVubmVjZXNzYXJ5IG92ZXJoZWFkLgo+IAo+IERvIHlv dSBoYXZlIGFueSBpZGVhcyBvciBwcmVmZXJlbmNlcz8KPiAKPiBDaGVlcnMsCj4gL2Z1YWQKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1r ZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpo dHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJu ZWwK