From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Matlack Date: Tue, 31 Oct 2023 11:24:07 -0700 Subject: [PATCH v13 16/35] KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory In-Reply-To: <20231027182217.3615211-17-seanjc@google.com> References: <20231027182217.3615211-1-seanjc@google.com> <20231027182217.3615211-17-seanjc@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On 2023-10-27 11:21 AM, Sean Christopherson wrote: > Introduce an ioctl(), KVM_CREATE_GUEST_MEMFD, to allow creating file-based > memory that is tied to a specific KVM virtual machine and whose primary > purpose is to serve guest memory. > > A guest-first memory subsystem allows for optimizations and enhancements > that are kludgy or outright infeasible to implement/support in a generic > memory subsystem. With guest_memfd, guest protections and mapping sizes > are fully decoupled from host userspace mappings. E.g. KVM currently > doesn't support mapping memory as writable in the guest without it also > being writable in host userspace, as KVM's ABI uses VMA protections to > define the allow guest protection. Userspace can fudge this by > establishing two mappings, a writable mapping for the guest and readable > one for itself, but that?s suboptimal on multiple fronts. > > Similarly, KVM currently requires the guest mapping size to be a strict > subset of the host userspace mapping size, e.g. KVM doesn?t support > creating a 1GiB guest mapping unless userspace also has a 1GiB guest > mapping. Decoupling the mappings sizes would allow userspace to precisely > map only what is needed without impacting guest performance, e.g. to > harden against unintentional accesses to guest memory. > > Decoupling guest and userspace mappings may also allow for a cleaner > alternative to high-granularity mappings for HugeTLB, which has reached a > bit of an impasse and is unlikely to ever be merged. > > A guest-first memory subsystem also provides clearer line of sight to > things like a dedicated memory pool (for slice-of-hardware VMs) and > elimination of "struct page" (for offload setups where userspace _never_ > needs to mmap() guest memory). All of these use-cases involve using guest_memfd for shared pages, but this entire series sets up KVM to only use guest_memfd for private pages. For example, the per-page attributes are a property of a KVM VM, not the underlying guest_memfd. So that implies we will need separate guest_memfds for private and shared pages. But a given memslot can have a mix of private and shared pages. So that implies a memslot will need to support 2 guest_memfds? But the UAPI only allows 1 and uses the HVA for shared mappings. My initial reaction after reading through this series is that the per-page private/shared should be a property of the guest_memfd, not the VM. Maybe it would even be cleaner in the long-run to make all memory attributes a property of the guest_memfd. That way we can scope the support to only guest_memfds and not have to worry about making per-page attributes work with "legacy" HVA-based memslots. Maybe can you sketch out how you see this proposal being extensible to using guest_memfd for shared mappings? From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7A02208CF for ; Tue, 31 Oct 2023 18:24:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KHl7teRk" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1cc1e1e74beso42095965ad.1 for ; Tue, 31 Oct 2023 11:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698776653; x=1699381453; darn=lists.linux.dev; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=KHl7teRkrXxTRwQxqPqAd0/PXp3HEl/k1muE3aGQ+1lQLF+vfKXCa31LKf5iDISOIZ ctWGH+1eUbIQ0Y+zwJYo6aGJFAo+o9H35Z/U35qK2vbvINjwArPxljaIkkZ95KroWuIw aYLhxpDBO6b1y79JA22wKlpRVglGF5QHbc5unF5HZ/s7/I1cohnxIkvzkC2EgKktRL3o 3ubsJ7VjkZwLDAzBP1hFLE/h5g+HFOvdSRwlh3jIq1OeGzWP/RTYH+gQj2HRjton+r7m HK0H82Jd4Ub6eJhtupCJtnvCFvmA853RGLi7GUqlBQeCrU09yn448rw/+hvsK3an3lUm LsyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698776653; x=1699381453; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=xJwafjpY89arCIiUvV5AfZoeMb/JSDnW/kpU36a0VpK3Unfb4nH/V7z9r3jPgvDpyM Oi90MwUH0jTaiFB8unp0Iwel/jrLHAgPKjJLEC/EAnk0cOq71v5MCxO15TA0SMjroeOO KZv/on2AxTwG5XTcGgOPOL+Xul4xmceOL5APnkU9oWAMpriQgi31Duj15Hb4oR5fnSM3 OV54Aw4oNIyW1EA/4ZucwwkfHL+9l93v4IrH2tAM4kdKsrvhdcdZwACX3wempjXNdUve 2glIfVaXkhu8/ABCqXStOiTUFHccwsAqxas1a3cMXvv5fJscOW/1qcngQutoWoWIlCiN H+9A== X-Gm-Message-State: AOJu0YyUk/jbfIUtLRIaw2h56O9vwg/fgkYkEEodMPzjAUUTK+NI2iLa y+Bb2DinbfX6Kp9lVJqqy+4EyQ== X-Google-Smtp-Source: AGHT+IFxkiDdZGGfHsdazjXYZcfW0cI1Huw/6T3VggKB9s0vUh/sA7Jolgtag5oCOtU2pTLnT1L5FA== X-Received: by 2002:a17:902:ecca:b0:1cc:54b5:b4fa with SMTP id a10-20020a170902ecca00b001cc54b5b4famr5681643plh.18.1698776652868; Tue, 31 Oct 2023 11:24:12 -0700 (PDT) Received: from google.com (175.199.125.34.bc.googleusercontent.com. [34.125.199.175]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c20d00b001c5dea67c26sm1620267pll.233.2023.10.31.11.24.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 11:24:11 -0700 (PDT) Date: Tue, 31 Oct 2023 11:24:07 -0700 From: David Matlack To: Sean Christopherson Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Xiaoyao Li , Xu Yilun , Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , =?iso-8859-1?Q?Micka=EBl_Sala=FCn?= , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Subject: Re: [PATCH v13 16/35] KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory Message-ID: References: <20231027182217.3615211-1-seanjc@google.com> <20231027182217.3615211-17-seanjc@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20231027182217.3615211-17-seanjc@google.com> On 2023-10-27 11:21 AM, Sean Christopherson wrote: > Introduce an ioctl(), KVM_CREATE_GUEST_MEMFD, to allow creating file-based > memory that is tied to a specific KVM virtual machine and whose primary > purpose is to serve guest memory. > > A guest-first memory subsystem allows for optimizations and enhancements > that are kludgy or outright infeasible to implement/support in a generic > memory subsystem. With guest_memfd, guest protections and mapping sizes > are fully decoupled from host userspace mappings. E.g. KVM currently > doesn't support mapping memory as writable in the guest without it also > being writable in host userspace, as KVM's ABI uses VMA protections to > define the allow guest protection. Userspace can fudge this by > establishing two mappings, a writable mapping for the guest and readable > one for itself, but that’s suboptimal on multiple fronts. > > Similarly, KVM currently requires the guest mapping size to be a strict > subset of the host userspace mapping size, e.g. KVM doesn’t support > creating a 1GiB guest mapping unless userspace also has a 1GiB guest > mapping. Decoupling the mappings sizes would allow userspace to precisely > map only what is needed without impacting guest performance, e.g. to > harden against unintentional accesses to guest memory. > > Decoupling guest and userspace mappings may also allow for a cleaner > alternative to high-granularity mappings for HugeTLB, which has reached a > bit of an impasse and is unlikely to ever be merged. > > A guest-first memory subsystem also provides clearer line of sight to > things like a dedicated memory pool (for slice-of-hardware VMs) and > elimination of "struct page" (for offload setups where userspace _never_ > needs to mmap() guest memory). All of these use-cases involve using guest_memfd for shared pages, but this entire series sets up KVM to only use guest_memfd for private pages. For example, the per-page attributes are a property of a KVM VM, not the underlying guest_memfd. So that implies we will need separate guest_memfds for private and shared pages. But a given memslot can have a mix of private and shared pages. So that implies a memslot will need to support 2 guest_memfds? But the UAPI only allows 1 and uses the HVA for shared mappings. My initial reaction after reading through this series is that the per-page private/shared should be a property of the guest_memfd, not the VM. Maybe it would even be cleaner in the long-run to make all memory attributes a property of the guest_memfd. That way we can scope the support to only guest_memfds and not have to worry about making per-page attributes work with "legacy" HVA-based memslots. Maybe can you sketch out how you see this proposal being extensible to using guest_memfd for shared mappings? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1622FC4332F for ; Tue, 31 Oct 2023 18:24:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xbe+fzLoPbw1ufok0/rM/Ms3l+Rx7+30I0OY5slv/Dg=; b=lFQV5HQdj68FYc t3hmhB2s0KgIRZxCo41BFjc1n2C0YqnncysVTuykZoTViih9FX/E9hBqkZhAkk+UymKu3IXz9X4yf rPzDFk8/T0X3I1JsACw4OgzPEd6n7LwYI4V7vf6q7QvOzz+fXMaCHOpS+lUGQTpIg3wUydvjljtcF K+hiIX9/LCX1shzqbWIf2Oiz3R6R+McwRCpY4P5gSu+wToBIBoqnLgJIgrZy/isVhKROqGGkhvnX7 RLaoMWtXg5PsCCidckadq1XL/L2Bj60cvxR5/ZPZYgz80RDYAd6ry4J3c1mewaT2LjoFP5OoCK2zz 913YA9YLzcifEyUkveNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qxtPb-005wtA-2Y; Tue, 31 Oct 2023 18:24:23 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qxtPY-005wqR-1o for linux-riscv@lists.infradead.org; Tue, 31 Oct 2023 18:24:22 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1cc1ee2d8dfso41800515ad.3 for ; Tue, 31 Oct 2023 11:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698776653; x=1699381453; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=ofL2l/62U8yrQ90hM9BCwDgXAR3/uuCpSfLhVmQhIXziXOZDIiBMLmkadApz1ed4DZ h1tnEQrZjnE1QCQJHodnIOnM3OU9oZBcf+k7UBVWexds3WwDRU6OmWtaHxClrXnfBion ZTlYaqcNyPT4kNp0myNVeQxix9Ar1/PlxMxn7whiJcJaAASnPU4G7O5Y4noAUJtbAMLB Tb8nt0fMBwXKZt3/7k/wS9VeZOYbJN9m0bHRETq80uZzLim7ncenzHMYPD/5HmVNi8aq kOGGhk2IfR/FbLBHPcTRDQkM2f7X06EIOp/VrR/eUnuSEtpniUd4LMJS4gFDtyFIvhaS /9tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698776653; x=1699381453; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=FhIouD4C1YOmJ4sXvLtKdQkHvO09zV3by4q4X6arq42JXSNAzWDFn2CKMT2sB9n4dO sFFA0/cXh1sOPGM53CI3tVCSIzLLJeMaCWWLVSlSNpfvKfUUzeNsShPxFrYKjDrvmCq6 fYdGFE93v3PqwcaXbojAyP6dMS1THwMvx0uocErllr3y59VwWs1e2+OyKN5vz6wTPrwZ JG2TDfCoXrkM8oq7cFk0M/qf5nuSPctK+3vJnblh4b2oQID1VYSxQAaPscCVSK/cB2gx GIKLwj4YqM+T1CcEMrjYz7Ysa8nmgEfRDgHPvoiFK9qoPhiMaW//8PYeFCbFDE3HXXRx vArQ== X-Gm-Message-State: AOJu0Yx0T9jd9StBBONcGDA5eQ2dQifEezYUOWpaXjG/F3iiYL+X1x1A QGe+08BcK4wvUWyvyN1EmeAcLA== X-Google-Smtp-Source: AGHT+IFxkiDdZGGfHsdazjXYZcfW0cI1Huw/6T3VggKB9s0vUh/sA7Jolgtag5oCOtU2pTLnT1L5FA== X-Received: by 2002:a17:902:ecca:b0:1cc:54b5:b4fa with SMTP id a10-20020a170902ecca00b001cc54b5b4famr5681643plh.18.1698776652868; Tue, 31 Oct 2023 11:24:12 -0700 (PDT) Received: from google.com (175.199.125.34.bc.googleusercontent.com. [34.125.199.175]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c20d00b001c5dea67c26sm1620267pll.233.2023.10.31.11.24.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 11:24:11 -0700 (PDT) Date: Tue, 31 Oct 2023 11:24:07 -0700 From: David Matlack To: Sean Christopherson Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Xiaoyao Li , Xu Yilun , Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , =?iso-8859-1?Q?Micka=EBl_Sala=FCn?= , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Subject: Re: [PATCH v13 16/35] KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory Message-ID: References: <20231027182217.3615211-1-seanjc@google.com> <20231027182217.3615211-17-seanjc@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027182217.3615211-17-seanjc@google.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231031_112420_601262_DDC4322D X-CRM114-Status: GOOD ( 19.61 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gMjAyMy0xMC0yNyAxMToyMSBBTSwgU2VhbiBDaHJpc3RvcGhlcnNvbiB3cm90ZToKPiBJbnRy b2R1Y2UgYW4gaW9jdGwoKSwgS1ZNX0NSRUFURV9HVUVTVF9NRU1GRCwgdG8gYWxsb3cgY3JlYXRp bmcgZmlsZS1iYXNlZAo+IG1lbW9yeSB0aGF0IGlzIHRpZWQgdG8gYSBzcGVjaWZpYyBLVk0gdmly dHVhbCBtYWNoaW5lIGFuZCB3aG9zZSBwcmltYXJ5Cj4gcHVycG9zZSBpcyB0byBzZXJ2ZSBndWVz dCBtZW1vcnkuCj4gCj4gQSBndWVzdC1maXJzdCBtZW1vcnkgc3Vic3lzdGVtIGFsbG93cyBmb3Ig b3B0aW1pemF0aW9ucyBhbmQgZW5oYW5jZW1lbnRzCj4gdGhhdCBhcmUga2x1ZGd5IG9yIG91dHJp Z2h0IGluZmVhc2libGUgdG8gaW1wbGVtZW50L3N1cHBvcnQgaW4gYSBnZW5lcmljCj4gbWVtb3J5 IHN1YnN5c3RlbS4gIFdpdGggZ3Vlc3RfbWVtZmQsIGd1ZXN0IHByb3RlY3Rpb25zIGFuZCBtYXBw aW5nIHNpemVzCj4gYXJlIGZ1bGx5IGRlY291cGxlZCBmcm9tIGhvc3QgdXNlcnNwYWNlIG1hcHBp bmdzLiAgIEUuZy4gS1ZNIGN1cnJlbnRseQo+IGRvZXNuJ3Qgc3VwcG9ydCBtYXBwaW5nIG1lbW9y eSBhcyB3cml0YWJsZSBpbiB0aGUgZ3Vlc3Qgd2l0aG91dCBpdCBhbHNvCj4gYmVpbmcgd3JpdGFi bGUgaW4gaG9zdCB1c2Vyc3BhY2UsIGFzIEtWTSdzIEFCSSB1c2VzIFZNQSBwcm90ZWN0aW9ucyB0 bwo+IGRlZmluZSB0aGUgYWxsb3cgZ3Vlc3QgcHJvdGVjdGlvbi4gIFVzZXJzcGFjZSBjYW4gZnVk Z2UgdGhpcyBieQo+IGVzdGFibGlzaGluZyB0d28gbWFwcGluZ3MsIGEgd3JpdGFibGUgbWFwcGlu ZyBmb3IgdGhlIGd1ZXN0IGFuZCByZWFkYWJsZQo+IG9uZSBmb3IgaXRzZWxmLCBidXQgdGhhdOKA mXMgc3Vib3B0aW1hbCBvbiBtdWx0aXBsZSBmcm9udHMuCj4gCj4gU2ltaWxhcmx5LCBLVk0gY3Vy cmVudGx5IHJlcXVpcmVzIHRoZSBndWVzdCBtYXBwaW5nIHNpemUgdG8gYmUgYSBzdHJpY3QKPiBz dWJzZXQgb2YgdGhlIGhvc3QgdXNlcnNwYWNlIG1hcHBpbmcgc2l6ZSwgZS5nLiBLVk0gZG9lc27i gJl0IHN1cHBvcnQKPiBjcmVhdGluZyBhIDFHaUIgZ3Vlc3QgbWFwcGluZyB1bmxlc3MgdXNlcnNw YWNlIGFsc28gaGFzIGEgMUdpQiBndWVzdAo+IG1hcHBpbmcuICBEZWNvdXBsaW5nIHRoZSBtYXBw aW5ncyBzaXplcyB3b3VsZCBhbGxvdyB1c2Vyc3BhY2UgdG8gcHJlY2lzZWx5Cj4gbWFwIG9ubHkg d2hhdCBpcyBuZWVkZWQgd2l0aG91dCBpbXBhY3RpbmcgZ3Vlc3QgcGVyZm9ybWFuY2UsIGUuZy4g dG8KPiBoYXJkZW4gYWdhaW5zdCB1bmludGVudGlvbmFsIGFjY2Vzc2VzIHRvIGd1ZXN0IG1lbW9y eS4KPiAKPiBEZWNvdXBsaW5nIGd1ZXN0IGFuZCB1c2Vyc3BhY2UgbWFwcGluZ3MgbWF5IGFsc28g YWxsb3cgZm9yIGEgY2xlYW5lcgo+IGFsdGVybmF0aXZlIHRvIGhpZ2gtZ3JhbnVsYXJpdHkgbWFw cGluZ3MgZm9yIEh1Z2VUTEIsIHdoaWNoIGhhcyByZWFjaGVkIGEKPiBiaXQgb2YgYW4gaW1wYXNz ZSBhbmQgaXMgdW5saWtlbHkgdG8gZXZlciBiZSBtZXJnZWQuCj4gCj4gQSBndWVzdC1maXJzdCBt ZW1vcnkgc3Vic3lzdGVtIGFsc28gcHJvdmlkZXMgY2xlYXJlciBsaW5lIG9mIHNpZ2h0IHRvCj4g dGhpbmdzIGxpa2UgYSBkZWRpY2F0ZWQgbWVtb3J5IHBvb2wgKGZvciBzbGljZS1vZi1oYXJkd2Fy ZSBWTXMpIGFuZAo+IGVsaW1pbmF0aW9uIG9mICJzdHJ1Y3QgcGFnZSIgKGZvciBvZmZsb2FkIHNl dHVwcyB3aGVyZSB1c2Vyc3BhY2UgX25ldmVyXwo+IG5lZWRzIHRvIG1tYXAoKSBndWVzdCBtZW1v cnkpLgoKQWxsIG9mIHRoZXNlIHVzZS1jYXNlcyBpbnZvbHZlIHVzaW5nIGd1ZXN0X21lbWZkIGZv ciBzaGFyZWQgcGFnZXMsIGJ1dAp0aGlzIGVudGlyZSBzZXJpZXMgc2V0cyB1cCBLVk0gdG8gb25s eSB1c2UgZ3Vlc3RfbWVtZmQgZm9yIHByaXZhdGUKcGFnZXMuCgpGb3IgZXhhbXBsZSwgdGhlIHBl ci1wYWdlIGF0dHJpYnV0ZXMgYXJlIGEgcHJvcGVydHkgb2YgYSBLVk0gVk0sIG5vdCB0aGUKdW5k ZXJseWluZyBndWVzdF9tZW1mZC4gU28gdGhhdCBpbXBsaWVzIHdlIHdpbGwgbmVlZCBzZXBhcmF0 ZQpndWVzdF9tZW1mZHMgZm9yIHByaXZhdGUgYW5kIHNoYXJlZCBwYWdlcy4gQnV0IGEgZ2l2ZW4g bWVtc2xvdCBjYW4gaGF2ZQphIG1peCBvZiBwcml2YXRlIGFuZCBzaGFyZWQgcGFnZXMuIFNvIHRo YXQgaW1wbGllcyBhIG1lbXNsb3Qgd2lsbCBuZWVkCnRvIHN1cHBvcnQgMiBndWVzdF9tZW1mZHM/ IEJ1dCB0aGUgVUFQSSBvbmx5IGFsbG93cyAxIGFuZCB1c2VzIHRoZSBIVkEKZm9yIHNoYXJlZCBt YXBwaW5ncy4KCk15IGluaXRpYWwgcmVhY3Rpb24gYWZ0ZXIgcmVhZGluZyB0aHJvdWdoIHRoaXMg c2VyaWVzIGlzIHRoYXQgdGhlCnBlci1wYWdlIHByaXZhdGUvc2hhcmVkIHNob3VsZCBiZSBhIHBy b3BlcnR5IG9mIHRoZSBndWVzdF9tZW1mZCwgbm90IHRoZQpWTS4gTWF5YmUgaXQgd291bGQgZXZl biBiZSBjbGVhbmVyIGluIHRoZSBsb25nLXJ1biB0byBtYWtlIGFsbCBtZW1vcnkKYXR0cmlidXRl cyBhIHByb3BlcnR5IG9mIHRoZSBndWVzdF9tZW1mZC4gVGhhdCB3YXkgd2UgY2FuIHNjb3BlIHRo ZQpzdXBwb3J0IHRvIG9ubHkgZ3Vlc3RfbWVtZmRzIGFuZCBub3QgaGF2ZSB0byB3b3JyeSBhYm91 dCBtYWtpbmcgcGVyLXBhZ2UKYXR0cmlidXRlcyB3b3JrIHdpdGggImxlZ2FjeSIgSFZBLWJhc2Vk IG1lbXNsb3RzLgoKTWF5YmUgY2FuIHlvdSBza2V0Y2ggb3V0IGhvdyB5b3Ugc2VlIHRoaXMgcHJv cG9zYWwgYmVpbmcgZXh0ZW5zaWJsZSB0bwp1c2luZyBndWVzdF9tZW1mZCBmb3Igc2hhcmVkIG1h cHBpbmdzPwoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K bGludXgtcmlzY3YgbWFpbGluZyBsaXN0CmxpbnV4LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcK aHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2136C4332F for ; Tue, 31 Oct 2023 18:25:13 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=i4Kry3Nf; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4SKdpw3YGbz3cVR for ; Wed, 1 Nov 2023 05:25:12 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=i4Kry3Nf; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=dmatlack@google.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4SKdnv550Lz2xwH for ; Wed, 1 Nov 2023 05:24:18 +1100 (AEDT) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1cc329ce84cso31059875ad.2 for ; Tue, 31 Oct 2023 11:24:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698776653; x=1699381453; darn=lists.ozlabs.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=i4Kry3NfAk1Lwqk9uJJcrwP5VQ1fkophVEf/Oq+z66NGNQZvS/B/zBG6VW7zCtUog4 mIaPRa2sSgXX0ZOTXpakFrgpCZxM0dQJlvFfksEbRbJjA7b7QLZG6Wn+oX+xsqBZ27Js 4rZSsEWD3DBZ2SuJhRrpuHg3sFjkHsbhLn0RS5nDeeCiQgu+DYE7GUtti83kZbqKCrun Ejh5qOcx3jJw7ZuX6GvQBtz9L+J5w8m34M5Euxn3LzxS0rYeYPLZzw5KdNKti9vxrUde bMLgHea0EIHACNEtRi6oeaNxEli/8CfxvI7cQWBm8+DV/vChwg3m+mH491AeUU5R94mY qBvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698776653; x=1699381453; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=QNB27aGwPLkldskMIiX0Ek9KeiWkWoz4waM3TlZbgZsCl6Kv5JNeM/wHcStVGKgbQG lwV4LJJ8stp2lL+87Grnj17STfhyilyxyGxfFCi4kZGRCho3BKisvyAXm5lhfRQQVJeu qo0lZDZfLM1hxw6kcM15ZbZ+HjgrceEObm93eNDKv/HlUoYxMpp0unaEau02mEeVkFMN WjtaUxCVmACPOAgPuCYGHtZ8WN29o4Sj254dnidVN88U6g9k5/gZq/hCtSEGue6VU/05 aOUSjxua83ibf//iI8YEpx272owoKOfyzhg/GjuBJ8THF2DgLgBcVZKl/OgDFTQqbsKS fFVg== X-Gm-Message-State: AOJu0YxOLDQ+FEqUPWlUs0JulWIqpvw8sGYOWBQ4fsj8c8nteg2FGAC4 1ppIgf5gep4JmkKFiIRThVqhYg== X-Google-Smtp-Source: AGHT+IFxkiDdZGGfHsdazjXYZcfW0cI1Huw/6T3VggKB9s0vUh/sA7Jolgtag5oCOtU2pTLnT1L5FA== X-Received: by 2002:a17:902:ecca:b0:1cc:54b5:b4fa with SMTP id a10-20020a170902ecca00b001cc54b5b4famr5681643plh.18.1698776652868; Tue, 31 Oct 2023 11:24:12 -0700 (PDT) Received: from google.com (175.199.125.34.bc.googleusercontent.com. [34.125.199.175]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c20d00b001c5dea67c26sm1620267pll.233.2023.10.31.11.24.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 11:24:11 -0700 (PDT) Date: Tue, 31 Oct 2023 11:24:07 -0700 From: David Matlack To: Sean Christopherson Subject: Re: [PATCH v13 16/35] KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory Message-ID: References: <20231027182217.3615211-1-seanjc@google.com> <20231027182217.3615211-17-seanjc@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20231027182217.3615211-17-seanjc@google.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chao Peng , linux-riscv@lists.infradead.org, Isaku Yamahata , Marc Zyngier , Huacai Chen , Xiaoyao Li , "Matthew Wilcox \(Oracle\)" , Wang , Fuad Tabba , Yu Zhang , Maciej Szmigiero , Albert Ou , Vlastimil Babka , Michael Roth , Ackerley Tng , Alexander Viro , Paul Walmsley , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, =?iso-8859-1?Q?Micka=EBl_Sala=FCn?= , Isaku Yamahata , Christian Brauner , Quentin Perret , Li am Merwick , linux-mips@vger.kernel.org, Oliver Upton , Jarkko Sakkinen , Palmer Dabbelt , "Kirill A . Shutemov" , kvm-riscv@lists.infradead.org, Anup Patel , linux-fsdevel@vger.kernel.org, Paolo Bonzini , Andrew Morton , Vishal Annapurve , linuxppc-dev@lists.ozlabs.org, Xu Yilun , Anish Moorthy Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 2023-10-27 11:21 AM, Sean Christopherson wrote: > Introduce an ioctl(), KVM_CREATE_GUEST_MEMFD, to allow creating file-based > memory that is tied to a specific KVM virtual machine and whose primary > purpose is to serve guest memory. > > A guest-first memory subsystem allows for optimizations and enhancements > that are kludgy or outright infeasible to implement/support in a generic > memory subsystem. With guest_memfd, guest protections and mapping sizes > are fully decoupled from host userspace mappings. E.g. KVM currently > doesn't support mapping memory as writable in the guest without it also > being writable in host userspace, as KVM's ABI uses VMA protections to > define the allow guest protection. Userspace can fudge this by > establishing two mappings, a writable mapping for the guest and readable > one for itself, but that’s suboptimal on multiple fronts. > > Similarly, KVM currently requires the guest mapping size to be a strict > subset of the host userspace mapping size, e.g. KVM doesn’t support > creating a 1GiB guest mapping unless userspace also has a 1GiB guest > mapping. Decoupling the mappings sizes would allow userspace to precisely > map only what is needed without impacting guest performance, e.g. to > harden against unintentional accesses to guest memory. > > Decoupling guest and userspace mappings may also allow for a cleaner > alternative to high-granularity mappings for HugeTLB, which has reached a > bit of an impasse and is unlikely to ever be merged. > > A guest-first memory subsystem also provides clearer line of sight to > things like a dedicated memory pool (for slice-of-hardware VMs) and > elimination of "struct page" (for offload setups where userspace _never_ > needs to mmap() guest memory). All of these use-cases involve using guest_memfd for shared pages, but this entire series sets up KVM to only use guest_memfd for private pages. For example, the per-page attributes are a property of a KVM VM, not the underlying guest_memfd. So that implies we will need separate guest_memfds for private and shared pages. But a given memslot can have a mix of private and shared pages. So that implies a memslot will need to support 2 guest_memfds? But the UAPI only allows 1 and uses the HVA for shared mappings. My initial reaction after reading through this series is that the per-page private/shared should be a property of the guest_memfd, not the VM. Maybe it would even be cleaner in the long-run to make all memory attributes a property of the guest_memfd. That way we can scope the support to only guest_memfds and not have to worry about making per-page attributes work with "legacy" HVA-based memslots. Maybe can you sketch out how you see this proposal being extensible to using guest_memfd for shared mappings? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20E47C4332F for ; Tue, 31 Oct 2023 18:24:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gYhmG/rFiT3JCKfC86lCj/FaTIdgtWq3RUajazPaWEA=; b=GAppC82qLXgfBc bWTG9o65pev2TSn9P2Qn/croW9ekvcXlk81fTi+xchCRhYhq0hAl9CSQKwIyGEDciDp7yYlfaOnQE n7331YEqHJpXBtVJXhCaEoQxyFJXUxzRuCn8LQ3vpxT5pRZFKwBxCLE988daM/MSgBR2iYIm5J6DY NxI0oWWOqZLePrpJ+lb/S18vWk9WAssLbUXQaZpzJLJd8eY+iGn+avVAm6HDhpcbTwZwZAjjAOoUb eKyteMQA+7MgIzYcsPYZNdE6uBpaUwhWG7mE7DsVu0WUh3fVyDis0XSxCteokQYNbTLv4bLwmkOCr WHuOZeH6Ara3h5aqqVBA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qxtPc-005wtc-2u; Tue, 31 Oct 2023 18:24:24 +0000 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qxtPY-005wqU-2h for linux-arm-kernel@lists.infradead.org; Tue, 31 Oct 2023 18:24:23 +0000 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1cc5b705769so17821375ad.0 for ; Tue, 31 Oct 2023 11:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698776653; x=1699381453; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=ofL2l/62U8yrQ90hM9BCwDgXAR3/uuCpSfLhVmQhIXziXOZDIiBMLmkadApz1ed4DZ h1tnEQrZjnE1QCQJHodnIOnM3OU9oZBcf+k7UBVWexds3WwDRU6OmWtaHxClrXnfBion ZTlYaqcNyPT4kNp0myNVeQxix9Ar1/PlxMxn7whiJcJaAASnPU4G7O5Y4noAUJtbAMLB Tb8nt0fMBwXKZt3/7k/wS9VeZOYbJN9m0bHRETq80uZzLim7ncenzHMYPD/5HmVNi8aq kOGGhk2IfR/FbLBHPcTRDQkM2f7X06EIOp/VrR/eUnuSEtpniUd4LMJS4gFDtyFIvhaS /9tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698776653; x=1699381453; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fA4BSuC8POmGjMQJ0k8vnna5BPAxY9Ekm8SCU7EDDBY=; b=d5UTjlMT5NpaHfcLAJGyfZBNflr0ykeDMwqlKzzBf3G3at38fMkFQEDPLcFhj3a+ps Ro2FY9/CnFpKFeGzUNyo5KtBllVYHp2UD8DnudGFxPrgsf2Ac3ceypHlKYxc2JJ7jJDl SOpulwKxF9ZRCjNu/p9VD/WtMTLy3F0McvvaVliavFZySH8NMa5pXrbngX0sYOVbSKHs Ec3YCoFl0wHIEMdknRoE8plCQPlmkDtfl/DPdVFwRa/a2ET1tboGpbAEmSxGUEIm+Sc5 LOewBqQBeRlF45Agpuqnne44Y8mwXTJ9rsoEzNQql0HgrA6KAD1NOZOLZKAOeJPHSWkO ZYJw== X-Gm-Message-State: AOJu0YwmFFTqEG6S+vlpNo9uYksWLs+sEO1gsGUT7jz1s3AJrw06Vgxn v9sr25uKfcbup9nQwc85KY9rUQ== X-Google-Smtp-Source: AGHT+IFxkiDdZGGfHsdazjXYZcfW0cI1Huw/6T3VggKB9s0vUh/sA7Jolgtag5oCOtU2pTLnT1L5FA== X-Received: by 2002:a17:902:ecca:b0:1cc:54b5:b4fa with SMTP id a10-20020a170902ecca00b001cc54b5b4famr5681643plh.18.1698776652868; Tue, 31 Oct 2023 11:24:12 -0700 (PDT) Received: from google.com (175.199.125.34.bc.googleusercontent.com. [34.125.199.175]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c20d00b001c5dea67c26sm1620267pll.233.2023.10.31.11.24.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 11:24:11 -0700 (PDT) Date: Tue, 31 Oct 2023 11:24:07 -0700 From: David Matlack To: Sean Christopherson Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Xiaoyao Li , Xu Yilun , Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , =?iso-8859-1?Q?Micka=EBl_Sala=FCn?= , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Subject: Re: [PATCH v13 16/35] KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory Message-ID: References: <20231027182217.3615211-1-seanjc@google.com> <20231027182217.3615211-17-seanjc@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027182217.3615211-17-seanjc@google.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231031_112420_897936_5C849C89 X-CRM114-Status: GOOD ( 21.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMjAyMy0xMC0yNyAxMToyMSBBTSwgU2VhbiBDaHJpc3RvcGhlcnNvbiB3cm90ZToKPiBJbnRy b2R1Y2UgYW4gaW9jdGwoKSwgS1ZNX0NSRUFURV9HVUVTVF9NRU1GRCwgdG8gYWxsb3cgY3JlYXRp bmcgZmlsZS1iYXNlZAo+IG1lbW9yeSB0aGF0IGlzIHRpZWQgdG8gYSBzcGVjaWZpYyBLVk0gdmly dHVhbCBtYWNoaW5lIGFuZCB3aG9zZSBwcmltYXJ5Cj4gcHVycG9zZSBpcyB0byBzZXJ2ZSBndWVz dCBtZW1vcnkuCj4gCj4gQSBndWVzdC1maXJzdCBtZW1vcnkgc3Vic3lzdGVtIGFsbG93cyBmb3Ig b3B0aW1pemF0aW9ucyBhbmQgZW5oYW5jZW1lbnRzCj4gdGhhdCBhcmUga2x1ZGd5IG9yIG91dHJp Z2h0IGluZmVhc2libGUgdG8gaW1wbGVtZW50L3N1cHBvcnQgaW4gYSBnZW5lcmljCj4gbWVtb3J5 IHN1YnN5c3RlbS4gIFdpdGggZ3Vlc3RfbWVtZmQsIGd1ZXN0IHByb3RlY3Rpb25zIGFuZCBtYXBw aW5nIHNpemVzCj4gYXJlIGZ1bGx5IGRlY291cGxlZCBmcm9tIGhvc3QgdXNlcnNwYWNlIG1hcHBp bmdzLiAgIEUuZy4gS1ZNIGN1cnJlbnRseQo+IGRvZXNuJ3Qgc3VwcG9ydCBtYXBwaW5nIG1lbW9y eSBhcyB3cml0YWJsZSBpbiB0aGUgZ3Vlc3Qgd2l0aG91dCBpdCBhbHNvCj4gYmVpbmcgd3JpdGFi bGUgaW4gaG9zdCB1c2Vyc3BhY2UsIGFzIEtWTSdzIEFCSSB1c2VzIFZNQSBwcm90ZWN0aW9ucyB0 bwo+IGRlZmluZSB0aGUgYWxsb3cgZ3Vlc3QgcHJvdGVjdGlvbi4gIFVzZXJzcGFjZSBjYW4gZnVk Z2UgdGhpcyBieQo+IGVzdGFibGlzaGluZyB0d28gbWFwcGluZ3MsIGEgd3JpdGFibGUgbWFwcGlu ZyBmb3IgdGhlIGd1ZXN0IGFuZCByZWFkYWJsZQo+IG9uZSBmb3IgaXRzZWxmLCBidXQgdGhhdOKA mXMgc3Vib3B0aW1hbCBvbiBtdWx0aXBsZSBmcm9udHMuCj4gCj4gU2ltaWxhcmx5LCBLVk0gY3Vy cmVudGx5IHJlcXVpcmVzIHRoZSBndWVzdCBtYXBwaW5nIHNpemUgdG8gYmUgYSBzdHJpY3QKPiBz dWJzZXQgb2YgdGhlIGhvc3QgdXNlcnNwYWNlIG1hcHBpbmcgc2l6ZSwgZS5nLiBLVk0gZG9lc27i gJl0IHN1cHBvcnQKPiBjcmVhdGluZyBhIDFHaUIgZ3Vlc3QgbWFwcGluZyB1bmxlc3MgdXNlcnNw YWNlIGFsc28gaGFzIGEgMUdpQiBndWVzdAo+IG1hcHBpbmcuICBEZWNvdXBsaW5nIHRoZSBtYXBw aW5ncyBzaXplcyB3b3VsZCBhbGxvdyB1c2Vyc3BhY2UgdG8gcHJlY2lzZWx5Cj4gbWFwIG9ubHkg d2hhdCBpcyBuZWVkZWQgd2l0aG91dCBpbXBhY3RpbmcgZ3Vlc3QgcGVyZm9ybWFuY2UsIGUuZy4g dG8KPiBoYXJkZW4gYWdhaW5zdCB1bmludGVudGlvbmFsIGFjY2Vzc2VzIHRvIGd1ZXN0IG1lbW9y eS4KPiAKPiBEZWNvdXBsaW5nIGd1ZXN0IGFuZCB1c2Vyc3BhY2UgbWFwcGluZ3MgbWF5IGFsc28g YWxsb3cgZm9yIGEgY2xlYW5lcgo+IGFsdGVybmF0aXZlIHRvIGhpZ2gtZ3JhbnVsYXJpdHkgbWFw cGluZ3MgZm9yIEh1Z2VUTEIsIHdoaWNoIGhhcyByZWFjaGVkIGEKPiBiaXQgb2YgYW4gaW1wYXNz ZSBhbmQgaXMgdW5saWtlbHkgdG8gZXZlciBiZSBtZXJnZWQuCj4gCj4gQSBndWVzdC1maXJzdCBt ZW1vcnkgc3Vic3lzdGVtIGFsc28gcHJvdmlkZXMgY2xlYXJlciBsaW5lIG9mIHNpZ2h0IHRvCj4g dGhpbmdzIGxpa2UgYSBkZWRpY2F0ZWQgbWVtb3J5IHBvb2wgKGZvciBzbGljZS1vZi1oYXJkd2Fy ZSBWTXMpIGFuZAo+IGVsaW1pbmF0aW9uIG9mICJzdHJ1Y3QgcGFnZSIgKGZvciBvZmZsb2FkIHNl dHVwcyB3aGVyZSB1c2Vyc3BhY2UgX25ldmVyXwo+IG5lZWRzIHRvIG1tYXAoKSBndWVzdCBtZW1v cnkpLgoKQWxsIG9mIHRoZXNlIHVzZS1jYXNlcyBpbnZvbHZlIHVzaW5nIGd1ZXN0X21lbWZkIGZv ciBzaGFyZWQgcGFnZXMsIGJ1dAp0aGlzIGVudGlyZSBzZXJpZXMgc2V0cyB1cCBLVk0gdG8gb25s eSB1c2UgZ3Vlc3RfbWVtZmQgZm9yIHByaXZhdGUKcGFnZXMuCgpGb3IgZXhhbXBsZSwgdGhlIHBl ci1wYWdlIGF0dHJpYnV0ZXMgYXJlIGEgcHJvcGVydHkgb2YgYSBLVk0gVk0sIG5vdCB0aGUKdW5k ZXJseWluZyBndWVzdF9tZW1mZC4gU28gdGhhdCBpbXBsaWVzIHdlIHdpbGwgbmVlZCBzZXBhcmF0 ZQpndWVzdF9tZW1mZHMgZm9yIHByaXZhdGUgYW5kIHNoYXJlZCBwYWdlcy4gQnV0IGEgZ2l2ZW4g bWVtc2xvdCBjYW4gaGF2ZQphIG1peCBvZiBwcml2YXRlIGFuZCBzaGFyZWQgcGFnZXMuIFNvIHRo YXQgaW1wbGllcyBhIG1lbXNsb3Qgd2lsbCBuZWVkCnRvIHN1cHBvcnQgMiBndWVzdF9tZW1mZHM/ IEJ1dCB0aGUgVUFQSSBvbmx5IGFsbG93cyAxIGFuZCB1c2VzIHRoZSBIVkEKZm9yIHNoYXJlZCBt YXBwaW5ncy4KCk15IGluaXRpYWwgcmVhY3Rpb24gYWZ0ZXIgcmVhZGluZyB0aHJvdWdoIHRoaXMg c2VyaWVzIGlzIHRoYXQgdGhlCnBlci1wYWdlIHByaXZhdGUvc2hhcmVkIHNob3VsZCBiZSBhIHBy b3BlcnR5IG9mIHRoZSBndWVzdF9tZW1mZCwgbm90IHRoZQpWTS4gTWF5YmUgaXQgd291bGQgZXZl biBiZSBjbGVhbmVyIGluIHRoZSBsb25nLXJ1biB0byBtYWtlIGFsbCBtZW1vcnkKYXR0cmlidXRl cyBhIHByb3BlcnR5IG9mIHRoZSBndWVzdF9tZW1mZC4gVGhhdCB3YXkgd2UgY2FuIHNjb3BlIHRo ZQpzdXBwb3J0IHRvIG9ubHkgZ3Vlc3RfbWVtZmRzIGFuZCBub3QgaGF2ZSB0byB3b3JyeSBhYm91 dCBtYWtpbmcgcGVyLXBhZ2UKYXR0cmlidXRlcyB3b3JrIHdpdGggImxlZ2FjeSIgSFZBLWJhc2Vk IG1lbXNsb3RzLgoKTWF5YmUgY2FuIHlvdSBza2V0Y2ggb3V0IGhvdyB5b3Ugc2VlIHRoaXMgcHJv cG9zYWwgYmVpbmcgZXh0ZW5zaWJsZSB0bwp1c2luZyBndWVzdF9tZW1mZCBmb3Igc2hhcmVkIG1h cHBpbmdzPwoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K bGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZy YWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGlu dXgtYXJtLWtlcm5lbAo=