From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Tue, 3 Oct 2023 13:51:52 -0700 Subject: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes In-Reply-To: References: <20230914015531.1419405-1-seanjc@google.com> <20230914015531.1419405-12-seanjc@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Tue, Oct 03, 2023, Fuad Tabba wrote: > On Tue, Oct 3, 2023 at 4:59?PM Sean Christopherson wrote: > > On Tue, Oct 03, 2023, Fuad Tabba wrote: > > > > +#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) > > > > + > > > > > > In pKVM, we don't want to allow setting (or clearing) of PRIVATE/SHARED > > > attributes from userspace. > > > > Why not? The whole thing falls apart if userspace doesn't *know* the state of a > > page, and the only way for userspace to know the state of a page at a given moment > > in time is if userspace controls the attributes. E.g. even if KVM were to provide > > a way for userspace to query attributes, the attributes exposed to usrspace would > > become stale the instant KVM drops slots_lock (or whatever lock protects the attributes) > > since userspace couldn't prevent future changes. > > I think I might not quite understand the purpose of the > KVM_SET_MEMORY_ATTRIBUTES ABI. In pKVM, all of a protected guest's memory is > private by default, until the guest shares it with the host (via a > hypercall), or another guest (future work). When the guest shares it, > userspace is notified via KVM_EXIT_HYPERCALL. In many use cases, userspace > doesn't need to keep track directly of all of this, but can reactively un/map > the memory being un/shared. Yes, and then userspace needs to tell KVM, via KVM_SET_MEMORY_ATTRIBUTES, that userspace has agreed to change the state of the page. Userspace may not need/want to explicitly track the state of pages, but userspace still needs to tell KVM what userspace wants. KVM is primarily an accelerator, e.g. KVM's role is to make things go fast (relative to doing things in userspace) and provide access to resources/instructions that require elevated privileges. As a general rule, we try to avoid defining the vCPU model, security policies, etc. in KVM, because hardcoding policy into KVM (and the kernel as a whole) eventually limits the utility of KVM. As it pertains to PRIVATE vs. SHARED, KVM's role is to define and enforce the basic rules, but KVM shouldn't do things like define when it is (il)legal to convert memory to/from SHARED, what pages can be converted, what happens if the guest and userspace disagree, etc. > > Why does pKVM need to prevent userspace from stating *its* view of attributes? > > > > If the goal is to reduce memory overhead, that can be solved by using an internal, > > non-ABI attributes flag to track pKVM's view of SHARED vs. PRIVATE. If the guest > > attempts to access memory where pKVM and userspace don't agree on the state, > > generate an exit to userspace. Or kill the guest. Or do something else entirely. > > For the pKVM hypervisor the guest's view of the attributes doesn't > matter. The hypervisor at the end of the day is the ultimate arbiter > for what is shared and with how. For pKVM (at least in my port of > guestmem), we use the memory attributes from guestmem essentially to > control which memory can be mapped by the host. The guest's view absolutely matters. The guest's view may not be expressed at access time, e.g. as you note below, pKVM and other software-protected VMs don't have a dedicated shared vs. private bit like TDX and SNP. But the view is still there, e.g. in the pKVM model, the guest expresses its desire for shared vs. private via hypercall, and IIRC, the guest's view is tracked by the hypervisor in the stage-2 PTEs. pKVM itself may track the guest's view on things, but the view is still the guest's. E.g. if the guest thinks a page is private, but in reality KVM and host userspace have it as shared, then the guest may unintentionally leak data to the untrusted world. IIUC, you have implemented guest_memfd support in pKVM by changing the attributes when the guest makes the hypercall. This can work, but only so long as the guest and userspace are well-behaved, and it will likely paint pKVM into a corner in the long run. E.g. if the guest makes a hypercall to convert memory to PRIVATE, but there is no memslot or the memslot doesn't support private memory, then unless there is policy baked into KVM, or an ABI for the guest<=>host hypercall interface that allows unwinding the program counter, you're stuck. Returning an error for the hypercall straight from KVM is undesirable as that would put policy into KVM that doesn't need to be there, e.g. that would prevent userspace from manipulating memslots in response to (un)share requests from the guest. It's a similar story if KVM marks the page as PRIVATE, as that would prevent userspace from returning an error for the hypercall, i.e. would prevent usersepace from denying the request to convert to PRIVATE. > One difference between pKVM and TDX (as I understand it), is that TDX > uses the msb of the guest's IPA to indicate whether memory is shared > or private, and that can generate a mismatch on guest memory access > between what it thinks the state is, and what the sharing state in > reality is. pKVM doesn't have that. Memory is private by default, and > can be shared in-place, both in the guest's IPA space as well as the > underlying physical page. TDX's shared bit and SNP's encryption bit are just a means of hardware enforcement. pKVM does have a hardware bit because hardware doesn't provide any enforcement. But as above, pKVM does have an equivalent *somewhere*. > > > The other thing, which we need for pKVM anyway, is to make > > > kvm_vm_set_mem_attributes() global, so that it can be called from outside of > > > kvm_main.c (already have a local patch for this that declares it in > > > kvm_host.h), > > > > That's no problem, but I am definitely opposed to KVM modifying attributes that > > are owned by userspace. > > > > > and not gate this function by KVM_GENERIC_MEMORY_ATTRIBUTES. > > > > As above, I am opposed to pKVM having a completely different ABI for managing > > PRIVATE vs. SHARED. I have no objection to pKVM using unclaimed flags in the > > attributes to store extra metadata, but if KVM_SET_MEMORY_ATTRIBUTES doesn't work > > for pKVM, then we've failed miserably and should revist the uAPI. > > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/SHARED, > just a way of tracking in the host kernel of what is shared (as opposed to > the hypervisor, which already has the knowledge). The solution could simply > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its own > tracking of the status of the guest pages, and only selects KVM_PRIVATE_MEM. At the risk of overstepping my bounds, I think that effectively giving the guest full control over what is shared vs. private is a mistake. It more or less locks pKVM into a single model, and even within that model, dealing with errors and/or misbehaving guests becomes unnecessarily problematic. Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the userspace side of pKVM could simply "reflect" all conversion hypercalls, and terminate the VM on errors. But the cost is very minimal, e.g. a single extra ioctl() per converion, and the upside is that pKVM won't be stuck if a use case comes along that wants to go beyond "all conversion requests either immediately succeed or terminate the guest". From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6ABA830FA6 for ; Tue, 3 Oct 2023 20:51:55 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-59b5a586da6so1991087b3.1 for ; Tue, 03 Oct 2023 13:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696366314; x=1696971114; darn=lists.linux.dev; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=lWjWgawJVQEjR4vx/Z1WQfAtKuL0vvIZ3uG1NdfEu5yuYvR7Nchwn3PKwyXyyDMa2t P5pJv02sr6JlNCH3f4iyR8WcFV86dK/rAHVfRn3WasrsMB8DECHz2U7FLfUf/MX6qqcr gY6ZBE2ssUldvoZYAjnzd9pPHZeyvyGLUgR28UhsAtu/6jkEuWs61EwgzYv1ULDhN1ju PJlGadn9v65pJIlKUXDwiEZ6ysNtvK/XRWVdJKNmWwLgYgINh/ChZSGWKhYQJk6H7mzl DOeveYH/lNG8qeub8O7SMAQI7UyixrsUH6PKAzr7yzIsV8VvMkdrXUiuHRu4bRWuOs21 0BHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696366314; x=1696971114; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=ovydS8nefLh8NUp9SfCGA+hC+TSaItdUN9GCamUYv+ahJlCT0XiVIqis3k3rzRzp4C BTNUj7d/06sJaQRxfQVuybvd1+f2WaWNZqUqhR/3FfC1GMX9yaFvyb111TOb0TGi7Lm8 NoTUO7v9UqAzfO/UZAR4O3Eq/Slqg0yMVc5WzIUbJy8kfqIrr6tktZjyRKqtgSHrUwyL zYaNd8mRf0BjVrGomUvZo1TKc+rriG6myPp1SETOh4aoBEeg1BYv8ZkwUqz0YPdoHv6A 3OcLCRASPrBAZem+0AhfeE/ZZE9WeJs/C9v2W2ziFnxP+dtxdyOGdpkFSEmI3aO7Y7nw mYlQ== X-Gm-Message-State: AOJu0YwLpqExuxu5v3Xt8oE7AScHEGqiwNrrVjH5r6llQh2V7TSAqGJ9 eyMkabBy1qsb7TkTJUkwvaQ8jZvVVa8= X-Google-Smtp-Source: AGHT+IH9SpWB5ZYki6B4kneRjXWww1fW5aqwjgYpBvf3LyPRvX5qN4iOqywaVkHZ9PW5pYyi5WCeWQUyODc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:b83:b0:59b:b0b1:d75a with SMTP id ck3-20020a05690c0b8300b0059bb0b1d75amr85511ywb.4.1696366314084; Tue, 03 Oct 2023 13:51:54 -0700 (PDT) Date: Tue, 3 Oct 2023 13:51:52 -0700 In-Reply-To: Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> <20230914015531.1419405-12-seanjc@google.com> Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, linux-kernel@vger.kernel.org, Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 03, 2023, Fuad Tabba wrote: > On Tue, Oct 3, 2023 at 4:59=E2=80=AFPM Sean Christopherson wrote: > > On Tue, Oct 03, 2023, Fuad Tabba wrote: > > > > +#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) > > > > + > > > > > > In pKVM, we don't want to allow setting (or clearing) of PRIVATE/SHAR= ED > > > attributes from userspace. > > > > Why not? The whole thing falls apart if userspace doesn't *know* the s= tate of a > > page, and the only way for userspace to know the state of a page at a g= iven moment > > in time is if userspace controls the attributes. E.g. even if KVM were= to provide > > a way for userspace to query attributes, the attributes exposed to usrs= pace would > > become stale the instant KVM drops slots_lock (or whatever lock protect= s the attributes) > > since userspace couldn't prevent future changes. >=20 > I think I might not quite understand the purpose of the > KVM_SET_MEMORY_ATTRIBUTES ABI. In pKVM, all of a protected guest's memory= is > private by default, until the guest shares it with the host (via a > hypercall), or another guest (future work). When the guest shares it, > userspace is notified via KVM_EXIT_HYPERCALL. In many use cases, userspac= e > doesn't need to keep track directly of all of this, but can reactively un= /map > the memory being un/shared. Yes, and then userspace needs to tell KVM, via KVM_SET_MEMORY_ATTRIBUTES, t= hat userspace has agreed to change the state of the page. Userspace may not ne= ed/want to explicitly track the state of pages, but userspace still needs to tell K= VM what userspace wants. KVM is primarily an accelerator, e.g. KVM's role is to make things go fast = (relative to doing things in userspace) and provide access to resources/instructions = that require elevated privileges. As a general rule, we try to avoid defining t= he vCPU model, security policies, etc. in KVM, because hardcoding policy into KVM (= and the kernel as a whole) eventually limits the utility of KVM. As it pertains to PRIVATE vs. SHARED, KVM's role is to define and enforce t= he basic rules, but KVM shouldn't do things like define when it is (il)legal to conv= ert memory to/from SHARED, what pages can be converted, what happens if the gue= st and userspace disagree, etc. > > Why does pKVM need to prevent userspace from stating *its* view of attr= ibutes? > > > > If the goal is to reduce memory overhead, that can be solved by using a= n internal, > > non-ABI attributes flag to track pKVM's view of SHARED vs. PRIVATE. If= the guest > > attempts to access memory where pKVM and userspace don't agree on the s= tate, > > generate an exit to userspace. Or kill the guest. Or do something els= e entirely. >=20 > For the pKVM hypervisor the guest's view of the attributes doesn't > matter. The hypervisor at the end of the day is the ultimate arbiter > for what is shared and with how. For pKVM (at least in my port of > guestmem), we use the memory attributes from guestmem essentially to > control which memory can be mapped by the host. The guest's view absolutely matters. The guest's view may not be expressed= at access time, e.g. as you note below, pKVM and other software-protected VMs = don't have a dedicated shared vs. private bit like TDX and SNP. But the view is = still there, e.g. in the pKVM model, the guest expresses its desire for shared vs= . private via hypercall, and IIRC, the guest's view is tracked by the hypervi= sor in the stage-2 PTEs. pKVM itself may track the guest's view on things, but= the view is still the guest's. E.g. if the guest thinks a page is private, but in reality KVM and host use= rspace have it as shared, then the guest may unintentionally leak data to the untr= usted world. IIUC, you have implemented guest_memfd support in pKVM by changing the attr= ibutes when the guest makes the hypercall. This can work, but only so long as the= guest and userspace are well-behaved, and it will likely paint pKVM into a corner= in the long run. E.g. if the guest makes a hypercall to convert memory to PRIVATE, but there= is no memslot or the memslot doesn't support private memory, then unless there= is policy baked into KVM, or an ABI for the guest<=3D>host hypercall interface= that allows unwinding the program counter, you're stuck. Returning an error for= the hypercall straight from KVM is undesirable as that would put policy into KV= M that doesn't need to be there, e.g. that would prevent userspace from manipulati= ng memslots in response to (un)share requests from the guest. It's a similar = story if KVM marks the page as PRIVATE, as that would prevent userspace from retu= rning an error for the hypercall, i.e. would prevent usersepace from denying the = request to convert to PRIVATE. > One difference between pKVM and TDX (as I understand it), is that TDX > uses the msb of the guest's IPA to indicate whether memory is shared > or private, and that can generate a mismatch on guest memory access > between what it thinks the state is, and what the sharing state in > reality is. pKVM doesn't have that. Memory is private by default, and > can be shared in-place, both in the guest's IPA space as well as the > underlying physical page. TDX's shared bit and SNP's encryption bit are just a means of hardware enfo= rcement. pKVM does have a hardware bit because hardware doesn't provide any enforcem= ent. But as above, pKVM does have an equivalent *somewhere*. > > > The other thing, which we need for pKVM anyway, is to make > > > kvm_vm_set_mem_attributes() global, so that it can be called from out= side of > > > kvm_main.c (already have a local patch for this that declares it in > > > kvm_host.h), > > > > That's no problem, but I am definitely opposed to KVM modifying attribu= tes that > > are owned by userspace. > > > > > and not gate this function by KVM_GENERIC_MEMORY_ATTRIBUTES. > > > > As above, I am opposed to pKVM having a completely different ABI for ma= naging > > PRIVATE vs. SHARED. I have no objection to pKVM using unclaimed flags = in the > > attributes to store extra metadata, but if KVM_SET_MEMORY_ATTRIBUTES do= esn't work > > for pKVM, then we've failed miserably and should revist the uAPI. >=20 > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/SHARE= D, > just a way of tracking in the host kernel of what is shared (as opposed t= o > the hypervisor, which already has the knowledge). The solution could simp= ly > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its own > tracking of the status of the guest pages, and only selects KVM_PRIVATE_M= EM. At the risk of overstepping my bounds, I think that effectively giving the = guest full control over what is shared vs. private is a mistake. It more or less= locks pKVM into a single model, and even within that model, dealing with errors a= nd/or misbehaving guests becomes unnecessarily problematic. Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the use= rspace side of pKVM could simply "reflect" all conversion hypercalls, and terminat= e the VM on errors. But the cost is very minimal, e.g. a single extra ioctl() pe= r converion, and the upside is that pKVM won't be stuck if a use case comes a= long that wants to go beyond "all conversion requests either immediately succeed= or terminate the guest". From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8C0EE8FDB6 for ; Tue, 3 Oct 2023 20:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=the2EuE+jDKSYYKQXaM0rLzecCMIyNP0rXyrZ5Im60A=; b=1tCcSJqvPNhanBxmBl9VwAZ1sx uFH0foLd9beieKy6YkTFqRJvXG/meHuyFbTln8QtjxprBnBYPizMRgDQNmwaF55qngejJpgc+YL9y eMWd0ShxVBYtPjnJh4zoCBqt8lZuz3eY5Bw2lLFX4t7my6O1No5hFh538ocYskKm2VK+ur1pDzyAp q2i2Bn08PRtElWIZSZ5vphDgWuFMdt5ub+Z6Rem3tz283DZhufy+/GAsHCnBGnAr/exZFf4MUsQtg 7b9LRLuIxNZK5ZzQLd8dM40pxYDPJQsMIYn6uvVsptyaiz3d3ktgcaAS4rtgoxY+WSzXFAMaCh0qy 2XFojQNw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qnmN9-00FH04-0L; Tue, 03 Oct 2023 20:52:03 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qnmN5-00FGwL-2V for linux-riscv@lists.infradead.org; Tue, 03 Oct 2023 20:52:01 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5a1f12cf1ddso2029187b3.0 for ; Tue, 03 Oct 2023 13:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696366314; x=1696971114; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=gBLRXfZoM+80FxhVym2ng0jITavuKXXpsHNJIaxxKsqco8zjhISRtsDXRutvtrs4Ce h5J1AwxtKJh4rQ+LTa1X9AGZMQyVDZVk/0P3llVsZm+8CP/YMccyXXnxYo/ZMRi9qmAt WfHUjIH5IKvab5BotKCFj/2PRoovvHDlkjUFLTMf9R+xlzcHp32ZsOUTjImLhTEHFAGa DdrgAlrDp9nQpXyRsQ9g/InVdRK6tznEvi59N0uMtDKbjpuTMab63XrC/cLRB/WCHDh5 NBqt25juWlUqeEcp9loigrDdv/5l6//NOKnDxEVaPRDIk+QFs7pw3v5NZy5U/9F4w+AM DyWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696366314; x=1696971114; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=ma8TjxTJZkB/uV3riM5WcDsfCPveJDfGdQyBd6/DGhdm2uw8MHRXEVFNNxZINCRSdc PLYwQWYlC0na9Etwpd8rUmKLN8kKXsRMGBYXkDxF6VhoO06BlvA5Ae1aY5Kcj3RHWBqg sT2sHVy5nmFaOVSn31KDWpl9iWl9sy6Ba2opzWmFN/VVg+pJlVVNHgBeWWFM9OPHeN5n tHBUmOamiJmdiswszBa8XaDrLTA4fJx0KxVUQEqLQVfqQM2xpEgY6F8BMvPOr3yGKVF8 Nw8abEJXVvw2Bl19ISKaQzKdUWF0l8Ga7Nt0gkodQRZo3PEmyGI3kyF9ZzKylbPeAVLb 7cDg== X-Gm-Message-State: AOJu0Yx3G9YXZZTxybXKt1AdzIWYPKhAIK03Y9NIGjgPJFeXlVvuCE1X WI36VeZsvut5PnVSCYT9HkL7mldZb3E= X-Google-Smtp-Source: AGHT+IH9SpWB5ZYki6B4kneRjXWww1fW5aqwjgYpBvf3LyPRvX5qN4iOqywaVkHZ9PW5pYyi5WCeWQUyODc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:b83:b0:59b:b0b1:d75a with SMTP id ck3-20020a05690c0b8300b0059bb0b1d75amr85511ywb.4.1696366314084; Tue, 03 Oct 2023 13:51:54 -0700 (PDT) Date: Tue, 3 Oct 2023 13:51:52 -0700 In-Reply-To: Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> <20230914015531.1419405-12-seanjc@google.com> Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, linux-kernel@vger.kernel.org, Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231003_135159_833023_C9E68F4D X-CRM114-Status: GOOD ( 45.88 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVHVlLCBPY3QgMDMsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gT24gVHVlLCBPY3QgMywg MjAyMyBhdCA0OjU54oCvUE0gU2VhbiBDaHJpc3RvcGhlcnNvbiA8c2VhbmpjQGdvb2dsZS5jb20+ IHdyb3RlOgo+ID4gT24gVHVlLCBPY3QgMDMsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gPiA+ ID4gKyNkZWZpbmUgS1ZNX01FTU9SWV9BVFRSSUJVVEVfUFJJVkFURSAgICAgICAgICAgKDFVTEwg PDwgMykKPiA+ID4gPiArCj4gPiA+Cj4gPiA+IEluIHBLVk0sIHdlIGRvbid0IHdhbnQgdG8gYWxs b3cgc2V0dGluZyAob3IgY2xlYXJpbmcpIG9mIFBSSVZBVEUvU0hBUkVECj4gPiA+IGF0dHJpYnV0 ZXMgZnJvbSB1c2Vyc3BhY2UuCj4gPgo+ID4gV2h5IG5vdD8gIFRoZSB3aG9sZSB0aGluZyBmYWxs cyBhcGFydCBpZiB1c2Vyc3BhY2UgZG9lc24ndCAqa25vdyogdGhlIHN0YXRlIG9mIGEKPiA+IHBh Z2UsIGFuZCB0aGUgb25seSB3YXkgZm9yIHVzZXJzcGFjZSB0byBrbm93IHRoZSBzdGF0ZSBvZiBh IHBhZ2UgYXQgYSBnaXZlbiBtb21lbnQKPiA+IGluIHRpbWUgaXMgaWYgdXNlcnNwYWNlIGNvbnRy b2xzIHRoZSBhdHRyaWJ1dGVzLiAgRS5nLiBldmVuIGlmIEtWTSB3ZXJlIHRvIHByb3ZpZGUKPiA+ IGEgd2F5IGZvciB1c2Vyc3BhY2UgdG8gcXVlcnkgYXR0cmlidXRlcywgdGhlIGF0dHJpYnV0ZXMg ZXhwb3NlZCB0byB1c3JzcGFjZSB3b3VsZAo+ID4gYmVjb21lIHN0YWxlIHRoZSBpbnN0YW50IEtW TSBkcm9wcyBzbG90c19sb2NrIChvciB3aGF0ZXZlciBsb2NrIHByb3RlY3RzIHRoZSBhdHRyaWJ1 dGVzKQo+ID4gc2luY2UgdXNlcnNwYWNlIGNvdWxkbid0IHByZXZlbnQgZnV0dXJlIGNoYW5nZXMu Cj4gCj4gSSB0aGluayBJIG1pZ2h0IG5vdCBxdWl0ZSB1bmRlcnN0YW5kIHRoZSBwdXJwb3NlIG9m IHRoZQo+IEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgQUJJLiBJbiBwS1ZNLCBhbGwgb2YgYSBw cm90ZWN0ZWQgZ3Vlc3QncyBtZW1vcnkgaXMKPiBwcml2YXRlIGJ5IGRlZmF1bHQsIHVudGlsIHRo ZSBndWVzdCBzaGFyZXMgaXQgd2l0aCB0aGUgaG9zdCAodmlhIGEKPiBoeXBlcmNhbGwpLCBvciBh bm90aGVyIGd1ZXN0IChmdXR1cmUgd29yaykuIFdoZW4gdGhlIGd1ZXN0IHNoYXJlcyBpdCwKPiB1 c2Vyc3BhY2UgaXMgbm90aWZpZWQgdmlhIEtWTV9FWElUX0hZUEVSQ0FMTC4gSW4gbWFueSB1c2Ug Y2FzZXMsIHVzZXJzcGFjZQo+IGRvZXNuJ3QgbmVlZCB0byBrZWVwIHRyYWNrIGRpcmVjdGx5IG9m IGFsbCBvZiB0aGlzLCBidXQgY2FuIHJlYWN0aXZlbHkgdW4vbWFwCj4gdGhlIG1lbW9yeSBiZWlu ZyB1bi9zaGFyZWQuCgpZZXMsIGFuZCB0aGVuIHVzZXJzcGFjZSBuZWVkcyB0byB0ZWxsIEtWTSwg dmlhIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMsIHRoYXQKdXNlcnNwYWNlIGhhcyBhZ3JlZWQg dG8gY2hhbmdlIHRoZSBzdGF0ZSBvZiB0aGUgcGFnZS4gIFVzZXJzcGFjZSBtYXkgbm90IG5lZWQv d2FudAp0byBleHBsaWNpdGx5IHRyYWNrIHRoZSBzdGF0ZSBvZiBwYWdlcywgYnV0IHVzZXJzcGFj ZSBzdGlsbCBuZWVkcyB0byB0ZWxsIEtWTSB3aGF0CnVzZXJzcGFjZSB3YW50cy4KCktWTSBpcyBw cmltYXJpbHkgYW4gYWNjZWxlcmF0b3IsIGUuZy4gS1ZNJ3Mgcm9sZSBpcyB0byBtYWtlIHRoaW5n cyBnbyBmYXN0IChyZWxhdGl2ZQp0byBkb2luZyB0aGluZ3MgaW4gdXNlcnNwYWNlKSBhbmQgcHJv dmlkZSBhY2Nlc3MgdG8gcmVzb3VyY2VzL2luc3RydWN0aW9ucyB0aGF0CnJlcXVpcmUgZWxldmF0 ZWQgcHJpdmlsZWdlcy4gIEFzIGEgZ2VuZXJhbCBydWxlLCB3ZSB0cnkgdG8gYXZvaWQgZGVmaW5p bmcgdGhlIHZDUFUKbW9kZWwsIHNlY3VyaXR5IHBvbGljaWVzLCBldGMuIGluIEtWTSwgYmVjYXVz ZSBoYXJkY29kaW5nIHBvbGljeSBpbnRvIEtWTSAoYW5kIHRoZQprZXJuZWwgYXMgYSB3aG9sZSkg ZXZlbnR1YWxseSBsaW1pdHMgdGhlIHV0aWxpdHkgb2YgS1ZNLgoKQXMgaXQgcGVydGFpbnMgdG8g UFJJVkFURSB2cy4gU0hBUkVELCBLVk0ncyByb2xlIGlzIHRvIGRlZmluZSBhbmQgZW5mb3JjZSB0 aGUgYmFzaWMKcnVsZXMsIGJ1dCBLVk0gc2hvdWxkbid0IGRvIHRoaW5ncyBsaWtlIGRlZmluZSB3 aGVuIGl0IGlzIChpbClsZWdhbCB0byBjb252ZXJ0Cm1lbW9yeSB0by9mcm9tIFNIQVJFRCwgd2hh dCBwYWdlcyBjYW4gYmUgY29udmVydGVkLCB3aGF0IGhhcHBlbnMgaWYgdGhlIGd1ZXN0IGFuZAp1 c2Vyc3BhY2UgZGlzYWdyZWUsIGV0Yy4KCj4gPiBXaHkgZG9lcyBwS1ZNIG5lZWQgdG8gcHJldmVu dCB1c2Vyc3BhY2UgZnJvbSBzdGF0aW5nICppdHMqIHZpZXcgb2YgYXR0cmlidXRlcz8KPiA+Cj4g PiBJZiB0aGUgZ29hbCBpcyB0byByZWR1Y2UgbWVtb3J5IG92ZXJoZWFkLCB0aGF0IGNhbiBiZSBz b2x2ZWQgYnkgdXNpbmcgYW4gaW50ZXJuYWwsCj4gPiBub24tQUJJIGF0dHJpYnV0ZXMgZmxhZyB0 byB0cmFjayBwS1ZNJ3MgdmlldyBvZiBTSEFSRUQgdnMuIFBSSVZBVEUuICBJZiB0aGUgZ3Vlc3QK PiA+IGF0dGVtcHRzIHRvIGFjY2VzcyBtZW1vcnkgd2hlcmUgcEtWTSBhbmQgdXNlcnNwYWNlIGRv bid0IGFncmVlIG9uIHRoZSBzdGF0ZSwKPiA+IGdlbmVyYXRlIGFuIGV4aXQgdG8gdXNlcnNwYWNl LiAgT3Iga2lsbCB0aGUgZ3Vlc3QuICBPciBkbyBzb21ldGhpbmcgZWxzZSBlbnRpcmVseS4KPiAK PiBGb3IgdGhlIHBLVk0gaHlwZXJ2aXNvciB0aGUgZ3Vlc3QncyB2aWV3IG9mIHRoZSBhdHRyaWJ1 dGVzIGRvZXNuJ3QKPiBtYXR0ZXIuIFRoZSBoeXBlcnZpc29yIGF0IHRoZSBlbmQgb2YgdGhlIGRh eSBpcyB0aGUgdWx0aW1hdGUgYXJiaXRlcgo+IGZvciB3aGF0IGlzIHNoYXJlZCBhbmQgd2l0aCBo b3cuIEZvciBwS1ZNIChhdCBsZWFzdCBpbiBteSBwb3J0IG9mCj4gZ3Vlc3RtZW0pLCB3ZSB1c2Ug dGhlIG1lbW9yeSBhdHRyaWJ1dGVzIGZyb20gZ3Vlc3RtZW0gZXNzZW50aWFsbHkgdG8KPiBjb250 cm9sIHdoaWNoIG1lbW9yeSBjYW4gYmUgbWFwcGVkIGJ5IHRoZSBob3N0LgoKVGhlIGd1ZXN0J3Mg dmlldyBhYnNvbHV0ZWx5IG1hdHRlcnMuICBUaGUgZ3Vlc3QncyB2aWV3IG1heSBub3QgYmUgZXhw cmVzc2VkIGF0CmFjY2VzcyB0aW1lLCBlLmcuIGFzIHlvdSBub3RlIGJlbG93LCBwS1ZNIGFuZCBv dGhlciBzb2Z0d2FyZS1wcm90ZWN0ZWQgVk1zIGRvbid0CmhhdmUgYSBkZWRpY2F0ZWQgc2hhcmVk IHZzLiBwcml2YXRlIGJpdCBsaWtlIFREWCBhbmQgU05QLiAgQnV0IHRoZSB2aWV3IGlzIHN0aWxs CnRoZXJlLCBlLmcuIGluIHRoZSBwS1ZNIG1vZGVsLCB0aGUgZ3Vlc3QgZXhwcmVzc2VzIGl0cyBk ZXNpcmUgZm9yIHNoYXJlZCB2cy4KcHJpdmF0ZSB2aWEgaHlwZXJjYWxsLCBhbmQgSUlSQywgdGhl IGd1ZXN0J3MgdmlldyBpcyB0cmFja2VkIGJ5IHRoZSBoeXBlcnZpc29yCmluIHRoZSBzdGFnZS0y IFBURXMuICBwS1ZNIGl0c2VsZiBtYXkgdHJhY2sgdGhlIGd1ZXN0J3MgdmlldyBvbiB0aGluZ3Ms IGJ1dCB0aGUKdmlldyBpcyBzdGlsbCB0aGUgZ3Vlc3Qncy4KCkUuZy4gaWYgdGhlIGd1ZXN0IHRo aW5rcyBhIHBhZ2UgaXMgcHJpdmF0ZSwgYnV0IGluIHJlYWxpdHkgS1ZNIGFuZCBob3N0IHVzZXJz cGFjZQpoYXZlIGl0IGFzIHNoYXJlZCwgdGhlbiB0aGUgZ3Vlc3QgbWF5IHVuaW50ZW50aW9uYWxs eSBsZWFrIGRhdGEgdG8gdGhlIHVudHJ1c3RlZAp3b3JsZC4KCklJVUMsIHlvdSBoYXZlIGltcGxl bWVudGVkIGd1ZXN0X21lbWZkIHN1cHBvcnQgaW4gcEtWTSBieSBjaGFuZ2luZyB0aGUgYXR0cmli dXRlcwp3aGVuIHRoZSBndWVzdCBtYWtlcyB0aGUgaHlwZXJjYWxsLiAgVGhpcyBjYW4gd29yaywg YnV0IG9ubHkgc28gbG9uZyBhcyB0aGUgZ3Vlc3QKYW5kIHVzZXJzcGFjZSBhcmUgd2VsbC1iZWhh dmVkLCBhbmQgaXQgd2lsbCBsaWtlbHkgcGFpbnQgcEtWTSBpbnRvIGEgY29ybmVyIGluCnRoZSBs b25nIHJ1bi4KCkUuZy4gaWYgdGhlIGd1ZXN0IG1ha2VzIGEgaHlwZXJjYWxsIHRvIGNvbnZlcnQg bWVtb3J5IHRvIFBSSVZBVEUsIGJ1dCB0aGVyZSBpcwpubyBtZW1zbG90IG9yIHRoZSBtZW1zbG90 IGRvZXNuJ3Qgc3VwcG9ydCBwcml2YXRlIG1lbW9yeSwgdGhlbiB1bmxlc3MgdGhlcmUgaXMKcG9s aWN5IGJha2VkIGludG8gS1ZNLCBvciBhbiBBQkkgZm9yIHRoZSBndWVzdDw9Pmhvc3QgaHlwZXJj YWxsIGludGVyZmFjZSB0aGF0CmFsbG93cyB1bndpbmRpbmcgdGhlIHByb2dyYW0gY291bnRlciwg eW91J3JlIHN0dWNrLiAgUmV0dXJuaW5nIGFuIGVycm9yIGZvciB0aGUKaHlwZXJjYWxsIHN0cmFp Z2h0IGZyb20gS1ZNIGlzIHVuZGVzaXJhYmxlIGFzIHRoYXQgd291bGQgcHV0IHBvbGljeSBpbnRv IEtWTSB0aGF0CmRvZXNuJ3QgbmVlZCB0byBiZSB0aGVyZSwgZS5nLiB0aGF0IHdvdWxkIHByZXZl bnQgdXNlcnNwYWNlIGZyb20gbWFuaXB1bGF0aW5nCm1lbXNsb3RzIGluIHJlc3BvbnNlIHRvICh1 bilzaGFyZSByZXF1ZXN0cyBmcm9tIHRoZSBndWVzdC4gIEl0J3MgYSBzaW1pbGFyIHN0b3J5Cmlm IEtWTSBtYXJrcyB0aGUgcGFnZSBhcyBQUklWQVRFLCBhcyB0aGF0IHdvdWxkIHByZXZlbnQgdXNl cnNwYWNlIGZyb20gcmV0dXJuaW5nCmFuIGVycm9yIGZvciB0aGUgaHlwZXJjYWxsLCBpLmUuIHdv dWxkIHByZXZlbnQgdXNlcnNlcGFjZSBmcm9tIGRlbnlpbmcgdGhlIHJlcXVlc3QKdG8gY29udmVy dCB0byBQUklWQVRFLgoKPiBPbmUgZGlmZmVyZW5jZSBiZXR3ZWVuIHBLVk0gYW5kIFREWCAoYXMg SSB1bmRlcnN0YW5kIGl0KSwgaXMgdGhhdCBURFgKPiB1c2VzIHRoZSBtc2Igb2YgdGhlIGd1ZXN0 J3MgSVBBIHRvIGluZGljYXRlIHdoZXRoZXIgbWVtb3J5IGlzIHNoYXJlZAo+IG9yIHByaXZhdGUs IGFuZCB0aGF0IGNhbiBnZW5lcmF0ZSBhIG1pc21hdGNoIG9uIGd1ZXN0IG1lbW9yeSBhY2Nlc3MK PiBiZXR3ZWVuIHdoYXQgaXQgdGhpbmtzIHRoZSBzdGF0ZSBpcywgYW5kIHdoYXQgdGhlIHNoYXJp bmcgc3RhdGUgaW4KPiByZWFsaXR5IGlzLiBwS1ZNIGRvZXNuJ3QgaGF2ZSB0aGF0LiBNZW1vcnkg aXMgcHJpdmF0ZSBieSBkZWZhdWx0LCBhbmQKPiBjYW4gYmUgc2hhcmVkIGluLXBsYWNlLCBib3Ro IGluIHRoZSBndWVzdCdzIElQQSBzcGFjZSBhcyB3ZWxsIGFzIHRoZQo+IHVuZGVybHlpbmcgcGh5 c2ljYWwgcGFnZS4KClREWCdzIHNoYXJlZCBiaXQgYW5kIFNOUCdzIGVuY3J5cHRpb24gYml0IGFy ZSBqdXN0IGEgbWVhbnMgb2YgaGFyZHdhcmUgZW5mb3JjZW1lbnQuCnBLVk0gZG9lcyBoYXZlIGEg aGFyZHdhcmUgYml0IGJlY2F1c2UgaGFyZHdhcmUgZG9lc24ndCBwcm92aWRlIGFueSBlbmZvcmNl bWVudC4KQnV0IGFzIGFib3ZlLCBwS1ZNIGRvZXMgaGF2ZSBhbiBlcXVpdmFsZW50ICpzb21ld2hl cmUqLgoKPiA+ID4gVGhlIG90aGVyIHRoaW5nLCB3aGljaCB3ZSBuZWVkIGZvciBwS1ZNIGFueXdh eSwgaXMgdG8gbWFrZQo+ID4gPiBrdm1fdm1fc2V0X21lbV9hdHRyaWJ1dGVzKCkgZ2xvYmFsLCBz byB0aGF0IGl0IGNhbiBiZSBjYWxsZWQgZnJvbSBvdXRzaWRlIG9mCj4gPiA+IGt2bV9tYWluLmMg KGFscmVhZHkgaGF2ZSBhIGxvY2FsIHBhdGNoIGZvciB0aGlzIHRoYXQgZGVjbGFyZXMgaXQgaW4K PiA+ID4ga3ZtX2hvc3QuaCksCj4gPgo+ID4gVGhhdCdzIG5vIHByb2JsZW0sIGJ1dCBJIGFtIGRl ZmluaXRlbHkgb3Bwb3NlZCB0byBLVk0gbW9kaWZ5aW5nIGF0dHJpYnV0ZXMgdGhhdAo+ID4gYXJl IG93bmVkIGJ5IHVzZXJzcGFjZS4KPiA+Cj4gPiA+IGFuZCBub3QgZ2F0ZSB0aGlzIGZ1bmN0aW9u IGJ5IEtWTV9HRU5FUklDX01FTU9SWV9BVFRSSUJVVEVTLgo+ID4KPiA+IEFzIGFib3ZlLCBJIGFt IG9wcG9zZWQgdG8gcEtWTSBoYXZpbmcgYSBjb21wbGV0ZWx5IGRpZmZlcmVudCBBQkkgZm9yIG1h bmFnaW5nCj4gPiBQUklWQVRFIHZzLiBTSEFSRUQuICBJIGhhdmUgbm8gb2JqZWN0aW9uIHRvIHBL Vk0gdXNpbmcgdW5jbGFpbWVkIGZsYWdzIGluIHRoZQo+ID4gYXR0cmlidXRlcyB0byBzdG9yZSBl eHRyYSBtZXRhZGF0YSwgYnV0IGlmIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgZG9lc24ndCB3 b3JrCj4gPiBmb3IgcEtWTSwgdGhlbiB3ZSd2ZSBmYWlsZWQgbWlzZXJhYmx5IGFuZCBzaG91bGQg cmV2aXN0IHRoZSB1QVBJLgo+IAo+IExpa2UgSSBzYWlkLCBwS1ZNIGRvZXNuJ3QgbmVlZCBhIHVz ZXJzcGFjZSBBQkkgZm9yIG1hbmFnaW5nIFBSSVZBVEUvU0hBUkVELAo+IGp1c3QgYSB3YXkgb2Yg dHJhY2tpbmcgaW4gdGhlIGhvc3Qga2VybmVsIG9mIHdoYXQgaXMgc2hhcmVkIChhcyBvcHBvc2Vk IHRvCj4gdGhlIGh5cGVydmlzb3IsIHdoaWNoIGFscmVhZHkgaGFzIHRoZSBrbm93bGVkZ2UpLiBU aGUgc29sdXRpb24gY291bGQgc2ltcGx5Cj4gYmUgdGhhdCBwS1ZNIGRvZXMgbm90IGVuYWJsZSBL Vk1fR0VORVJJQ19NRU1PUllfQVRUUklCVVRFUywgaGFzIGl0cyBvd24KPiB0cmFja2luZyBvZiB0 aGUgc3RhdHVzIG9mIHRoZSBndWVzdCBwYWdlcywgYW5kIG9ubHkgc2VsZWN0cyBLVk1fUFJJVkFU RV9NRU0uCgpBdCB0aGUgcmlzayBvZiBvdmVyc3RlcHBpbmcgbXkgYm91bmRzLCBJIHRoaW5rIHRo YXQgZWZmZWN0aXZlbHkgZ2l2aW5nIHRoZSBndWVzdApmdWxsIGNvbnRyb2wgb3ZlciB3aGF0IGlz IHNoYXJlZCB2cy4gcHJpdmF0ZSBpcyBhIG1pc3Rha2UuICBJdCBtb3JlIG9yIGxlc3MgbG9ja3MK cEtWTSBpbnRvIGEgc2luZ2xlIG1vZGVsLCBhbmQgZXZlbiB3aXRoaW4gdGhhdCBtb2RlbCwgZGVh bGluZyB3aXRoIGVycm9ycyBhbmQvb3IKbWlzYmVoYXZpbmcgZ3Vlc3RzIGJlY29tZXMgdW5uZWNl c3NhcmlseSBwcm9ibGVtYXRpYy4KClVzaW5nIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgbWF5 IG5vdCBwcm92aWRlIHZhbHVlICp0b2RheSosIGUuZy4gdGhlIHVzZXJzcGFjZQpzaWRlIG9mIHBL Vk0gY291bGQgc2ltcGx5ICJyZWZsZWN0IiBhbGwgY29udmVyc2lvbiBoeXBlcmNhbGxzLCBhbmQg dGVybWluYXRlIHRoZQpWTSBvbiBlcnJvcnMuICBCdXQgdGhlIGNvc3QgaXMgdmVyeSBtaW5pbWFs LCBlLmcuIGEgc2luZ2xlIGV4dHJhIGlvY3RsKCkgcGVyCmNvbnZlcmlvbiwgYW5kIHRoZSB1cHNp ZGUgaXMgdGhhdCBwS1ZNIHdvbid0IGJlIHN0dWNrIGlmIGEgdXNlIGNhc2UgY29tZXMgYWxvbmcK dGhhdCB3YW50cyB0byBnbyBiZXlvbmQgImFsbCBjb252ZXJzaW9uIHJlcXVlc3RzIGVpdGhlciBp bW1lZGlhdGVseSBzdWNjZWVkIG9yCnRlcm1pbmF0ZSB0aGUgZ3Vlc3QiLgoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtcmlzY3YgbWFpbGluZyBs aXN0CmxpbnV4LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVh ZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01680E8FDB7 for ; Tue, 3 Oct 2023 20:52:54 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=umbpPuCX; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S0VQF277pz3cgf for ; Wed, 4 Oct 2023 07:52:53 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=umbpPuCX; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--seanjc.bounces.google.com (client-ip=2607:f8b0:4864:20::1149; helo=mail-yw1-x1149.google.com; envelope-from=36n4czqykdik5rn0wpt11tyr.p1zyv07a22p-qr8yv565.1cyno5.14t@flex--seanjc.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S0VPC1qwkz30f8 for ; Wed, 4 Oct 2023 07:51:57 +1100 (AEDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-59b5a586da6so1991097b3.1 for ; Tue, 03 Oct 2023 13:51:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696366314; x=1696971114; darn=lists.ozlabs.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=umbpPuCXaAGA7P920lVc9pT2Ux0v5kzNdpHoWOUJ0LXBPLjzx+ipFTebT2A+i5DQbx 5s2sLQIWf0+ACwr6FdhSy1BByqn/wgEVxCYeQo9zl6fQ6cEx7QKqlARSFfw1gl4ZMhvw DLSc9OqDPYGhF5pHN2on5yZMuATY37puCMJFh5Zq6dXKdg4CTvHXdq5gqGpbuorlCCkr KJ67HWNOAUKFDB3GF6cAZGFXCc7nNOTNlMpcK/k3B+xJWGkHQcQsN9WA9SzNFTtLKPfo aKw0Ebd47QEqtuEc8dN24W7DXcnIDDJEM+cmAwqW6QTO1a9+GNZHw880F2mvrI1xonW5 FGwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696366314; x=1696971114; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=evKvWGkCT2Rqpx4FVUI27aAY5mwxjv1v3n97ZQ03zyeIJidWK1njMFNVD5ipNoZB9b WIl0zpvAnWnaj5mB9sAwpKGm3/txQ6S8Eehg7T3RV/H0UqzbsX/Zescc4ICGDWrk8DDq 67ZIqBbDJF/OXtljvuB9qLeEFY0xNNkCQ3YYAGMAP7DF6o16KGR+PD11wv/sdEUInDFM 1lzOEZqK2zAtCx34WPKz0CBfDMAGt499aGxjo8ulBZhsWW2VEvINe4Z/jotbcnpKaND4 aEPPRM7oJMXCK+uIxsXiiJwT+S7i+5K6c4GqSlc3BjBCiKKL3Pu35udwNFsz79jVEZ60 wMMg== X-Gm-Message-State: AOJu0YwYXFuRqjFvGk8KjzKKqS1XUyIWYrzo4PoWAbWUvQazB1WO9rxK WN4Epf8qchs/Pma7JFvctMdOUNRAqPA= X-Google-Smtp-Source: AGHT+IH9SpWB5ZYki6B4kneRjXWww1fW5aqwjgYpBvf3LyPRvX5qN4iOqywaVkHZ9PW5pYyi5WCeWQUyODc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:b83:b0:59b:b0b1:d75a with SMTP id ck3-20020a05690c0b8300b0059bb0b1d75amr85511ywb.4.1696366314084; Tue, 03 Oct 2023 13:51:54 -0700 (PDT) Date: Tue, 3 Oct 2023 13:51:52 -0700 In-Reply-To: Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> <20230914015531.1419405-12-seanjc@google.com> Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , Yu Zhang , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chao Peng , linux-riscv@lists.infradead.org, Isaku Yamahata , Paul Moore , Marc Zyngier , Huacai Chen , James Morris , "Matthew Wilcox \(Oracle\)" , Wang , Vlastimil Babka , Jarkko Sakkinen , "Serge E. Hallyn" , Maciej Szmigiero , Albert Ou , Michael Roth , Ackerley Tng , Paul Walmsley , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Isaku Yamahata , Quentin Perret , Liam Merwick , linux-mips@vger.kernel .org, Oliver Upton , linux-security-module@vger.kernel.org, Palmer Dabbelt , "Kirill A . Shutemov" , kvm-riscv@lists.infradead.org, Anup Patel , linux-fsdevel@vger.kernel.org, Paolo Bonzini , Andrew Morton , Vishal Annapurve , linuxppc-dev@lists.ozlabs.org, Xu Yilun , Anish Moorthy Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Tue, Oct 03, 2023, Fuad Tabba wrote: > On Tue, Oct 3, 2023 at 4:59=E2=80=AFPM Sean Christopherson wrote: > > On Tue, Oct 03, 2023, Fuad Tabba wrote: > > > > +#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) > > > > + > > > > > > In pKVM, we don't want to allow setting (or clearing) of PRIVATE/SHAR= ED > > > attributes from userspace. > > > > Why not? The whole thing falls apart if userspace doesn't *know* the s= tate of a > > page, and the only way for userspace to know the state of a page at a g= iven moment > > in time is if userspace controls the attributes. E.g. even if KVM were= to provide > > a way for userspace to query attributes, the attributes exposed to usrs= pace would > > become stale the instant KVM drops slots_lock (or whatever lock protect= s the attributes) > > since userspace couldn't prevent future changes. >=20 > I think I might not quite understand the purpose of the > KVM_SET_MEMORY_ATTRIBUTES ABI. In pKVM, all of a protected guest's memory= is > private by default, until the guest shares it with the host (via a > hypercall), or another guest (future work). When the guest shares it, > userspace is notified via KVM_EXIT_HYPERCALL. In many use cases, userspac= e > doesn't need to keep track directly of all of this, but can reactively un= /map > the memory being un/shared. Yes, and then userspace needs to tell KVM, via KVM_SET_MEMORY_ATTRIBUTES, t= hat userspace has agreed to change the state of the page. Userspace may not ne= ed/want to explicitly track the state of pages, but userspace still needs to tell K= VM what userspace wants. KVM is primarily an accelerator, e.g. KVM's role is to make things go fast = (relative to doing things in userspace) and provide access to resources/instructions = that require elevated privileges. As a general rule, we try to avoid defining t= he vCPU model, security policies, etc. in KVM, because hardcoding policy into KVM (= and the kernel as a whole) eventually limits the utility of KVM. As it pertains to PRIVATE vs. SHARED, KVM's role is to define and enforce t= he basic rules, but KVM shouldn't do things like define when it is (il)legal to conv= ert memory to/from SHARED, what pages can be converted, what happens if the gue= st and userspace disagree, etc. > > Why does pKVM need to prevent userspace from stating *its* view of attr= ibutes? > > > > If the goal is to reduce memory overhead, that can be solved by using a= n internal, > > non-ABI attributes flag to track pKVM's view of SHARED vs. PRIVATE. If= the guest > > attempts to access memory where pKVM and userspace don't agree on the s= tate, > > generate an exit to userspace. Or kill the guest. Or do something els= e entirely. >=20 > For the pKVM hypervisor the guest's view of the attributes doesn't > matter. The hypervisor at the end of the day is the ultimate arbiter > for what is shared and with how. For pKVM (at least in my port of > guestmem), we use the memory attributes from guestmem essentially to > control which memory can be mapped by the host. The guest's view absolutely matters. The guest's view may not be expressed= at access time, e.g. as you note below, pKVM and other software-protected VMs = don't have a dedicated shared vs. private bit like TDX and SNP. But the view is = still there, e.g. in the pKVM model, the guest expresses its desire for shared vs= . private via hypercall, and IIRC, the guest's view is tracked by the hypervi= sor in the stage-2 PTEs. pKVM itself may track the guest's view on things, but= the view is still the guest's. E.g. if the guest thinks a page is private, but in reality KVM and host use= rspace have it as shared, then the guest may unintentionally leak data to the untr= usted world. IIUC, you have implemented guest_memfd support in pKVM by changing the attr= ibutes when the guest makes the hypercall. This can work, but only so long as the= guest and userspace are well-behaved, and it will likely paint pKVM into a corner= in the long run. E.g. if the guest makes a hypercall to convert memory to PRIVATE, but there= is no memslot or the memslot doesn't support private memory, then unless there= is policy baked into KVM, or an ABI for the guest<=3D>host hypercall interface= that allows unwinding the program counter, you're stuck. Returning an error for= the hypercall straight from KVM is undesirable as that would put policy into KV= M that doesn't need to be there, e.g. that would prevent userspace from manipulati= ng memslots in response to (un)share requests from the guest. It's a similar = story if KVM marks the page as PRIVATE, as that would prevent userspace from retu= rning an error for the hypercall, i.e. would prevent usersepace from denying the = request to convert to PRIVATE. > One difference between pKVM and TDX (as I understand it), is that TDX > uses the msb of the guest's IPA to indicate whether memory is shared > or private, and that can generate a mismatch on guest memory access > between what it thinks the state is, and what the sharing state in > reality is. pKVM doesn't have that. Memory is private by default, and > can be shared in-place, both in the guest's IPA space as well as the > underlying physical page. TDX's shared bit and SNP's encryption bit are just a means of hardware enfo= rcement. pKVM does have a hardware bit because hardware doesn't provide any enforcem= ent. But as above, pKVM does have an equivalent *somewhere*. > > > The other thing, which we need for pKVM anyway, is to make > > > kvm_vm_set_mem_attributes() global, so that it can be called from out= side of > > > kvm_main.c (already have a local patch for this that declares it in > > > kvm_host.h), > > > > That's no problem, but I am definitely opposed to KVM modifying attribu= tes that > > are owned by userspace. > > > > > and not gate this function by KVM_GENERIC_MEMORY_ATTRIBUTES. > > > > As above, I am opposed to pKVM having a completely different ABI for ma= naging > > PRIVATE vs. SHARED. I have no objection to pKVM using unclaimed flags = in the > > attributes to store extra metadata, but if KVM_SET_MEMORY_ATTRIBUTES do= esn't work > > for pKVM, then we've failed miserably and should revist the uAPI. >=20 > Like I said, pKVM doesn't need a userspace ABI for managing PRIVATE/SHARE= D, > just a way of tracking in the host kernel of what is shared (as opposed t= o > the hypervisor, which already has the knowledge). The solution could simp= ly > be that pKVM does not enable KVM_GENERIC_MEMORY_ATTRIBUTES, has its own > tracking of the status of the guest pages, and only selects KVM_PRIVATE_M= EM. At the risk of overstepping my bounds, I think that effectively giving the = guest full control over what is shared vs. private is a mistake. It more or less= locks pKVM into a single model, and even within that model, dealing with errors a= nd/or misbehaving guests becomes unnecessarily problematic. Using KVM_SET_MEMORY_ATTRIBUTES may not provide value *today*, e.g. the use= rspace side of pKVM could simply "reflect" all conversion hypercalls, and terminat= e the VM on errors. But the cost is very minimal, e.g. a single extra ioctl() pe= r converion, and the upside is that pKVM won't be stuck if a use case comes a= long that wants to go beyond "all conversion requests either immediately succeed= or terminate the guest". From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 036A7E8FDB6 for ; Tue, 3 Oct 2023 20:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rEQyk5BdqCXswwdLsf7O9KlV9DvjbdSGKo0EbCpjYH8=; b=ev75G9bpwxPyy/DHGY2OOViAft Blqzu/P7ixqPDRLC5Bxh4RAyV1QKPEmspE0oislKg3TRtaWTyvNyugGw8T5sH52k8lRY8BwwcZCZH +U2HqScEkepvOi4iaRXYzjf1wjKjM0iCrXcl1tC+RBF0kEDpcOiPJWd+7GsY69AelG2CihQV5blUG 79WL/VwD8Knc9F6nRt6oKZufXO1EbfFMSE7xZR2qx6pmx1Tl4gRubIfU4eONdSaYGJrbzJBw2OxFg ShlFkpCUSnXra9FjAvFl9HOQr/fGsvFjW4f/65HyqRn1c/jj0msW2DGyTGEojSAZKEFXUtHKekBAH 9h67ZAxA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qnmN8-00FGza-1r; Tue, 03 Oct 2023 20:52:02 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qnmN5-00FGwK-2E for linux-arm-kernel@lists.infradead.org; Tue, 03 Oct 2023 20:52:01 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59b5a586da6so1991057b3.1 for ; Tue, 03 Oct 2023 13:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696366314; x=1696971114; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=gBLRXfZoM+80FxhVym2ng0jITavuKXXpsHNJIaxxKsqco8zjhISRtsDXRutvtrs4Ce h5J1AwxtKJh4rQ+LTa1X9AGZMQyVDZVk/0P3llVsZm+8CP/YMccyXXnxYo/ZMRi9qmAt WfHUjIH5IKvab5BotKCFj/2PRoovvHDlkjUFLTMf9R+xlzcHp32ZsOUTjImLhTEHFAGa DdrgAlrDp9nQpXyRsQ9g/InVdRK6tznEvi59N0uMtDKbjpuTMab63XrC/cLRB/WCHDh5 NBqt25juWlUqeEcp9loigrDdv/5l6//NOKnDxEVaPRDIk+QFs7pw3v5NZy5U/9F4w+AM DyWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696366314; x=1696971114; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=iN51NFJCcwM1r+zCqreeXDah/vTq4bXHZX6Qt4tveaY=; b=jAMssfT7bqddvMosT5g7U5DeD5UaHcyKzXtsbAjsa4uZXsvyp8nXs2VmTt/TLnYDMM KWEUTHYOY1O/NR9oucG2gd7BQqC1ycVz2UAfuxJHXap4MFosgijG6yeyj36ctjZTyP8f 7c+CxOfQeZjavFPnhuFhqLvrxWpT/GhlNftvW+XvpEMBu1a0Os02U/aJOH3/e65YUNCb Bvr8raQ2682y1KdMzE9RXjgTROMiOVDSmf3X59I0sK6OJgrm/rbg7seFs0MkiiKaLjzF 7yWs5qfGLb7ZQeFiIrkr3h70EJxMqWg767KftG0I2X+b0ILruGSHeBEyCI3UPufBHRJK lKGA== X-Gm-Message-State: AOJu0Yzyynid0zlA2beQVAJOYuwNyQj4eeIxJxYpwHiVvMkmu52XpFiW zTXypVTVW89j87tYzTKk6FjMLJ+DPe4= X-Google-Smtp-Source: AGHT+IH9SpWB5ZYki6B4kneRjXWww1fW5aqwjgYpBvf3LyPRvX5qN4iOqywaVkHZ9PW5pYyi5WCeWQUyODc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:b83:b0:59b:b0b1:d75a with SMTP id ck3-20020a05690c0b8300b0059bb0b1d75amr85511ywb.4.1696366314084; Tue, 03 Oct 2023 13:51:54 -0700 (PDT) Date: Tue, 3 Oct 2023 13:51:52 -0700 In-Reply-To: Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> <20230914015531.1419405-12-seanjc@google.com> Message-ID: Subject: Re: [RFC PATCH v12 11/33] KVM: Introduce per-page memory attributes From: Sean Christopherson To: Fuad Tabba Cc: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, linux-kernel@vger.kernel.org, Chao Peng , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231003_135159_733383_A8DFD9DE X-CRM114-Status: GOOD ( 47.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBPY3QgMDMsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gT24gVHVlLCBPY3QgMywg MjAyMyBhdCA0OjU54oCvUE0gU2VhbiBDaHJpc3RvcGhlcnNvbiA8c2VhbmpjQGdvb2dsZS5jb20+ IHdyb3RlOgo+ID4gT24gVHVlLCBPY3QgMDMsIDIwMjMsIEZ1YWQgVGFiYmEgd3JvdGU6Cj4gPiA+ ID4gKyNkZWZpbmUgS1ZNX01FTU9SWV9BVFRSSUJVVEVfUFJJVkFURSAgICAgICAgICAgKDFVTEwg PDwgMykKPiA+ID4gPiArCj4gPiA+Cj4gPiA+IEluIHBLVk0sIHdlIGRvbid0IHdhbnQgdG8gYWxs b3cgc2V0dGluZyAob3IgY2xlYXJpbmcpIG9mIFBSSVZBVEUvU0hBUkVECj4gPiA+IGF0dHJpYnV0 ZXMgZnJvbSB1c2Vyc3BhY2UuCj4gPgo+ID4gV2h5IG5vdD8gIFRoZSB3aG9sZSB0aGluZyBmYWxs cyBhcGFydCBpZiB1c2Vyc3BhY2UgZG9lc24ndCAqa25vdyogdGhlIHN0YXRlIG9mIGEKPiA+IHBh Z2UsIGFuZCB0aGUgb25seSB3YXkgZm9yIHVzZXJzcGFjZSB0byBrbm93IHRoZSBzdGF0ZSBvZiBh IHBhZ2UgYXQgYSBnaXZlbiBtb21lbnQKPiA+IGluIHRpbWUgaXMgaWYgdXNlcnNwYWNlIGNvbnRy b2xzIHRoZSBhdHRyaWJ1dGVzLiAgRS5nLiBldmVuIGlmIEtWTSB3ZXJlIHRvIHByb3ZpZGUKPiA+ IGEgd2F5IGZvciB1c2Vyc3BhY2UgdG8gcXVlcnkgYXR0cmlidXRlcywgdGhlIGF0dHJpYnV0ZXMg ZXhwb3NlZCB0byB1c3JzcGFjZSB3b3VsZAo+ID4gYmVjb21lIHN0YWxlIHRoZSBpbnN0YW50IEtW TSBkcm9wcyBzbG90c19sb2NrIChvciB3aGF0ZXZlciBsb2NrIHByb3RlY3RzIHRoZSBhdHRyaWJ1 dGVzKQo+ID4gc2luY2UgdXNlcnNwYWNlIGNvdWxkbid0IHByZXZlbnQgZnV0dXJlIGNoYW5nZXMu Cj4gCj4gSSB0aGluayBJIG1pZ2h0IG5vdCBxdWl0ZSB1bmRlcnN0YW5kIHRoZSBwdXJwb3NlIG9m IHRoZQo+IEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgQUJJLiBJbiBwS1ZNLCBhbGwgb2YgYSBw cm90ZWN0ZWQgZ3Vlc3QncyBtZW1vcnkgaXMKPiBwcml2YXRlIGJ5IGRlZmF1bHQsIHVudGlsIHRo ZSBndWVzdCBzaGFyZXMgaXQgd2l0aCB0aGUgaG9zdCAodmlhIGEKPiBoeXBlcmNhbGwpLCBvciBh bm90aGVyIGd1ZXN0IChmdXR1cmUgd29yaykuIFdoZW4gdGhlIGd1ZXN0IHNoYXJlcyBpdCwKPiB1 c2Vyc3BhY2UgaXMgbm90aWZpZWQgdmlhIEtWTV9FWElUX0hZUEVSQ0FMTC4gSW4gbWFueSB1c2Ug Y2FzZXMsIHVzZXJzcGFjZQo+IGRvZXNuJ3QgbmVlZCB0byBrZWVwIHRyYWNrIGRpcmVjdGx5IG9m IGFsbCBvZiB0aGlzLCBidXQgY2FuIHJlYWN0aXZlbHkgdW4vbWFwCj4gdGhlIG1lbW9yeSBiZWlu ZyB1bi9zaGFyZWQuCgpZZXMsIGFuZCB0aGVuIHVzZXJzcGFjZSBuZWVkcyB0byB0ZWxsIEtWTSwg dmlhIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMsIHRoYXQKdXNlcnNwYWNlIGhhcyBhZ3JlZWQg dG8gY2hhbmdlIHRoZSBzdGF0ZSBvZiB0aGUgcGFnZS4gIFVzZXJzcGFjZSBtYXkgbm90IG5lZWQv d2FudAp0byBleHBsaWNpdGx5IHRyYWNrIHRoZSBzdGF0ZSBvZiBwYWdlcywgYnV0IHVzZXJzcGFj ZSBzdGlsbCBuZWVkcyB0byB0ZWxsIEtWTSB3aGF0CnVzZXJzcGFjZSB3YW50cy4KCktWTSBpcyBw cmltYXJpbHkgYW4gYWNjZWxlcmF0b3IsIGUuZy4gS1ZNJ3Mgcm9sZSBpcyB0byBtYWtlIHRoaW5n cyBnbyBmYXN0IChyZWxhdGl2ZQp0byBkb2luZyB0aGluZ3MgaW4gdXNlcnNwYWNlKSBhbmQgcHJv dmlkZSBhY2Nlc3MgdG8gcmVzb3VyY2VzL2luc3RydWN0aW9ucyB0aGF0CnJlcXVpcmUgZWxldmF0 ZWQgcHJpdmlsZWdlcy4gIEFzIGEgZ2VuZXJhbCBydWxlLCB3ZSB0cnkgdG8gYXZvaWQgZGVmaW5p bmcgdGhlIHZDUFUKbW9kZWwsIHNlY3VyaXR5IHBvbGljaWVzLCBldGMuIGluIEtWTSwgYmVjYXVz ZSBoYXJkY29kaW5nIHBvbGljeSBpbnRvIEtWTSAoYW5kIHRoZQprZXJuZWwgYXMgYSB3aG9sZSkg ZXZlbnR1YWxseSBsaW1pdHMgdGhlIHV0aWxpdHkgb2YgS1ZNLgoKQXMgaXQgcGVydGFpbnMgdG8g UFJJVkFURSB2cy4gU0hBUkVELCBLVk0ncyByb2xlIGlzIHRvIGRlZmluZSBhbmQgZW5mb3JjZSB0 aGUgYmFzaWMKcnVsZXMsIGJ1dCBLVk0gc2hvdWxkbid0IGRvIHRoaW5ncyBsaWtlIGRlZmluZSB3 aGVuIGl0IGlzIChpbClsZWdhbCB0byBjb252ZXJ0Cm1lbW9yeSB0by9mcm9tIFNIQVJFRCwgd2hh dCBwYWdlcyBjYW4gYmUgY29udmVydGVkLCB3aGF0IGhhcHBlbnMgaWYgdGhlIGd1ZXN0IGFuZAp1 c2Vyc3BhY2UgZGlzYWdyZWUsIGV0Yy4KCj4gPiBXaHkgZG9lcyBwS1ZNIG5lZWQgdG8gcHJldmVu dCB1c2Vyc3BhY2UgZnJvbSBzdGF0aW5nICppdHMqIHZpZXcgb2YgYXR0cmlidXRlcz8KPiA+Cj4g PiBJZiB0aGUgZ29hbCBpcyB0byByZWR1Y2UgbWVtb3J5IG92ZXJoZWFkLCB0aGF0IGNhbiBiZSBz b2x2ZWQgYnkgdXNpbmcgYW4gaW50ZXJuYWwsCj4gPiBub24tQUJJIGF0dHJpYnV0ZXMgZmxhZyB0 byB0cmFjayBwS1ZNJ3MgdmlldyBvZiBTSEFSRUQgdnMuIFBSSVZBVEUuICBJZiB0aGUgZ3Vlc3QK PiA+IGF0dGVtcHRzIHRvIGFjY2VzcyBtZW1vcnkgd2hlcmUgcEtWTSBhbmQgdXNlcnNwYWNlIGRv bid0IGFncmVlIG9uIHRoZSBzdGF0ZSwKPiA+IGdlbmVyYXRlIGFuIGV4aXQgdG8gdXNlcnNwYWNl LiAgT3Iga2lsbCB0aGUgZ3Vlc3QuICBPciBkbyBzb21ldGhpbmcgZWxzZSBlbnRpcmVseS4KPiAK PiBGb3IgdGhlIHBLVk0gaHlwZXJ2aXNvciB0aGUgZ3Vlc3QncyB2aWV3IG9mIHRoZSBhdHRyaWJ1 dGVzIGRvZXNuJ3QKPiBtYXR0ZXIuIFRoZSBoeXBlcnZpc29yIGF0IHRoZSBlbmQgb2YgdGhlIGRh eSBpcyB0aGUgdWx0aW1hdGUgYXJiaXRlcgo+IGZvciB3aGF0IGlzIHNoYXJlZCBhbmQgd2l0aCBo b3cuIEZvciBwS1ZNIChhdCBsZWFzdCBpbiBteSBwb3J0IG9mCj4gZ3Vlc3RtZW0pLCB3ZSB1c2Ug dGhlIG1lbW9yeSBhdHRyaWJ1dGVzIGZyb20gZ3Vlc3RtZW0gZXNzZW50aWFsbHkgdG8KPiBjb250 cm9sIHdoaWNoIG1lbW9yeSBjYW4gYmUgbWFwcGVkIGJ5IHRoZSBob3N0LgoKVGhlIGd1ZXN0J3Mg dmlldyBhYnNvbHV0ZWx5IG1hdHRlcnMuICBUaGUgZ3Vlc3QncyB2aWV3IG1heSBub3QgYmUgZXhw cmVzc2VkIGF0CmFjY2VzcyB0aW1lLCBlLmcuIGFzIHlvdSBub3RlIGJlbG93LCBwS1ZNIGFuZCBv dGhlciBzb2Z0d2FyZS1wcm90ZWN0ZWQgVk1zIGRvbid0CmhhdmUgYSBkZWRpY2F0ZWQgc2hhcmVk IHZzLiBwcml2YXRlIGJpdCBsaWtlIFREWCBhbmQgU05QLiAgQnV0IHRoZSB2aWV3IGlzIHN0aWxs CnRoZXJlLCBlLmcuIGluIHRoZSBwS1ZNIG1vZGVsLCB0aGUgZ3Vlc3QgZXhwcmVzc2VzIGl0cyBk ZXNpcmUgZm9yIHNoYXJlZCB2cy4KcHJpdmF0ZSB2aWEgaHlwZXJjYWxsLCBhbmQgSUlSQywgdGhl IGd1ZXN0J3MgdmlldyBpcyB0cmFja2VkIGJ5IHRoZSBoeXBlcnZpc29yCmluIHRoZSBzdGFnZS0y IFBURXMuICBwS1ZNIGl0c2VsZiBtYXkgdHJhY2sgdGhlIGd1ZXN0J3MgdmlldyBvbiB0aGluZ3Ms IGJ1dCB0aGUKdmlldyBpcyBzdGlsbCB0aGUgZ3Vlc3Qncy4KCkUuZy4gaWYgdGhlIGd1ZXN0IHRo aW5rcyBhIHBhZ2UgaXMgcHJpdmF0ZSwgYnV0IGluIHJlYWxpdHkgS1ZNIGFuZCBob3N0IHVzZXJz cGFjZQpoYXZlIGl0IGFzIHNoYXJlZCwgdGhlbiB0aGUgZ3Vlc3QgbWF5IHVuaW50ZW50aW9uYWxs eSBsZWFrIGRhdGEgdG8gdGhlIHVudHJ1c3RlZAp3b3JsZC4KCklJVUMsIHlvdSBoYXZlIGltcGxl bWVudGVkIGd1ZXN0X21lbWZkIHN1cHBvcnQgaW4gcEtWTSBieSBjaGFuZ2luZyB0aGUgYXR0cmli dXRlcwp3aGVuIHRoZSBndWVzdCBtYWtlcyB0aGUgaHlwZXJjYWxsLiAgVGhpcyBjYW4gd29yaywg YnV0IG9ubHkgc28gbG9uZyBhcyB0aGUgZ3Vlc3QKYW5kIHVzZXJzcGFjZSBhcmUgd2VsbC1iZWhh dmVkLCBhbmQgaXQgd2lsbCBsaWtlbHkgcGFpbnQgcEtWTSBpbnRvIGEgY29ybmVyIGluCnRoZSBs b25nIHJ1bi4KCkUuZy4gaWYgdGhlIGd1ZXN0IG1ha2VzIGEgaHlwZXJjYWxsIHRvIGNvbnZlcnQg bWVtb3J5IHRvIFBSSVZBVEUsIGJ1dCB0aGVyZSBpcwpubyBtZW1zbG90IG9yIHRoZSBtZW1zbG90 IGRvZXNuJ3Qgc3VwcG9ydCBwcml2YXRlIG1lbW9yeSwgdGhlbiB1bmxlc3MgdGhlcmUgaXMKcG9s aWN5IGJha2VkIGludG8gS1ZNLCBvciBhbiBBQkkgZm9yIHRoZSBndWVzdDw9Pmhvc3QgaHlwZXJj YWxsIGludGVyZmFjZSB0aGF0CmFsbG93cyB1bndpbmRpbmcgdGhlIHByb2dyYW0gY291bnRlciwg eW91J3JlIHN0dWNrLiAgUmV0dXJuaW5nIGFuIGVycm9yIGZvciB0aGUKaHlwZXJjYWxsIHN0cmFp Z2h0IGZyb20gS1ZNIGlzIHVuZGVzaXJhYmxlIGFzIHRoYXQgd291bGQgcHV0IHBvbGljeSBpbnRv IEtWTSB0aGF0CmRvZXNuJ3QgbmVlZCB0byBiZSB0aGVyZSwgZS5nLiB0aGF0IHdvdWxkIHByZXZl bnQgdXNlcnNwYWNlIGZyb20gbWFuaXB1bGF0aW5nCm1lbXNsb3RzIGluIHJlc3BvbnNlIHRvICh1 bilzaGFyZSByZXF1ZXN0cyBmcm9tIHRoZSBndWVzdC4gIEl0J3MgYSBzaW1pbGFyIHN0b3J5Cmlm IEtWTSBtYXJrcyB0aGUgcGFnZSBhcyBQUklWQVRFLCBhcyB0aGF0IHdvdWxkIHByZXZlbnQgdXNl cnNwYWNlIGZyb20gcmV0dXJuaW5nCmFuIGVycm9yIGZvciB0aGUgaHlwZXJjYWxsLCBpLmUuIHdv dWxkIHByZXZlbnQgdXNlcnNlcGFjZSBmcm9tIGRlbnlpbmcgdGhlIHJlcXVlc3QKdG8gY29udmVy dCB0byBQUklWQVRFLgoKPiBPbmUgZGlmZmVyZW5jZSBiZXR3ZWVuIHBLVk0gYW5kIFREWCAoYXMg SSB1bmRlcnN0YW5kIGl0KSwgaXMgdGhhdCBURFgKPiB1c2VzIHRoZSBtc2Igb2YgdGhlIGd1ZXN0 J3MgSVBBIHRvIGluZGljYXRlIHdoZXRoZXIgbWVtb3J5IGlzIHNoYXJlZAo+IG9yIHByaXZhdGUs IGFuZCB0aGF0IGNhbiBnZW5lcmF0ZSBhIG1pc21hdGNoIG9uIGd1ZXN0IG1lbW9yeSBhY2Nlc3MK PiBiZXR3ZWVuIHdoYXQgaXQgdGhpbmtzIHRoZSBzdGF0ZSBpcywgYW5kIHdoYXQgdGhlIHNoYXJp bmcgc3RhdGUgaW4KPiByZWFsaXR5IGlzLiBwS1ZNIGRvZXNuJ3QgaGF2ZSB0aGF0LiBNZW1vcnkg aXMgcHJpdmF0ZSBieSBkZWZhdWx0LCBhbmQKPiBjYW4gYmUgc2hhcmVkIGluLXBsYWNlLCBib3Ro IGluIHRoZSBndWVzdCdzIElQQSBzcGFjZSBhcyB3ZWxsIGFzIHRoZQo+IHVuZGVybHlpbmcgcGh5 c2ljYWwgcGFnZS4KClREWCdzIHNoYXJlZCBiaXQgYW5kIFNOUCdzIGVuY3J5cHRpb24gYml0IGFy ZSBqdXN0IGEgbWVhbnMgb2YgaGFyZHdhcmUgZW5mb3JjZW1lbnQuCnBLVk0gZG9lcyBoYXZlIGEg aGFyZHdhcmUgYml0IGJlY2F1c2UgaGFyZHdhcmUgZG9lc24ndCBwcm92aWRlIGFueSBlbmZvcmNl bWVudC4KQnV0IGFzIGFib3ZlLCBwS1ZNIGRvZXMgaGF2ZSBhbiBlcXVpdmFsZW50ICpzb21ld2hl cmUqLgoKPiA+ID4gVGhlIG90aGVyIHRoaW5nLCB3aGljaCB3ZSBuZWVkIGZvciBwS1ZNIGFueXdh eSwgaXMgdG8gbWFrZQo+ID4gPiBrdm1fdm1fc2V0X21lbV9hdHRyaWJ1dGVzKCkgZ2xvYmFsLCBz byB0aGF0IGl0IGNhbiBiZSBjYWxsZWQgZnJvbSBvdXRzaWRlIG9mCj4gPiA+IGt2bV9tYWluLmMg KGFscmVhZHkgaGF2ZSBhIGxvY2FsIHBhdGNoIGZvciB0aGlzIHRoYXQgZGVjbGFyZXMgaXQgaW4K PiA+ID4ga3ZtX2hvc3QuaCksCj4gPgo+ID4gVGhhdCdzIG5vIHByb2JsZW0sIGJ1dCBJIGFtIGRl ZmluaXRlbHkgb3Bwb3NlZCB0byBLVk0gbW9kaWZ5aW5nIGF0dHJpYnV0ZXMgdGhhdAo+ID4gYXJl IG93bmVkIGJ5IHVzZXJzcGFjZS4KPiA+Cj4gPiA+IGFuZCBub3QgZ2F0ZSB0aGlzIGZ1bmN0aW9u IGJ5IEtWTV9HRU5FUklDX01FTU9SWV9BVFRSSUJVVEVTLgo+ID4KPiA+IEFzIGFib3ZlLCBJIGFt IG9wcG9zZWQgdG8gcEtWTSBoYXZpbmcgYSBjb21wbGV0ZWx5IGRpZmZlcmVudCBBQkkgZm9yIG1h bmFnaW5nCj4gPiBQUklWQVRFIHZzLiBTSEFSRUQuICBJIGhhdmUgbm8gb2JqZWN0aW9uIHRvIHBL Vk0gdXNpbmcgdW5jbGFpbWVkIGZsYWdzIGluIHRoZQo+ID4gYXR0cmlidXRlcyB0byBzdG9yZSBl eHRyYSBtZXRhZGF0YSwgYnV0IGlmIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgZG9lc24ndCB3 b3JrCj4gPiBmb3IgcEtWTSwgdGhlbiB3ZSd2ZSBmYWlsZWQgbWlzZXJhYmx5IGFuZCBzaG91bGQg cmV2aXN0IHRoZSB1QVBJLgo+IAo+IExpa2UgSSBzYWlkLCBwS1ZNIGRvZXNuJ3QgbmVlZCBhIHVz ZXJzcGFjZSBBQkkgZm9yIG1hbmFnaW5nIFBSSVZBVEUvU0hBUkVELAo+IGp1c3QgYSB3YXkgb2Yg dHJhY2tpbmcgaW4gdGhlIGhvc3Qga2VybmVsIG9mIHdoYXQgaXMgc2hhcmVkIChhcyBvcHBvc2Vk IHRvCj4gdGhlIGh5cGVydmlzb3IsIHdoaWNoIGFscmVhZHkgaGFzIHRoZSBrbm93bGVkZ2UpLiBU aGUgc29sdXRpb24gY291bGQgc2ltcGx5Cj4gYmUgdGhhdCBwS1ZNIGRvZXMgbm90IGVuYWJsZSBL Vk1fR0VORVJJQ19NRU1PUllfQVRUUklCVVRFUywgaGFzIGl0cyBvd24KPiB0cmFja2luZyBvZiB0 aGUgc3RhdHVzIG9mIHRoZSBndWVzdCBwYWdlcywgYW5kIG9ubHkgc2VsZWN0cyBLVk1fUFJJVkFU RV9NRU0uCgpBdCB0aGUgcmlzayBvZiBvdmVyc3RlcHBpbmcgbXkgYm91bmRzLCBJIHRoaW5rIHRo YXQgZWZmZWN0aXZlbHkgZ2l2aW5nIHRoZSBndWVzdApmdWxsIGNvbnRyb2wgb3ZlciB3aGF0IGlz IHNoYXJlZCB2cy4gcHJpdmF0ZSBpcyBhIG1pc3Rha2UuICBJdCBtb3JlIG9yIGxlc3MgbG9ja3MK cEtWTSBpbnRvIGEgc2luZ2xlIG1vZGVsLCBhbmQgZXZlbiB3aXRoaW4gdGhhdCBtb2RlbCwgZGVh bGluZyB3aXRoIGVycm9ycyBhbmQvb3IKbWlzYmVoYXZpbmcgZ3Vlc3RzIGJlY29tZXMgdW5uZWNl c3NhcmlseSBwcm9ibGVtYXRpYy4KClVzaW5nIEtWTV9TRVRfTUVNT1JZX0FUVFJJQlVURVMgbWF5 IG5vdCBwcm92aWRlIHZhbHVlICp0b2RheSosIGUuZy4gdGhlIHVzZXJzcGFjZQpzaWRlIG9mIHBL Vk0gY291bGQgc2ltcGx5ICJyZWZsZWN0IiBhbGwgY29udmVyc2lvbiBoeXBlcmNhbGxzLCBhbmQg dGVybWluYXRlIHRoZQpWTSBvbiBlcnJvcnMuICBCdXQgdGhlIGNvc3QgaXMgdmVyeSBtaW5pbWFs LCBlLmcuIGEgc2luZ2xlIGV4dHJhIGlvY3RsKCkgcGVyCmNvbnZlcmlvbiwgYW5kIHRoZSB1cHNp ZGUgaXMgdGhhdCBwS1ZNIHdvbid0IGJlIHN0dWNrIGlmIGEgdXNlIGNhc2UgY29tZXMgYWxvbmcK dGhhdCB3YW50cyB0byBnbyBiZXlvbmQgImFsbCBjb252ZXJzaW9uIHJlcXVlc3RzIGVpdGhlciBp bW1lZGlhdGVseSBzdWNjZWVkIG9yCnRlcm1pbmF0ZSB0aGUgZ3Vlc3QiLgoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWls aW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0 cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=