From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF16C2BB85 for ; Thu, 20 Jun 2024 16:33:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 103E86B0282; Thu, 20 Jun 2024 12:33:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B3758D00C1; Thu, 20 Jun 2024 12:33:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6F408D00AF; Thu, 20 Jun 2024 12:33:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C900F6B0282 for ; Thu, 20 Jun 2024 12:33:21 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 75A30C03A7 for ; Thu, 20 Jun 2024 16:33:21 +0000 (UTC) X-FDA: 82251812202.18.BC5A00A Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf06.hostedemail.com (Postfix) with ESMTP id 6EAA2180011 for ; Thu, 20 Jun 2024 16:33:19 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tUSMUuao; spf=pass (imf06.hostedemail.com: domain of smostafa@google.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=smostafa@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718901191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4B53UjI6PoAGWaCsloRkiHZtuTpGHM2ZWrCUhiNKWNY=; b=a+D3KMcWsD5QcusYYmchlsqCFIdSzRIygfhMuL9+uMIQDLed34ogETOL06i5pwfkNkXsm/ C6G4/uGviKW61iwB2cGjpU/A5qXBlaal14wi2yiHxcV0gfm7yZhnWD9H20GSyoNUt3JoUm X4ZQ/FEJQxAVkle3q3LlkDICzCfk12o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718901191; a=rsa-sha256; cv=none; b=mpDVdkhj9vbeaQsl4ZyOuRpFsJi2O2KdvQeNKGGOh8dlAHznKl9PQY+1zc1x0iCJgROaFI i7buFiJI5l0L3agQanEUkIgu1EVK8fwjd8d+77DtuoTd/yYDmjtfmoPyRhFUWQy5A8GO/A njdvDQzkKne2H2E9lW//AOcVdBLLceA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tUSMUuao; spf=pass (imf06.hostedemail.com: domain of smostafa@google.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=smostafa@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-424786e4056so85e9.0 for ; Thu, 20 Jun 2024 09:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718901198; x=1719505998; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=4B53UjI6PoAGWaCsloRkiHZtuTpGHM2ZWrCUhiNKWNY=; b=tUSMUuaoW+xGs7eWjmPo08/6RJzj7WrAm5O5mhyV5/UULlThYfe2MZtL76xxZckXxJ r9NZJ/8N4shx2gL4yywZunbKY726YKKLm1r+MAz/qJY3XSXSxcvavahlZYYDiTxI9omX uC3TdHQOTYDPd7ZNiiUsojPRmk4/vYvR8830YT/d3XN4EPw9gd7neJtnqLjXoWSogZbu B5oJyi0mjFaBC3N6LkgEtleY0VeekDL8T/i2Lv5mn8dEsktcOIGZxf3cWC7Ca+suKVLt +dNUpxud/jccLEt6tt9m0MAOE+MjmehDElUlAU6DyyWQspz51yu7tcohvqV9+5w8cE7B WZ3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718901198; x=1719505998; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4B53UjI6PoAGWaCsloRkiHZtuTpGHM2ZWrCUhiNKWNY=; b=kdCXiw2PSKWqTMHiNZMzIAZ8ZWlvk+/wTqIeeJxA7DCW6ztquTQrDv4haRkg9dU3/z YHg5xjukOPPthYhSjDb5QtTmW0eimfWIf01THu/IjZiHVuaQWX28r14OxVqLd9vyMUgx PAX9kvacpQ7hTd69+sb06kvESbgIBxyif1W21OEnHbmi8/WOJEye+aVwcUXhkACcw9uh 3cIaxFLSeFu3Ih/iyZ6q69bOUtEd/O+yxxAOSvIJwuJf/ShZlLU7QDaoV1eaGnBFKM74 do2QuTs4y06PcKOQ2J7s2vNSmcouolQU9CZ41sbzmIKvAVpyit59E+W+PHUcUGCGzjts 520g== X-Forwarded-Encrypted: i=1; AJvYcCXLSTpcRH39a2tVxvu2V7NdR5qu/9wL24RDLtptpzX8skXF1aCNjYeNSeTEPjfKB9qDfiJzL9S1jrkp74RhGYrdpb4= X-Gm-Message-State: AOJu0YwbCrYWxDb653EvC39sNHMeEISEXKmku1T+s/OMHeCn4tVbVyEG j49FdTm2/Srrxg3hjDo4+8b0cMqxdoYfCSsJlj0lylJqY339m4zTT+f93NrMKA== X-Google-Smtp-Source: AGHT+IEz2SgAboLf42+BQNW/d36kQQhVc7F1UusAvGsQvMHx+tzIbLyscYR5TmIlxga6H6/BKgdcwQ== X-Received: by 2002:a05:600c:34d6:b0:424:7ac9:4d9a with SMTP id 5b1f17b1804b1-4247ac95619mr2095555e9.1.1718901197463; Thu, 20 Jun 2024 09:33:17 -0700 (PDT) Received: from google.com (205.215.190.35.bc.googleusercontent.com. [35.190.215.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4247d0c9e27sm30767025e9.23.2024.06.20.09.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Jun 2024 09:33:16 -0700 (PDT) Date: Thu, 20 Jun 2024 16:33:12 +0000 From: Mostafa Saleh To: David Hildenbrand Cc: John Hubbard , Elliot Berman , Andrew Morton , Shuah Khan , Matthew Wilcox , maz@kernel.org, kvm@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, pbonzini@redhat.com, Fuad Tabba , Jason Gunthorpe Subject: Re: [PATCH RFC 0/5] mm/gup: Introduce exclusive GUP pinning Message-ID: References: <20240618-exclusive-gup-v1-0-30472a19c5d1@quicinc.com> <7fb8cc2c-916a-43e1-9edf-23ed35e42f51@nvidia.com> <14bd145a-039f-4fb9-8598-384d6a051737@redhat.com> <1ab73f42-9397-4fc7-8e62-2627b945f729@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1ab73f42-9397-4fc7-8e62-2627b945f729@redhat.com> X-Rspamd-Queue-Id: 6EAA2180011 X-Stat-Signature: c7qa84h3hw86sog87upqiqp86a8u3itj X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1718901199-891870 X-HE-Meta: U2FsdGVkX18mpmI0/lc7R+ei0bDB73csx/KnP5HErl/TMH03Vs4AZMgXC7meg12weu+TMMNMtqwqRYajNJ6T6eJO4raAb4Z5r1hgq4neKw9ZcFd/K4K5owTpyS675YOWUu+Ug1KfDq+VLj5HwbAl+RdHRb7SnsTPev2aZagvJxhknS1J+Ym5DVpcagqLzh3IjOxJYiypURANVV4zMZ8X2Qur6b5E5Yqda9yqtLcqgzzE+rpLmV93yYp19HXfaBayMJvNgsLWda7p/cfnJfzT4be5sfifTdm6XLYgdTFN5fdGhs2atjjDwxSgLE204J3CjcqMUhIe6yyjpEMzhbNbYiIG49mTjGEpapd+CJnQiyQsojKjNLF4H1WHx0vQOkBwalABXYVkctFQWwdHGo3jxBnGoG2ujKbC4Uor2KXU38uKi/L1yj72sh06cKLUtoq9DIibWNiTT2MGahxYG7AAE3LTxMReCi4FE86Ao62DtCXbZRDuJ89+4hb4OSt0cXUQsLht+g3euBsXb2ypilW2gVIXEM7mM6i/UaY2JQk+GKpskOl2EXhVcDQskrO12/I24dUcKBkSn6rBXDz9F++Xdt67C/wYWqj0VVzHnWKPUaBphxXfBd9NE+Qb70KctXUlga1aZwgRC4yDqAAAZJT6JaUzQ6f8rvIsi/JnqAocOno011gF5yAXx1A9d4DYzuN8wJpH3ZLemRwWmctLQxbqe7q/SuNxLAqWKwQAbVBka+FzLfMMNAgkkA7nW98VTwbWF3IslaA6N62FmSbaBtr5k/pfiwqggG2XPP7Di6LWipM4F0nNuQ90c9BT++rjwv9KeVqL4e6waPMk8p1XdiMWlvEHzUC8ve85k2fb495iVi0sw/KnkP4PY6aT6nByccFR6DqFarsqqeEVuf4pqJYFEIa9fyXGp9El6p/aMwIzdbJDw054gq84M5xtfTKnM5LIdLTEyCQNeDNxlwK5yYt f0Xd04Jn OZdxFho9XZBGteH6AiUcqjedgj26qs/1Hz1ICLol7wIxKMzqvH7p/SuTGhtSujM1+p3LLgN3/ain9KoxqvuHuuC09WLaCJkAAYyJ1gunj1cmmu4Ff3MNB/Zc2sjExHBb80atVGGmYV9ARbmFJEakr438LxM6hdCAATeuK4K9lhC5Qa+xlPjtOwTF+wTu3HUnLzhmpPYoa+XBalm9J6VWyNd6WXA3S8wty/uxmFoSel4XBY927r4mQjc2s2GP/jbckou/hUFs0E1VXxmTONh0tXB8RSiZl0sIZYRk/fhquaTNzI7FUzXVlrEHY++lExNo1rB7klWc4dBD/nBA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi David, On Thu, Jun 20, 2024 at 04:14:23PM +0200, David Hildenbrand wrote: > On 20.06.24 15:08, Mostafa Saleh wrote: > > Hi David, > > > > On Wed, Jun 19, 2024 at 09:37:58AM +0200, David Hildenbrand wrote: > > > Hi, > > > > > > On 19.06.24 04:44, John Hubbard wrote: > > > > On 6/18/24 5:05 PM, Elliot Berman wrote: > > > > > In arm64 pKVM and QuIC's Gunyah protected VM model, we want to support > > > > > grabbing shmem user pages instead of using KVM's guestmemfd. These > > > > > hypervisors provide a different isolation model than the CoCo > > > > > implementations from x86. KVM's guest_memfd is focused on providing > > > > > memory that is more isolated than AVF requires. Some specific examples > > > > > include ability to pre-load data onto guest-private pages, dynamically > > > > > sharing/isolating guest pages without copy, and (future) migrating > > > > > guest-private pages. In sum of those differences after a discussion in > > > > > [1] and at PUCK, we want to try to stick with existing shmem and extend > > > > > GUP to support the isolation needs for arm64 pKVM and Gunyah. > > > > > > The main question really is, into which direction we want and can develop > > > guest_memfd. At this point (after talking to Jason at LSF/MM), I wonder if > > > guest_memfd should be our new target for guest memory, both shared and > > > private. There are a bunch of issues to be sorted out though ... > > > > > > As there is interest from Red Hat into supporting hugetlb-style huge pages > > > in confidential VMs for real-time workloads, and wasting memory is not > > > really desired, I'm going to think some more about some of the challenges > > > (shared+private in guest_memfd, mmap support, migration of !shared folios, > > > hugetlb-like support, in-place shared<->private conversion, interaction with > > > page pinning). Tricky. > > > > > > Ideally, we'd have one way to back guest memory for confidential VMs in the > > > future. > > > > > > > > > Can you comment on the bigger design goal here? In particular: > > > > > > 1) Who would get the exclusive PIN and for which reason? When would we > > > pin, when would we unpin? > > > > > > 2) What would happen if there is already another PIN? Can we deal with > > > speculative short-term PINs from GUP-fast that could introduce > > > errors? > > > > > > 3) How can we be sure we don't need other long-term pins (IOMMUs?) in > > > the future? > > > > Can you please clarify more about the IOMMU case? > > > > pKVM has no merged upstream IOMMU support at the moment, although > > there was an RFC a while a go [1], also there would be a v2 soon. > > > > In the patches KVM (running in EL2) will manage the IOMMUs including > > the page tables and all pages used in that are allocated from the > > kernel. > > > > These patches don't support IOMMUs for guests. However, I don't see > > why would that be different from the CPU? as once the page is pinned > > it can be owned by a guest and that would be reflected in the > > hypervisor tracking, the CPU stage-2 and IOMMU page tables as well. > > So this is my thinking, it might be flawed: > > In the "normal" world (e.g., vfio), we FOLL_PIN|FOLL_LONGTERM the pages to > be accessible by a dedicated device. We look them up in the page tables to > pin them, then we can map them into the IOMMU. > > Devices that cannot speak "private memory" should only access shared memory. > So we must not have "private memory" mapped into their IOMMU. > > Devices that can speak "private memory" may either access shared or private > memory. So we may have"private memory" mapped into their IOMMU. > Private pages must not be accessible to devices owned by the host, and for that we have the same rules as the CPU: A) The hypervisor doesn’t trust the host, and must enforce that using the CPU stage-2 MMU. B) It’s preferable that userspace doesn’t, and hence these patches (or guest_memfd...) We need the same rules for DMA, otherwise it is "simple" to instrument a DMA attack, so we need a protection by the IOMMU. pKVM at the moment provides 2 ways of establishing that (each has their own trade-off which are not relevant here): 1) pKVM manages the IOMMUs and provides a hypercall interface to map/unmap in the IOMMU, looking at the rules For A), pKVM has its own per-page metadata which tracks page state, which can prevent mapping private pages in the IOMMU and transitioning pages to private if they are mapped in the IOMMU. For B), userspace won’t be able to map private pages(through VFIO/IOMMUFD), as the hypercall interface would fail if the pages are private. This proposal is the one on the list. 2) pKVM manages a second stage of the IOMMU (as SMMUv3), and let the kernel map what it wants in stage-1 and pKVM would use a mirrored page table of the CPU MMU stage-2. For A) Similar to the CPU, stage-2 IOMMU will protect the private pages. For B) userspace can map private pages in the first stage IOMMU, and that would result in stage-2 fault, AFAIK, SMMUv3 is the only Arm implementation that supports nesting in Linux, for that the driver would only print a page fault, and ideally the kernel wouldn’t crash, although that is really hardware dependant how it handle faults, and I guess assigning a device through VFIO to userspace comes with similar risks already (bogus MMIO access can crash the system). This proposal only exists in Android at the moment(However I am working on getting an SMMUv3 compliant implementation that can be posted upstream). > > What I see (again, I might be just wrong): > > 1) How would the device be able to grab/access "private memory", if not > via the user page tables? I hope the above answers the question, but just to confirmn, a device owned by the host shouldn’t access the memory as the host kernel is not trusted and can instrument DMA attacks. Device assignment (passthrough) is another story. > 2) How would we be able to convert shared -> private, if there is a > longterm pin from that IOMMU? We must dynamically unmap it from the > IOMMU. Depending on which solution from the above, for 1) The transition from shared -> private would fail 2) The private page would be unmapped from the stage-2 IOMMU (similar to the stage-2 CPU MMU) > > I assume when you're saying "In the patches KVM (running in EL2) will manage > the IOMMUs including the page tables", this is easily solved by not relying > on pinning: KVM just knows what to update and where. (which is a very > different model than what VFIO does) > Yes, that's is not required to protect private memory. Thanks, Mostafa > Thanks! > > -- > Cheers, > > David / dhildenb >