From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84DFAC4332F for ; Sat, 8 Oct 2022 16:15:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229841AbiJHQPW (ORCPT ); Sat, 8 Oct 2022 12:15:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229609AbiJHQPV (ORCPT ); Sat, 8 Oct 2022 12:15:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ED1431DC7; Sat, 8 Oct 2022 09:15:20 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 34BE160A37; Sat, 8 Oct 2022 16:15:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A207C433D6; Sat, 8 Oct 2022 16:15:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1665245719; bh=NIgNigZexN067z2ad+UvtpUHM+JSZc0JYM0evS6yNx0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=VAGeVVDf6KIOhTWVugZOJTuKPzDIFSGBDrGjIzZCQaYbWM1BU4NBt3yHdh/p+QB8Z Mx6iHaLe+DEGh+bDFLK83w/Rr0LUpsfWq5pCWNpvvFsJhEEnvolwGToq7kVMYm16C7 QcypgNRE4zg87Wqm9iJT8rOYJDbQxSGv57JHugrmcHqPoIxRdFRiHZJqRPFSyCDMtn VuP1L+Zw3IgC/rMVlFF4fXoapnhUapUQB/vBk7S49rXo0meL1hSUbsDRuB/MFvpsc1 IrKiCJ6IemXXuaxmEzEV+cS22e0yEOJl6KXc6pQBJg1bl7a6cOxdRgEbitVsWeCajN r4dGxRV6nD9YQ== Date: Sat, 8 Oct 2022 19:15:13 +0300 From: Jarkko Sakkinen To: Sean Christopherson Cc: Chao Peng , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, qemu-devel@nongnu.org, Paolo Bonzini , Jonathan Corbet , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H . Peter Anvin" , Hugh Dickins , Jeff Layton , "J . Bruce Fields" , Andrew Morton , Shuah Khan , Mike Rapoport , Steven Price , "Maciej S . Szmigiero" , Vlastimil Babka , Vishal Annapurve , Yu Zhang , "Kirill A . Shutemov" , luto@kernel.org, jun.nakajima@intel.com, dave.hansen@intel.com, ak@linux.intel.com, david@redhat.com, aarcange@redhat.com, ddutile@redhat.com, dhildenb@redhat.com, Quentin Perret , Michael Roth , mhocko@suse.com, Muchun Song , wei.w.wang@intel.com Subject: Re: [PATCH v8 2/8] KVM: Extend the memslot to support fd-based private memory Message-ID: References: <20220915142913.2213336-1-chao.p.peng@linux.intel.com> <20220915142913.2213336-3-chao.p.peng@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-api@vger.kernel.org On Sat, Oct 08, 2022 at 12:54:32AM +0300, Jarkko Sakkinen wrote: > On Fri, Oct 07, 2022 at 02:58:54PM +0000, Sean Christopherson wrote: > > On Fri, Oct 07, 2022, Jarkko Sakkinen wrote: > > > On Thu, Oct 06, 2022 at 03:34:58PM +0000, Sean Christopherson wrote: > > > > On Thu, Oct 06, 2022, Jarkko Sakkinen wrote: > > > > > On Thu, Oct 06, 2022 at 05:58:03PM +0300, Jarkko Sakkinen wrote: > > > > > > On Thu, Sep 15, 2022 at 10:29:07PM +0800, Chao Peng wrote: > > > > > > > This new extension, indicated by the new flag KVM_MEM_PRIVATE, adds two > > > > > > > additional KVM memslot fields private_fd/private_offset to allow > > > > > > > userspace to specify that guest private memory provided from the > > > > > > > private_fd and guest_phys_addr mapped at the private_offset of the > > > > > > > private_fd, spanning a range of memory_size. > > > > > > > > > > > > > > The extended memslot can still have the userspace_addr(hva). When use, a > > > > > > > single memslot can maintain both private memory through private > > > > > > > fd(private_fd/private_offset) and shared memory through > > > > > > > hva(userspace_addr). Whether the private or shared part is visible to > > > > > > > guest is maintained by other KVM code. > > > > > > > > > > > > What is anyway the appeal of private_offset field, instead of having just > > > > > > 1:1 association between regions and files, i.e. one memfd per region? > > > > > > > > Modifying memslots is slow, both in KVM and in QEMU (not sure about Google's VMM). > > > > E.g. if a vCPU converts a single page, it will be forced to wait until all other > > > > vCPUs drop SRCU, which can have severe latency spikes, e.g. if KVM is faulting in > > > > memory. KVM's memslot updates also hold a mutex for the entire duration of the > > > > update, i.e. conversions on different vCPUs would be fully serialized, exacerbating > > > > the SRCU problem. > > > > > > > > KVM also has historical baggage where it "needs" to zap _all_ SPTEs when any > > > > memslot is deleted. > > > > > > > > Taking both a private_fd and a shared userspace address allows userspace to convert > > > > between private and shared without having to manipulate memslots. > > > > > > Right, this was really good explanation, thank you. > > > > > > Still wondering could this possibly work (or not): > > > > > > 1. Union userspace_addr and private_fd. > > > > No, because userspace needs to be able to provide both userspace_addr (shared > > memory) and private_fd (private memory) for a single memslot. > > Got it, thanks for clearing my misunderstandings on this topic, and it > is quite obviously visible in 5/8 and 7/8. I.e. if I got it right, > memblock can be partially private, and you dig the shared holes with > KVM_MEMORY_ENCRYPT_UNREG_REGION. We have (in Enarx) ATM have memblock > per host mmap, I was looking into this dilated by that mindset but makes > definitely sense to support that. For me the most useful reference with this feature is kvm_set_phys_mem() implementation in privmem-v8 branch. Took while to find it because I did not have much experience with QEMU code base. I'd even recommend to mention that function in the cover letter because it is really good reference on how this feature is supposed to be used. BR, Jarkko