From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12AD4C433EF for ; Wed, 15 Sep 2021 20:04:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 84F55610A6 for ; Wed, 15 Sep 2021 20:04:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 84F55610A6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0191E6B0071; Wed, 15 Sep 2021 16:04:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F094F6B0072; Wed, 15 Sep 2021 16:04:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD0FA6B0073; Wed, 15 Sep 2021 16:04:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id CDDD76B0071 for ; Wed, 15 Sep 2021 16:04:52 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4409D184008B6 for ; Wed, 15 Sep 2021 20:04:52 +0000 (UTC) X-FDA: 78590886024.07.1883CCA Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by imf14.hostedemail.com (Postfix) with ESMTP id EDE966001983 for ; Wed, 15 Sep 2021 20:04:51 +0000 (UTC) Received: by mail-lf1-f50.google.com with SMTP id y28so9105067lfb.0 for ; Wed, 15 Sep 2021 13:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=PJmcziiXmV8RIDbZ5D5UGsw521UNmkikWzjsy2xatzI=; b=qYfsUO72jzT4TWWeJIe6bnU4+xaGNqkuwhxq7qPUJWGcVmmE5nrH7w4PwgOVP/45i6 D6JCUo6tVAYsJy51Xvn+OiLGsnYNha2MvjzXv8QYcO3NdMMAV2fcBogsClVqHnnVc+HB EUuJGikXwdM6Ga3rUvHG502bJiXyBCWfkOnQ1JuFr5XXVCh+Zxfk9Q1F10pmP0vagj41 bHQvOeDYwZh2UkF9bB58Wc/gHQwSTrRs4iZLWMinpicl1khA6fit3JvQSwMDRvmXheqF wkLv1G99JxZdvfklwZSY6QAuLXEdLfO2HVMQvfBXlR4G/HjFD5sqRF1Ow6kcW2PlrVYg +AZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=PJmcziiXmV8RIDbZ5D5UGsw521UNmkikWzjsy2xatzI=; b=CBz3I3zSsY53RNmDf1O5FtwQ12HZ0n/ExkebbxRFbAb84fRCyTjezTHPun0trNp2wQ 7Zn3ABSCnJkA/MLJG9u3bEMEgy/xtbGEEJUO2/yMUK7d7QbbYSE8yA8q6SE+wwr6jIwB mDfSG4iqkM+uay3oY0teECWUk2+/wn0JJk6RMjK5St49sKaOPD41GOi2QuasUGo6BqpI GeDDAq1pDkCqUtbbiNWBhRio6V9KLv8m0Wpr3Kb9V7SIQgscBVJs2ggrY5vhzvMrJlPx WVSLqgt1whGFjbtkYgDPCb083uoveix2SRY06f3SoNPSj5DyDXNLkoeZ/8hYxqOfVXAo NuGg== X-Gm-Message-State: AOAM531i5iao2Bg1zdC/M669qNYgG3MkCfv97eYM+P+wdi2q9gbZyH+B RjsWCFtr7YkH8lIXFuBi5+f63g== X-Google-Smtp-Source: ABdhPJxjPfwH2g9ak/eK9haf/qlkdtExND36Ul9ulD+4sIjr7HF+ctNlv6IsgFBwf9d9RA8jxd6Xjg== X-Received: by 2002:a05:6512:1329:: with SMTP id x41mr1241997lfu.9.1631736289779; Wed, 15 Sep 2021 13:04:49 -0700 (PDT) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id t15sm93908ljo.102.2021.09.15.13.04.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 13:04:48 -0700 (PDT) Received: by box.localdomain (Postfix, from userid 1000) id 26905102F9E; Wed, 15 Sep 2021 23:04:52 +0300 (+03) Date: Wed, 15 Sep 2021 23:04:52 +0300 From: "Kirill A. Shutemov" To: David Hildenbrand Cc: "Kirill A. Shutemov" , Chao Peng , Andy Lutomirski , Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Borislav Petkov , Andrew Morton , Joerg Roedel , Andi Kleen , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Varad Gautam , Dario Faggioli , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, Kuppuswamy Sathyanarayanan , Dave Hansen , Yu Zhang Subject: Re: [RFC] KVM: mm: fd-based approach for supporting KVM guest private memory Message-ID: <20210915200452.wp6ippdvjz6zpv6a@box.shutemov.name> References: <20210824005248.200037-1-seanjc@google.com> <20210902184711.7v65p5lwhpr2pvk7@box.shutemov.name> <20210903191414.g7tfzsbzc7tpkx37@box.shutemov.name> <02806f62-8820-d5f9-779c-15c0e9cd0e85@kernel.org> <20210910171811.xl3lms6xoj3kx223@box.shutemov.name> <20210915195857.GA52522@chaop.bj.intel.com> <51a6f74f-6c05-74b9-3fd7-b7cd900fb8cc@redhat.com> <20210915142921.bxxsap6xktkt4bek@black.fi.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EDE966001983 X-Stat-Signature: 3tatdpiuxuzmjzsm7dq15hog89peufu7 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=shutemov-name.20150623.gappssmtp.com header.s=20150623 header.b=qYfsUO72; spf=none (imf14.hostedemail.com: domain of kirill@shutemov.name has no SPF policy when checking 209.85.167.50) smtp.mailfrom=kirill@shutemov.name; dmarc=none X-HE-Tag: 1631736291-571028 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Sep 15, 2021 at 04:59:46PM +0200, David Hildenbrand wrote: > > > > I don't think we are, it still feels like we are in the early prototype > > > phase (even way before a PoC). I'd be happy to see something "cleaner" so to > > > say -- it still feels kind of hacky to me, especially there seem to be many > > > pieces of the big puzzle missing so far. Unfortunately, this series hasn't > > > caught the attention of many -MM people so far, maybe because other people > > > miss the big picture as well and are waiting for a complete design proposal. > > > > > > For example, what's unclear to me: we'll be allocating pages with > > > GFP_HIGHUSER_MOVABLE, making them land on MIGRATE_CMA or ZONE_MOVABLE; then > > > we silently turn them unmovable, which breaks these concepts. Who'd migrate > > > these pages away just like when doing long-term pinning, or how is that > > > supposed to work? > > > > That's fair point. We can fix it by changing mapping->gfp_mask. > > That's essentially what secretmem does when setting up a file. > > > > > > Also unclear to me is how refcount and mapcount will be handled to prevent > > > swapping, > > > > refcount and mapcount are unchanged. Pages not pinned per se. Swapping > > prevented with the change in shmem_writepage(). > > So when mapping into the guest, we'd increment the refcount but not the > mapcount I assume? No. The only refcount hold page cache. But we inform KVM via callback before removing the page from the page cache. It is similar to mmu_notifier scheme KVM uses at the moment. > > > > > > who will actually do some kind of gfn-epfn etc. mapping, how we'll > > > forbid access to this memory e.g., via /proc/kcore or when dumping memory > > > > It's not aimed to prevent root to shoot into his leg. Root do root. > > IMHO being root is not an excuse to read some random file (actually used in > production environments) to result in the machine crashing. Not acceptable > for distributions. Reading does not cause problems. Writing does. > I'm still missing the whole gfn-epfn 1:1 mapping discussion we identified as > requirements. Is that supposed to be done by KVM? How? KVM memslots that represents a range of GFNs refers to memfd (and holds file pin) plus offset in the file. This info enough to calculate offset in the file and find PFN. memfd tied 1:1 to struct kvm and KVM would make sure that there's only one possible gfn for a file offset. > > > ... and how it would ever work with migration/swapping/rmap (it's clearly > > > future work, but it's been raised that this would be the way to make it > > > work, I don't quite see how it would all come together). > > > > Given that hardware supports it migration and swapping can be implemented > > by providing new callbacks in guest_ops. Like ->migrate_page would > > transfer encrypted data between pages and ->swapout would provide > > encrypted blob that can be put on disk or handled back to ->swapin to > > bring back to memory. > > Again, I'm missing the complete picture. To make swapping decisions vmscan > code needs track+handle dirty+reference information. How would we be able to > track references? Does the hardware allow for temporary unmapping of > encrypted memory and faulting on it? How would page_referenced() continue > working? "we can add callbacks" is not a satisfying answer, at least for me. > Especially, when it comes to eventual locking problems and races. HW doesn't support swapping yet, so details will be just speculation. IIUC, there's an accessed bit in EPT that can be used for tracking. > Maybe saying "migration+swap is not supported" is clearer than "we can add > callbacks" and missing some details on the bigger picture. > > Again, a complete design proposal would be highly valuable, especially to > get some more review from other -MM folks. Otherwise there is a high chance > that this will be rejected late when trying to upstream and -MM people > stumbling over it (we've had some similar thing happening just recently > unfortunately ...). I only work on core-mm side of the story. We will definitely need to look at whole picture again once all pieces are somewhat ready. -- Kirill A. Shutemov