From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 706DCC4167B for ; Wed, 29 Nov 2023 22:40:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAE108D0022; Wed, 29 Nov 2023 17:40:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5E308D0001; Wed, 29 Nov 2023 17:40:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFE448D0022; Wed, 29 Nov 2023 17:40:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A07F58D0001 for ; Wed, 29 Nov 2023 17:40:25 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6A5FE8061F for ; Wed, 29 Nov 2023 22:40:25 +0000 (UTC) X-FDA: 81512462010.14.1FA8B72 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 1D91C4000A for ; Wed, 29 Nov 2023 22:40:21 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=U2l7dOCe; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 31b1nZQYKCCIQC8LHAEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=31b1nZQYKCCIQC8LHAEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--seanjc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701297622; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LDVeTl61QPCCxPz+sfODK89nbVJMVG6TQMGABDOMFAw=; b=PvA2uoAv096UvkE6pjideUohjyXL2C9mh3R8u0EG8pbMVULWcm8umEGe1JLubWkd9zRFx3 0Y/QDX6zz6ULR3I+Y1hThzA7qe0XXc0tXwG6fsuzCWTXJqOaObz/OwbmbDsvUGDyg15Brf DiN+TTMKoEvQArMROKc6gBqws/DdfaM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=U2l7dOCe; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 31b1nZQYKCCIQC8LHAEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=31b1nZQYKCCIQC8LHAEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--seanjc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701297622; a=rsa-sha256; cv=none; b=GizROhGpn86+lupqp2XSM/MvHLNKGfdvkv8ulYBCB7pOx3ZxWdrbdgKMDL9gKEhZ/6+DNT sHMFdql2s9hibTta6WPNJWXbjRUXve/b4Yb+FKSqFQO1WsPJ6feg8+r1rSPseYhDBqiHZk mBaXSPRkjE1JfBj72YxkUeIaXCkgf+A= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5d1b2153ba1so5736997b3.2 for ; Wed, 29 Nov 2023 14:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701297621; x=1701902421; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=LDVeTl61QPCCxPz+sfODK89nbVJMVG6TQMGABDOMFAw=; b=U2l7dOCe89hMtID2dRl6OADaILfnK+hYJKZQzSV/5hsBaWpcOeSiWEUglkld4BOy5V yLmZgvbXoyQ9WUmghknDzFF7SsYGLZLF92Frr2lcvop5nO27oEd5z91u/7peTDz+7Zpk sZfAdA66hKjwd132AZVnRfPXM3/nZjK37K1ZusMD4q9cDfSwfQCq/RcUBa7bYMwUBR6A v1MbzjxjxBPk3B6LFUwEAfNSPXbYZqhJPJ/oCwzds5JXRZV/Ew18NFUPco3KmTgawpfG NhThLXUf+psOZHl6S1nVOebJYaLPVm4xBmtE8t+rrUJMFxHPv8+mr1uqO4MAdFGqhJXe v6rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701297621; x=1701902421; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=LDVeTl61QPCCxPz+sfODK89nbVJMVG6TQMGABDOMFAw=; b=NBLqqJ1ar+jtB1sRpW3t2W3Sazc86DBEFC19bS+KKK3VMmN6XkVAEP3N9YYtYvWAqP 4Uh/4vQOmGF+hKRZ5QpeSvVmIlmaDfA1hVUzzW+Uybo6xF4Fm/7+wJ1eBLELFQ8kb07r uNvGGOojGdciGq5osJ1sv8wQPbAb/k3rO/u/1RFSYbvjCMCpt3u4bOTkawqPhKiihNVi scq1Z1IbDhtJRT+htgOUm2RM3x2DAlkLBYY4jnB0k5tWKvMpVwJpCB4H8rmqzbEOWfU4 WurK21V1OJ6t1gby54KqevsVzrzvxWR6r5kPtftNnerYtiobXp9BaFouqdAFgrjg+3uc aZsQ== X-Gm-Message-State: AOJu0Yyne81iYwkcfkpTC+aZBcSsp/eQhz0bFd5ms4z9e3/KsVC9vv2f AMm17TDliTDy796DLqObff2y9Tc5tGw= X-Google-Smtp-Source: AGHT+IH9RxzmajTLifcNJivXHXF15GYt6RNI6hyz68HWdvvEFKxSVfujRMym9vZDynLvRqNw698SRRht3dY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:842:b0:5cc:cd5e:8f0e with SMTP id bz2-20020a05690c084200b005cccd5e8f0emr592433ywb.0.1701297621195; Wed, 29 Nov 2023 14:40:21 -0800 (PST) Date: Wed, 29 Nov 2023 14:40:19 -0800 In-Reply-To: <81628606-ca9b-866f-5e71-91001e856871@suse.cz> Mime-Version: 1.0 References: <92ba7ddd-2bc8-4a8d-bd67-d6614b21914f@intel.com> <4ca2253d-276f-43c5-8e9f-0ded5d5b2779@redhat.com> <81628606-ca9b-866f-5e71-91001e856871@suse.cz> Message-ID: Subject: Re: [PATCH v13 17/35] KVM: Add transparent hugepage support for dedicated guest memory From: Sean Christopherson To: Vlastimil Babka Cc: Paolo Bonzini , Xiaoyao Li , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Xu Yilun , Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , David Matlack , Yu Zhang , Isaku Yamahata , "=?utf-8?Q?Micka=C3=ABl_Sala=C3=BCn?=" , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1D91C4000A X-Stat-Signature: ete56153qpfrtafc5r799zjo379ophnq X-HE-Tag: 1701297621-318617 X-HE-Meta: U2FsdGVkX1/V63jU0O8XdhYJ8dr6tkozNlFiMGR1FFPMC5YdBl0MTRqUGRaEsj9qb4QyVUQeDXDgU+RSLIJOs/hKu1pKE2mFTtD/Nkoz0SfzCmYbBGovORbxnQBxIDU4cVbmtZfMWklSkce3TuQ0nWdtxex0h2eS6qDlq/DMgMeLavO29P45sJlf4Tq3CA2iOnzhbrlPLkMd6Z7NMFicWZLh42x/4wP1w/0y+j9xO8nAYXGqfF1q9jT8+oX+ews1uLYSzhc4MLxMcIITBxtD97uInwVcPnSiSiWrHM58NBoW8qO1Dxodnamcn6zAa4SKWVPk8Cw+xDTdTTx1rzItkv4MoSlb3VWkTvPen7s1G8PtjGU/SIk4J+F6O3sgED8f7oErtB0HX+1z77GYNq1r0oXB5qMMHEbYEvaNXFPTO2VAYeSo+qFC549HhYgGRxj8c7rTnWfgmA1dbXpEgX8/QUf2m2Jd5us4D4aM1FJbHNb8JF6kNBIo6gElWeWTqQOrfGTfqBMQmfYlmvB2/46xSNhWyAzmQDSmUWf8EdFw0cKOUNLJP8BpzHDPFtGBysjzkHqg69WzEFlSNVZl3bWzbDCkj6UjMX666QXCTUCG61uE9ZEekZr5VxYthPakbuou4X/zg+9DKy2+dOqXAtRF0C8FLPqW7yY4UDy+Fgq2tMhbvi5q3O37OHyUzH73lMrX9BHd6MPiobPR/WnPbYITPMhTNKF4y2doTrj51Jem70eg86c9XlS26za6Ne1lM8ACS+g54nXMIYSeUo0MmpH153S0vM4P0ehXbdSjv0pP7ODYnN9jNBEMPZe+XCg68Zgika9t+hZkP1CjIW25XnwsBbDMRFzny9YVqwKxF3/ZqioKWm1LxAcX3u1Z1hojwFC55o7eKTCWpuTuaa/5SRBXXLg5fkLJNKUNXyCK2eYSOd2OTkOJ24GOWwJJaktm57Zk+wSZiyPUpYdjUTHQfLD 00rnuLe/ p5yKsghCmhNjlkp/ttDOx49TJJNZOsbdMBkXLMRG3AV0YdbCrsKk/hc32lh2vh05nq+yWoUFlPZ01kTIY0MzQ7AeBHsiMqmRvVY8txIpUBO7ZvWKwlgBzZWmjSzv9dZrl2yL/Csmuid2TkOggJeUyRiilMZwm/YyrrQ1ApEc1VItsI0VtxRBvNo0lsa9un/s5aP3UVJm6NoKZwjXH7nJSP/7TaT8hW1leT+anC5F0p/0H6LKNflgkJW9BquQLF7KQCqMIoTOeAnmjuGbDJ18pG//4NbDya9yWdLm6hwb2fsPH8gcMoIyyyME+NgbRxXRwOQsDZDhBS9IYdbvtYgRTayCKF+1D/wH9md5WFse7FpVVo++zdljQRGr1kk7pkhHbWgWNlqJMKa1YSS2GeQiyZL+FbZw4U8lFZGjygt/BF1DIsR5T0vjdH2hIPtxk4GonmauLpLBwQeIOI6nCbUBxAeDAB+C4T8Pm1nH+vVVAu2jNwsDYiafydEH7knJrzsKuNyXSeoP+tdbKuJYHnvlDqMkno747UaW3Ykle46SFP1HO8gw6kET3ueVOsJk+Mx3NaF9eOd24xF1/LClBN6nF0CBbQNYRFF78zxeVltSpD4NtdTLOt1Qe2g76iA7Q5YBVdfNkwzvey9rHQdZQSC7NSq3Zwv5vx0FVrzusuDEU1N47J3KCI1fWKVNQew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Nov 27, 2023, Vlastimil Babka wrote: > On 11/2/23 16:46, Paolo Bonzini wrote: > > On Thu, Nov 2, 2023 at 4:38=E2=80=AFPM Sean Christopherson wrote: > >> Actually, looking that this again, there's not actually a hard depende= ncy on THP. > >> A THP-enabled kernel _probably_ gives a higher probability of using h= ugepages, > >> but mostly because THP selects COMPACTION, and I suppose because using= THP for > >> other allocations reduces overall fragmentation. > >=20 > > Yes, that's why I didn't even bother enabling it unless THP is > > enabled, but it makes even more sense to just try. > >=20 > >> So rather than honor KVM_GUEST_MEMFD_ALLOW_HUGEPAGE iff THP is enabled= , I think > >> we should do the below (I verified KVM can create hugepages with THP= =3Dn). We'll > >> need another capability, but (a) we probably should have that anyways = and (b) it > >> provides a cleaner path to adding PUD-sized hugepage support in the fu= ture. > >=20 > > I wonder if we need KVM_CAP_GUEST_MEMFD_HUGEPAGE_PMD_SIZE though. This > > should be a generic kernel API and in fact the sizes are available in > > a not-so-friendly format in /sys/kernel/mm/hugepages. > >=20 > > We should just add /sys/kernel/mm/hugepages/sizes that contains > > "2097152 1073741824" on x86 (only the former if 1G pages are not > > supported). > >=20 > > Plus: is this the best API if we need something else for 1G pages? > >=20 > > Let's drop *this* patch and proceed incrementally. (Again, this is > > what I want to do with this final review: identify places that are > > stil sticky, and don't let them block the rest). > >=20 > > Coincidentially we have an open spot next week at plumbers. Let's > > extend Fuad's section to cover more guestmem work. >=20 > Hi, >=20 > was there any outcome wrt this one? No, we punted on hugepage support for the initial guest_memfd merge. We de= finitely plan on adding hugeapge support sooner than later, but we haven't yet agree= d on exactly what that will look like. > Based on my experience with THP's it would be best if userspace didn't ha= ve > to opt-in, nor care about the supported size. If the given size is unalig= ned, > provide a mix of large pages up to an aligned size, and for the rest fall= back > to base pages, which should be better than -EINVAL on creation (is it > possible with the current implementation? I'd hope so so?). guest_memfd serves a different use case than THP. For modern VMs, and espe= cially for slice-of-hardware VMs that are one of the main targets for guest_memfd,= if not _the_ main target, guest memory should _always_ be backed by hugepages in t= he physical domain. The actual guest mappings might not be huge, e.g. x86 nee= ds to do partial mappings to skip over (legacy) memory holes, but KVM already gra= cefully handles that. In other words, for most guest_memfd use cases, if userspace wants hugepage= s but KVM can't provide hugepages, then it is much more desirable to return an er= ror than to silently fall back to small pages. I 100% agree that having to opt-in is suboptimal, but IMO providing "error = on an incompatible configuration" semantics without requiring userspace to opt-in= is an even worse experience for userspace. > A way to opt-out from huge pages could be useful although there's always = the > risk of some initial troubles resulting in various online sources cargo-c= ult > recommending to opt-out forever.