From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCAE7C04E69 for ; Tue, 15 Aug 2023 17:12:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238685AbjHORLd (ORCPT ); Tue, 15 Aug 2023 13:11:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238682AbjHORLU (ORCPT ); Tue, 15 Aug 2023 13:11:20 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49F02E52 for ; Tue, 15 Aug 2023 10:11:19 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-68872e445cdso670982b3a.3 for ; Tue, 15 Aug 2023 10:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692119479; x=1692724279; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cv2n5WJJQIuL/aVi/GBknUHwNAlL9ubXLjVa7Ggcv7I=; b=juBpYIQZEyCC6Jxj3rJ8WnAzIPrIzPVj5tnkYhXpIeMCc4oO2/fQGumrrkXbGCk3g8 89rFQ8JFJ7jAi+RKDOoGZu0lboERBLSeqj26Du2aiiZT3HUXr++j5MsFNimW/4tMY2+g rhai3gOe8SafhKvRJ/5b9PjsXezcxE8GcsL2AIP6DNYOol1ymPNSpzGKX4N4U2270Cds PN1LqsVigNzgBLwqPpwepEwsM8KimXnw2TOdhRx1KWDbFUKPeDRaFClMO9+2T2z1icRJ wmeoUCm00GDLdP2w0bHV7FmLWFYd4+xi5/TBn+q9x2KyZBLOZopWH3LIlJtxQ8Z+T6b0 OarA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692119479; x=1692724279; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cv2n5WJJQIuL/aVi/GBknUHwNAlL9ubXLjVa7Ggcv7I=; b=W24MjBy2WOirFsC25zxg/rcGtUelIkhQpMnBCUNvBiPDpiXx4r++/6/bQR76WznW0k ++uEXW+Sma92ErZRnaC1GGKfjCOIoN1doidSu2rYlUM4bHybO2/UoQOdBlPvVJUJ6Bkz dMFci0+/JcWhLfmHLK9mjXZIzvWPq1GAtH37eEghiuhWBLssv6qYnWJDo0mqM1hNiKw+ QTvZwXOH5FSf3oMFdfSZLg4BML9lRxL2q9FSmtqU69/kNbMYOzmaNLDLjj2ZLJXLpCJo r7e6OEiq1utTF4tsa92SR3dJH0nn3344Z1NnheXC/osTRMePALGbQlGxZMVwULYfeYCD l4gQ== X-Gm-Message-State: AOJu0YxtHpeif67479MQN8iyPMilXTrq+bQLAAesO5FrkkyoiwETe4Cf bLXeAvJvGIidRqtpaoSgQKS9/21ZYbY= X-Google-Smtp-Source: AGHT+IHbOv08Ksb8i4GH5D8SvhTbeb288EfGwKpVHkJw1xaHbiUdJmgJiWqHwrAr1/byOS/72K0ihXyR9Hs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:398a:b0:687:5d7c:82b8 with SMTP id fi10-20020a056a00398a00b006875d7c82b8mr5796779pfb.2.1692119478722; Tue, 15 Aug 2023 10:11:18 -0700 (PDT) Date: Tue, 15 Aug 2023 10:11:17 -0700 In-Reply-To: <20230815153537.113861-1-kyle.meyer@hpe.com> Mime-Version: 1.0 References: <20230815153537.113861-1-kyle.meyer@hpe.com> Message-ID: Subject: Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 4096 From: Sean Christopherson To: Kyle Meyer Cc: pbonzini@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hasen@linux.intel.com, x86@kernel.org, hpa@zytor.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, vkuznets@redhat.com, dmatlack@google.com, russ.anderson@hpe.com, dimitri.sivanich@hpe.com, steve.wahl@hpe.com Content-Type: text/plain; charset="us-ascii" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 15, 2023, Kyle Meyer wrote: > Increase KVM_MAX_VCPUS to 4096 when MAXSMP is enabled. > > Notable changes (when MAXSMP is enabled): > > * KMV_MAX_VCPUS will increase from 1024 to 4096. > * KVM_MAX_VCPU_IDS will increase from 4096 to 16384. > * KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 64. > * CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (0x40000005)].EAX will now be 4096. > > * struct kvm will increase from 39408 B to 39792 B. > * struct kvm_ioapic will increase from 5240 B to 19064 B. > > * The following (on-stack) bitmaps will increase from 128 B to 512 B: > * dest_vcpu_bitmap in kvm_irq_delivery_to_apic. > * vcpu_mask in kvm_hv_flush_tlb. > * vcpu_bitmap in ioapic_write_indirect. > * vp_bitmap in sparse_set_to_vcpu_mask. > > Signed-off-by: Kyle Meyer > --- > Virtual machines with 4096 virtual CPUs have been created on 32 socket > Cascade Lake and Sapphire Rapids systems. > > 4096 is the current maximum value because of the Hyper-V TLFS. See > BUILD_BUG_ON in arch/x86/kvm/hyperv.c, commit 79661c3, and Vitaly's > comment on https://lore.kernel.org/all/87r136shcc.fsf@redhat.com. Mostly out of curiosity, do you care about Hyper-V support? If not, at some point it'd probably be worth exploring a CONFIG_KVM_HYPERV option to allow disabling KVM's Hyper-V support at compile time so that we're not bound by the restrictions of the TLFS. > arch/x86/include/asm/kvm_host.h | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h > index 3bc146dfd38d..91a01fa17fa7 100644 > --- a/arch/x86/include/asm/kvm_host.h > +++ b/arch/x86/include/asm/kvm_host.h > @@ -39,7 +39,11 @@ > > #define __KVM_HAVE_ARCH_VCPU_DEBUGFS > > +#ifdef CONFIG_MAXSMP > +#define KVM_MAX_VCPUS 4096 > +#else > #define KVM_MAX_VCPUS 1024 > +#endif Rather than tightly couple this to MAXSMP, what if we add a Kconfig? I know of at least one scenario, SVM's AVIC/x2AVIC, where it would be desirable to configure KVM to a much smaller maximum. The biggest downside I can think of is that KVM selftests would need to be updated (they assume the max is >=512), and some of the tests might be completely invalid if KVM_MAX_VCPUS is too low (<256?). E.g. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 60d430b4650f..8704748e35d9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -39,7 +39,7 @@ #define __KVM_HAVE_ARCH_VCPU_DEBUGFS -#define KVM_MAX_VCPUS 1024 +#define KVM_MAX_VCPUS CONFIG_KVM_MAX_NR_VCPUS /* * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index ed90f148140d..b0f92eb77f78 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -151,6 +151,17 @@ config KVM_PROVE_MMU If in doubt, say "N". +config KVM_MAX_NR_VCPUS + int "Maximum vCPUs per VM" + default "4096" if MAXSMP + default "1024" + range 1 4096 + depends on KVM + help + Set the maximum number of vCPUs for a single VM. Larger values + increase the memory footprint of each VM regardless of how many vCPUs + are actually created (though the memory increase is relatively small). + config KVM_EXTERNAL_WRITE_TRACKING bool