* [PATCH] x86: Raise the hard VCPU count limit
@ 2011-07-09 12:25 Sasha Levin
2011-07-09 13:04 ` Pekka Enberg
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Sasha Levin @ 2011-07-09 12:25 UTC (permalink / raw)
To: kvm; +Cc: Sasha Levin, Avi Kivity, Ingo Molnar, Marcelo Tosatti,
Pekka Enberg
The patch raises the hard limit of VCPU count to 1024.
This will allow developers to easily work on scalability
and will allow users to test high VCPU setups easily without
patching the kernel.
To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
now returns the recommended VCPU limit (which is still 64) - this
should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
returns the hard limit which is now 1024.
Cc: Avi Kivity <avi@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Pekka Enberg <penberg@kernel.org>
Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi>
Signed-off-by: Sasha Levin <levinsasha928@gmail.com>
---
Documentation/virtual/kvm/api.txt | 11 +++++++++--
arch/x86/include/asm/kvm_host.h | 3 ++-
arch/x86/kvm/x86.c | 3 +++
include/linux/kvm.h | 3 ++-
4 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 42542eb..84883a0 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -175,10 +175,17 @@ Parameters: vcpu id (apic id on x86)
Returns: vcpu fd on success, -1 on error
This API adds a vcpu to a virtual machine. The vcpu id is a small integer
-in the range [0, max_vcpus). You can use KVM_CAP_NR_VCPUS of the
-KVM_CHECK_EXTENSION ioctl() to determine the value for max_vcpus at run-time.
+in the range [0, max_vcpus).
+
+The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of
+the KVM_CHECK_EXTENSION ioctl() at run-time.
+The maximum possible value for max_vcpus can be retrieved using the
+KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time.
+
If the KVM_CAP_NR_VCPUS does not exist, you should assume that max_vcpus is 4
cpus max.
+If the KVM_CAP_MAX_VCPUS does not exist, you should assume that max_vcpus is
+same as the value returned from KVM_CAP_NR_VCPUS.
4.8 KVM_GET_DIRTY_LOG (vm ioctl)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d2ac8e2..7db3e66 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -26,7 +26,8 @@
#include <asm/mtrr.h>
#include <asm/msr-index.h>
-#define KVM_MAX_VCPUS 64
+#define KVM_MAX_VCPUS 1024
+#define KVM_SOFT_MAX_VCPUS 64
#define KVM_MEMORY_SLOTS 32
/* memory slots that does not exposed to userspace */
#define KVM_PRIVATE_MEM_SLOTS 4
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 77c9d86..9766f46 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2006,6 +2006,9 @@ int kvm_dev_ioctl_check_extension(long ext)
r = !kvm_x86_ops->cpu_has_accelerated_tpr();
break;
case KVM_CAP_NR_VCPUS:
+ r = KVM_SOFT_MAX_VCPUS;
+ break;
+ case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
break;
case KVM_CAP_NR_MEMSLOTS:
diff --git a/include/linux/kvm.h b/include/linux/kvm.h
index 55ef181..077c193 100644
--- a/include/linux/kvm.h
+++ b/include/linux/kvm.h
@@ -457,7 +457,7 @@ struct kvm_ppc_pvinfo {
#define KVM_CAP_VAPIC 6
#define KVM_CAP_EXT_CPUID 7
#define KVM_CAP_CLOCKSOURCE 8
-#define KVM_CAP_NR_VCPUS 9 /* returns max vcpus per vm */
+#define KVM_CAP_NR_VCPUS 9 /* returns recommended max vcpus per vm */
#define KVM_CAP_NR_MEMSLOTS 10 /* returns max memory slots per vm */
#define KVM_CAP_PIT 11
#define KVM_CAP_NOP_IO_DELAY 12
@@ -544,6 +544,7 @@ struct kvm_ppc_pvinfo {
#define KVM_CAP_TSC_CONTROL 60
#define KVM_CAP_GET_TSC_KHZ 61
#define KVM_CAP_PPC_BOOKE_SREGS 62
+#define KVM_CAP_MAX_VCPUS 63 /* returns max vcpus per vm */
#ifdef KVM_CAP_IRQ_ROUTING
--
1.7.6
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] x86: Raise the hard VCPU count limit
2011-07-09 12:25 [PATCH] x86: Raise the hard VCPU count limit Sasha Levin
@ 2011-07-09 13:04 ` Pekka Enberg
2011-07-12 13:49 ` Sasha Levin
2011-07-13 13:30 ` Avi Kivity
2 siblings, 0 replies; 6+ messages in thread
From: Pekka Enberg @ 2011-07-09 13:04 UTC (permalink / raw)
To: Sasha Levin; +Cc: kvm, Avi Kivity, Ingo Molnar, Marcelo Tosatti
On Sat, Jul 9, 2011 at 3:25 PM, Sasha Levin <levinsasha928@gmail.com> wrote:
> The patch raises the hard limit of VCPU count to 1024.
>
> This will allow developers to easily work on scalability
> and will allow users to test high VCPU setups easily without
> patching the kernel.
>
> To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
> now returns the recommended VCPU limit (which is still 64) - this
> should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
> returns the hard limit which is now 1024.
>
> Cc: Avi Kivity <avi@redhat.com>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Marcelo Tosatti <mtosatti@redhat.com>
> Cc: Pekka Enberg <penberg@kernel.org>
Acked-by: Pekka Enbeg <penberg@kernel.org>
> Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi>
> Signed-off-by: Sasha Levin <levinsasha928@gmail.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86: Raise the hard VCPU count limit
2011-07-09 12:25 [PATCH] x86: Raise the hard VCPU count limit Sasha Levin
2011-07-09 13:04 ` Pekka Enberg
@ 2011-07-12 13:49 ` Sasha Levin
2011-07-13 13:30 ` Avi Kivity
2 siblings, 0 replies; 6+ messages in thread
From: Sasha Levin @ 2011-07-12 13:49 UTC (permalink / raw)
To: kvm; +Cc: Avi Kivity, Ingo Molnar, Marcelo Tosatti, Pekka Enberg
On Sat, 2011-07-09 at 15:25 +0300, Sasha Levin wrote:
> The patch raises the hard limit of VCPU count to 1024.
>
> This will allow developers to easily work on scalability
> and will allow users to test high VCPU setups easily without
> patching the kernel.
>
> To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
> now returns the recommended VCPU limit (which is still 64) - this
> should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
> returns the hard limit which is now 1024.
>
> Cc: Avi Kivity <avi@redhat.com>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Marcelo Tosatti <mtosatti@redhat.com>
> Cc: Pekka Enberg <penberg@kernel.org>
> Suggested-by: Pekka Enberg <penberg@cs.helsinki.fi>
> Signed-off-by: Sasha Levin <levinsasha928@gmail.com>
> ---
Ping?
> Documentation/virtual/kvm/api.txt | 11 +++++++++--
> arch/x86/include/asm/kvm_host.h | 3 ++-
> arch/x86/kvm/x86.c | 3 +++
> include/linux/kvm.h | 3 ++-
> 4 files changed, 16 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 42542eb..84883a0 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -175,10 +175,17 @@ Parameters: vcpu id (apic id on x86)
> Returns: vcpu fd on success, -1 on error
>
> This API adds a vcpu to a virtual machine. The vcpu id is a small integer
> -in the range [0, max_vcpus). You can use KVM_CAP_NR_VCPUS of the
> -KVM_CHECK_EXTENSION ioctl() to determine the value for max_vcpus at run-time.
> +in the range [0, max_vcpus).
> +
> +The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of
> +the KVM_CHECK_EXTENSION ioctl() at run-time.
> +The maximum possible value for max_vcpus can be retrieved using the
> +KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time.
> +
> If the KVM_CAP_NR_VCPUS does not exist, you should assume that max_vcpus is 4
> cpus max.
> +If the KVM_CAP_MAX_VCPUS does not exist, you should assume that max_vcpus is
> +same as the value returned from KVM_CAP_NR_VCPUS.
>
> 4.8 KVM_GET_DIRTY_LOG (vm ioctl)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index d2ac8e2..7db3e66 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -26,7 +26,8 @@
> #include <asm/mtrr.h>
> #include <asm/msr-index.h>
>
> -#define KVM_MAX_VCPUS 64
> +#define KVM_MAX_VCPUS 1024
> +#define KVM_SOFT_MAX_VCPUS 64
> #define KVM_MEMORY_SLOTS 32
> /* memory slots that does not exposed to userspace */
> #define KVM_PRIVATE_MEM_SLOTS 4
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 77c9d86..9766f46 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -2006,6 +2006,9 @@ int kvm_dev_ioctl_check_extension(long ext)
> r = !kvm_x86_ops->cpu_has_accelerated_tpr();
> break;
> case KVM_CAP_NR_VCPUS:
> + r = KVM_SOFT_MAX_VCPUS;
> + break;
> + case KVM_CAP_MAX_VCPUS:
> r = KVM_MAX_VCPUS;
> break;
> case KVM_CAP_NR_MEMSLOTS:
> diff --git a/include/linux/kvm.h b/include/linux/kvm.h
> index 55ef181..077c193 100644
> --- a/include/linux/kvm.h
> +++ b/include/linux/kvm.h
> @@ -457,7 +457,7 @@ struct kvm_ppc_pvinfo {
> #define KVM_CAP_VAPIC 6
> #define KVM_CAP_EXT_CPUID 7
> #define KVM_CAP_CLOCKSOURCE 8
> -#define KVM_CAP_NR_VCPUS 9 /* returns max vcpus per vm */
> +#define KVM_CAP_NR_VCPUS 9 /* returns recommended max vcpus per vm */
> #define KVM_CAP_NR_MEMSLOTS 10 /* returns max memory slots per vm */
> #define KVM_CAP_PIT 11
> #define KVM_CAP_NOP_IO_DELAY 12
> @@ -544,6 +544,7 @@ struct kvm_ppc_pvinfo {
> #define KVM_CAP_TSC_CONTROL 60
> #define KVM_CAP_GET_TSC_KHZ 61
> #define KVM_CAP_PPC_BOOKE_SREGS 62
> +#define KVM_CAP_MAX_VCPUS 63 /* returns max vcpus per vm */
>
> #ifdef KVM_CAP_IRQ_ROUTING
>
--
Sasha.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86: Raise the hard VCPU count limit
2011-07-09 12:25 [PATCH] x86: Raise the hard VCPU count limit Sasha Levin
2011-07-09 13:04 ` Pekka Enberg
2011-07-12 13:49 ` Sasha Levin
@ 2011-07-13 13:30 ` Avi Kivity
2011-07-13 20:00 ` Sasha Levin
2 siblings, 1 reply; 6+ messages in thread
From: Avi Kivity @ 2011-07-13 13:30 UTC (permalink / raw)
To: Sasha Levin; +Cc: kvm, Ingo Molnar, Marcelo Tosatti, Pekka Enberg
On 07/09/2011 03:25 PM, Sasha Levin wrote:
> The patch raises the hard limit of VCPU count to 1024.
>
> This will allow developers to easily work on scalability
> and will allow users to test high VCPU setups easily without
> patching the kernel.
>
> To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
> now returns the recommended VCPU limit (which is still 64) - this
> should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
> returns the hard limit which is now 1024.
>
Can 1024 vcpus even work without interrupt remapping?
Looks like the patch will break coalesced mmio:
static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev,
gpa_t addr, int len)
{
struct kvm_coalesced_mmio_zone *zone;
struct kvm_coalesced_mmio_ring *ring;
unsigned avail;
int i;
/* Are we able to batch it ? */
/* last is the first free entry
* check if we don't meet the first used entry
* there is always one unused entry in the buffer
*/
ring = dev->kvm->coalesced_mmio_ring;
avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX;
if (avail < KVM_MAX_VCPUS) {
/* full */
return 0;
}
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86: Raise the hard VCPU count limit
2011-07-13 13:30 ` Avi Kivity
@ 2011-07-13 20:00 ` Sasha Levin
2011-07-14 8:14 ` Avi Kivity
0 siblings, 1 reply; 6+ messages in thread
From: Sasha Levin @ 2011-07-13 20:00 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm, Ingo Molnar, Marcelo Tosatti, Pekka Enberg
On Wed, 2011-07-13 at 16:30 +0300, Avi Kivity wrote:
> On 07/09/2011 03:25 PM, Sasha Levin wrote:
> > The patch raises the hard limit of VCPU count to 1024.
> >
> > This will allow developers to easily work on scalability
> > and will allow users to test high VCPU setups easily without
> > patching the kernel.
> >
> > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
> > now returns the recommended VCPU limit (which is still 64) - this
> > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
> > returns the hard limit which is now 1024.
> >
>
> Can 1024 vcpus even work without interrupt remapping?
>
I'm not sure. I've successfully tried it with 255 vcpus.
> Looks like the patch will break coalesced mmio:
>
> static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev,
> gpa_t addr, int len)
> {
> struct kvm_coalesced_mmio_zone *zone;
> struct kvm_coalesced_mmio_ring *ring;
> unsigned avail;
> int i;
>
> /* Are we able to batch it ? */
>
> /* last is the first free entry
> * check if we don't meet the first used entry
> * there is always one unused entry in the buffer
> */
> ring = dev->kvm->coalesced_mmio_ring;
> avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX;
> if (avail < KVM_MAX_VCPUS) {
> /* full */
> return 0;
> }
>
>
I don't quite understand what KVM_MAX_VCPUS has to do with that if ().
Shouldn't it check whether theres more than one buffer between first and
last? What role does KVM_MAX_VCPUS play there?
--
Sasha.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86: Raise the hard VCPU count limit
2011-07-13 20:00 ` Sasha Levin
@ 2011-07-14 8:14 ` Avi Kivity
0 siblings, 0 replies; 6+ messages in thread
From: Avi Kivity @ 2011-07-14 8:14 UTC (permalink / raw)
To: Sasha Levin; +Cc: kvm, Ingo Molnar, Marcelo Tosatti, Pekka Enberg
On 07/13/2011 11:00 PM, Sasha Levin wrote:
> On Wed, 2011-07-13 at 16:30 +0300, Avi Kivity wrote:
> > On 07/09/2011 03:25 PM, Sasha Levin wrote:
> > > The patch raises the hard limit of VCPU count to 1024.
> > >
> > > This will allow developers to easily work on scalability
> > > and will allow users to test high VCPU setups easily without
> > > patching the kernel.
> > >
> > > To prevent possible issues with current setups, KVM_CAP_NR_VCPUS
> > > now returns the recommended VCPU limit (which is still 64) - this
> > > should be a safe value for everybody, while a new KVM_CAP_MAX_VCPUS
> > > returns the hard limit which is now 1024.
> > >
> >
> > Can 1024 vcpus even work without interrupt remapping?
> >
>
> I'm not sure. I've successfully tried it with 255 vcpus.
>
Even 255 is problematic. One APIC ID is consumed by the IO-APIC, and ID
255 is reserved for broadcast IIRC. So at most 254 vcpus can be addressed.
> > Looks like the patch will break coalesced mmio:
> >
> > static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev,
> > gpa_t addr, int len)
> > {
> > struct kvm_coalesced_mmio_zone *zone;
> > struct kvm_coalesced_mmio_ring *ring;
> > unsigned avail;
> > int i;
> >
> > /* Are we able to batch it ? */
> >
> > /* last is the first free entry
> > * check if we don't meet the first used entry
> > * there is always one unused entry in the buffer
> > */
> > ring = dev->kvm->coalesced_mmio_ring;
> > avail = (ring->first - ring->last - 1) % KVM_COALESCED_MMIO_MAX;
> > if (avail< KVM_MAX_VCPUS) {
> > /* full */
> > return 0;
> > }
> >
> >
>
> I don't quite understand what KVM_MAX_VCPUS has to do with that if ().
> Shouldn't it check whether theres more than one buffer between first and
> last? What role does KVM_MAX_VCPUS play there?
At most KVM_MAX_VCPUS can be writing simultaneously. Since we're
checking outside the spinlock, we have to consider the worst case.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-07-14 8:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-07-09 12:25 [PATCH] x86: Raise the hard VCPU count limit Sasha Levin
2011-07-09 13:04 ` Pekka Enberg
2011-07-12 13:49 ` Sasha Levin
2011-07-13 13:30 ` Avi Kivity
2011-07-13 20:00 ` Sasha Levin
2011-07-14 8:14 ` Avi Kivity
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox