[PATCH] KVM: make uevents configurable

[PATCH] KVM: make uevents configurable

[Bernhard Kauer]

Handling of uevents in userlevel is a bottleneck for tiny VMs.

Running 10_000 VMs keeps one and a half cores busy for 5.4 seconds to let
systemd-udevd handle all messages.  That is roughly 27x longer than
the 0.2 seconds needed for running the VMs without them.

We choose a read-only module parameter here due to its simplicity and
ease of maintenance.

Signed-off-by: Bernhard Kauer <bk@alpico.io>
---
 virt/kvm/kvm_main.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 609e0bd68e8e..6139cd67a96a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -97,6 +97,9 @@ EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
 bool debugfs_per_vm = true;
 module_param(debugfs_per_vm, bool, 0644);
 
+bool disable_uevent_notify;
+module_param(disable_uevent_notify, bool, 0444);
+
 /*
  * Ordering of locks:
  *
@@ -6276,7 +6279,7 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
 	struct kobj_uevent_env *env;
 	unsigned long long created, active;
 
-	if (!kvm_dev.this_device || !kvm)
+	if (!kvm_dev.this_device || !kvm || disable_uevent_notify)
 		return;
 
 	mutex_lock(&kvm_lock);
-- 
2.45.2

Re: [PATCH] KVM: make uevents configurable

[Marc Zyngier]

On Fri, 22 Nov 2024 09:58:02 +0000,
Bernhard Kauer <bk@alpico.io> wrote:
> 
> Handling of uevents in userlevel is a bottleneck for tiny VMs.
> 
> Running 10_000 VMs keeps one and a half cores busy for 5.4 seconds to let
> systemd-udevd handle all messages.  That is roughly 27x longer than
> the 0.2 seconds needed for running the VMs without them.
> 
> We choose a read-only module parameter here due to its simplicity and
> ease of maintenance.
> 
> Signed-off-by: Bernhard Kauer <bk@alpico.io>
> ---
>  virt/kvm/kvm_main.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 609e0bd68e8e..6139cd67a96a 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -97,6 +97,9 @@ EXPORT_SYMBOL_GPL(halt_poll_ns_shrink);
>  bool debugfs_per_vm = true;
>  module_param(debugfs_per_vm, bool, 0644);
>  
> +bool disable_uevent_notify;
> +module_param(disable_uevent_notify, bool, 0444);
> +
>  /*
>   * Ordering of locks:
>   *
> @@ -6276,7 +6279,7 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
>  	struct kobj_uevent_env *env;
>  	unsigned long long created, active;
>  
> -	if (!kvm_dev.this_device || !kvm)
> +	if (!kvm_dev.this_device || !kvm || disable_uevent_notify)
>  		return;
>  
>  	mutex_lock(&kvm_lock);

Thanks for this. It was on my list of things to investigate, as this
is a bottleneck when running a lot of concurrent syzkaller tests which
create and destroy VMs repeatedly.

I'm not overly keen on the command-line flag though, as this is the
sort of things you'd like to be able to control more finely. Or at
least without having to trigger a reboot.

How about something such as a sysctl? with the kvm namespace?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: make uevents configurable

[Bernhard Kauer]

On Mon, Nov 25, 2024 at 05:31:32PM +0000, Marc Zyngier wrote:
> On Fri, 22 Nov 2024 09:58:02 +0000, Bernhard Kauer <bk@alpico.io> wrote:
> > Handling of uevents in userlevel is a bottleneck for tiny VMs.
> > 
> > Running 10_000 VMs keeps one and a half cores busy for 5.4 seconds to let
> > systemd-udevd handle all messages.  That is roughly 27x longer than
> > the 0.2 seconds needed for running the VMs without them.
> > 
> > We choose a read-only module parameter here due to its simplicity and
> > ease of maintenance.
> Thanks for this. It was on my list of things to investigate, as this
> is a bottleneck when running a lot of concurrent syzkaller tests which
> create and destroy VMs repeatedly.

That is interesting.  How many tests do you have?


> I'm not overly keen on the command-line flag though, as this is the
> sort of things you'd like to be able to control more finely. Or at
> least without having to trigger a reboot.

I compile KVM as module, so I can reload it with different parameters
easily.


> How about something such as a sysctl? with the kvm namespace?

I have not seen a sysctl in KVM yet. I would probably not introduce
another config method here.  

But I can make the module parameter read-write so that it is modifiable
during runtime via /sys/module/kvm/parameters/ even when KVM is compiled
into the kernel.


	Bernhard

Re: [PATCH] KVM: make uevents configurable

[Marc Zyngier]

On Tue, 26 Nov 2024 10:20:02 +0000,
Bernhard Kauer <bk@alpico.io> wrote:
> 
> On Mon, Nov 25, 2024 at 05:31:32PM +0000, Marc Zyngier wrote:
> > On Fri, 22 Nov 2024 09:58:02 +0000, Bernhard Kauer <bk@alpico.io> wrote:
> > > Handling of uevents in userlevel is a bottleneck for tiny VMs.
> > > 
> > > Running 10_000 VMs keeps one and a half cores busy for 5.4 seconds to let
> > > systemd-udevd handle all messages.  That is roughly 27x longer than
> > > the 0.2 seconds needed for running the VMs without them.
> > > 
> > > We choose a read-only module parameter here due to its simplicity and
> > > ease of maintenance.
> > Thanks for this. It was on my list of things to investigate, as this
> > is a bottleneck when running a lot of concurrent syzkaller tests which
> > create and destroy VMs repeatedly.
> 
> That is interesting.  How many tests do you have?

It's not the number of tests. It is the number of VMs. Each test can
create several hundred of them per second. Per CPU.

> > I'm not overly keen on the command-line flag though, as this is the
> > sort of things you'd like to be able to control more finely. Or at
> > least without having to trigger a reboot.
> 
> I compile KVM as module, so I can reload it with different parameters
> easily.

arm64 doesn't (and cannot) build KVM as a module. Also, the prospect
of killing all pre-existing VMs just to flip this thing is a
bit... extreme?

> > How about something such as a sysctl? with the kvm namespace?
> 
> I have not seen a sysctl in KVM yet. I would probably not introduce
> another config method here.

Well, that's the standard way for changing kernel parameters at
runtime. Very last century, I know.

> 
> But I can make the module parameter read-write so that it is modifiable
> during runtime via /sys/module/kvm/parameters/ even when KVM is compiled
> into the kernel.

I guess that'd be the next best thing.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] KVM: make uevents configurable

[Bernhard Kauer]

On Tue, Nov 26, 2024 at 11:23:10AM +0000, Marc Zyngier wrote:
> > > I'm not overly keen on the command-line flag though, as this is the
> > > sort of things you'd like to be able to control more finely. Or at
> > > least without having to trigger a reboot.
> > 
> > I compile KVM as module, so I can reload it with different parameters
> > easily.
> 
> arm64 doesn't (and cannot) build KVM as a module. 

Ah, I didn't realize it.  But being able to patch EL2 from EL1 is probably
more complicated than what is gained with it.


> > But I can make the module parameter read-write so that it is modifiable
> > during runtime via /sys/module/kvm/parameters/ even when KVM is compiled
> > into the kernel.
> 
> I guess that'd be the next best thing.

Ok, I will do that in v2 of the patch to be sent next week.


	Bernhard

Re: [PATCH] KVM: make uevents configurable

[Marc Zyngier]

On Wed, 27 Nov 2024 08:38:59 +0000,
Bernhard Kauer <bk@alpico.io> wrote:
> 
> On Tue, Nov 26, 2024 at 11:23:10AM +0000, Marc Zyngier wrote:
> > > > I'm not overly keen on the command-line flag though, as this is the
> > > > sort of things you'd like to be able to control more finely. Or at
> > > > least without having to trigger a reboot.
> > > 
> > > I compile KVM as module, so I can reload it with different parameters
> > > easily.
> > 
> > arm64 doesn't (and cannot) build KVM as a module. 
> 
> Ah, I didn't realize it.  But being able to patch EL2 from EL1 is probably
> more complicated than what is gained with it.

It is mostly pulling the rug from under your feet if you do that on
non-VHE systems, and make absolutely for things like pKVM.

> > > But I can make the module parameter read-write so that it is modifiable
> > > during runtime via /sys/module/kvm/parameters/ even when KVM is compiled
> > > into the kernel.
> > 
> > I guess that'd be the next best thing.
> 
> Ok, I will do that in v2 of the patch to be sent next week.

Thanks.

	M.

-- 
Without deviation from the norm, progress is not possible.