[PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Paolo Bonzini]

While KVM_XEN_EVTCHN_RESET is usually called with no vCPUs running,
if that happened it could cause a deadlock.  This is due to
kvm_xen_eventfd_reset() doing a synchronize_srcu() inside
a kvm->lock critical section.

To avoid this, first collect all the evtchnfd objects in an
array and free all of them once the kvm->lock critical section
is over and th SRCU grace period has expired.

Reported-by: Michal Luczaj <mhal@rbox.co>
Cc: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/xen.c                            | 30 +++++++++++++++++--
 .../selftests/kvm/x86_64/xen_shinfo_test.c    |  6 ++++
 2 files changed, 33 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index b178f40bd863..2e29bdc2949c 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1942,18 +1942,42 @@ static int kvm_xen_eventfd_deassign(struct kvm *kvm, u32 port)
 
 static int kvm_xen_eventfd_reset(struct kvm *kvm)
 {
-	struct evtchnfd *evtchnfd;
+	struct evtchnfd *evtchnfd, **all_evtchnfds;
 	int i;
+	int n = 0;
 
 	mutex_lock(&kvm->lock);
+
+	/*
+	 * Because synchronize_srcu() cannot be called inside the
+	 * critical section, first collect all the evtchnfd objects
+	 * in an array as they are removed from evtchn_ports.
+	 */
+	idr_for_each_entry(&kvm->arch.xen.evtchn_ports, evtchnfd, i)
+		n++;
+
+	all_evtchnfds = kmalloc_array(n, sizeof(struct evtchnfd *), GFP_KERNEL);
+	if (!all_evtchnfds) {
+		mutex_unlock(&kvm->lock);
+		return -ENOMEM;
+	}
+
+	n = 0;
 	idr_for_each_entry(&kvm->arch.xen.evtchn_ports, evtchnfd, i) {
+		all_evtchnfds[n++] = evtchnfd;
 		idr_remove(&kvm->arch.xen.evtchn_ports, evtchnfd->send_port);
-		synchronize_srcu(&kvm->srcu);
+	}
+	mutex_unlock(&kvm->lock);
+
+	synchronize_srcu(&kvm->srcu);
+
+	while (n--) {
+		evtchnfd = all_evtchnfds[n];
 		if (!evtchnfd->deliver.port.port)
 			eventfd_ctx_put(evtchnfd->deliver.eventfd.ctx);
 		kfree(evtchnfd);
 	}
-	mutex_unlock(&kvm->lock);
+	kfree(all_evtchnfds);
 
 	return 0;
 }
diff --git a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
index 721f6a693799..dae510c263b4 100644
--- a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
+++ b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
@@ -962,6 +962,12 @@ int main(int argc, char *argv[])
 	}
 
  done:
+	struct kvm_xen_hvm_attr evt_reset = {
+		.type = KVM_XEN_ATTR_TYPE_EVTCHN,
+		.u.evtchn.flags = KVM_XEN_EVTCHN_RESET,
+	};
+	vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
+
 	alarm(0);
 	clock_gettime(CLOCK_REALTIME, &max_ts);
 
-- 
2.31.1

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Yu Zhang]

Hi Paolo,

> diff --git a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> index 721f6a693799..dae510c263b4 100644
> --- a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> +++ b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> @@ -962,6 +962,12 @@ int main(int argc, char *argv[])
>  	}
>  
>   done:
> +	struct kvm_xen_hvm_attr evt_reset = {
> +		.type = KVM_XEN_ATTR_TYPE_EVTCHN,
> +		.u.evtchn.flags = KVM_XEN_EVTCHN_RESET,
> +	};
> +	vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
> +
>  	alarm(0);
>  	clock_gettime(CLOCK_REALTIME, &max_ts);
>  

This change generates a build failure with error message: 
"error: a label can only be part of a statement and a declaration is not a statement".

Moving the definition of evt_reset to the beginning of main() can fix it:

diff --git a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
index dae510c263b4..d71f1508bb21 100644
--- a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
+++ b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
@@ -623,6 +623,10 @@ int main(int argc, char *argv[])
        rs->state = 0x5a;

        bool evtchn_irq_expected = false;
+       struct kvm_xen_hvm_attr evt_reset = {
+               .type = KVM_XEN_ATTR_TYPE_EVTCHN,
+               .u.evtchn.flags = KVM_XEN_EVTCHN_RESET,
+       };

        for (;;) {
                volatile struct kvm_run *run = vcpu->run;
@@ -962,10 +966,6 @@ int main(int argc, char *argv[])
        }

  done:
-       struct kvm_xen_hvm_attr evt_reset = {
-               .type = KVM_XEN_ATTR_TYPE_EVTCHN,
-               .u.evtchn.flags = KVM_XEN_EVTCHN_RESET,
-       };
        vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);

        alarm(0);

B.R.
Yu

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Sean Christopherson]

On Thu, Jan 19, 2023, Yu Zhang wrote:
> Hi Paolo,
> 
> > diff --git a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> > index 721f6a693799..dae510c263b4 100644
> > --- a/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> > +++ b/tools/testing/selftests/kvm/x86_64/xen_shinfo_test.c
> > @@ -962,6 +962,12 @@ int main(int argc, char *argv[])
> >  	}
> >  
> >   done:
> > +	struct kvm_xen_hvm_attr evt_reset = {
> > +		.type = KVM_XEN_ATTR_TYPE_EVTCHN,
> > +		.u.evtchn.flags = KVM_XEN_EVTCHN_RESET,
> > +	};
> > +	vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
> > +
> >  	alarm(0);
> >  	clock_gettime(CLOCK_REALTIME, &max_ts);
> >  
> 
> This change generates a build failure with error message: 
> "error: a label can only be part of a statement and a declaration is not a statement".

And other flavors too, e.g.

x86_64/xen_shinfo_test.c:965:2: error: expected expression
        struct kvm_xen_hvm_attr evt_reset = {
        ^
x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
        vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
                                            ^
x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
3 errors generated.
make: *** [../lib.mk:145: tools/testing/selftests/kvm/x86_64/xen_shinfo_test] Error 1
make: *** Waiting for unfinished jobs....

I'm surprised bots haven't complained about this, haven't seen any reports.

> Moving the definition of evt_reset to the beginning of main() can fix it:

I'll queue a patch, this is already in Linus' tree and I've collected a few other
tiny fixes for v6.2-rcwhatever that I'll send to Paolo.

Thanks!

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Paolo Bonzini]

On Thu, Jan 19, 2023 at 6:57 PM Sean Christopherson <seanjc@google.com> wrote:
> > This change generates a build failure with error message:
> > "error: a label can only be part of a statement and a declaration is not a statement".
>
> And other flavors too, e.g.
>
> x86_64/xen_shinfo_test.c:965:2: error: expected expression
>         struct kvm_xen_hvm_attr evt_reset = {
>         ^
> x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
>         vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
>                                             ^
> x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
> 3 errors generated.
> make: *** [../lib.mk:145: tools/testing/selftests/kvm/x86_64/xen_shinfo_test] Error 1
> make: *** Waiting for unfinished jobs....
>
> I'm surprised bots haven't complained about this, haven't seen any reports.

It's clang only; GCC only warns with -Wpedantic. Plus, bots probably
don't compile tools/ that much.

Paolo

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Sean Christopherson]

On Thu, Jan 19, 2023, Paolo Bonzini wrote:
> On Thu, Jan 19, 2023 at 6:57 PM Sean Christopherson <seanjc@google.com> wrote:
> > > This change generates a build failure with error message:
> > > "error: a label can only be part of a statement and a declaration is not a statement".
> >
> > And other flavors too, e.g.
> >
> > x86_64/xen_shinfo_test.c:965:2: error: expected expression
> >         struct kvm_xen_hvm_attr evt_reset = {
> >         ^
> > x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
> >         vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt_reset);
> >                                             ^
> > x86_64/xen_shinfo_test.c:969:38: error: use of undeclared identifier 'evt_reset'
> > 3 errors generated.
> > make: *** [../lib.mk:145: tools/testing/selftests/kvm/x86_64/xen_shinfo_test] Error 1
> > make: *** Waiting for unfinished jobs....
> >
> > I'm surprised bots haven't complained about this, haven't seen any reports.
> 
> It's clang only; GCC only warns with -Wpedantic. Plus, bots probably
> don't compile tools/ that much.

/wave

Want to queue Yu's fix directly Paolo?  I was assuming you'd be offline until
sometime tomorrow.

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Paolo Bonzini]

On Thu, Jan 19, 2023 at 7:04 PM Sean Christopherson <seanjc@google.com> wrote:
> > It's clang only; GCC only warns with -Wpedantic. Plus, bots probably
> > don't compile tools/ that much.
>
> /wave
>
> Want to queue Yu's fix directly Paolo?  I was assuming you'd be offline until
> sometime tomorrow.

Yes, I can, but what other patches were you meaning to send?

Paolo

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Sean Christopherson]

On Thu, Jan 19, 2023, Paolo Bonzini wrote:
> On Thu, Jan 19, 2023 at 7:04 PM Sean Christopherson <seanjc@google.com> wrote:
> > > It's clang only; GCC only warns with -Wpedantic. Plus, bots probably
> > > don't compile tools/ that much.
> >
> > /wave
> >
> > Want to queue Yu's fix directly Paolo?  I was assuming you'd be offline until
> > sometime tomorrow.
> 
> Yes, I can, but what other patches were you meaning to send?

A minor selftest fix

  https://lore.kernel.org/all/20230111183408.104491-1-vipinsh@google.com

and a fix for a longstanding VMX bug that seems problematic enough that it
warrants going into this cycle.

  https://lore.kernel.org/all/20221114164823.69555-1-hborghor@amazon.de

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Paolo Bonzini]

On Thu, Jan 19, 2023 at 7:15 PM Sean Christopherson <seanjc@google.com> wrote:
> A minor selftest fix
>
>   https://lore.kernel.org/all/20230111183408.104491-1-vipinsh@google.com
>
> and a fix for a longstanding VMX bug that seems problematic enough that it
> warrants going into this cycle.
>
>   https://lore.kernel.org/all/20221114164823.69555-1-hborghor@amazon.de

Ok, I had seen the latter so I'll put together a pull request.

Paolo

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Michal Luczaj]

On 28/12/2022 12:04, Paolo Bonzini wrote:
> While KVM_XEN_EVTCHN_RESET is usually called with no vCPUs running,
> if that happened it could cause a deadlock.  This is due to
> kvm_xen_eventfd_reset() doing a synchronize_srcu() inside
> a kvm->lock critical section.
>
> [...]
>
> +	/*
> +	 * Because synchronize_srcu() cannot be called inside the
> +	 * critical section, first collect all the evtchnfd objects
> +	 * in an array as they are removed from evtchn_ports.
> +	 */

With the recent changes regarding the locking order (locking.rst:
"synchronize_srcu(&kvm->srcu) is called inside critical sections for kvm->lock,
vcpu->mutex and kvm->slots_lock"), is this comment still valid?

Or is there a rule that forbids synchronize_srcu() under the newly introduced
kvm->arch.xen.xen_lock?

thanks,
Michal

Re: [PATCH] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET

[Paolo Bonzini]

On 12/28/22 12:55, Woodhouse, David wrote:
> 
> IIRC the send_port and type aren't used from hcall_send so we could 
> stick those in a union with a 'next' pointer and use it to build a list 
> instead of having to allocate the array (pathological worst case 32KiB).

Yeah I thought about using a list.  The union is doable as you say, but 
I wasn't sure if things were going to change in kvm_xen_evtchn_send later.

And the worst case is order 3, which is within PAGE_ALLOC_COSTLY_ORDER, 
so I decided to go with the array which only consumes memory on reset 
rather than always; it's much more likely that the array will be smaller 
than a page.

Paolo

> Or if the union is a bit icky, we could just add the 'next' pointer 
> unconditionally.
>