* Re: [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set
2022-05-04 18:08 [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set Seth Forshee
@ 2022-05-04 18:16 ` Eric W. Biederman
2022-05-05 0:35 ` Jens Axboe
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Eric W. Biederman @ 2022-05-04 18:16 UTC (permalink / raw)
To: Seth Forshee
Cc: Thomas Gleixner, Peter Zijlstra, Andy Lutomirski, Josh Poimboeuf,
Jiri Kosina, Miroslav Benes, Petr Mladek, Paolo Bonzini,
Sean Christopherson, Jens Axboe, linux-kernel, live-patching, kvm
Seth Forshee <sforshee@digitalocean.com> writes:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <ebiederm@xmission.com>
> Cc: Petr Mladek <pmladek@suse.com>
> Signed-off-by: Seth Forshee <sforshee@digitalocean.com>
Acked-by: "Eric W. Biederman" <ebiederm@xmission.com>
> ---
> kernel/entry/kvm.c | 6 ------
> 1 file changed, 6 deletions(-)
>
> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
> index 9d09f489b60e..2e0f75bcb7fd 100644
> --- a/kernel/entry/kvm.c
> +++ b/kernel/entry/kvm.c
> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
> int ret;
>
> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
> - clear_notify_signal();
> - if (task_work_pending(current))
> - task_work_run();
> - }
> -
> - if (ti_work & _TIF_SIGPENDING) {
> kvm_handle_signal_exit(vcpu);
> return -EINTR;
> }
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set
2022-05-04 18:08 [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set Seth Forshee
2022-05-04 18:16 ` Eric W. Biederman
@ 2022-05-05 0:35 ` Jens Axboe
2022-05-05 7:30 ` Petr Mladek
2022-06-06 14:13 ` Seth Forshee
3 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2022-05-05 0:35 UTC (permalink / raw)
To: Seth Forshee, Thomas Gleixner, Peter Zijlstra, Andy Lutomirski
Cc: Josh Poimboeuf, Jiri Kosina, Miroslav Benes, Petr Mladek,
Paolo Bonzini, Sean Christopherson, linux-kernel, live-patching,
kvm, Eric W. Biederman
On 5/4/22 12:08 PM, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
Reviewed-by: Jens Axboe <axboe@kernel.dk>
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set
2022-05-04 18:08 [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set Seth Forshee
2022-05-04 18:16 ` Eric W. Biederman
2022-05-05 0:35 ` Jens Axboe
@ 2022-05-05 7:30 ` Petr Mladek
2022-06-06 14:13 ` Seth Forshee
3 siblings, 0 replies; 6+ messages in thread
From: Petr Mladek @ 2022-05-05 7:30 UTC (permalink / raw)
To: Seth Forshee
Cc: Thomas Gleixner, Peter Zijlstra, Andy Lutomirski, Josh Poimboeuf,
Jiri Kosina, Miroslav Benes, Paolo Bonzini, Sean Christopherson,
Jens Axboe, linux-kernel, live-patching, kvm, Eric W. Biederman
On Wed 2022-05-04 13:08:40, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <ebiederm@xmission.com>
> Cc: Petr Mladek <pmladek@suse.com>
> Signed-off-by: Seth Forshee <sforshee@digitalocean.com>
Acked-by: Petr Mladek <pmladek@suse.com>
Thanks Seth for discovering the problem.
Thanks everyone who helped to find the right solution.
Best Regards.
Petr
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set
2022-05-04 18:08 [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set Seth Forshee
` (2 preceding siblings ...)
2022-05-05 7:30 ` Petr Mladek
@ 2022-06-06 14:13 ` Seth Forshee
2022-06-06 16:20 ` Paolo Bonzini
3 siblings, 1 reply; 6+ messages in thread
From: Seth Forshee @ 2022-06-06 14:13 UTC (permalink / raw)
To: Thomas Gleixner, Peter Zijlstra, Andy Lutomirski
Cc: Josh Poimboeuf, Jiri Kosina, Miroslav Benes, Petr Mladek,
Paolo Bonzini, Sean Christopherson, Jens Axboe, linux-kernel,
live-patching, kvm, Eric W. Biederman
On Wed, May 04, 2022 at 01:08:40PM -0500, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <ebiederm@xmission.com>
> Cc: Petr Mladek <pmladek@suse.com>
> Signed-off-by: Seth Forshee <sforshee@digitalocean.com>
Friendly reminder as it seems like this patch may have been forgotten.
Thanks,
Seth
> ---
> kernel/entry/kvm.c | 6 ------
> 1 file changed, 6 deletions(-)
>
> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
> index 9d09f489b60e..2e0f75bcb7fd 100644
> --- a/kernel/entry/kvm.c
> +++ b/kernel/entry/kvm.c
> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
> int ret;
>
> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
> - clear_notify_signal();
> - if (task_work_pending(current))
> - task_work_run();
> - }
> -
> - if (ti_work & _TIF_SIGPENDING) {
> kvm_handle_signal_exit(vcpu);
> return -EINTR;
> }
> --
> 2.32.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] entry/kvm: Exit to user mode when TIF_NOTIFY_SIGNAL is set
2022-06-06 14:13 ` Seth Forshee
@ 2022-06-06 16:20 ` Paolo Bonzini
0 siblings, 0 replies; 6+ messages in thread
From: Paolo Bonzini @ 2022-06-06 16:20 UTC (permalink / raw)
To: Seth Forshee, Thomas Gleixner, Peter Zijlstra, Andy Lutomirski
Cc: Josh Poimboeuf, Jiri Kosina, Miroslav Benes, Petr Mladek,
Sean Christopherson, Jens Axboe, linux-kernel, live-patching, kvm,
Eric W. Biederman
On 6/6/22 16:13, Seth Forshee wrote:
> On Wed, May 04, 2022 at 01:08:40PM -0500, Seth Forshee wrote:
>> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
>> loaded. To the host, the vCPU task is a user thread which is spending a
>> very long time in the ioctl(KVM_RUN) syscall. During livepatch
>> transition, set_notify_signal() will be called on such tasks to
>> interrupt the syscall so that the task can be transitioned. This
>> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
>> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
>> exit to user mode is unnecessary, and guest execution is resumed without
>> transitioning the task for the livepatch.
>>
>> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
>> is expected to break tasks out of interruptible kernel loops and cause
>> them to return to userspace. Change xfer_to_guest_mode_work() to handle
>> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
>> loop that an exit to userpsace is needed. Any pending task_work will be
>> run when get_signal() is called from exit_to_user_mode_loop(), so there
>> is no longer any need to run task work from xfer_to_guest_mode_work().
>>
>> Suggested-by: "Eric W. Biederman" <ebiederm@xmission.com>
>> Cc: Petr Mladek <pmladek@suse.com>
>> Signed-off-by: Seth Forshee <sforshee@digitalocean.com>
>
> Friendly reminder as it seems like this patch may have been forgotten.
Probably AB-BA maintainer deadlock. I have queued it now.
Paolo
> Thanks,
> Seth
>
>> ---
>> kernel/entry/kvm.c | 6 ------
>> 1 file changed, 6 deletions(-)
>>
>> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
>> index 9d09f489b60e..2e0f75bcb7fd 100644
>> --- a/kernel/entry/kvm.c
>> +++ b/kernel/entry/kvm.c
>> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
>> int ret;
>>
>> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
>> - clear_notify_signal();
>> - if (task_work_pending(current))
>> - task_work_run();
>> - }
>> -
>> - if (ti_work & _TIF_SIGPENDING) {
>> kvm_handle_signal_exit(vcpu);
>> return -EINTR;
>> }
>> --
>> 2.32.0
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread