[PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[Gaurav Kohli]

Hyperv cleanup codes comes under panic path where preemption and irq
is already disabled. So calling of unregister_syscore_ops which has mutex
from hyperv cleanup might schedule out the thread and never comes back.

To prevent the same remove unwanted unregister_syscore_ops function call.

Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>

diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index f49bc3ec76e6..c050de69dfde 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -537,7 +537,12 @@ void hyperv_cleanup(void)
 	union hv_x64_msr_hypercall_contents hypercall_msr;
 	union hv_reference_tsc_msr tsc_msr;
 
-	unregister_syscore_ops(&hv_syscore_ops);
+	/*
+	 * Avoid unregister_syscore_ops(&hv_syscore_ops) from cleanup code,
+	 * as this is only called in crash path where irq and preemption disabled.
+	 * If we add this, there is a chance that this get scheduled out due to mutex
+	 * in unregister_syscore_ops and never comes back.
+	 */
 
 	/* Reset our OS id */
 	wrmsrl(HV_X64_MSR_GUEST_OS_ID, 0);
-- 
2.17.1

Re: [PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[Wei Liu]

On Wed, Nov 23, 2022 at 09:23:11PM -0800, Gaurav Kohli wrote:
> Hyperv cleanup codes comes under panic path where preemption and irq

Please use "Hyper-V" throughout.

> is already disabled. So calling of unregister_syscore_ops which has mutex
> from hyperv cleanup might schedule out the thread and never comes back.
> 

While on paper this looks problematic -- have you seen this issue
triggered in real life?

This looks to be only triggered when there is another thread already
holding the mutex, which seems rather rare in the life cycle of the
machine?

> To prevent the same remove unwanted unregister_syscore_ops function call.
> 
> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>
> 
> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> index f49bc3ec76e6..c050de69dfde 100644
> --- a/arch/x86/hyperv/hv_init.c
> +++ b/arch/x86/hyperv/hv_init.c
> @@ -537,7 +537,12 @@ void hyperv_cleanup(void)
>  	union hv_x64_msr_hypercall_contents hypercall_msr;
>  	union hv_reference_tsc_msr tsc_msr;
>  
> -	unregister_syscore_ops(&hv_syscore_ops);
> +	/*
> +	 * Avoid unregister_syscore_ops(&hv_syscore_ops) from cleanup code,
> +	 * as this is only called in crash path where irq and preemption disabled.
> +	 * If we add this, there is a chance that this get scheduled out due to mutex
> +	 * in unregister_syscore_ops and never comes back.
> +	 */

There is no need to document things we don't do, right?

>  
>  	/* Reset our OS id */
>  	wrmsrl(HV_X64_MSR_GUEST_OS_ID, 0);
> -- 
> 2.17.1
> 

Re: [PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[Gaurav Kohli]

On 11/25/2022 8:58 PM, Wei Liu wrote:
> On Wed, Nov 23, 2022 at 09:23:11PM -0800, Gaurav Kohli wrote:
>> Hyperv cleanup codes comes under panic path where preemption and irq
> Please use "Hyper-V" throughout.
Thanks for the comment, sure will do on v2.
>
>> is already disabled. So calling of unregister_syscore_ops which has mutex
>> from hyperv cleanup might schedule out the thread and never comes back.
>>
> While on paper this looks problematic -- have you seen this issue
> triggered in real life?
>
> This looks to be only triggered when there is another thread already
> holding the mutex, which seems rather rare in the life cycle of the
> machine?


Earlier we also suspected the same that someone was holding the lock, 
but actually there

was no owner of lock and it got scheduled out due to might sleep code in 
mutex_lock.

Looks like where voluntary preemption config is on, there it is getting 
scheduled out in might sleep.

But there is no need of unregister_syscore_ops as this is in crash path 
only, So removing the same.

>
>> To prevent the same remove unwanted unregister_syscore_ops function call.
>>
>> Signed-off-by: Gaurav Kohli <gauravkohli@linux.microsoft.com>
>>
>> diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
>> index f49bc3ec76e6..c050de69dfde 100644
>> --- a/arch/x86/hyperv/hv_init.c
>> +++ b/arch/x86/hyperv/hv_init.c
>> @@ -537,7 +537,12 @@ void hyperv_cleanup(void)
>>   	union hv_x64_msr_hypercall_contents hypercall_msr;
>>   	union hv_reference_tsc_msr tsc_msr;
>>   
>> -	unregister_syscore_ops(&hv_syscore_ops);
>> +	/*
>> +	 * Avoid unregister_syscore_ops(&hv_syscore_ops) from cleanup code,
>> +	 * as this is only called in crash path where irq and preemption disabled.
>> +	 * If we add this, there is a chance that this get scheduled out due to mutex
>> +	 * in unregister_syscore_ops and never comes back.
>> +	 */
> There is no need to document things we don't do, right?

Yes, we have added so to avoid the same in future.

>
>>   
>>   	/* Reset our OS id */
>>   	wrmsrl(HV_X64_MSR_GUEST_OS_ID, 0);
>> -- 
>> 2.17.1
>>

Re: [PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[Wei Liu]

On Fri, Nov 25, 2022 at 09:09:52PM +0530, Gaurav Kohli wrote:
> 
> On 11/25/2022 8:58 PM, Wei Liu wrote:
> > On Wed, Nov 23, 2022 at 09:23:11PM -0800, Gaurav Kohli wrote:
> > > Hyperv cleanup codes comes under panic path where preemption and irq
> > Please use "Hyper-V" throughout.
> Thanks for the comment, sure will do on v2.
> > 
> > > is already disabled. So calling of unregister_syscore_ops which has mutex
> > > from hyperv cleanup might schedule out the thread and never comes back.
> > > 
> > While on paper this looks problematic -- have you seen this issue
> > triggered in real life?
> > 
> > This looks to be only triggered when there is another thread already
> > holding the mutex, which seems rather rare in the life cycle of the
> > machine?
> 
> 
> Earlier we also suspected the same that someone was holding the lock, but
> actually there
> 
> was no owner of lock and it got scheduled out due to might sleep code in
> mutex_lock.
> 
> Looks like where voluntary preemption config is on, there it is getting
> scheduled out in might sleep.

If there is only one CPU online and the mutex is free, then
rescheduling will not have any adverse effect, right? Does it not get
scheduled on the only available CPU and make further progress?

> 
> But there is no need of unregister_syscore_ops as this is in crash path
> only, So removing the same.

I'm not against removing it, but the reasoning left in the commit
message and comment should reflect what happens.

Thanks,
Wei.

Re: [PATCH] x86/hyperv: Remove unregister syscore call from hyperv cleanup

[Gaurav Kohli]

On 11/25/2022 9:30 PM, Wei Liu wrote:
> On Fri, Nov 25, 2022 at 09:09:52PM +0530, Gaurav Kohli wrote:
>> On 11/25/2022 8:58 PM, Wei Liu wrote:
>>> On Wed, Nov 23, 2022 at 09:23:11PM -0800, Gaurav Kohli wrote:
>>>> Hyperv cleanup codes comes under panic path where preemption and irq
>>> Please use "Hyper-V" throughout.
>> Thanks for the comment, sure will do on v2.
>>>> is already disabled. So calling of unregister_syscore_ops which has mutex
>>>> from hyperv cleanup might schedule out the thread and never comes back.
>>>>
>>> While on paper this looks problematic -- have you seen this issue
>>> triggered in real life?
>>>
>>> This looks to be only triggered when there is another thread already
>>> holding the mutex, which seems rather rare in the life cycle of the
>>> machine?
>>
>> Earlier we also suspected the same that someone was holding the lock, but
>> actually there
>>
>> was no owner of lock and it got scheduled out due to might sleep code in
>> mutex_lock.
>>
>> Looks like where voluntary preemption config is on, there it is getting
>> scheduled out in might sleep.
> If there is only one CPU online and the mutex is free, then
> rescheduling will not have any adverse effect, right? Does it not get
> scheduled on the only available CPU and make further progress?

As this is in panic path where preemption and irq is disabled, so looks 
like it is not able

to schedule back to this cpu once scheduled out.

>> But there is no need of unregister_syscore_ops as this is in crash path
>> only, So removing the same.
> I'm not against removing it, but the reasoning left in the commit
> message and comment should reflect what happens.
thanks, will update the comment.
>
> Thanks,
> Wei.