[PATCH 0/2] KVM: fix and cleanup: kvm_lock and hardware_disable

[PATCH 0/2] KVM: fix and cleanup: kvm_lock and hardware_disable

[Takuya Yoshikawa]

Hello!

During investigating kvm's mutual exclusions, starting from checking
kvm's srcu grace periods, I could not understand some of the locking rules.

This one is an example which I doubt.
But I'm not so sure. Please check!

Thanks,
  Takuya

[PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Takuya Yoshikawa]

In kvm_cpu_hotplug(), only CPU_STARTING case is protected by kvm_lock.
This patch adds missing protection for CPU_DYING case.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
---
 virt/kvm/kvm_main.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 339dd43..0fdd911 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
 	case CPU_DYING:
 		printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
 		       cpu);
+		spin_lock(&kvm_lock);
 		hardware_disable(NULL);
+		spin_unlock(&kvm_lock);
 		break;
 	case CPU_STARTING:
 		printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
-- 
1.7.1

[PATCH 2/2] KVM: rename hardware_[dis|en]able() to *_nolock() and add locking wrappers

[Takuya Yoshikawa]

The naming convension of hardware_[dis|en]able family is little bit confusing
because only hardware_[dis|en]able_all are using _nolock suffix.

Renaming current hardware_[dis|en]able() to *_nolock() and using
hardware_[dis|en]able() as wrapper functions which take kvm_lock for them
reduces extra confusion.

Signed-off-by: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
---
 virt/kvm/kvm_main.c |   34 ++++++++++++++++++++++------------
 1 files changed, 22 insertions(+), 12 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 0fdd911..fb93ff9 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2067,7 +2067,7 @@ static struct miscdevice kvm_dev = {
 	&kvm_chardev_ops,
 };
 
-static void hardware_enable(void *junk)
+static void hardware_enable_nolock(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 	int r;
@@ -2087,7 +2087,14 @@ static void hardware_enable(void *junk)
 	}
 }
 
-static void hardware_disable(void *junk)
+static void hardware_enable(void *junk)
+{
+	spin_lock(&kvm_lock);
+	hardware_enable_nolock(junk);
+	spin_unlock(&kvm_lock);
+}
+
+static void hardware_disable_nolock(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 
@@ -2097,13 +2104,20 @@ static void hardware_disable(void *junk)
 	kvm_arch_hardware_disable(NULL);
 }
 
+static void hardware_disable(void *junk)
+{
+	spin_lock(&kvm_lock);
+	hardware_disable_nolock(junk);
+	spin_unlock(&kvm_lock);
+}
+
 static void hardware_disable_all_nolock(void)
 {
 	BUG_ON(!kvm_usage_count);
 
 	kvm_usage_count--;
 	if (!kvm_usage_count)
-		on_each_cpu(hardware_disable, NULL, 1);
+		on_each_cpu(hardware_disable_nolock, NULL, 1);
 }
 
 static void hardware_disable_all(void)
@@ -2122,7 +2136,7 @@ static int hardware_enable_all(void)
 	kvm_usage_count++;
 	if (kvm_usage_count == 1) {
 		atomic_set(&hardware_enable_failed, 0);
-		on_each_cpu(hardware_enable, NULL, 1);
+		on_each_cpu(hardware_enable_nolock, NULL, 1);
 
 		if (atomic_read(&hardware_enable_failed)) {
 			hardware_disable_all_nolock();
@@ -2148,16 +2162,12 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
 	case CPU_DYING:
 		printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
 		       cpu);
-		spin_lock(&kvm_lock);
 		hardware_disable(NULL);
-		spin_unlock(&kvm_lock);
 		break;
 	case CPU_STARTING:
 		printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
 		       cpu);
-		spin_lock(&kvm_lock);
 		hardware_enable(NULL);
-		spin_unlock(&kvm_lock);
 		break;
 	}
 	return NOTIFY_OK;
@@ -2188,7 +2198,7 @@ static int kvm_reboot(struct notifier_block *notifier, unsigned long val,
 	 */
 	printk(KERN_INFO "kvm: exiting hardware virtualization\n");
 	kvm_rebooting = true;
-	on_each_cpu(hardware_disable, NULL, 1);
+	on_each_cpu(hardware_disable_nolock, NULL, 1);
 	return NOTIFY_OK;
 }
 
@@ -2358,7 +2368,7 @@ static void kvm_exit_debug(void)
 static int kvm_suspend(struct sys_device *dev, pm_message_t state)
 {
 	if (kvm_usage_count)
-		hardware_disable(NULL);
+		hardware_disable_nolock(NULL);
 	return 0;
 }
 
@@ -2366,7 +2376,7 @@ static int kvm_resume(struct sys_device *dev)
 {
 	if (kvm_usage_count) {
 		WARN_ON(spin_is_locked(&kvm_lock));
-		hardware_enable(NULL);
+		hardware_enable_nolock(NULL);
 	}
 	return 0;
 }
@@ -2543,7 +2553,7 @@ void kvm_exit(void)
 	sysdev_class_unregister(&kvm_sysdev_class);
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
-	on_each_cpu(hardware_disable, NULL, 1);
+	on_each_cpu(hardware_disable_nolock, NULL, 1);
 	kvm_arch_hardware_unsetup();
 	kvm_arch_exit();
 	free_cpumask_var(cpus_hardware_enabled);
-- 
1.7.1

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Zachary Amsden]

On 11/15/2010 10:35 PM, Takuya Yoshikawa wrote:
> In kvm_cpu_hotplug(), only CPU_STARTING case is protected by kvm_lock.
> This patch adds missing protection for CPU_DYING case.
>
> Signed-off-by: Takuya Yoshikawa<yoshikawa.takuya@oss.ntt.co.jp>
> ---
>   virt/kvm/kvm_main.c |    2 ++
>   1 files changed, 2 insertions(+), 0 deletions(-)
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 339dd43..0fdd911 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
>   	case CPU_DYING:
>   		printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
>   		       cpu);
> +		spin_lock(&kvm_lock);
>   		hardware_disable(NULL);
> +		spin_unlock(&kvm_lock);
>   		break;
>   	case CPU_STARTING:
>   		printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
>    

I believe this is correct.

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Takuya Yoshikawa]

(2010/11/18 10:59), Zachary Amsden wrote:
> On 11/15/2010 10:35 PM, Takuya Yoshikawa wrote:
>> In kvm_cpu_hotplug(), only CPU_STARTING case is protected by kvm_lock.
>> This patch adds missing protection for CPU_DYING case.
>>
>> Signed-off-by: Takuya Yoshikawa<yoshikawa.takuya@oss.ntt.co.jp>
>> ---
>> virt/kvm/kvm_main.c | 2 ++
>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 339dd43..0fdd911 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
>> case CPU_DYING:
>> printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
>> cpu);
>> + spin_lock(&kvm_lock);
>> hardware_disable(NULL);
>> + spin_unlock(&kvm_lock);
>> break;
>> case CPU_STARTING:
>> printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
>
> I believe this is correct.

You mean lock is not necessary?

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Zachary Amsden]

On 11/17/2010 04:04 PM, Takuya Yoshikawa wrote:
> (2010/11/18 10:59), Zachary Amsden wrote:
>> On 11/15/2010 10:35 PM, Takuya Yoshikawa wrote:
>>> In kvm_cpu_hotplug(), only CPU_STARTING case is protected by kvm_lock.
>>> This patch adds missing protection for CPU_DYING case.
>>>
>>> Signed-off-by: Takuya Yoshikawa<yoshikawa.takuya@oss.ntt.co.jp>
>>> ---
>>> virt/kvm/kvm_main.c | 2 ++
>>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>>> index 339dd43..0fdd911 100644
>>> --- a/virt/kvm/kvm_main.c
>>> +++ b/virt/kvm/kvm_main.c
>>> @@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct 
>>> notifier_block *notifier, unsigned long val,
>>> case CPU_DYING:
>>> printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
>>> cpu);
>>> + spin_lock(&kvm_lock);
>>> hardware_disable(NULL);
>>> + spin_unlock(&kvm_lock);
>>> break;
>>> case CPU_STARTING:
>>> printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
>>
>> I believe this is correct.
>
> You mean lock is not necessary?

No, I believe your patch is correct and the lock should be there.  Did 
you test with spinlock debugging just to be sure?

Thanks,

Zach

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Takuya Yoshikawa]

(2010/11/18 11:33), Zachary Amsden wrote:
> On 11/17/2010 04:04 PM, Takuya Yoshikawa wrote:
>> (2010/11/18 10:59), Zachary Amsden wrote:
>>> On 11/15/2010 10:35 PM, Takuya Yoshikawa wrote:
>>>> In kvm_cpu_hotplug(), only CPU_STARTING case is protected by kvm_lock.
>>>> This patch adds missing protection for CPU_DYING case.
>>>>
>>>> Signed-off-by: Takuya Yoshikawa<yoshikawa.takuya@oss.ntt.co.jp>
>>>> ---
>>>> virt/kvm/kvm_main.c | 2 ++
>>>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>>>> index 339dd43..0fdd911 100644
>>>> --- a/virt/kvm/kvm_main.c
>>>> +++ b/virt/kvm/kvm_main.c
>>>> @@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
>>>> case CPU_DYING:
>>>> printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
>>>> cpu);
>>>> + spin_lock(&kvm_lock);
>>>> hardware_disable(NULL);
>>>> + spin_unlock(&kvm_lock);
>>>> break;
>>>> case CPU_STARTING:
>>>> printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
>>>
>>> I believe this is correct.
>>
>> You mean lock is not necessary?
>
> No, I believe your patch is correct and the lock should be there. Did you test with spinlock debugging just to be sure?
>

Sorry but no.

I have no experience with cpu hotplug.

So I thought it would take too much time to do real test by myself and reported like this this time.

Any easy way to test?

   Takuya

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Zachary Amsden]

On 11/17/2010 04:41 PM, Takuya Yoshikawa wrote:
> (2010/11/18 11:33), Zachary Amsden wrote:
>> On 11/17/2010 04:04 PM, Takuya Yoshikawa wrote:
>>> (2010/11/18 10:59), Zachary Amsden wrote:
>>>> On 11/15/2010 10:35 PM, Takuya Yoshikawa wrote:
>>>>> In kvm_cpu_hotplug(), only CPU_STARTING case is protected by 
>>>>> kvm_lock.
>>>>> This patch adds missing protection for CPU_DYING case.
>>>>>
>>>>> Signed-off-by: Takuya Yoshikawa<yoshikawa.takuya@oss.ntt.co.jp>
>>>>> ---
>>>>> virt/kvm/kvm_main.c | 2 ++
>>>>> 1 files changed, 2 insertions(+), 0 deletions(-)
>>>>>
>>>>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>>>>> index 339dd43..0fdd911 100644
>>>>> --- a/virt/kvm/kvm_main.c
>>>>> +++ b/virt/kvm/kvm_main.c
>>>>> @@ -2148,7 +2148,9 @@ static int kvm_cpu_hotplug(struct 
>>>>> notifier_block *notifier, unsigned long val,
>>>>> case CPU_DYING:
>>>>> printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
>>>>> cpu);
>>>>> + spin_lock(&kvm_lock);
>>>>> hardware_disable(NULL);
>>>>> + spin_unlock(&kvm_lock);
>>>>> break;
>>>>> case CPU_STARTING:
>>>>> printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
>>>>
>>>> I believe this is correct.
>>>
>>> You mean lock is not necessary?
>>
>> No, I believe your patch is correct and the lock should be there. Did 
>> you test with spinlock debugging just to be sure?
>>
>
> Sorry but no.
>
> I have no experience with cpu hotplug.
>
> So I thought it would take too much time to do real test by myself and 
> reported like this this time.
>
> Any easy way to test?

Yes, quite easy.  Some systems may not let cpu0 go offline, but you can 
manually disable and re-enable the other processors:

[root@mysore ~]# echo "0" > /sys/devices/system/cpu/cpu1/online
[root@mysore ~]# echo "1" > /sys/devices/system/cpu/cpu1/online

Cheers,

Zach

Re: [PATCH 1/2] KVM: take kvm_lock for hardware_disable() during cpu hotplug

[Takuya Yoshikawa]

(2010/11/18 14:45), Zachary Amsden wrote:

>>> No, I believe your patch is correct and the lock should be there. Did you test with spinlock debugging just to be sure?
>>>
>>
>> Sorry but no.
>>
>> I have no experience with cpu hotplug.
>>
>> So I thought it would take too much time to do real test by myself and reported like this this time.
>>
>> Any easy way to test?
>
> Yes, quite easy. Some systems may not let cpu0 go offline, but you can manually disable and re-enable the other processors:
>
> [root@mysore ~]# echo "0" > /sys/devices/system/cpu/cpu1/online
> [root@mysore ~]# echo "1" > /sys/devices/system/cpu/cpu1/online
>
> Cheers,
>
> Zach


Thanks a lot!

I tried and got a log like this:

kernel: [  422.084620] kvm: disabling virtualization on CPU1
kernel: [  422.085757] CPU 1 is now offline
kernel: [  422.085766] lockdep: fixing up alternatives.
kernel: [  422.085780] SMP alternatives: switching to UP code
kernel: [  472.081069] lockdep: fixing up alternatives.
kernel: [  472.081080] SMP alternatives: switching to SMP code
kernel: [  472.099182] Booting Node 0 Processor 1 APIC 0x1
kernel: [  422.104799] kvm: enabling virtualization on CPU1

Working correctly, I think.


   Takuya

Re: [PATCH 0/2] KVM: fix and cleanup: kvm_lock and hardware_disable

[Marcelo Tosatti]

On Tue, Nov 16, 2010 at 05:32:44PM +0900, Takuya Yoshikawa wrote:
> Hello!
> 
> During investigating kvm's mutual exclusions, starting from checking
> kvm's srcu grace periods, I could not understand some of the locking rules.
> 
> This one is an example which I doubt.
> But I'm not so sure. Please check!
> 
> Thanks,
>   Takuya

Applied, thanks.