[PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Rusty Russell]

This changes cpus_hardware_enabled from a cpumask_t to a cpumask_var_t:
equivalent for CONFIG_CPUMASKS_OFFSTACK=n, otherwise dynamically allocated.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
---
 virt/kvm/kvm_main.c |   20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -63,7 +63,7 @@ DEFINE_SPINLOCK(kvm_lock);
 DEFINE_SPINLOCK(kvm_lock);
 LIST_HEAD(vm_list);
 
-static cpumask_t cpus_hardware_enabled;
+static cpumask_var_t cpus_hardware_enabled;
 
 struct kmem_cache *kvm_vcpu_cache;
 EXPORT_SYMBOL_GPL(kvm_vcpu_cache);
@@ -1796,9 +1796,9 @@ static void hardware_enable(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 
-	if (cpu_isset(cpu, cpus_hardware_enabled))
+	if (cpumask_test_cpu(cpu, cpus_hardware_enabled))
 		return;
-	cpu_set(cpu, cpus_hardware_enabled);
+	cpumask_set_cpu(cpu, cpus_hardware_enabled);
 	kvm_arch_hardware_enable(NULL);
 }
 
@@ -1806,9 +1806,9 @@ static void hardware_disable(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 
-	if (!cpu_isset(cpu, cpus_hardware_enabled))
+	if (!cpumask_test_cpu(cpu, cpus_hardware_enabled))
 		return;
-	cpu_clear(cpu, cpus_hardware_enabled);
+	cpumask_clear_cpu(cpu, cpus_hardware_enabled);
 	kvm_arch_hardware_disable(NULL);
 }
 
@@ -2042,9 +2042,14 @@ int kvm_init(void *opaque, unsigned int 
 
 	bad_pfn = page_to_pfn(bad_page);
 
+	if (!alloc_cpumask_var(&cpus_hardware_enabled, GFP_KERNEL)) {
+		r = -ENOMEM;
+		goto out_free_0;
+	}
+
 	r = kvm_arch_hardware_setup();
 	if (r < 0)
-		goto out_free_0;
+		goto out_free_0a;
 
 	for_each_online_cpu(cpu) {
 		smp_call_function_single(cpu,
@@ -2103,6 +2108,8 @@ out_free_2:
 	on_each_cpu(hardware_disable, NULL, 1);
 out_free_1:
 	kvm_arch_hardware_unsetup();
+out_free_0a:
+	free_cpumask_var(cpus_hardware_enabled);
 out_free_0:
 	__free_page(bad_page);
 out:
@@ -2126,6 +2133,7 @@ void kvm_exit(void)
 	kvm_arch_hardware_unsetup();
 	kvm_arch_exit();
 	kvm_exit_debug();
+	free_cpumask_var(cpus_hardware_enabled);
 	__free_page(bad_page);
 }
 EXPORT_SYMBOL_GPL(kvm_exit);

Re: [PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Avi Kivity]

Rusty Russell wrote:
> This changes cpus_hardware_enabled from a cpumask_t to a cpumask_var_t:
> equivalent for CONFIG_CPUMASKS_OFFSTACK=n, otherwise dynamically allocated.
>
>  
> -static cpumask_t cpus_hardware_enabled;
> +static cpumask_var_t cpus_hardware_enabled

This isn't on stack, so it isn't buying us anything.

Is the plan to drop cpumask_t?  If so, we're penalizing non-stack users 
by forcing them to go through another pointer (and cacheline).



-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Rusty Russell]

On Monday 08 December 2008 02:25:50 Avi Kivity wrote:
> Rusty Russell wrote:
> > This changes cpus_hardware_enabled from a cpumask_t to a cpumask_var_t:
> > equivalent for CONFIG_CPUMASKS_OFFSTACK=n, otherwise dynamically allocated.
> >
> >  
> > -static cpumask_t cpus_hardware_enabled;
> > +static cpumask_var_t cpus_hardware_enabled
> 
> This isn't on stack, so it isn't buying us anything.

It's the CONFIG_NR_CPUS=4096 but nr_cpu_ids=4 case which we win using
dynamic allocation.  Gotta love distribution kernels.

> Is the plan to drop cpumask_t?

Yes.  And undefine 'struct cpumask' if CONFIG_CPUMASK_OFFSTACK.  That
will stop assignment and on-stack declarations for all but the most
determined.

> If so, we're penalizing non-stack users 
> by forcing them to go through another pointer (and cacheline).

Not quite.  If !CONFIG_CPUMASK_OFFSTACK, cpumask_var_t == cpumask_t[1].
Blame Linus :)

Cheers,
Rusty.

Re: [PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Avi Kivity]

Rusty Russell wrote:
>> This isn't on stack, so it isn't buying us anything.
>>     
>
> It's the CONFIG_NR_CPUS=4096 but nr_cpu_ids=4 case which we win using
> dynamic allocation.  Gotta love distribution kernels.
>
>   

What does it buy? 4096/8 = 512 bytes statically allocated?

I understand passing things as pointers, but allocating everything 
dynamically is unCish.

>> Is the plan to drop cpumask_t?
>>     
>
> Yes.  And undefine 'struct cpumask' if CONFIG_CPUMASK_OFFSTACK.  That
> will stop assignment and on-stack declarations for all but the most
> determined.
>
>   
>> If so, we're penalizing non-stack users 
>> by forcing them to go through another pointer (and cacheline).
>>     
>
> Not quite.  If !CONFIG_CPUMASK_OFFSTACK, cpumask_var_t == cpumask_t[1].
> Blame Linus :)
>   

Hm, is there a C trick which will error out when allocating something on 
the stack, but work when allocating statically?  I can think of 
something to do the reverse, but that doesn't help.

Maybe a weak or visibility attribute?  These don't make sense on 
function locals.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Rusty Russell]

On Monday 08 December 2008 20:16:44 Avi Kivity wrote:
> Rusty Russell wrote:
> >> This isn't on stack, so it isn't buying us anything.
> >>     
> >
> > It's the CONFIG_NR_CPUS=4096 but nr_cpu_ids=4 case which we win using
> > dynamic allocation.  Gotta love distribution kernels.
> >
> >   
> 
> What does it buy? 4096/8 = 512 bytes statically allocated?

It adds up, and 4096 seems to be only the start of the insanityH^H^Hfun.

> > Not quite.  If !CONFIG_CPUMASK_OFFSTACK, cpumask_var_t == cpumask_t[1].
> > Blame Linus :)
> >   
> 
> Hm, is there a C trick which will error out when allocating something on 
> the stack, but work when allocating statically?  I can think of 
> something to do the reverse, but that doesn't help.

We also need to prevent assignment, eg:

	*foo = *bar;

Because when we allocate them, we'll cut them to size.

Cheers,
Rusty.

Re: [PATCH 2/2] kvm: use cpumask_var_t for cpus_hardware_enabled

[Mike Travis]

Rusty Russell wrote:
> On Monday 08 December 2008 20:16:44 Avi Kivity wrote:
>> Rusty Russell wrote:
>>>> This isn't on stack, so it isn't buying us anything.
>>>>     
>>> It's the CONFIG_NR_CPUS=4096 but nr_cpu_ids=4 case which we win using
>>> dynamic allocation.  Gotta love distribution kernels.
>>>
>>>   
>> What does it buy? 4096/8 = 512 bytes statically allocated?
> 
> It adds up, and 4096 seems to be only the start of the insanityH^H^Hfun.

The real win though is when cpumask_size represents the actual size of the
cpumask (based on # of possible cpus) instead of the pre-configured size
of NR_CPUS.  So for 99.9% of the systems (having 64 or fewer cpus), the
savings will be 504 bytes not allocated.

> 
>>> Not quite.  If !CONFIG_CPUMASK_OFFSTACK, cpumask_var_t == cpumask_t[1].
>>> Blame Linus :)
>>>   
>> Hm, is there a C trick which will error out when allocating something on 
>> the stack, but work when allocating statically?  I can think of 
>> something to do the reverse, but that doesn't help.
> 
> We also need to prevent assignment, eg:
> 
> 	*foo = *bar;
> 
> Because when we allocate them, we'll cut them to size.
> 
> Cheers,
> Rusty.