RE: Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)

RE: Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)

[Magenheimer, Dan (HP Labs Fort Collins)]

I understand and sympathize with the need for dom0 to
sometimes get and use information from each processor
that is only available if dom0 is running on each processor.

However, AFAIK, SMP guests are always gang-scheduled, correct?
(If not, aren't there some very knotty research issues related
to locking and forward progress?)

So on a 16-processor system, every time dom0 needs to
run (e.g. to handle backend I/O for any one of perhaps hundreds
of domains), *every* domain gets descheduled so that dom0
can be (gang-)scheduled on all 16 processors?

If true, this sounds like a _horrible_ performance hit, so
I hope I'm misunderstanding something...

Dan

> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
> Sent: Tuesday, April 04, 2006 1:26 AM
> To: Tian, Kevin
> Cc: xen-devel; xen-ia64-devel@lists.xensource.com; 
> Magenheimer, Dan (HP Labs Fort Collins); Tristan Gingold
> Subject: Re: [Xen-devel] Does dom0 see all physical 
> processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)
> 
> 
> On 4 Apr 2006, at 03:17, Tian, Kevin wrote:
> 
> >
> > Then consider your question about a large box with many processors.
> > How about the real environment? Is it the case to provide a 
> 16-way SMP
> > box, or a 16-way NUMA box? I prefer to the latter. If it's 
> a NUMA box,
> > dom0 sees physical ACPI table and can be configured as NUMA aware.
> 
> This is a model we must support if we are to have domain0 
> handle other 
> processor-related ACPI activities (e.g., power management). The power 
> information and available settings won't make much sense to the user 
> unless there's an equivalence between VCPUs and PCPUs for domain0.
> 
>   -- Keir
> 
> 

Re: Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)

[Ryan Harper]

* Magenheimer, Dan (HP Labs Fort Collins) <dan.magenheimer@hp.com> [2006-04-07 01:43]:
> I understand and sympathize with the need for dom0 to
> sometimes get and use information from each processor
> that is only available if dom0 is running on each processor.
> 
> However, AFAIK, SMP guests are always gang-scheduled, correct?

I don't believe either the bvt or sedf scheduler in Xen provide any gang
scheduling support.  Each physical cpu has its own runqueue and it
schedules VCPUs independently.  The scheduling parameters are set on a
per-domain basis but end up being the same for each VCPU.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)

[Ian Pratt]

> I believe ppc has "paravirtualized spinlocks" in their Linux 
> kernel, though even this won't necessarily help with a poorly 
> written SMP application.

We have an equivalent of this ("bad pre-emption mitigation"), along with
an alternative ("bad pre-emption avoidance"). Both have various pros and
cons, and can be shown to offer significant benefits in various
contrived benchmarks.

We have benchmark code that records when these bad pre-emptions happen,
and the workloads we've looked at we haven't been moved to check either
scheme in.

That's not to say that we won't have to at some point in the future, but
the limits to scalability are currently elsewhere. 

> > > So on a 16-processor system, every time dom0 needs to run 
> (e.g. to 
> > > handle backend I/O for any one of perhaps hundreds of domains), 
> > > *every* domain gets descheduled so that dom0 can be 
> (gang-)scheduled 
> > > on all 16 processors?
> > > 
> > > If true, this sounds like a _horrible_ performance hit, so I hope 
> > > I'm misunderstanding something...
> > 
> > This isn't an issue.
> > 
> > After booting you probably want dom0 to give up all but 1 
> vCPU anyway.
> 
> Unless of course the PCPU's have data that change over time, 
> such as variable cycle rate (for power management) or 
> hot-plug memory...

Such events don't happen very often -- dom0 is still free to run
something on a given CPU whenever it wants to.

Ian

RE: Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)

[Magenheimer, Dan (HP Labs Fort Collins)]

(Orran/Jimi cc'ed, see question below...)

> > I understand and sympathize with the need for dom0 to 
> > sometimes get and use information from each processor that is 
> > only available if dom0 is running on each processor.
> > 
> > However, AFAIK, SMP guests are always gang-scheduled, correct?
> 
> No, there's no need to strictly gang schedule, and the 
> current scheduler makes no attempt to do so. It may generally 
> be a decent thing to do, though.
> 
> > (If not, aren't there some very knotty research issues 
> > related to locking and forward progress?)
> 
> You could end up preempting a vCPU holding a lock which could 
> lead to daft behaviour of naïve spin locks. A number of 
> possible workarounds have been prototyped, but since it 
> doesn't seem to be much of a problem in practice nothing has 
> been checked in.

I wonder if "not a problem in practice" is more of an indication
of lack of practice than lack of problem.  I can see that the
problem would be unlikely to occur with small numbers of
processors and one SMP guests running a highly scalable SMP app
(such as a web server), but I'll bet a real enterprise load
of home-grown SMP apps running in a IT shop that's had big SMP
boxes for years would see the problem more quickly, especially
after multiple SMP guests are consolidated onto a single box.

I believe ppc has "paravirtualized spinlocks" in their Linux
kernel, though even this won't necessarily help with a poorly
written SMP application.

No data, admittedly, but perhaps our good buddies at
Watson could comment?

> > So on a 16-processor system, every time dom0 needs to run 
> > (e.g. to handle backend I/O for any one of perhaps hundreds 
> > of domains), *every* domain gets descheduled so that dom0 can 
> > be (gang-)scheduled on all 16 processors?
> > 
> > If true, this sounds like a _horrible_ performance hit, so I 
> > hope I'm misunderstanding something...
> 
> This isn't an issue.
> 
> After booting you probably want dom0 to give up all but 1 vCPU anyway.

Unless of course the PCPU's have data that change over time, such
as variable cycle rate (for power management) or hot-plug memory...
 
> Ian
>

Dan

Re: Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)

[Keir Fraser]

On 5 Apr 2006, at 15:07, Magenheimer, Dan (HP Labs Fort Collins) wrote:

> I believe ppc has "paravirtualized spinlocks" in their Linux
> kernel, though even this won't necessarily help with a poorly
> written SMP application.
>
> No data, admittedly, but perhaps our good buddies at
> Watson could comment?

IBM did some investigation into this a while back. The conclusion then 
was that, even in a microbenchmark somewhat contrived to try and 
indicate 'worst case' spinlock behaviour, the benefit of smarter 
spinlocks was negligible. Maybe the benchmark was bad, maybe there are 
other pathological worst cases out there that the benchmark did not 
represent, or maybe the smart spinlocks should have been smarter. 
Whatever: the numbers we have so far do not indicate that there's a 
problem to be solved, and these potential multi-processor optimisations 
have to be empirically driven because intuition is so frequently off 
the mark.

  -- Keir

RE: Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)

[Stephen C. Tweedie]

Hi,

On Wed, 2006-04-05 at 07:07 -0700, Magenheimer, Dan (HP Labs Fort
Collins) wrote:

> I believe ppc has "paravirtualized spinlocks" in their Linux
> kernel

ppc64 does.  From include/asm-ppc64/spinlock.h:

/*
 * On a system with shared processors (that is, where a physical
 * processor is multiplexed between several virtual processors),
 * there is no point spinning on a lock if the holder of the lock
 * isn't currently scheduled on a physical processor.  Instead
 * we detect this situation and ask the hypervisor to give the
 * rest of our timeslice to the lock holder.
 *
 * So that we can tell which virtual processor is holding a lock,
 * we put 0x80000000 | smp_processor_id() in the lock when it is
 * held.  Conveniently, we have a word in the paca that holds this
 * value.
 */

A quick hack for this which simply did a yield sched_op when it detected
a collision could be done in the same way from a paravirtualised kernel
without any HV changes; but I think it would take extra scheduler help
to do a directed yield to a specific CPU.

--Stephen

RE: Does dom0 see all physical processors? (RE:[Xen-ia64-devel] SAL INFO virtualization)

[Ian Pratt]

> I understand and sympathize with the need for dom0 to 
> sometimes get and use information from each processor that is 
> only available if dom0 is running on each processor.
> 
> However, AFAIK, SMP guests are always gang-scheduled, correct?

No, there's no need to strictly gang schedule, and the current scheduler makes no attempt to do so. It may generally be a decent thing to do, though.

> (If not, aren't there some very knotty research issues 
> related to locking and forward progress?)

You could end up preempting a vCPU holding a lock which could lead to daft behaviour of naïve spin locks. A number of possible workarounds have been prototyped, but since it doesn't seem to be much of a problem in practice nothing has been checked in.

> So on a 16-processor system, every time dom0 needs to run 
> (e.g. to handle backend I/O for any one of perhaps hundreds 
> of domains), *every* domain gets descheduled so that dom0 can 
> be (gang-)scheduled on all 16 processors?
> 
> If true, this sounds like a _horrible_ performance hit, so I 
> hope I'm misunderstanding something...

This isn't an issue.

After booting you probably want dom0 to give up all but 1 vCPU anyway.

Ian

RE: Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)

[Tian, Kevin]

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2006年4月4日 15:26
>
>
>On 4 Apr 2006, at 03:17, Tian, Kevin wrote:
>
>>
>> Then consider your question about a large box with many processors.
>> How about the real environment? Is it the case to provide a 16-way
>SMP
>> box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box,
>> dom0 sees physical ACPI table and can be configured as NUMA
>aware.
>
>This is a model we must support if we are to have domain0 handle other
>processor-related ACPI activities (e.g., power management). The power
>information and available settings won't make much sense to the user
>unless there's an equivalence between VCPUs and PCPUs for domain0.
>
>  -- Keir

Yes, and thanks for making it clearer. (Originally I only came up
performance reason for this model.)

Thanks,
Kevin

Does dom0 see all physical processors? (RE: SAL INFO virtualization)

[Tian, Kevin]

>From: Magenheimer, Dan (HP Labs Fort Collins)
>[mailto:dan.magenheimer@hp.com]
>Sent: 2006年4月3日 23:55
>>
>> Just an unrelated comment, Dom0 should always run on every
>physical
>> processor, which is the base requirement for performance
>> reason. It's not
>> "therefore" result of this specific issue and instead you can
>> take it as the
>> basic assumption for your thought. :-)
>>
>> Thanks,
>> Kevin
>
>Hmmm... are you saying that on a 16-processor Xen system,
>domain0 will run as a 16-way SMP guest?

OK, seems that I didn't describe the case clearly. :-) For current xen
model, dom0 default runs on all physical cpus which is however 
configurable.:

    if ( opt_dom0_max_vcpus == 0 )
        opt_dom0_max_vcpus = num_online_cpus();
    if ( opt_dom0_max_vcpus > MAX_VIRT_CPUS )
        opt_dom0_max_vcpus = MAX_VIRT_CPUS;
    printk("Dom0 has maximum %u VCPUs\n", opt_dom0_max_vcpus);

    for ( i = 1; i < opt_dom0_max_vcpus; i++ )
        (void)alloc_vcpu(d, i, i); /* vcpu N runs on physical cpu N */

In this case, max vcpus of dom0 is configurable but each vcpu is running 
at different physical processor once allocated. For a typical SMP box, 
such model can provide better performance or else dom0 will become the 
bottle neck if only owning one physical processor.

Then consider your question about a large box with many processors. 
How about the real environment? Is it the case to provide a 16-way SMP 
box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box, 
dom0 sees physical ACPI table and can be configured as NUMA aware.

Yes, such binding model may change later but it's always bad to have 
dom0 only running on single physical processor.

CC to a wider list for comments. :-)

Thanks,
Kevin

>
>This isn't the case today with Xen (x86 or ia64) and I
>don't expect that it will work this way in the future
>either.
>
>Dan

Re: Does dom0 see all physical processors? (RE: [Xen-ia64-devel] SAL INFO virtualization)

[Keir Fraser]

On 4 Apr 2006, at 03:17, Tian, Kevin wrote:

>
> Then consider your question about a large box with many processors.
> How about the real environment? Is it the case to provide a 16-way SMP
> box, or a 16-way NUMA box? I prefer to the latter. If it's a NUMA box,
> dom0 sees physical ACPI table and can be configured as NUMA aware.

This is a model we must support if we are to have domain0 handle other 
processor-related ACPI activities (e.g., power management). The power 
information and available settings won't make much sense to the user 
unless there's an equivalence between VCPUs and PCPUs for domain0.

  -- Keir