pv_ops smp support

pv_ops smp support

[Jeremy Fitzhardinge]

I'm looking at adding Xen SMP support, so I'm trying to work out what 
pv_ops we need, and how to cut into the existing smp stuff.

smpboot.c has a mixture of stuff which is generally useful for SMP stuff 
(the various CPU sets, and presumably the sibling relationships are 
useful in principle), but also a whole pile of APIC stuff which is 
irrelevent to Xen.  It has these exported symbols, with my first pass 
comments:

    00000644 T __cpu_die			-- need pv_op
    000008a5 T __cpu_disable		-- need pv_op
    000006aa T __cpu_up			-- need pv_op
    00000000 T cpu_coregroup_map		-- ? ignore
    00000868 T cpu_exit_clear		-- ?
    00000d9f T initialize_secondary		-- Xen no-op
    000008ee T smp_alloc_memory		-- Xen no-op (doesn't matter if it gets called)
    00000057 T smp_cpus_done		-- unwanted for Xen
    00000015 T smp_intr_init		-- need something to set up IPIs, but APIC independent
    0000082a T smp_prepare_boot_cpu		-- looks OK for Xen, I think
    00000066 T smp_prepare_cpus		-- need pv_op
      


So at first pass, it looks like we need 6-8 new pv_ops for SMP, which 
isn't very appealing.

Any thoughts about how to come up with a more elegant interface?  I'm 
digging through smpboot.c and friends, but there's a lot of goo in there...

It's not clear to me what we should do with all the topology stuff.  
Clearly a VCPU won't have any fixed physical relationship with other 
VCPUs unless they're pinned; I'm wondering if the existing code will 
confuse itself if it gets basically random topology info from cpuid.

    J

Re: pv_ops smp support

[Zachary Amsden]

Jeremy Fitzhardinge wrote:
> I'm looking at adding Xen SMP support, so I'm trying to work out what 
> pv_ops we need, and how to cut into the existing smp stuff.
>
> So at first pass, it looks like we need 6-8 new pv_ops for SMP, which 
> isn't very appealing.
>
> Any thoughts about how to come up with a more elegant interface?  I'm 
> digging through smpboot.c and friends, but there's a lot of goo in 
> there...

Most of the goo is apic related goo, which can be conveniently be tucked 
out of the way by just ignoring or putting in appropriate hooks.  Before 
we try to dissect this into operations, I think there are two bigger 
fundamental questions you need to answer -

1) What do you plan to do to address per-cpu data structures?
2) What is your remote TLB shootdown model?

Both of these could have major impacts on how you want to carve up smp.c 
and smpboot.c.

Zach

Re: pv_ops smp support

[Jeremy Fitzhardinge]

Zachary Amsden wrote:
> 1) What do you plan to do to address per-cpu data structures?

Er, what's there at the moment, more or less.  The main thing is that 
the secondary CPU get the PDA set up (init gdt and %gs) before anyone 
wants to use it (which is generally the first use of smp_processor_id() 
or current). At some point we'll probably fold the PDA and PER_CPU 
together.  Xen can more or less completely initialize the VCPU state 
before it is brought up, so there's little or no need for any kind of 
bootstrap code.

> 2) What is your remote TLB shootdown model?

Xen has a hypercall to shoot down a set of CPU's TLBs, so it doesn't 
need to do an IPI (we'll need to extend the flush_tlb interface to make 
good use of this).  It will still need IPIs for reschedule and remote 
function calls or course.

> Both of these could have major impacts on how you want to carve up 
> smp.c and smpboot.c.

Re: pv_ops smp support

[Zachary Amsden]

Jeremy Fitzhardinge wrote:
> Zachary Amsden wrote:
>> 1) What do you plan to do to address per-cpu data structures?
>
> Er, what's there at the moment, more or less.  The main thing is that 
> the secondary CPU get the PDA set up (init gdt and %gs) before anyone 
> wants to use it (which is generally the first use of 
> smp_processor_id() or current). At some point we'll probably fold the 
> PDA and PER_CPU together.  Xen can more or less completely initialize 
> the VCPU state before it is brought up, so there's little or no need 
> for any kind of bootstrap code.

No, I don't mean the Linux PDA - how do you access the Xen PDA?  Or have 
they conjoined somehow?



>> 2) What is your remote TLB shootdown model?
>
> Xen has a hypercall to shoot down a set of CPU's TLBs, so it doesn't 
> need to do an IPI (we'll need to extend the flush_tlb interface to 
> make good use of this).  It will still need IPIs for reschedule and 
> remote function calls or course.

So your invalidate "IPI" is actually a hypercall, and you can use the 
existing flush_tlb interface for the most part.  You just need a 
paravirt-op then for the IPI itself, which takes a CPU mask - and this 
seems to match nicely onto your hypercall.  I think you might want to 
optimize this a bit more, however, since in some cases you will issue 
implicit shootdown IPIs during a pte update hypercall.

This might require some reworking of the MMU interfaces in paravirt-ops 
to accomodate providing such a record, and perhaps adding a function to 
flush the tlb_gather interface so that pte updates which have such 
properties can record the shootdown.

This seems much cleaner than designing the shootdown semantic directly 
into such PTE updates, which is I believe what some of the older Xen 
patches did, although I could have misread them.

Zach

Re: pv_ops smp support

[Jeremy Fitzhardinge]

Zachary Amsden wrote:
> No, I don't mean the Linux PDA - how do you access the Xen PDA?  Or 
> have they conjoined somehow?

You could put it that way I guess.  There's a generic Linux PDA; a 
paravirt patch adds a union for pv use, and a Xen patch adds a 
Xen-specific element to that union.  That's how it has been from the 
start, so there wasn't really anything to conjoin (there was never a Xen 
PDA per se).

> So your invalidate "IPI" is actually a hypercall, and you can use the 
> existing flush_tlb interface for the most part.  You just need a 
> paravirt-op then for the IPI itself, which takes a CPU mask - and this 
> seems to match nicely onto your hypercall.
Yep.  There are calls for flushing the whole tlb, and for just a page; 
both take CPU masks.

> I think you might want to optimize this a bit more, however, since in 
> some cases you will issue implicit shootdown IPIs during a pte update 
> hypercall.

Not that I've seen, at least none that doesn't also exist in baseline.

> This seems much cleaner than designing the shootdown semantic directly 
> into such PTE updates, which is I believe what some of the older Xen 
> patches did, although I could have misread them.

I was starting on SMP with the idea that it would be relatively isolated 
and simple, but it seems I should probably do the MMU stuff first to see 
what impact it has on SMP.

    J

Re: pv_ops smp support

[Zachary Amsden]

Jeremy Fitzhardinge wrote:
>
>> I think you might want to optimize this a bit more, however, since in 
>> some cases you will issue implicit shootdown IPIs during a pte update 
>> hypercall.
>
> Not that I've seen, at least none that doesn't also exist in baseline.

But if Xen uses direct page tables, the update to a page table through a 
hypercall can cause other CPUs with that direct page table loaded to 
require an invalidation.  <Pauses>.  Never mind, I see that you don't 
require an immediate invalidation.  Instead, you use page type 
enforcement to guarantee that page remappings can not be used to subvert 
hypervisor protection, and would either disallow or issue TLB 
invalidations on page type changes from PT to writable pages.  In that 
case, the guest must use TLB shootdown to guarantee correctness, but no 
extra TLB shootdowns are required to guarantee hypervisor security, as I 
had feared.

>
>> This seems much cleaner than designing the shootdown semantic 
>> directly into such PTE updates, which is I believe what some of the 
>> older Xen patches did, although I could have misread them.
>
> I was starting on SMP with the idea that it would be relatively 
> isolated and simple, but it seems I should probably do the MMU stuff 
> first to see what impact it has on SMP.

I think I was imagining some complications which really don't exist, but 
it certainly doesn't hurt to consider the Xen MMU interfaces as you iron 
out the SMP issues.

So perhaps SMP will be more straightforward than I expected.

Zach