[RFC] Genapic in x86_64 Dom0

[RFC] Genapic in x86_64 Dom0

[Puthiyaparambil, Aravindh]

I am working on a patch on to accommodate clustered APICs in x86_64
Dom0/U Genapic. At the moment it defaults to flat. I am thinking of
splitting genapic_xen into genapic_xen_flat and genapic_xen_cluster. 

My question is that since the most of the code that checks for clustered
vs flat has been IFDEFed out what is the best way to inform the Dom that
it needs to run in clustered/flat APIC mode? Should I be thinking about
introducing a new hypercall that does this or is there another standard
way by which information is passed between Xen and Domains during boot
that I can make use of?

Thanks
Aravindh

Re: [RFC] Genapic in x86_64 Dom0

[Keir Fraser]

> I am working on a patch on to accommodate clustered APICs in x86_64
> Dom0/U Genapic. At the moment it defaults to flat. I am thinking of
> splitting genapic_xen into genapic_xen_flat and genapic_xen_cluster. 
> 
> My question is that since the most of the code that checks for clustered
> vs flat has been IFDEFed out what is the best way to inform the Dom that
> it needs to run in clustered/flat APIC mode? Should I be thinking about
> introducing a new hypercall that does this or is there another standard
> way by which information is passed between Xen and Domains during boot
> that I can make use of?

Isn't flat/clustered mostly a local APIC issue? In which case domain 0
needn't care, as local APICs are entirely maintained by Xen.

As for IO APICs, although domain 0 writes the routing entries, Xen is
responsible for stamping in appropriate destination fields, based on
knowledge of physical CPU IDs and which domains are running on which
physical CPUs. This code can be extended to handle writing
clustering-compatible routing entries.

Perhaps genapic_xen is currently being too smart for its own good? :-)
On i386 we lobotomised local apic handling.

 -- Keir

Re: [RFC] Genapic in x86_64 Dom0

[Chris Wright]

* Keir Fraser (Keir.Fraser@cl.cam.ac.uk) wrote:
> Perhaps genapic_xen is currently being too smart for its own good? :-)

Hehe, quite possible ;-)

> On i386 we lobotomised local apic handling.

My goal was to avoid (equivalent of) #if 0 where possible, by backfilling
functionality where possible.  I also had filled out mpc_config_processor
during mp_register_lapic and tried to maintain the physid map (that bit
isn't merged).  Do you think there's value in maintaining that level of
code compatibility?

thanks,
-chris

Re: [RFC] Genapic in x86_64 Dom0

[Keir Fraser]

> * Keir Fraser (Keir.Fraser@cl.cam.ac.uk) wrote:
> > Perhaps genapic_xen is currently being too smart for its own good? :-)
> 
> Hehe, quite possible ;-)
> 
> > On i386 we lobotomised local apic handling.
> 
> My goal was to avoid (equivalent of) #if 0 where possible, by backfilling
> functionality where possible.  I also had filled out mpc_config_processor
> during mp_register_lapic and tried to maintain the physid map (that bit
> isn't merged).  Do you think there's value in maintaining that level of
> code compatibility?

I suspect we might be best off just diverging completely and having
our own file that implements enough of the local APIC "interface" to
get everything else to build, but is basically a set of no-ops. That's
basically what we have for arch/xen/i386.

Maintaining processor and lapic info is pretty pointless since those
things aren't under domain 0's control, although it may be as easy to
do so as not. 

 -- Keir

Re: [RFC] Genapic in x86_64 Dom0

[Chris Wright]

* Keir Fraser (Keir.Fraser@cl.cam.ac.uk) wrote:
> I suspect we might be best off just diverging completely and having
> our own file that implements enough of the local APIC "interface" to
> get everything else to build, but is basically a set of no-ops. That's
> basically what we have for arch/xen/i386.

OK, that's more or less what I was thinkg (re: no-ops).

> Maintaining processor and lapic info is pretty pointless since those
> things aren't under domain 0's control, although it may be as easy to
> do so as not. 

My intention was more about being able to run more code paths without
having #if CONFIG_XEN if possible to hand to trivial or no-op stub
instead.

RE: [RFC] Genapic in x86_64 Dom0

[Puthiyaparambil, Aravindh]

> 
> Isn't flat/clustered mostly a local APIC issue? In which case domain 0
> needn't care, as local APICs are entirely maintained by Xen.

Yes you are right. The way Chris has written xen_send_IPI_shortcut()
should take care of the clustered case too.

> 
> As for IO APICs, although domain 0 writes the routing entries, Xen is
> responsible for stamping in appropriate destination fields, based on
> knowledge of physical CPU IDs and which domains are running on which
> physical CPUs. This code can be extended to handle writing
> clustering-compatible routing entries.

This sounds like a better idea and less work ;-) I will submit a patch.

Aravindh