IRQs delivery.

IRQs delivery.

[Tristan Gingold]

Hi,

on xen-ia64-devel, we are currently discussing on IRQ delivery.
Xen/x86 delivers IRQs using the event channel.
Xen/ia64 currently delivers IRQs using the virtualized hardware way.

Is there major drawbacks about not using event channel (from a XEN POV) ?

How xen/ppc delivers interrupts ?

Thanks,
Tristan.

Re: IRQs delivery.

[Hollis Blanchard]

On Friday 10 March 2006 04:07, Tristan Gingold wrote:
> 
> How xen/ppc delivers interrupts ?

Ignoring hypervisors for a moment, the PPC model for external interrupts is 
this: the hardware resets the PC to a fixed entry point (0x500) and changes 
the MSR to put the processor into real (untranslated) mode. (The old PC and 
MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The 
exception handler is then responsible for querying the interrupt 
controller(s) to get the interrupt vector, and then call the appropriate 
driver.

The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of 
querying hardware, the kernel makes an hcall to find the incoming vector. If 
the interrupt is not in fact for this domain, the hypervisor queues the 
interrupt for later and tells the current domain "nothing was pending after 
all". Note that in this model, interrupts are delivered by the hardware 
directly to the current domain without hypervisor involvement.

With Xen, the hypervisor simulates hardware delivering an interrupt (jumping 
to 0x500), and then the kernel looks at the event channel structures (instead 
of making an hcall) to figure out what happened.

Important note: in our current implementation, interrupts are always delivered 
by hardware to the hypervisor, and then we simulate the interrupt delivery to 
the domain. Because of the increased latency, it's possible that in the 
future we'll want to adopt a more rhypish model. However, since the 
hypervisor itself takes interrupts, we need to be able to simulate domain 
interrupt delivery anyways.

> on xen-ia64-devel, we are currently discussing on IRQ delivery.
> Xen/x86 delivers IRQs using the event channel.
> Xen/ia64 currently delivers IRQs using the virtualized hardware way.
> 
> Is there major drawbacks about not using event channel (from a XEN POV) ?

I'm not sure what "the virtualized hardware way" is. You probably need to 
virtualize the PIC *somehow*, and the Xen event channels are as good a way as 
any to do that. If IA64 already has an unrelated way of virtualizing the PIC 
(as PPC did with the rhype model), then you get to choose. If that's the 
case, I would recommend adopting the event channels to share more Xen code 
and design...

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: Re: IRQs delivery.

[Tristan Gingold]

Le Vendredi 10 Mars 2006 16:30, Hollis Blanchard a écrit :
> On Friday 10 March 2006 04:07, Tristan Gingold wrote:
> > How xen/ppc delivers interrupts ?
[...]
Thank you for the vast amount of details.
> > on xen-ia64-devel, we are currently discussing on IRQ delivery.
> > Xen/x86 delivers IRQs using the event channel.
> > Xen/ia64 currently delivers IRQs using the virtualized hardware way.
> >
> > Is there major drawbacks about not using event channel (from a XEN POV) ?
>
> I'm not sure what "the virtualized hardware way" is. You probably need to
> virtualize the PIC *somehow*, and the Xen event channels are as good a way
> as any to do that. If IA64 already has an unrelated way of virtualizing the
> PIC (as PPC did with the rhype model), then you get to choose. If that's
> the case, I would recommend adopting the event channels to share more Xen
> code and design...
On ia64, PIC is part of the processor and is already fully-virtualized.
We are considering to switch to event channel.

Thanks,
Tristan.

Re: [XenPPC] Re: IRQs delivery.

[Olof Johansson]

On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote:
> On Friday 10 March 2006 04:07, Tristan Gingold wrote:
> > 
> > How xen/ppc delivers interrupts ?
> 
> Ignoring hypervisors for a moment, the PPC model for external interrupts is 
> this: the hardware resets the PC to a fixed entry point (0x500) and changes 
> the MSR to put the processor into real (untranslated) mode. (The old PC and 
> MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The 
> exception handler is then responsible for querying the interrupt 
> controller(s) to get the interrupt vector, and then call the appropriate 
> driver.
> 
> The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of 
> querying hardware, the kernel makes an hcall to find the incoming vector. If 
> the interrupt is not in fact for this domain, the hypervisor queues the 
> interrupt for later and tells the current domain "nothing was pending after 
> all". Note that in this model, interrupts are delivered by the hardware 
> directly to the current domain without hypervisor involvement.

Not that you have a choice on PPC970, but there's a drawback to this: If
you let all interrupts be delivered directly to the domain, then it can
hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off),
causing interrupts to other domains to be delayed.

Some PPC processors have a feature called "mediated external interrupts",
where they will be delivered to the hypervisor instead, and if the domain
can't service it then (MSR_EE off), the HV can request to be notified
when the domain can take it. The extra code path for re-delivering to a
domain can be made short.

rHype has some support for this already, but it's unclear which hardware
has it. If I had to guess I would say at least Cell, since those parts
of rhype are ripped out in the sources but some of the infrastructure
is still in there.


I'm just mentioning this in case the IA64 guys have something like
mediated interrupts as an option, they might want to go with that for
the fairness/latency reasons.


-Olof

Re: [XenPPC] Re: IRQs delivery.

[Michal Ostrowski]

On Fri, 2006-03-10 at 11:00 -0600, Olof Johansson wrote:
> On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote:
> > On Friday 10 March 2006 04:07, Tristan Gingold wrote:
> > > 
> > > How xen/ppc delivers interrupts ?
> > 
> > Ignoring hypervisors for a moment, the PPC model for external interrupts is 
> > this: the hardware resets the PC to a fixed entry point (0x500) and changes 
> > the MSR to put the processor into real (untranslated) mode. (The old PC and 
> > MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The 
> > exception handler is then responsible for querying the interrupt 
> > controller(s) to get the interrupt vector, and then call the appropriate 
> > driver.
> > 
> > The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of 
> > querying hardware, the kernel makes an hcall to find the incoming vector. If 
> > the interrupt is not in fact for this domain, the hypervisor queues the 
> > interrupt for later and tells the current domain "nothing was pending after 
> > all". Note that in this model, interrupts are delivered by the hardware 
> > directly to the current domain without hypervisor involvement.
> 

Acutally, rhype handles all 0x500 exceptions directly and then simulates
0x500 exceptions to partitions when they are to actually see an
interrupt.  Thus if an interrupt occurs for a partition that is not
currently running, the partition does not see the exception.  PHYP
implements it differently; 0x500 exceptions are always seen by the
currently running partitions even if the interrupt is not for them.

> Not that you have a choice on PPC970, but there's a drawback to this: If
> you let all interrupts be delivered directly to the domain, then it can
> hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off),
> causing interrupts to other domains to be delayed.
> 


PPC970 gives you the choice of where external exceptions are directed
to; hypervisor or partition (e.g. do external interrupts turn on
MSR.HV?).

> Some PPC processors have a feature called "mediated external interrupts",
> where they will be delivered to the hypervisor instead, and if the domain
> can't service it then (MSR_EE off), the HV can request to be notified
> when the domain can take it. The extra code path for re-delivering to a
> domain can be made short.
> 
> rHype has some support for this already, but it's unclear which hardware
> has it. If I had to guess I would say at least Cell, since those parts
> of rhype are ripped out in the sources but some of the infrastructure
> is still in there.
> 

rhype interrupt handling  code has been design with mediated interrupts
in mind.  If there are no mediated interrupts certain branches are never
taken and certain things become no-ops.


-- 
Michal Ostrowski <mostrows@watson.ibm.com>

Re: Re: IRQs delivery.

[Hollis Blanchard]

Tristan Gingold wrote:

>On ia64, PIC is part of the processor and is already fully-virtualized.
>We are considering to switch to event channel.
>  
>
What do you mean by "already fully virtualized" -- you mean before Xen, 
this was already part of the architecture? Could you elaborate on that?

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: [XenPPC] Re: IRQs delivery.

[Jimi Xenidis]

On Mar 10, 2006, at 12:00 PM, Olof Johansson wrote:

> On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote:
>>
>> The IBM Research hypervisor (rhype) virtualized the PIC, so that  
>> instead of
>> querying hardware, the kernel makes an hcall to find the incoming  
>> vector. If
>> the interrupt is not in fact for this domain, the hypervisor  
>> queues the
>> interrupt for later and tells the current domain "nothing was  
>> pending after
>> all". Note that in this model, interrupts are delivered by the  
>> hardware
>> directly to the current domain without hypervisor involvement.

Actually rHype and xen configure the processor to deliver external  
interrupts directly to the hypervisor.

> Not that you have a choice on PPC970,

This _is_ the case for 970 as well (it was not the case for POWER4).

> but there's a drawback to this: If
> you let all interrupts be delivered directly to the domain, then it  
> can
> hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off),
> causing interrupts to other domains to be delayed.

The Hostage (MSR_EE) bit problem exists does exist with 970.

>
> Some PPC processors have a feature called "mediated external  
> interrupts",
> where they will be delivered to the hypervisor instead,

MEI does not make this descision.. the LPES does.
The problem MEI solves (as you mention below) is the case that the HV  
wants to deliver an interrupt but MSR_EE for the domain is off.  MEI  
allows the hypervisor to force the processor a special form of  
external when the domain sets MSR_EE.

> and if the domain
> can't service it then (MSR_EE off), the HV can request to be notified
> when the domain can take it. The extra code path for re-delivering  
> to a
> domain can be made short.
>
> rHype has some support for this already, but it's unclear which  
> hardware
> has it. If I had to guess I would say at least Cell, since those parts
> of rhype are ripped out in the sources but some of the infrastructure
> is still in there.
Yes it is for Cell and other forth-comming processors.  It is really  
for latency issues of RTOS (and games), server workloads are not  
expected to ever need this.

>
>
> I'm just mentioning this in case the IA64 guys have something like
> mediated interrupts as an option, they might want to go with that for
> the fairness/latency reasons.

Does IA64 indeed have MEI like abilities or the ability to route the  
interrupt?
-JX

Re: [XenPPC] Re: IRQs delivery.

[Olof Johansson]

On Fri, Mar 10, 2006 at 09:27:50PM -0500, Jimi Xenidis wrote:
> 
> >Not that you have a choice on PPC970,
> 
> This _is_ the case for 970 as well (it was not the case for POWER4).

Since POWER4 didn't have a hypervisor that allowed shared processors,
I guess this wasn't much of a problem. (Well, maybe seen the other
way the lack of the processor feature kept them from adding shared
processor support).

> >Some PPC processors have a feature called "mediated external  
> >interrupts",
> >where they will be delivered to the hypervisor instead,
> 
> MEI does not make this descision.. the LPES does.

Right. Hollis threw me off by saying that interrupts go to the domain,
leading me to believe the two features were introduced at the same time.
Especially since both MEI and HV/SV delivery are configured via LPES.

> Yes it is for Cell and other forth-comming processors.  It is really  
> for latency issues of RTOS (and games), server workloads are not  
> expected to ever need this.

Yep, you'd definitely want it on a desktop running a hypervisor too.
Classic server workloads are obviously not nearly as latency sensitive.


-Olof

RE: Re: IRQs delivery.

[Tian, Kevin]

>From: Hollis Blanchard
>Sent: 2006年3月11日 1:27
>
>Tristan Gingold wrote:
>
>>On ia64, PIC is part of the processor and is already fully-virtualized.
>>We are considering to switch to event channel.
>>
>>
>What do you mean by "already fully virtualized" -- you mean before Xen,
>this was already part of the architecture? Could you elaborate on that?
>

I may put some word on this, since simple "PIC" is not clear term here. 
The interrupt architecture on IA64 is called SAPIC (Steamlined Advanced 
Programmable Interrupt Controller), which is mostly like APIC architecture 
on X86. For example, there're also two parts: IOSAPIC and Local SAPIC, 
with IOSAPIC sitting in the chipset, while Local SAPIC part of processor. 
Part of Local SAPIC is virtualized due to processor virtualization, like 
registers. MMIO of Local SAPIC like PIB/XTP and IOSAPIC are handled 
differently, and that's the part we're proposing strongly to reuse same 
xen/x86 event channel mechanism on xen-ia64 mailing list now.

Hope helps.

Thanks,
Kevin

Re: Re: IRQs delivery.

[Tristan Gingold]

Le Vendredi 10 Mars 2006 18:27, Hollis Blanchard a écrit :
> Tristan Gingold wrote:
> >On ia64, PIC is part of the processor and is already fully-virtualized.
> >We are considering to switch to event channel.
>
> What do you mean by "already fully virtualized" -- you mean before Xen,
> this was already part of the architecture? Could you elaborate on that?
I think so.
Before working on Xen, Dan Magenheimer worked on an HP project, vBlades.
It seems he has imported ideas and sources files from vBlades to Xen/ia64.

Tristan.

Re: [XenPPC] Re: IRQs delivery.

[Tristan Gingold]

Le Samedi 11 Mars 2006 03:27, Jimi Xenidis a écrit :
[...]
> > I'm just mentioning this in case the IA64 guys have something like
> > mediated interrupts as an option, they might want to go with that for
> > the fairness/latency reasons.
>
> Does IA64 indeed have MEI like abilities or the ability to route the
> interrupt?
Well, I am a little bit lost with these ppc features.
Currently on ia64, interrupts are caught by Xen and then injected into dom0.
I don't see any fairness or latency problems [except when we are dealing with
event channel].

On ia64, there are two PIC: one on each processor (LSAPIC ~= LAPIC) and one 
(or several) for IO (IOSAPIC ~= IO_APIC).

Tristan.

Re: Re: [XenPPC] Re: IRQs delivery.

[Jimi Xenidis]

On Mar 13, 2006, at 3:41 AM, Tristan Gingold wrote:

> I don't see any fairness or latency problems [except when we are  
> dealing with
> event channel].

Problem exists:
If you defer the delivery of an IO interrupt because the VCPU is  
masking them
--and--
If the domain can unmask interrupts without the hypervisor knowing

-JX

Re: Re: [XenPPC] Re: IRQs delivery.

[Tristan Gingold]

Le Lundi 13 Mars 2006 21:44, Jimi Xenidis a écrit :
> On Mar 13, 2006, at 3:41 AM, Tristan Gingold wrote:
> > I don't see any fairness or latency problems [except when we are
> > dealing with
> > event channel].
>
> Problem exists:
> If you defer the delivery of an IO interrupt because the VCPU is
> masking them
> --and--
> If the domain can unmask interrupts without the hypervisor knowing
Sure, but this cannot happen on ia64 since masking/unmasking interrupts is a 
privilegied operation which is always trapped.

Tristan.