RE: Question about evtchn_callback reentry

RE: Question about evtchn_callback reentry

[Xu, Anthony]

Keir Fraser write on 2006年12月18日 17:32:
> On 18/12/06 5:28 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 

Thanks for your quick response

>> If there is no special handling, low privileged interrupt may
>> "interrupt" high privileged interrupt.
> 
> There's no concept of privilege among event channels. Since Xen simply
> allocates ExtInts to vectors in a round-robin manner, there is no
> meaningful priority comparsion to be made between vectors anyway.

Yes there's no concept of privilege among event channel,

But all device interrupts have privilege and  device interrupts are delivered through event channel.

That means ide interrupt may interrupt time interrup in dom0.
It maybe not reasonable.


> 
>> BTW,
>> I do see evtchn_callback reentry happen on IPF dom0, and that incur
>> about 5% performance degradation
>> with KB on VTI-domain compared to non_reentry.
> 
> KB == Keyboard?!
Sorry for confusion,
KB means Kernal Build.

> 
> Reentry is allowed -- it shouldn't happen often unless the ISRs are
> really long (bad) or we are hammering the interrupt line for some
> reason (bad). 

If evtchn_callback reentry is allowed.
There are 1024 event channels.
In theory, dom0 kernal stack may be overflowed.
Is there any mechanism to prevent this?

Since event channels have no concept of privilege.
Why we allow evtchn_callback reentered ?

I think evtchn_callback can handle all events in batch.
This is good for performance.


-- Anthony

> 
>  -- Keir

Re: Question about evtchn_callback reentry

[Keir Fraser]

> But all device interrupts have privilege and  device interrupts are delivered
> through event channel.

Do they? Architecturally defined? I don't think so.

> That means ide interrupt may interrupt time interrup in dom0.
> It maybe not reasonable.

Linux doesn't care. If it did we could apply some form of prioritisation in
Linux's event-channel subsystem.

> If evtchn_callback reentry is allowed.
> There are 1024 event channels.
> In theory, dom0 kernal stack may be overflowed.
> Is there any mechanism to prevent this?

When running natively there are over 200 vectors available for allocation to
interrupt sources. In theory the kernel stack may be overflowed, and there
is no mechanism in Linux to prevent this.

> Since event channels have no concept of privilege.
> Why we allow evtchn_callback reentered ?

Mainly because it follows the model of interrupt delivery on native Linux.
We could deviate this default behaviour when running on Xen -- interrupt
handlers should be quick to run so really I would expect no performance
difference one way or the other. Do you understand why you are getting so
much reentrancy, and hence such a horrible performance degradation, on ia64?

 -- Keir

Re: Question about evtchn_callback reentry

[Keir Fraser]

On 18/12/06 11:24, "Keir Fraser" <keir@xensource.com> wrote:

>> If evtchn_callback reentry is allowed.
>> There are 1024 event channels.
>> In theory, dom0 kernal stack may be overflowed.
>> Is there any mechanism to prevent this?
> 
> When running natively there are over 200 vectors available for allocation to
> interrupt sources. In theory the kernel stack may be overflowed, and there
> is no mechanism in Linux to prevent this.

Okay, so I remember now that of course the APIC divides these into 16
priority levels so that does rather restrict the maximum nesting. I'll be
happy to consider adding prioritisation (or just complete disallow of
nesting) for 3.0.5. But I think you should also work to understand the root
cause of the performance drop on ia64.

 -- Keir

RE: Question about evtchn_callback reentry

[Xu, Anthony]

Keir Fraser write on 2006年12月20日 16:31:
> On 20/12/06 3:03 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
> How did you implement nesting avoidance? It'll be a bit subtle to get
> right. 

I added another global_event_mask in share page.

1. before injecting to event_callback_handler, set global_event_mask to 1 to 
   disable event inject..
2. after event_callback_handler is executed, it will return to ia64_leave_kernel
    assembly code in dom0, in this assembly code, it sets global_event_mask to 0
    to enable event inject.

There is no other code operating the global_event_mask.
Global_event_mask is initiated to 0(unmask).

Only global_event_mask is 0 and vcpu->vcpu_info->evtchn_upcall_mask is 0,
Can XEN injects to event_callback_handler of dom0.

--Anthony





> 
>  -- Keir

Re: Question about evtchn_callback reentry

[Keir Fraser]

On 20/12/06 10:08, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> There is no other code operating the global_event_mask.
> Global_event_mask is initiated to 0(unmask).
> 
> Only global_event_mask is 0 and vcpu->vcpu_info->evtchn_upcall_mask is 0,
> Can XEN injects to event_callback_handler of dom0.

How much does injecting a stack frame on ia64 actually cost? It sounds like
it must be quite hideously expensive... I would be inclined to implement
this for x86 by adding the nesting detection into the callback handler
(i.e., the extra callback injection would not be avoided). Keeps the
hypervisor interface clean.

 -- Keir

RE: [Xen-devel] Question about evtchn_callback reentry

[Xu, Anthony]

Keir Fraser write on 2006年12月18日 21:42:
> On 18/12/06 11:24, "Keir Fraser" <keir@xensource.com> wrote:
> 
> Okay, so I remember now that of course the APIC divides these into 16
> priority levels so that does rather restrict the maximum nesting.
> I'll be happy to consider adding prioritisation (or just complete
> disallow of nesting) for 3.0.5. 

Hi Keir,

I did a experiment,
If we completely disallow of nesting.
Even dom0 can't boot.

I suspected timer interrupt is blocked by other events.

--Anthony



> 
>  -- Keir

Re: Question about evtchn_callback reentry

[Keir Fraser]

On 20/12/06 3:03 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:

>> Okay, so I remember now that of course the APIC divides these into 16
>> priority levels so that does rather restrict the maximum nesting.
>> I'll be happy to consider adding prioritisation (or just complete
>> disallow of nesting) for 3.0.5.
> 
> Hi Keir,
> 
> I did a experiment,
> If we completely disallow of nesting.
> Even dom0 can't boot.
> 
> I suspected timer interrupt is blocked by other events.

How did you implement nesting avoidance? It'll be a bit subtle to get right.

 -- Keir

[Xen-devel] Question about evtchn_callback reentry

[Xu, Anthony]

Hi Keir,

I have some questions about evtchn_callback,

As we know all external interrupts and events are deliver to dom0 by
evtchn_callback,

Are evtchn_callback reentered?

I mean when evtchn_callbacak is executing on dom0,

Another external interrupt or event may happen, can this event be
deliverred immediately?

I found __sti will set vcpu_info-> evtchn_upcall_mask=0, which permits
event deliverring.

and __sti may be called at any device driver. 

If evtchn_callback can be reentered, how does it hanlde external
interrupt privilege?

If there is no special handling, low privileged interrupt may
"interrupt" high privileged interrupt.


BTW,
I do see evtchn_callback reentry happen on IPF dom0, and that incur
about 5% performance degradation
with KB on VTI-domain compared to non_reentry.



Thanks,
Anthony

Re: Question about evtchn_callback reentry

[Keir Fraser]

On 18/12/06 5:28 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> If there is no special handling, low privileged interrupt may
> "interrupt" high privileged interrupt.

There's no concept of privilege among event channels. Since Xen simply
allocates ExtInts to vectors in a round-robin manner, there is no meaningful
priority comparsion to be made between vectors anyway.

> BTW,
> I do see evtchn_callback reentry happen on IPF dom0, and that incur
> about 5% performance degradation
> with KB on VTI-domain compared to non_reentry.

KB == Keyboard?!

Reentry is allowed -- it shouldn't happen often unless the ISRs are really
long (bad) or we are hammering the interrupt line for some reason (bad).

 -- Keir