[RFC] Deferred interrupt handling.

[RFC] Deferred interrupt handling.

[Or Sagi]

Gentlepeople,

We're currently working on PCI device pass-through support for KVM. In this
model all physical hardware access to specific devices will be performed by
the VM, and not by the host.

In particular, this requires interrupt handling to be done by the guest --
The host shouldn't load the corresponding device driver or otherwise access
the device. Since the host kernel is not aware of the device semantics it
cannot acknowledge the interrupt at the device level.

As far as the host kernel is concerned the VM is a user level process. We
require the ability to forward interrupt handling to such entities. The
current kernel interrupt handling path doesn't allow deferring interrupt
handling _and_ acknowledgement.

We believe that this requires changing the current interrupt path in several
fashions:

0. Adding an IRQ_DEFERRED mechanism to the interrupt handling path. ISRs
returning IRQ_DEFERRED will keep the interrupt masked until future
acknowledge.
1. Deferred acknowledge mechanism which would acknowledge the APIC and
unmask
the interrupt.

An alternative idea was suggested at
http://lists.xensource.com/archives/html/xen-devel/2007-05/msg01148.html ---
which is based on reversing the polarity of the received interrupt. Once
the guest acknowledges the interrupt at the device level, the polarity is
reversed again by the host.

This may not be the best choice performance wise -- twice the interrupt
count.

Another issue we are aware of is the effect deferred handling will have on
other devices sharing the interrupt. This seems to be a problem regardless
of the solution.

We wouldn't like to assume VT-d availability right now.

Any ideas ? Thoughts ?

-- Ors

Re: [RFC] Deferred interrupt handling.

[Alan Cox]

> In particular, this requires interrupt handling to be done by the guest --
> The host shouldn't load the corresponding device driver or otherwise access
> the device. Since the host kernel is not aware of the device semantics it
> cannot acknowledge the interrupt at the device level.

Tricky indeed.

> As far as the host kernel is concerned the VM is a user level process. We
> require the ability to forward interrupt handling to such entities. The
> current kernel interrupt handling path doesn't allow deferring interrupt
> handling _and_ acknowledgement.

We don't support this model at all, and it doesn't appear to work anyway.

> 0. Adding an IRQ_DEFERRED mechanism to the interrupt handling path. ISRs
> returning IRQ_DEFERRED will keep the interrupt masked until future
> acknowledge.

Deadlock. If you get an IRQ for a guest and you block the IRQ until the
guest handles it you may (eg if the IRQ is shared) get priority inversion
with another interrupt source on the same line the guest requires first
(eg disks and other I/O)

> Another issue we are aware of is the effect deferred handling will have on
> other devices sharing the interrupt. This seems to be a problem regardless
> of the solution.

Yes. This is the big reason Linux has never supported this kind of model
for anything but old ISA IRQ handling in vm86.
> 
> We wouldn't like to assume VT-d availability right now.
> 
> Any ideas ? Thoughts ?

Mask the interrupt in the main kernel, pass an event of some kind to the
guest. You can describe most devices from guest to kernel in a safe form
as

device, bar, offset, register size, mask, bits to set, bits to clear

(or bits to test when deciding if it is the irq source)

RE: [kvm-devel] [RFC] Deferred interrupt handling.

[Dor Laor]

>> In particular, this requires interrupt handling to be done by the
>guest --
>> The host shouldn't load the corresponding device driver or otherwise
>access
>> the device. Since the host kernel is not aware of the device
semantics
>it
>> cannot acknowledge the interrupt at the device level.
>
>Tricky indeed.
>
>> As far as the host kernel is concerned the VM is a user level
process.
>We
>> require the ability to forward interrupt handling to such entities.
>The
>> current kernel interrupt handling path doesn't allow deferring
>interrupt
>> handling _and_ acknowledgement.
>
>We don't support this model at all, and it doesn't appear to work
>anyway.
>
>> 0. Adding an IRQ_DEFERRED mechanism to the interrupt handling path.
>ISRs
>> returning IRQ_DEFERRED will keep the interrupt masked until future
>> acknowledge.
>
>Deadlock. If you get an IRQ for a guest and you block the IRQ until the
>guest handles it you may (eg if the IRQ is shared) get priority
>inversion
>with another interrupt source on the same line the guest requires first
>(eg disks and other I/O)

What if we will force the specific device to the end of the list. Once
IRQ_NONE was returned by the other devices, we will mask the irq,
forward the irq to the guest, issue a timer for 1msec. Motivation:
1msec is long enough for the guest to ack the irq + host unmask the irq
+
cancell the timer. (ping round-trip for a guest is about 100msec)
If the timer poped, it will unmask irqs + run over the device list to
check
whether one of them has a pending irq.

This will solve the deadlock possiblity in a small price of potential
latency.

...

>> Any ideas ? Thoughts ?
>
>Mask the interrupt in the main kernel, pass an event of some kind to
the
>guest. You can describe most devices from guest to kernel in a safe
form
>as
>
>device, bar, offset, register size, mask, bits to set, bits to clear
>
>(or bits to test when deciding if it is the irq source)
>

The problem is that each device has its own bits and it cannot be a
general solution.
Except that the device driver inside the guest should be changed because
the host already
disabled the irq/status for them.


I know the above solution in not neat but we do want to contribute it.
Any other ideas are welcome,
10x, Dor.

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Alan Cox]

> What if we will force the specific device to the end of the list. Once
> IRQ_NONE was returned by the other devices, we will mask the irq,
> forward the irq to the guest, issue a timer for 1msec. Motivation:
> 1msec is long enough for the guest to ack the irq + host unmask the irq

It makes no difference. The deadlock isn't fixable by timing hacks.
Consider the following sequence


	Guest0	-	blocked on I/O

	IRQ14 from your hardware
		Block IRQ14
		Sent to guest (guest is blocked)

	IRQ14 from hard disk
		Ignored (as blocked)

	Deadlock

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Avi Kivity]

Alan Cox wrote:
>> What if we will force the specific device to the end of the list. Once
>> IRQ_NONE was returned by the other devices, we will mask the irq,
>> forward the irq to the guest, issue a timer for 1msec. Motivation:
>> 1msec is long enough for the guest to ack the irq + host unmask the irq
>>     
>
> It makes no difference. The deadlock isn't fixable by timing hacks.
> Consider the following sequence
>
>
> 	Guest0	-	blocked on I/O
>
> 	IRQ14 from your hardware
> 		Block IRQ14
> 		Sent to guest (guest is blocked)
>
> 	IRQ14 from hard disk
> 		Ignored (as blocked)
>
> 	Deadlock
>   

IMO the only reasonable solution is to disallow interrupt forwarding
with shared irqs.  If someone later comes up with a bright idea, we can
implement it.  Otherwise the problem will solve itself with hardware
moving to msi.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Alan Cox]

> IMO the only reasonable solution is to disallow interrupt forwarding
> with shared irqs.  If someone later comes up with a bright idea, we can

Which means you are back to ISA bus devices. Even checking if an IRQ is
currently unshared isn't simple as with hotplug this may change.

> implement it.  Otherwise the problem will solve itself with hardware
> moving to msi.

Only if MSI ever works properly on the bridges and hardware 8(

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Avi Kivity]

Alan Cox wrote:
>> IMO the only reasonable solution is to disallow interrupt forwarding
>> with shared irqs.  If someone later comes up with a bright idea, we can
>>     
>
> Which means you are back to ISA bus devices. Even checking if an IRQ is
> currently unshared isn't simple as with hotplug this may change.
>
>   

Hotplug is user-controllable, so if the user refrains from adding pci
devices after assigning a device to the guest, it should work.  I think
that USB interrupts are assigned to the controller, not the device, so
USB hotplug can be ruled out.

I admit this is fairly weak.

>> implement it.  Otherwise the problem will solve itself with hardware
>> moving to msi.
>>     
>
> Only if MSI ever works properly on the bridges and hardware 8(
>   

Oh, I've no doubt it will be made to work -- there's money to be saved
in those irq lines.  And msi makes sense technically as well.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Alan Cox]

> Hotplug is user-controllable, so if the user refrains from adding pci
> devices after assigning a device to the guest, it should work.  I think
> that USB interrupts are assigned to the controller, not the device, so
> USB hotplug can be ruled out.

Cardbus is more problematic.

Alan

RE: [kvm-devel] [RFC] Deferred interrupt handling.

[Dor Laor]

>Alan Cox wrote:
>>> What if we will force the specific device to the end of the list.
>Once
>>> IRQ_NONE was returned by the other devices, we will mask the irq,
>>> forward the irq to the guest, issue a timer for 1msec. Motivation:
>>> 1msec is long enough for the guest to ack the irq + host unmask the
>irq
>>>
>>
>> It makes no difference. The deadlock isn't fixable by timing hacks.
>> Consider the following sequence
>>
>>
>> 	Guest0	-	blocked on I/O
>>
>> 	IRQ14 from your hardware
>> 		Block IRQ14
>> 		Sent to guest (guest is blocked)
>>
>> 	IRQ14 from hard disk
>> 		Ignored (as blocked)
>>


But now the timer will pop and the hard disk will get its irq.
The guest will be released right after.

>> 	Deadlock
>>
>
>IMO the only reasonable solution is to disallow interrupt forwarding
>with shared irqs.  If someone later comes up with a bright idea, we can
>implement it.  Otherwise the problem will solve itself with hardware
>moving to msi.
>

I though of that but the problem is that we'd like to use it with
current hardware
devices that are shared.
:(

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Alan Cox]

> >> 	Guest0	-	blocked on I/O
> >>
> >> 	IRQ14 from your hardware
> >> 		Block IRQ14
> >> 		Sent to guest (guest is blocked)
> >>
> >> 	IRQ14 from hard disk
> >> 		Ignored (as blocked)
> >>
> 
> 
> But now the timer will pop and the hard disk will get its irq.
> The guest will be released right after.

How do you plan to do this ? If you unmask the interrupt then it will
immediately jam solid with IRQs from your hardware and the line will be
disabled.

RE: [kvm-devel] [RFC] Deferred interrupt handling.

[Dor Laor]

>> >> 	Guest0	-	blocked on I/O
>> >>
>> >> 	IRQ14 from your hardware
>> >> 		Block IRQ14
>> >> 		Sent to guest (guest is blocked)
>> >>
>> >> 	IRQ14 from hard disk
>> >> 		Ignored (as blocked)
>> >>
>>
>>
>> But now the timer will pop and the hard disk will get its irq.
>> The guest will be released right after.
>
>How do you plan to do this ? If you unmask the interrupt then it will
>immediately jam solid with IRQs from your hardware and the line will be
>disabled.

Hope it should work like the following [Please correct me if I'm wrong]:
- Make the device the last irqaction in the list
- Our dummy handler will always return IRQ_HANDLED in case any other
previous
  irqaction did not return such. It will also issue the timer and mask
the irq in this case.
  The line is temporarily jammed but not disabled - note_interrupt()
will not consider 
  our irq unhandled and won't disable it.
  btw, if I'm not mistaken only after bad 99900/100000 the irq is
disabled.
- If the timer pops before the guest acks the irq, the timer handler
will
  ack the irq and unmask it. The timer's job is only to prevent
deadlocks.

Maybe it's better to code it first then send RFC.
Or wanted to get a feed back before hand to hear opinions and to know
whether to use the 
deferred option or the irq polarity option. Both of them can lead to the
above deadlock 
without the timer hack.
Best regards, Dor.

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Alan Cox]

> - Make the device the last irqaction in the list
> - Our dummy handler will always return IRQ_HANDLED in case any other
> previous
>   irqaction did not return such. It will also issue the timer and mask
> the irq in this case.

Ok

>   btw, if I'm not mistaken only after bad 99900/100000 the irq is
> disabled.

(and with GIT providing that they occurred in a short time period)

> - If the timer pops before the guest acks the irq, the timer handler
> will
>   ack the irq and unmask it. The timer's job is only to prevent
> deadlocks.

Ok I see what you are doing. It's either inspired or insane and I am not
quite sure which of the two. I agree it should work although may cause
performance crunches now and then and will also need care getting the
locking right.

Alan

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Pavel Machek]

Hi!

I guess this will also allow UIO to work without _any_ kernel parts,
with only slight performance penalty in 'almost-never-happens'
deadlock case?

(Greg, details are below, and better description is in the lkml
thread).
							Pavel

> > - Our dummy handler will always return IRQ_HANDLED in case any other
> > previous
> >   irqaction did not return such. It will also issue the timer and mask
> > the irq in this case.
> 
> Ok
> 
> >   btw, if I'm not mistaken only after bad 99900/100000 the irq is
> > disabled.
> 
> (and with GIT providing that they occurred in a short time period)
> 
> > - If the timer pops before the guest acks the irq, the timer handler
> > will
> >   ack the irq and unmask it. The timer's job is only to prevent
> > deadlocks.
> 
> Ok I see what you are doing. It's either inspired or insane and I am not
> quite sure which of the two. I agree it should work although may cause
> performance crunches now and then and will also need care getting the
> locking right.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

RE: [kvm-devel] [RFC] Deferred interrupt handling.

[Or Sagi]

> Hope it should work like the following [Please correct me if I'm
> wrong]:
> - Make the device the last irqaction in the list
> - Our dummy handler will always return IRQ_HANDLED in case any other
> previous
>   irqaction did not return such. It will also issue the timer and mask
> the irq in this case.
>   The line is temporarily jammed but not disabled - note_interrupt()
> will not consider
>   our irq unhandled and won't disable it.
>   btw, if I'm not mistaken only after bad 99900/100000 the irq is
> disabled.
> - If the timer pops before the guest acks the irq, the timer handler
> will
>   ack the irq and unmask it. The timer's job is only to prevent
> deadlocks.
> 
> Maybe it's better to code it first then send RFC.

Definitely. We've already agreed that it's correct for the (rare, 
unfortunately) non shared scenario. Let's see how much do we have to bend 
reality to fit once it's in code form.

-- Ors.

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Lennart Sorensen]

On Wed, Jul 18, 2007 at 07:46:13PM +0300, Avi Kivity wrote:
> IMO the only reasonable solution is to disallow interrupt forwarding
> with shared irqs.  If someone later comes up with a bright idea, we can
> implement it.  Otherwise the problem will solve itself with hardware
> moving to msi.

Disallowing shared IRQs means disallowing PCI devices in general, so now
you are back to ISA only devices, which it sounds like were the only
things already allowed by vm86 or whatever it was for passing through.

MSI would be good, but how long until all devices support that?
Certainly does appear to have potential though for allowing more
interesting virtual machines.

--
Len Sorensen

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Avi Kivity]

Lennart Sorensen wrote:
> On Wed, Jul 18, 2007 at 07:46:13PM +0300, Avi Kivity wrote:
>   
>> IMO the only reasonable solution is to disallow interrupt forwarding
>> with shared irqs.  If someone later comes up with a bright idea, we can
>> implement it.  Otherwise the problem will solve itself with hardware
>> moving to msi.
>>     
>
> Disallowing shared IRQs means disallowing PCI devices in general, so now
> you are back to ISA only devices, which it sounds like were the only
> things already allowed by vm86 or whatever it was for passing through.
>
>   

No, it means disallowing pci devices that use shared irqs, and allowing 
pci devices that use non-shared irqs.

> Certainly does appear to have potential though for allowing more
> interesting virtual machines.
>   

IMO pass-through is counter to most of the benefits of virtualization.  
But users seem to want it.

-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Lennart Sorensen]

On Thu, Jul 19, 2007 at 12:23:24PM +0300, Avi Kivity wrote:
> No, it means disallowing pci devices that use shared irqs, and allowing 
> pci devices that use non-shared irqs.

Most machiens I see today have almost no chance of having PCI devices
without shared IRQs.  This probably means any implementation will only
work on a small set of machines with very specific setup in terms of
which PCI slots they install cards in, and only as long as you don't
allow any type of hotplugging of devices (or ever changing hardware at
all).  May not be worth implementing if it has such a limited use case.
The MSI setup on the other hand does sound like it might have potential
for working in general.

> IMO pass-through is counter to most of the benefits of virtualization.  
> But users seem to want it.

Certainly if you can have the host manage the device and provide virutal
access instead, that is much better.

--
Len Sorensen

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Avi Kivity]

Lennart Sorensen wrote:
> On Thu, Jul 19, 2007 at 12:23:24PM +0300, Avi Kivity wrote:
>   
>> No, it means disallowing pci devices that use shared irqs, and allowing 
>> pci devices that use non-shared irqs.
>>     
>
> Most machiens I see today have almost no chance of having PCI devices
> without shared IRQs.  This probably means any implementation will only
> work on a small set of machines with very specific setup in terms of
> which PCI slots they install cards in, and only as long as you don't
> allow any type of hotplugging of devices (or ever changing hardware at
> all).  May not be worth implementing if it has such a limited use case.
> The MSI setup on the other hand does sound like it might have potential
> for working in general.
>
>   

Looking at two random servers here and a desktop, interrupts are 
unshared except for usb.  A laptop was not so lucky.  So "no chance" is 
a bit extreme.

I agree it's far from optimal, but it is less limited than you imply.


-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] [RFC] Deferred interrupt handling.

[Lennart Sorensen]

On Thu, Jul 19, 2007 at 04:38:09PM +0300, Avi Kivity wrote:
> Looking at two random servers here and a desktop, interrupts are 
> unshared except for usb.  A laptop was not so lucky.  So "no chance" is 
> a bit extreme.
> 
> I agree it's far from optimal, but it is less limited than you imply.

Well with 4 PCI interrupts, it doesn't take that many PCI devices before
you must be sharing.  Of course it seems some higher end systems split
up the PCI bus in some way and run different IRQs from the APIC to
different slots.  I suspect that is how I see IRQ's of 177, 185, 193,
201, 209, etc on some systems.

If you have a server with an IO APIC then I suppose you have a rather
good chance of avoiding shared IRQs for the most part.

--
Len Sorensen

Re: [RFC] Deferred interrupt handling.

[Lennart Sorensen]

On Wed, Jul 18, 2007 at 12:53:19PM +0300, Or Sagi wrote:
> Gentlepeople,
> 
> We're currently working on PCI device pass-through support for KVM. In this
> model all physical hardware access to specific devices will be performed by
> the VM, and not by the host.
> 
> In particular, this requires interrupt handling to be done by the guest --
> The host shouldn't load the corresponding device driver or otherwise access
> the device. Since the host kernel is not aware of the device semantics it
> cannot acknowledge the interrupt at the device level.
> 
> As far as the host kernel is concerned the VM is a user level process. We
> require the ability to forward interrupt handling to such entities. The
> current kernel interrupt handling path doesn't allow deferring interrupt
> handling _and_ acknowledgement.
> 
> We believe that this requires changing the current interrupt path in several
> fashions:
> 
> 0. Adding an IRQ_DEFERRED mechanism to the interrupt handling path. ISRs
> returning IRQ_DEFERRED will keep the interrupt masked until future
> acknowledge.
> 1. Deferred acknowledge mechanism which would acknowledge the APIC and
> unmask
> the interrupt.
> 
> An alternative idea was suggested at
> http://lists.xensource.com/archives/html/xen-devel/2007-05/msg01148.html ---
> which is based on reversing the polarity of the received interrupt. Once
> the guest acknowledges the interrupt at the device level, the polarity is
> reversed again by the host.
> 
> This may not be the best choice performance wise -- twice the interrupt
> count.
> 
> Another issue we are aware of is the effect deferred handling will have on
> other devices sharing the interrupt. This seems to be a problem regardless
> of the solution.
> 
> We wouldn't like to assume VT-d availability right now.
> 
> Any ideas ? Thoughts ?

What if you have two PCI devices sharing an interrupt, unless the host
OS knows about all the devices that can generate that interrupt, how it
is supposed to deal with it?  It can't know that it should pass in IRQ
to a guest OS, unless it can determine which device actually generated
the interrupt, which probably requires some knowledge of the devices
involved.  Should the guest be able to register as a handler of that
interrupt, so that the host has to call the guest's interrupt handler
everytime an interrupt occours that the guest has registered as a
potential handler of?  There might not be any other choice.  Maybe a
driver that simply registers as using the interrupt and then passes the
call to the guest somehow so the guest can determine if the device
generated the interrupt and handle it, and then return back to the main
interrupt handler.  Of course this would mean the guest has to be
scheduled and run during the host's interrupt handler which might just
not be an option.

Sounds tricky.

--
Len Sorensen

Re: [RFC] Deferred interrupt handling.

[Peter Chubb]

The problem you're having is essentially the same as the user-level
interrupt handler problem I've been dealing with for ages.

The basic rule is: don't share interrupts between devices on the host
and devices in the guest.  But you *can* share interrupts between
devices in a single guest.

If you want the code, see
http://www.gelato.unsw.edu.au/cgi-bin/viewvc.cgi/cvs/kernel/usrdrivers/latest/
and look at generic-irq.patch and fasync (which adds asynchronous notifications)

For the KVM work it'll need modifying a little, but the basic
infrastructure is there.

We've currently got this working to pass interrupts to a type-II (hosted)
virtual machine monitor running a guest kernel with native drivers.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia