Multiple IRQ's in HVM for Windows

Multiple IRQ's in HVM for Windows

[James Harper]

I really need to have the ability to tie event channel port's to
interrupts for my gplpv drivers under Windows. Is anyone working on
anything like this? Does MSI allow more than one interrupt per PCI
device?

Thanks

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 26/12/2008 03:29, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I really need to have the ability to tie event channel port's to
> interrupts for my gplpv drivers under Windows. Is anyone working on
> anything like this? Does MSI allow more than one interrupt per PCI
> device?

What kind of interface do you want?

 -- Keir

> Thanks
> 
> James
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 26/12/2008 03:29, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > I really need to have the ability to tie event channel port's to
> > interrupts for my gplpv drivers under Windows. Is anyone working on
> > anything like this? Does MSI allow more than one interrupt per PCI
> > device?
> 
> What kind of interface do you want?
> 

I'm not entirely sure. Windows makes it hard.

Essentially, I need to be able to tie an event channel to a PIC or APIC
interrupt, and have a way of checking the event channel to see if it was
responsible for the interrupt and then clearing the pending flag for
only that event channel. I don't think that in itself is that hard -
what will be hard is making Windows happy to have those interrupt lines
as resources and to program the APIC correctly to allow them to be
delivered.

The above probably shows a lack of understanding on my part about how
events are delivered into hvm domains as interrupts - can you point me
to what files I need to look at to understand this better? Then maybe
I'll have a better idea about what I'm asking for (or I'll understand
why it won't work :)

Thanks

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 26/12/2008 10:15, "James Harper" <james.harper@bendigoit.com.au> wrote:

> The above probably shows a lack of understanding on my part about how
> events are delivered into hvm domains as interrupts - can you point me
> to what files I need to look at to understand this better? Then maybe
> I'll have a better idea about what I'm asking for (or I'll understand
> why it won't work :)

Currently all event channels are tied together as a single interrupt line,
which can be configured to appear to come from the xen-platform fake PCI
device. We could duplicate that device so you could have a PCI INTx line per
CPU. Or duplicate it so that you could have a PCI INTx line per event
channel (or at least barber-pole onto such a set of INTx lines).

Or we could bypass the INTx emulation entirely, and deliver per-cpu
event_pending status to that HVM VCPU's local APIC, on a pre-registered
interrupt vector (this is most like MSI, except the interrupt wouldn't be
coming from a PCI device, although we could perhaps even fake that out too).

There's quite a few options.

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> Or we could bypass the INTx emulation entirely, and deliver per-cpu
> event_pending status to that HVM VCPU's local APIC, on a
pre-registered
> interrupt vector (this is most like MSI, except the interrupt wouldn't
be
> coming from a PCI device, although we could perhaps even fake that out
> too).
> 
> There's quite a few options.
> 

How many interrupts do we have to choose from here? I was able to get
Windows to use up to (I think) IRQ31.

As I understand it, most of the 'protocol' we are talking about is based
around the shared_info_t structure, which appears to make the assumption
that there is a single point of entry for event delivery into a domain.
To make use of multiple IRQ's we'd have to check every single event bit
right? Is that a performance problem.

The reason all of this comes up btw, is that Windows is a PITA. What I
do in my PV drivers is attach each device (xen platform pci driver, vbd
driver, net driver, scsi driver, etc) to the same interrupt. The
platform driver attaches first and gets called first. It clears the
pending flag etc in the normal way, but where the event channel is
registered as wanting delivery via an ISR, it sets a corresponding event
channel bit in a private area, and then tells Windows that the interrupt
was not actually for that device afterall, which prompts Windows to call
the next ISR in the chain. That ISR (eg the vbd driver) check's its bit
in the private area and does it's ISR code if the bit is set, and then
tells Windows that the interrupt was not for that device, and so on.
When Windows is using one of the ACPI HAL's all this works fine. However
when Windows is using the 'Standard PC' HAL, I stop seeing interrupts
for the network device very soon after boot. I think that because of all
this 'interrupt not for me'-ness going on, Windows thinks that spurious
interrupts are occurring and stops calling the network ISR except for
every 3-20 interrupts. I can work around it, but that kind of behaviour
makes me think that maybe I'm going around things the wrong way...

For the network device I have gone back to using a direct callback, but
if I ever want to get these drivers WHQL certified (and I do
eventually), that sort of thing won't fly without a whole lot of
explaining. And for storage devices using the scsiport framework, an
interrupt is the only way to do it. The storport framework is a
possibility but currently it seems too broken and isn't available under
XP.

Thanks

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 26/12/2008 10:54, "James Harper" <james.harper@bendigoit.com.au> wrote:

> How many interrupts do we have to choose from here? I was able to get
> Windows to use up to (I think) IRQ31.

If going via the virtual PIC, then there are as many interrupts as there are
non-legacy IO-APIC pins. I think currently we have 32 of them.

> As I understand it, most of the 'protocol' we are talking about is based
> around the shared_info_t structure, which appears to make the assumption
> that there is a single point of entry for event delivery into a domain.
> To make use of multiple IRQ's we'd have to check every single event bit
> right? Is that a performance problem.

No, currently we call into HVM interrupt emulation code to assert an IO-APIC
pin when CPU0's event_pending flag is asserted. We could do that also when
other CPU's event_pending flags become asserted, or when individual event
channels become asserted. We just have to hook into event-channel code in a
different place. Or indeed we can just send 'messages' to HVM virtual local
APICs to trigger interrupts on the virtual CPUs directly, without any
integration with the virtual PCI or PIC subsystems.

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 26/12/2008 10:54, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > How many interrupts do we have to choose from here? I was able to
get
> > Windows to use up to (I think) IRQ31.
> 
> If going via the virtual PIC, then there are as many interrupts as
there
> are
> non-legacy IO-APIC pins. I think currently we have 32 of them.
> 
> > As I understand it, most of the 'protocol' we are talking about is
based
> > around the shared_info_t structure, which appears to make the
assumption
> > that there is a single point of entry for event delivery into a
domain.
> > To make use of multiple IRQ's we'd have to check every single event
bit
> > right? Is that a performance problem.
> 
> No, currently we call into HVM interrupt emulation code to assert an
IO-
> APIC
> pin when CPU0's event_pending flag is asserted. We could do that also
when
> other CPU's event_pending flags become asserted, or when individual
event
> channels become asserted. We just have to hook into event-channel code
in
> a
> different place. Or indeed we can just send 'messages' to HVM virtual
> local
> APICs to trigger interrupts on the virtual CPUs directly, without any
> integration with the virtual PCI or PIC subsystems.
> 

What about when we are running on a non-APIC HAL? People keep telling me
that the 'Standard PC' HAL (all IRQ's < 16) under Windows 2003 is higher
performance under Xen, but I've only done a quick test via iperf and
maybe there is 10% higher throughput with 'Standard PC', but that could
easily be within the level of background noise on that machine.

The original problem I am having is that Windows (NDIS actually) doesn't
always call my ISR when an interrupt is shared if it suspects that the
interrupts might be spurious. To work around that I would need for the
interrupt to be triggered if the 'pending' flag for that event wasn't
cleared, which is the 'check each event bit' that I was talking about.
This would also happen if two events are bound to the same IRQ and are
both triggered at the same time, eg:

. Event channels 2, 5, and 8 are bound to IRQ 29
. Event channels 2 and 8 are both triggered simultaneously (or close
enough that by the time the ISR happens they are both set)
. Windows will call the ISR for the device using event channel 2
. The device will see that its event is responsible for the interrupt,
clear the 'pending' flag _for that port only_ and then schedule a DPC to
do the work. It will return TRUE from its ISR and windows will not call
any other ISR for that interrupt
. Xen/qemu will need to notice that there is still an outstanding
interrupt for that event and will trigger the ISRs again
. ISR for the device using event channel 2 will get called again, but
will return FALSE this time meaning 'interrupt was not for me'
. ditto for the device using event channel 5
. Finally, the ISR for the device using event channel 8 will be called
and will do its thing

So instead of checking for one 'pending' flag, I think the pending flags
for each port will need to be called. If that's not a problem for
performance (only need to do it when returning from the interrupt I
think...) then I think we can proceed...

I can think of the following pieces of the puzzle:
. A mechanism to assign an event channel to an IRQ - a new type of
parameter for HYPERVISOR_hvm_op?
. At the end of each interrupt, re-trigger the interrupt if the event
bits are set rather than if the 'evtchn_upcall_pending' flag is set
(where is this done currently? I couldn't find it)
. Some way to make this backwards compatible... maybe assume the old
behaviour until the new HYPERVISOR_hvm_op parameter is set?

What else would be required?

Thanks

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 27/12/2008 09:19, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I can think of the following pieces of the puzzle:
> . A mechanism to assign an event channel to an IRQ - a new type of
> parameter for HYPERVISOR_hvm_op?

That's part of the problem. What do you mean by 'IRQ' here? What's the IRQ
namespace that you're indexing into?

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 27/12/2008 09:19, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > I can think of the following pieces of the puzzle:
> > . A mechanism to assign an event channel to an IRQ - a new type of
> > parameter for HYPERVISOR_hvm_op?
> 
> That's part of the problem. What do you mean by 'IRQ' here? What's the
IRQ
> namespace that you're indexing into?
> 

I'm not sure if I understand the question. Under my drivers I just do:

hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number);

where irq_number is what Windows assigns the platform PCI device, and is
normally 28 when the HAL is APIC, otherwise it is 5, assuming no qemu
network adapters or passed through PCI devices.

I think there is another way of setting the IRQ using the PCI address,
but that doesn't appear to be required here.

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 27/12/2008 09:35, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I'm not sure if I understand the question. Under my drivers I just do:
> 
> hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number);
> 
> where irq_number is what Windows assigns the platform PCI device, and is
> normally 28 when the HAL is APIC, otherwise it is 5, assuming no qemu
> network adapters or passed through PCI devices.
> 
> I think there is another way of setting the IRQ using the PCI address,
> but that doesn't appear to be required here.

So where will irq_number come from when every event channel needs one?

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 27/12/2008 09:35, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > I'm not sure if I understand the question. Under my drivers I just
do:
> >
> > hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number);
> >
> > where irq_number is what Windows assigns the platform PCI device,
and is
> > normally 28 when the HAL is APIC, otherwise it is 5, assuming no
qemu
> > network adapters or passed through PCI devices.
> >
> > I think there is another way of setting the IRQ using the PCI
address,
> > but that doesn't appear to be required here.
> 
> So where will irq_number come from when every event channel needs one?
> 

The driver for the xen platform PCI device is a 'bus driver' under
windows, and enumerates child devices. When it enumerates a child
device, I can say 'and allocate me an interrupt line'. During past
experimentation I did this with the criteria 'allocate me an interrupt
line somewhere between 16 and 31', and then I would call that interrupt
via a software interrupt mechanism (asm {int 0xXX} ), so I know that the
allocation and setup side of things work. What I don't know is if
Windows is programming the APIC correctly to match that physical IRQ
number with the vector... the 'int 0xXX' instruction just calls the
vector.

With enough devices, event channels are going to end up having to share
interrupts, but the number of event channels would be around 15 before
this would be required I think (windows would never allocate me an IRQ >
31), which I don't think is an unreasonable limitation.

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au> wrote:

> The driver for the xen platform PCI device is a 'bus driver' under
> windows, and enumerates child devices. When it enumerates a child
> device, I can say 'and allocate me an interrupt line'.

So these child devices don't have to have a physical manifestation in PCI
space? And you can really request an arbitrary IRQ and then you are expected
to plumb it through? That sounds weird, but actually quite helpful for us.

Probably we'd implement it with an hvm_op to associate an event channel with
an IO-APIC pin or a local APIC vector. If implemented as wire-OR into a set
of IO-APIC pins, we'd need logic to deassert wires when event channels
become not-pending, before the wire gets resampled by the PIC/IO-APIC. It's
all easier if we can directly deliver to a LAPIC vector because those are
inherently edge-triggered / message-based (which I think is what we really
want; although it's more complicated if we need to be able to share a LAPIC
vector among several event channels without 'losing' edges).

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > The driver for the xen platform PCI device is a 'bus driver' under
> > windows, and enumerates child devices. When it enumerates a child
> > device, I can say 'and allocate me an interrupt line'.
> 
> So these child devices don't have to have a physical manifestation in
PCI
> space? And you can really request an arbitrary IRQ and then you are
> expected
> to plumb it through? That sounds weird, but actually quite helpful for
us.
> 
> Probably we'd implement it with an hvm_op to associate an event
channel
> with
> an IO-APIC pin or a local APIC vector. If implemented as wire-OR into
a
> set
> of IO-APIC pins, we'd need logic to deassert wires when event channels
> become not-pending, before the wire gets resampled by the PIC/IO-APIC.
> It's
> all easier if we can directly deliver to a LAPIC vector because those
are
> inherently edge-triggered / message-based (which I think is what we
really
> want; although it's more complicated if we need to be able to share a
> LAPIC vector among several event channels without 'losing' edges).
> 

Well... the 'old' way would probably still have to work (or would it?),
so we could just keep allocating IRQ's until we run out and any leftover
devices just have to use the old way.

I've mentioned the possibility of using MSI before... would that work?
I'm not yet sure if they are supported across all windows versions, but
we get lots more 'interrupt channels'...

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au> wrote:

> Well... the 'old' way would probably still have to work (or would it?),
> so we could just keep allocating IRQ's until we run out and any leftover
> devices just have to use the old way.

Yes, that did occur to me. Might be a nice fallback while still allowing up
to 16 or whatever devices to have their interrupts distributed across VCPUs.
The old mechanism does still need to work, so making it a fallback in this
new mechanism would be probably not too difficult.

> I've mentioned the possibility of using MSI before... would that work?
> I'm not yet sure if they are supported across all windows versions, but
> we get lots more 'interrupt channels'...

Well, would Windows need to see more fake PCI devices (where these MSIs
would emanate from) for this to work? It would be nice, though perhaps not
essential, to avoid this since it needs backwards-compatible changes to
qemu-dm, and also possibly our vBIOS.

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > Well... the 'old' way would probably still have to work (or would
it?),
> > so we could just keep allocating IRQ's until we run out and any
leftover
> > devices just have to use the old way.
> 
> Yes, that did occur to me. Might be a nice fallback while still
allowing
> up to 16 or whatever devices to have their interrupts distributed
across
> VCPUs.
> The old mechanism does still need to work, so making it a fallback in
this
> new mechanism would be probably not too difficult.

Is Windows limited to 31 IRQ's? Linux appears to allow much more than
that, but maybe I'm not comparing apples with apples here.

Still... an average Windows system under Xen would have a few disks, a
few network adapters, and maybe a scsi passthrough. I think it would be
unusual to see a requirement for more than 6 IRQ's, so I don't think
that a limit of 16 is going to cripple anyone.

So are you suggesting that these interrupts be non-shareable? Would that
remove the need to check for a pending interrupt status in Xen, and for
the virtual device driver to clear any flags?

> 
> > I've mentioned the possibility of using MSI before... would that
work?
> > I'm not yet sure if they are supported across all windows versions,
but
> > we get lots more 'interrupt channels'...
> 
> Well, would Windows need to see more fake PCI devices (where these
MSIs
> would emanate from) for this to work? It would be nice, though perhaps
not
> essential, to avoid this since it needs backwards-compatible changes
to
> qemu-dm, and also possibly our vBIOS.
> 

Unfortunately I don't know enough about MSI and how Windows handles MSI
to be able to answer that. Lets give it a miss :)

James

RE: Multiple IRQ's in HVM for Windows

[James Harper]

> On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > Well... the 'old' way would probably still have to work (or would
it?),
> > so we could just keep allocating IRQ's until we run out and any
leftover
> > devices just have to use the old way.
> 
> Yes, that did occur to me. Might be a nice fallback while still
allowing
> up
> to 16 or whatever devices to have their interrupts distributed across
> VCPUs.
> The old mechanism does still need to work, so making it a fallback in
this
> new mechanism would be probably not too difficult.

I've been reading up on APIC's, and it appears that (depending on the
model) they can support up to 64 interrupts. Is the emulated APIC under
Xen of the 16 interrupt variety?

James

Re: Multiple IRQ's in HVM for Windows

[Keir Fraser]

On 28/12/2008 03:22, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Yes, that did occur to me. Might be a nice fallback while still
> allowing
>> up
>> to 16 or whatever devices to have their interrupts distributed across
>> VCPUs.
>> The old mechanism does still need to work, so making it a fallback in
> this
>> new mechanism would be probably not too difficult.
> 
> I've been reading up on APIC's, and it appears that (depending on the
> model) they can support up to 64 interrupts. Is the emulated APIC under
> Xen of the 16 interrupt variety?

The IO-APIC has 48 pins, 16 of which are mapped for legacy ISA, so there are
32 non-legacy pins. Actually we could have more, but that creates
backward-compatibility issues now. 32 should be enough for most uses!

 -- Keir

RE: Multiple IRQ's in HVM for Windows

[Ross Philipson]

> I'm not yet sure if they are supported across all windows versions

I believe MSI is supported on Vista and later for workstation and server
2003 and later.

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of James Harper
Sent: Saturday, December 27, 2008 5:28 AM
To: Keir Fraser; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] Multiple IRQ's in HVM for Windows

> On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > The driver for the xen platform PCI device is a 'bus driver' under
> > windows, and enumerates child devices. When it enumerates a child
> > device, I can say 'and allocate me an interrupt line'.
> 
> So these child devices don't have to have a physical manifestation in
PCI
> space? And you can really request an arbitrary IRQ and then you are
> expected
> to plumb it through? That sounds weird, but actually quite helpful for
us.
> 
> Probably we'd implement it with an hvm_op to associate an event
channel
> with
> an IO-APIC pin or a local APIC vector. If implemented as wire-OR into
a
> set
> of IO-APIC pins, we'd need logic to deassert wires when event channels
> become not-pending, before the wire gets resampled by the PIC/IO-APIC.
> It's
> all easier if we can directly deliver to a LAPIC vector because those
are
> inherently edge-triggered / message-based (which I think is what we
really
> want; although it's more complicated if we need to be able to share a
> LAPIC vector among several event channels without 'losing' edges).
> 

Well... the 'old' way would probably still have to work (or would it?),
so we could just keep allocating IRQ's until we run out and any leftover
devices just have to use the old way.

I've mentioned the possibility of using MSI before... would that work?
I'm not yet sure if they are supported across all windows versions, but
we get lots more 'interrupt channels'...

James


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel