Re: take2: [PATCH] Vector sharing (Large I/O system support)

[Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>]

> Kenji,
> Inside HP I've suggested linux use multiple vector domains
> (one CPU could belong in only one domain). Would that sufficiently
> solve this same problem or is someone hard coded to share a particular
> vector?

Hi Grant,

As far as I know, there are three methods to handle many
interrupt sources:

(1) Share a single RTE with multiple level-triggered interrupts (e.g.
multiple PCI devices share the same interrupt line). Of course,
current linux can handle it. Whether to use this method depend on
hardware design.

(2) Share a single vector with multiple RTEs. This is what my vector
sharing patch is doing.

(3) Use multiple vector domains on the multi-node system. This method
strongly depends on hardware design, and it will be implemented in the
architecture specific code. I guess SGI machine is using this method.

I guess the method you mentioned is extension of method (3), which
use multiple vector domains on a generic SMP machine (I'll call this
"method (4)" below). Is it correct?

I think method (4) is interesting and it would be able to solve the
same problem. I have discussed it a little with Bjorn Helgaas before.
(please see http://www.gelato.unsw.edu.au/linux-ia64/0404/9363.html)

However, method (4) will not work if system has only a few CPUs (or
it is an UP machine). So vector sharing is still needed. In addition,
though I have not investigated it much, I think there would be a lot
of issues need to be considered for method (4).
For example:

    o How to separate CPUs and devices into multiple vector domains
    o How to associate vector number with IRQ number
    o How to prepare multiple 'irq_desc' arrays for each vector domains
    o How to install interrupt handlers into each 'irq_desc'
    o Need to consider the case some CPUs are hot-removed
    o How to display the IRQ infomation through /proc filesystem

    and so on...

Most of those need a much time and a lot of changes to kernel.

After all, I think the good way is to implement vector sharing first,
and then consider the method (4) next. I beleave vector sharing and
method (4) can work together.

By the way, do you have specific reasons to suggest method (4)?
Performance issue?
And do you already have a patch for method (4)? If so, can I see
it?

Thanks,
Kenji Kaneshige