Very sparse and high interrupt map with SPARSE_IRQ

Very sparse and high interrupt map with SPARSE_IRQ

[Michael Bohan]

Hi,

I am working with a Qualcomm SPMI device that supports up to 32768 
interrupts. Now, in practice, not nearly all of these interrupts will be 
populated on a real device. Most likely on the order of 200-300 
interrupts will be specified in Device Tree. The problem is the set of 
active interrupts will change on future devices sharing the same 
architecture, and there's really no predicting which of this range will 
be active. Ideally, the supporting code should never have to change.

To support such a device, I am considering using SPARSE_IRQ and 
allocating the irq_desc at runtime as necessary while walking the Device 
Tree. To keep the mapping function simple and fast, I was thinking of 
using discontinuous, high system interrupt numbers that can be computed 
with a simple O(1) operation. Alternatively, I could use a radix tree or 
hash to map these to more traditional, lower and contiguous interrupt 
numbers, but I'm not aware of any significant benefit in doing so.

As far as I can tell, the only potential problems with using such high 
interrupt numbers (eg. 33102) are:

1. IRQ_BITMAP_BITS must be expanded to support the entire possible range 
(eg. ~0-33500). IRQ_BITMAP_BITS is defined as NR_IRQS. This will waste 
~3 KB on such a range. To me 3 KB can be justified if it speeds up the 
fast path interrupt handling.
2. NR_IRQS will increase beyond the HARDIRQ_BITS limitation, which 
governs the number of nested interrupts. But as mentioned, we won't 
actually have more real interrupts than the maximum setting (10 bit) -- 
it's just that our NR_IRQ definition will be so high to trip an older 
ARM check for NR_IRQS going beyond HARDIRQ_BITS.

So basically, I'm asking whether this analysis is correct, and what I'm 
doing seems reasonable. I'd also like to propose a couple changes as a 
consequence of what I mentioned above:

1. Add another macro to distinguish between the actual number of 
interrupts a system supports and the *highest* number it supports. Eg. 
NR_IRQS seems to imply a quantity - not a maximum value. But it's 
currently being used to cover both constraints.
2. Remove the check in arch/arm/include/asm/hardirq.h for HARDIRQ_BITS 
being too low. Actually, if 1) were really implemented, then most likely 
NR_IRQS would be below 1024, and this check would not be violated. But 
regardless, per the comment in kernel/irq/internals.h, we're probably 
bound by interrupt stack for such systems, anyways.

Thanks,
Mike

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: Very sparse and high interrupt map with SPARSE_IRQ

[Rob Herring]

On 12/15/2011 03:18 PM, Michael Bohan wrote:
> Hi,
> 
> I am working with a Qualcomm SPMI device that supports up to 32768
> interrupts. Now, in practice, not nearly all of these interrupts will be
> populated on a real device. Most likely on the order of 200-300
> interrupts will be specified in Device Tree. The problem is the set of
> active interrupts will change on future devices sharing the same
> architecture, and there's really no predicting which of this range will
> be active. Ideally, the supporting code should never have to change.
> 
> To support such a device, I am considering using SPARSE_IRQ and
> allocating the irq_desc at runtime as necessary while walking the Device
> Tree. To keep the mapping function simple and fast, I was thinking of
> using discontinuous, high system interrupt numbers that can be computed
> with a simple O(1) operation. Alternatively, I could use a radix tree or
> hash to map these to more traditional, lower and contiguous interrupt
> numbers, but I'm not aware of any significant benefit in doing so.
> 
> As far as I can tell, the only potential problems with using such high
> interrupt numbers (eg. 33102) are:
> 
> 1. IRQ_BITMAP_BITS must be expanded to support the entire possible range
> (eg. ~0-33500). IRQ_BITMAP_BITS is defined as NR_IRQS. This will waste
> ~3 KB on such a range. To me 3 KB can be justified if it speeds up the
> fast path interrupt handling.
> 2. NR_IRQS will increase beyond the HARDIRQ_BITS limitation, which
> governs the number of nested interrupts. But as mentioned, we won't
> actually have more real interrupts than the maximum setting (10 bit) --
> it's just that our NR_IRQ definition will be so high to trip an older
> ARM check for NR_IRQS going beyond HARDIRQ_BITS.
> 
> So basically, I'm asking whether this analysis is correct, and what I'm
> doing seems reasonable. I'd also like to propose a couple changes as a
> consequence of what I mentioned above:
> 
> 1. Add another macro to distinguish between the actual number of
> interrupts a system supports and the *highest* number it supports. Eg.
> NR_IRQS seems to imply a quantity - not a maximum value. But it's
> currently being used to cover both constraints.
> 2. Remove the check in arch/arm/include/asm/hardirq.h for HARDIRQ_BITS
> being too low. Actually, if 1) were really implemented, then most likely
> NR_IRQS would be below 1024, and this check would not be violated. But
> regardless, per the comment in kernel/irq/internals.h, we're probably
> bound by interrupt stack for such systems, anyways.

Have you looked at irq_domain (kernel/irq/irqdomain.c). This is meant to
support complex mappings like this. Although in its current form it
needs some work to support this. Is there no sub-grouping of interrupts
at all?

The hwirq # is stored in irq_data, so converting from Linux irq to hwirq
# is O(1). Going the other way is implemented per domain and would
depend on the implementation of .to_irq.

Rob

Re: Very sparse and high interrupt map with SPARSE_IRQ

[Michael Bohan]

On 12/15/2011 3:24 PM, Rob Herring wrote:
> Have you looked at irq_domain (kernel/irq/irqdomain.c). This is meant to
> support complex mappings like this. Although in its current form it
> needs some work to support this. Is there no sub-grouping of interrupts
> at all?
>
> The hwirq # is stored in irq_data, so converting from Linux irq to hwirq
> # is O(1). Going the other way is implemented per domain and would
> depend on the implementation of .to_irq.

Thanks for the advice. The chip driver houses a number of devices shared 
over a SPMI bus. The SPMI bus supports up to 16 slave IDs, and each 
slave can have 256 devices. Each device can have up to 8 interrupts.

So this is pretty easy to support with irq_domains if we treat all 32768 
interrupts as one domain and allow the chip driver to own the entire 
32768 block of interrupts. Each translation is merely a O(1) calculation 
as you mentioned. And with SPARSE_IRQ, we only allocate the interrupts 
we use on the bus, and so there's very little cost associated with the 
large number of interrupts.

Were you thinking of supporting these sub-groups (eg. group of devices) 
in separate irq_domains? I'm somewhat curious if there's future work 
planned around this area. As it currently stands, it seems like this 
would only complicate things.

There is one issue that needs to be resolved in order to support even 
one irq_domain with this configuration. The problem is that we allocate 
irq_descs dynamically based on Device Tree information, and so 
irq_domain_add() needs to be modified to support this. In fact, there's 
even a comment in the function regarding this limitation. There's two 
problems with the implementation of irq_domain_add() for my usage:

1. It assumes irq_descs are already allocated. In my case, I would like 
to add the irq_domain at init time through of_irq_init(). But the actual 
descriptors are not allocated until much later.

2. It assumes that every hw_irq is available within the range of 
interrupts the domain manages, which is not true in my case. In my case, 
they are truly sparse.

Thus I am proposing we have a more simple irq_domain_add() function that 
solely adds the irq_domain to the list. Then we can create registration 
interface that does the extra initialization for the irq_data of each 
specified hw_irq. What do you think about this? I can submit a patch if 
others think this is reasonable.

Thanks,
Mike

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum