Re: sgi-gru drivers need of up to 8192 irqs

[Ingo Molnar <mingo@elte.hu>]

(Cc:-ed Yinghai and Mike Travis.)

* Dean Nelson <dcn@sgi.com> wrote:

> Hi Ingo,
> 
> You may remember a patch that I submitted not so long ago that added 
> uv_setup_irq() and uv_teardown_irq() functions for UV on x86.
> 
> It gave us (SGI) a way to 'allocate' an irq/vector pair and write an 
> MMR with a CPU's APCID and the vector number. (Two functions were 
> added to /arch/x86/kernel/io_apic.c to accomplish this -- 
> arch_enable_uv_irq() and arch_disable_uv_irq().)
> 
> This solution was what was suggested by the community after much 
> discussion.
> 
> I'm now trying to modify the GRU driver to call uv_setup_irq(). One 
> of the issues that is arising is that we need to be able to allocate 
> lots of irqs. For the GRU alone we need up to 32 irqs per blade 
> (based on the number of CPUs) and there can be up to 256 blades per 
> SSI. (A total of 8192 irqs per SSI.)
> 
> Currently I'm finding that NR_IRQS=4352. (It is constrained to the 
> lesser of NR_CPUS and MAX_IO_APICS.) But the system really runs off 
> of nr_irqs which gets set down by probe_nr_irqs() to 96 in one 
> configuration I've been running. This function bases its answer on 
> io_apic_get_redir_entries().
> 
> For the GRU there isn't an io_apic component, its needs are based on 
> NR_CPUS. So there isn't something readily available to cause 
> probe_nr_irqs() to generate a larger number.
> 
> Anyway, I'm looking for any suggestions as to what we should do. 
> (Which may include abandoning the using of irqs for the GRU driver 
> and just go with two hardwired system vectors?
> 
> Also, what has happened to the sparse irq patch? It doesn't seem to 
> exist anywhere. Has it been abandoned? I thought it was the means by 
> which we would be able to have huge numbers of irqs.

it's being worked on, with the first half of the changes upstream 
already, and the latest version of the second half of sparseirq (patch 
v11) was posted by Yinghai to lkml today, under this thread:

Subject: [PATCH] sparse_irq aka dyn_irq v11

	Ingo