NIC interrupt assignments under UltraSPARC-T1

NIC interrupt assignments under UltraSPARC-T1

[Rick Jones]

 From time to time I "play" with netperf on different systems.  I happen 
to have occasion to play with a T2000.  Under Solaris 10 I am able to 
coerce the interrupts of the different core GbEs to be on different 
cores rather than strands of the same core.

Under a 2.6.15 kernel (Ubuntu Dapper) it would appear that the old 
standby of "echo affinity mask to the IRQ" doesn't "work" - no matter 
how I change the mask for a NIC, running a netperf TCP_RR test seems to 
show the interrupts happening on the same strand.

Is it indeed not possible to alter the interrupt assignments or have I 
(as I'm wont to do) missed something quasi-obvious?

thanks,

rick jones

Re: NIC interrupt assignments under UltraSPARC-T1

[David Miller]

From: Rick Jones <rick.jones2@hp.com>
Date: Wed, 13 Sep 2006 17:47:27 -0700

> Is it indeed not possible to alter the interrupt assignments or have I 
> (as I'm wont to do) missed something quasi-obvious?

It should work, perhaps there is a bug somewhere.

I think it's a bad idea, btw, to asking sparc port questions
on the netdev list :-)  sparclinux@vger would be a much
better place.

I'm travelling currently, and won't be back until next week,
so I have no way to plug in and power on my T1000 machine to
try and diagnose this.  Please try and debug this yourself by
poking around in arch/sparc64/kernel/irq.c where this is supposed
to be happening.

Thanks.

NIC interrupt assignments under UltraSPARC-T1

[Rick Jones]

 From time to time I "play" with netperf on different systems.  I happen 
to have occasion to play with a T2000.  Under Solaris 10 I am able to 
coerce the interrupts of the different core GbEs to be on different 
cores rather than strands of the same core.

Under a 2.6.15 kernel (Ubuntu Dapper) it would appear that the old 
standby of "echo affinity mask to the /proc/irq/mumble" I've used with 
Linux on other (non-SPARC) hardware doesn't "work" - no matter how I 
change the mask for a NIC, running a netperf TCP_RR test seems to show 
the interrupts happening on the same strand.

Is it indeed not possible to alter the interrupt assignments or have I 
(as I'm wont to do) missed something quasi-obvious?

It has been suggested that I should be poking around in 
arch/sparc64/kernel/irq.c and i've looked there, I'm still a bit lost as 
I've not learned the heirarchy of the code in that area.  With my 
netperf stuff I've had the good fortune to not have to go any deeper 
than getting a call to sched_setaffinity() right...  I have noticed that 
down in that file everything seems to be assuming a specific "CPU" 
rather than a set of them (mask)

thanks,

rick jones
http://www.netperf.org/

Re: NIC interrupt assignments under UltraSPARC-T1

[David Miller]

From: Rick Jones <rick.jones2@hp.com>
Date: Thu, 14 Sep 2006 13:17:15 -0700

> Is it indeed not possible to alter the interrupt assignments or have
> I (as I'm wont to do) missed something quasi-obvious?

Unfortunately this feature had to be removed for the time
being.  The reason is that the "switch" used to occur
dynamically on the next received interrupt.  Ie. when
the next instance of that interrupt arrived, we would
program it to something in the newly configured affinity
mask.

That proved to be race prone and could jam up an interrupt,
so until I am able to diagnose that problem and re-add the
necessary logic, interrupt distribution is currently fixed.