Soft CPU Affinity

[arlie@worldash.org (Arlie Stephens)]

Hi Folks,

I'm totally failing to come up with the right terms to google, so I'm
hoping the collective wisdom of this list can help me once again....

What I'm trying to understand is how the linux kernel decides where to
run things, in the absence of calls to explicitly set cpu affinity. 

Here's the scenario - 
- 16 core system (not a huge number; not small either)
- hard irq that's been assigned to a particular cpu by
  irqbalance. Let's say it's running on cpu 8.
- the hard irq triggers (probably) some kind of softirq activity,
  which runs ... where? ...  [by the looks of it preferentially on cpus
  8 and 9]
- this results in some task(s) becoming ready to run, which run
  ... where?... [by the looks of it, preferentially on cpus 8-11]

There are NUMA considerations - cpus 0-7 are on one socket, with cpus
8-15 on another. AFAIK, 8 and 9 or 8 and 10 are no "closer" together
than 8 and 15. Due to BIOS issues, the kernel does not have a fully
accurate picture of the NUMA layout, but it appears to know where the
cores are. 

What I'm looking for is explanations of the design and/or
implementation affecting this. It makes sense to me that the kernel
would run the soft irq on cpu 8, but why cpu 9? And why wouldn't the
user space processes be running on the same cpu where they'd
previously been? 

The system I'm looking at is running an elderly kernel ... RHEL 6.1,
aka 2.6.32. But explanations of how this worked/works in any version
of linux would be a step forward. 

Thanks for any pointers, and my apologies for using the list's
collective wisdom in place of google, but none of the terms I've tried
have given me anything very useful. 

--
Arlie

(Arlie Stephens						 arlie at worldash.org)