3 ways to represent cpu affinity in /sys and counting

3 ways to represent cpu affinity in /sys and counting

[Anton Blanchard]

Hi,

We have a patch to change pcibus_to_cpumask to pcibus_to_node. This makes
it more consistent with cpu_to_node, and when you want a cpumask you
use node_to_cpumask.

The one real user of pcibus_to_cpumask in arch/x86-64/kernel/pci-gart.c
highlights this:

	cpumask_t mask;
	mask = pcibus_to_cpumask(to_pci_dev(dev)->bus->number);
	node = cpu_to_node(first_cpu(mask));

We go around in circles to get back to the node. I then started looking
at how we export cpu affinity and node information in sysfs:

A pci device has a local_cpus property:

/sys/devices/pci000a:00/000a:00:02.6/local_cpus

A pci_bus has a cpuaffinity property:

/sys/class/pci_bus/000d:d8/cpuaffinity

A node has a cpumap property:

/sys/devices/system/node/node3/cpumap

Can we standardize on a single property name for this? :)

Furthermore, looking at node linkages:

A node has symlinks to cpus:

/sys/devices/system/node/node0/cpu0 -> /sys/devices/system/cpu/cpu0

But doesnt have symlinks to pci devices.

A cpu doesnt have a cpumask of its node, but a pci device and pci bus
both do. Is there some way we can stardardize this too? 

Ideally we want to be able to lookup a device -> node -> cpumask
relationship in a consistent way. Assuming we fix up the 3 different
names for cpu affinity properties, we still have 2 methods of looking
that up, and no easy way of going from pci devices to nodes.

Anton

Re: 3 ways to represent cpu affinity in /sys and counting

[Jesse Barnes]

On Saturday, December 25, 2004 4:27 pm, Anton Blanchard wrote:
> Hi,
>
> We have a patch to change pcibus_to_cpumask to pcibus_to_node. This makes
> it more consistent with cpu_to_node, and when you want a cpumask you
> use node_to_cpumask.

Great!  I think Matt Dobson said he was going to do something similar, but he 
got sidetracked (and iirc, ak also didn't like the idea).  FWIW, I like it, 
with the caveat that the node returned from pcibus_to_node may not have any 
memory or CPUs associated with it.

> A pci device has a local_cpus property:
>
> /sys/devices/pci000a:00/000a:00:02.6/local_cpus
>
> A pci_bus has a cpuaffinity property:
>
> /sys/class/pci_bus/000d:d8/cpuaffinity

I don't know how these two got different names...

> A node has a cpumap property:
>
> /sys/devices/system/node/node3/cpumap
>
> Can we standardize on a single property name for this? :)

Seems like nodes should have a cpumap and PCI busses should have a node.

> Furthermore, looking at node linkages:
>
> A node has symlinks to cpus:
>
> /sys/devices/system/node/node0/cpu0 -> /sys/devices/system/cpu/cpu0
>
> But doesnt have symlinks to pci devices.

Yep, this would be nice to have.

Jesse

Re: 3 ways to represent cpu affinity in /sys and counting

[Andi Kleen]

On Sun, Dec 26, 2004 at 11:27:44AM +1100, Anton Blanchard wrote:
> 
> A pci device has a local_cpus property:
> 
> /sys/devices/pci000a:00/000a:00:02.6/local_cpus

We should have both node and cpus here because both
are needed in user space (one for CPU affinity, the other
for memory affinity). While it's possible to convert
in user space it's quite cumbersome. 

I plan to add functions to libnuma to make this easy,
but I'm not sure everybody will want to use libnuma just
for this.

> 
> A pci_bus has a cpuaffinity property:
> 
> /sys/class/pci_bus/000d:d8/cpuaffinity

I'm not sure what that is anyways. I don't think x86-64 sets it up
and does anybody really use it?
> 
> A node has a cpumap property:
> 
> /sys/devices/system/node/node3/cpumap
> 
> Can we standardize on a single property name for this? :)

I suspect it's already too late. 

> 
> Furthermore, looking at node linkages:
> 
> A node has symlinks to cpus:
> 
> /sys/devices/system/node/node0/cpu0 -> /sys/devices/system/cpu/cpu0
> 
> But doesnt have symlinks to pci devices.

I'm not sure how useful that would be anyways.

> 
> A cpu doesnt have a cpumask of its node, but a pci device and pci bus
> both do. Is there some way we can stardardize this too? 

You mean a nodemask?

> 
> Ideally we want to be able to lookup a device -> node -> cpumask
> relationship in a consistent way. Assuming we fix up the 3 different
> names for cpu affinity properties, we still have 2 methods of looking
> that up, and no easy way of going from pci devices to nodes.

You mean the user space interface?

Getting it from sysfs will be always a mess, best is to have a standard
library that offers this in nice functions and a command whose output
can be parsed by shell scripts. I plan to add this eventually to 
numactl/libnuma.

-Andi