detecting hyperthreading in linux 2.4.19

detecting hyperthreading in linux 2.4.19

[Jason Lunz]

Is there a way for a userspace program running on linux 2.4.19 to tell
the difference between a single hyperthreaded xeon P4 with HT enabled
and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
for the two cases are indistinguishable.

Jason

Re: detecting hyperthreading in linux 2.4.19

[James Cleverdon]

On Thursday 09 January 2003 12:02 pm, Jason Lunz wrote:
> Is there a way for a userspace program running on linux 2.4.19 to tell
> the difference between a single hyperthreaded xeon P4 with HT enabled
> and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
> for the two cases are indistinguishable.
>
> Jason
>
> -

In the kernel that's no problem:

A) If the BIOS writers followed Intel's guidelines, just look at the physical 
APIC IDs.  HT siblings have odd IDs, the real ones have even.

B) Check the siblings table built up at boot time and used by the scheduler.

I don't know of any way to do this in userland.  The whole point is that the 
sibling processors are supposed to look like real ones.

You _could_ try running two processes simultaneously in tight spin loops for 
100 million cycles and comparing the amount of real time consumed.  That 
would be rather unreliable and kludgey though.

-- 
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com

Re: detecting hyperthreading in linux 2.4.19

[John Bradford]

> > Is there a way for a userspace program running on linux 2.4.19 to tell
> > the difference between a single hyperthreaded xeon P4 with HT enabled
> > and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
> > for the two cases are indistinguishable.
> 
> I don't know of any way to do this in userland.  The whole point is that the 
> sibling processors are supposed to look like real ones.
> 
> You _could_ try running two processes simultaneously in tight spin loops for 
> 100 million cycles and comparing the amount of real time consumed.  That 
> would be rather unreliable and kludgey though.

If /proc/interrupts shows a processor is handling interrupts then it
is definitely a 'real' one.  If it isn't handling interrupts, it may
or may not be a 'real' one.  That's another unreliable and kludgey way
to tell the difference :-).

John.

Re: detecting hyperthreading in linux 2.4.19

[James Cleverdon]

On Thursday 09 January 2003 01:54 pm, John Bradford wrote:
> > > Is there a way for a userspace program running on linux 2.4.19 to tell
> > > the difference between a single hyperthreaded xeon P4 with HT enabled
> > > and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
> > > for the two cases are indistinguishable.
> >
> > I don't know of any way to do this in userland.  The whole point is that
> > the sibling processors are supposed to look like real ones.
> >
> > You _could_ try running two processes simultaneously in tight spin loops
> > for 100 million cycles and comparing the amount of real time consumed. 
> > That would be rather unreliable and kludgey though.
>
> If /proc/interrupts shows a processor is handling interrupts then it
> is definitely a 'real' one.  If it isn't handling interrupts, it may
> or may not be a 'real' one.  That's another unreliable and kludgey way
> to tell the difference :-).
>
> John.
> -

Not quite.  The "logical" or "sibling" processor has its own local APIC.  This 
is necessary to send it the Startup IPI and soft IPIs while the system is 
running.

However, it is a full function APIC and can receive I/O interrupts.  I have 
seen this happen many times.

-- 
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com

Re: detecting hyperthreading in linux 2.4.19

[Mike Dresser]

On Thu, 9 Jan 2003, John Bradford wrote:

> If /proc/interrupts shows a processor is handling interrupts then it
> is definitely a 'real' one.  If it isn't handling interrupts, it may
> or may not be a 'real' one.  That's another unreliable and kludgey way
> to tell the difference :-).

What about something like lmsensors?  Wouldn't the motherboard normally
report different temperatures for each cpu, since each has its own temp
diode?

(this originally was supposed to be a in-jest idea to run two threads and
see if one cpu or two cpu's heat up, but then i realized you might as well
just count the temperature diodes :D)

Mike

Re: detecting hyperthreading in linux 2.4.19

[Jason Lunz]

jamesclv@us.ibm.com said:
> I don't know of any way to do this in userland.  The whole point is
> that the sibling processors are supposed to look like real ones.

That's unfortunately not always true. I'm writing a program that will
run on a system that will be doing high-load routing. Testing has shown
that we get better performance when binding each NIC's interrupts to a
separate physical processor using /proc/irq/*/smp_affinity (especially
when all the interrupts would hit the first CPU, another problem i've
yet to address). That only works for real processors, though, not
HT siblings.

I'm writing a program to run on machines of unknown (by me)
configuration, that will spread out the NIC interrupts appropriately.
So userspace needs to know the difference, at least until interrupts can
be automatically distributed by the kernel in a satisfactory way.

Jason

Re: detecting hyperthreading in linux 2.4.19

[Mark Hounschell]

Jason Lunz wrote:
> 
> jamesclv@us.ibm.com said:
> > I don't know of any way to do this in userland.  The whole point is
> > that the sibling processors are supposed to look like real ones.
> 
> That's unfortunately not always true. I'm writing a program that will
> run on a system that will be doing high-load routing. Testing has shown
> that we get better performance when binding each NIC's interrupts to a
> separate physical processor using /proc/irq/*/smp_affinity (especially
> when all the interrupts would hit the first CPU, another problem i've
> yet to address). That only works for real processors, though, not
> HT siblings.

Strange, I'm doing the very same thing (not with NICs though) using the
local_irq_desc
at the driver level or via /proc/irq/*/smp_affinity method and both work fine.
2.4.18 and 2.4.20 both work here but haven't actually used 2.4.19.
 
> 
> I'm writing a program to run on machines of unknown (by me)
> configuration, that will spread out the NIC interrupts appropriately.
> So userspace needs to know the difference, at least until interrupts can
> be automatically distributed by the kernel in a satisfactory way.

My userland app doesn't know the difference and works fine whether HT or not.
At least according to /proc/interrupts, xosview, and the actual performance of
my app.

Mark

Re: detecting hyperthreading in linux 2.4.19

[Mikael Pettersson]

James Cleverdon writes:
 > On Thursday 09 January 2003 12:02 pm, Jason Lunz wrote:
 > > Is there a way for a userspace program running on linux 2.4.19 to tell
 > > the difference between a single hyperthreaded xeon P4 with HT enabled
 > > and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
 > > for the two cases are indistinguishable.
 > >
 > > Jason
 > >
 > > -
 > 
 > In the kernel that's no problem:
 > 
 > A) If the BIOS writers followed Intel's guidelines, just look at the physical 
 > APIC IDs.  HT siblings have odd IDs, the real ones have even.
 > 
 > B) Check the siblings table built up at boot time and used by the scheduler.
 > 
 > I don't know of any way to do this in userland.  The whole point is that the 
 > sibling processors are supposed to look like real ones.

If the kernel has sched_setaffinity() or some other way of binding a process
to a given CPU (as numbered by the kernel, which may or may not be related
to any physical CPU numbers), then this will do it: execute CPUID on each
CPU and check the initial APIC ID field. If you find one that's non-zero,
then HT is enabled.

My performance monitoring counters driver uses this approach in kernel-space
using smp_call_function(). I don't use the siblings tables because they suck :-)
[I don't think they distinguish between logical CPUs #0 and #1, and they aren't
exported to modules. The CPUID check is simple and portable across kernel versions.]

Re: detecting hyperthreading in linux 2.4.19

[Dave Jones]

On Fri, Jan 10, 2003 at 08:05:07AM +0100, Mikael Pettersson wrote:

 > If the kernel has sched_setaffinity() or some other way of binding a process
 > to a given CPU (as numbered by the kernel, which may or may not be related
 > to any physical CPU numbers), then this will do it: execute CPUID on each
 > CPU and check the initial APIC ID field. If you find one that's non-zero,
 > then HT is enabled.

That's a horrible way of reimplementing /dev/cpu/x/cpuid  8-)

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: detecting hyperthreading in linux 2.4.19

[Mikael Pettersson]

Dave Jones writes:
 > On Fri, Jan 10, 2003 at 08:05:07AM +0100, Mikael Pettersson wrote:
 > 
 >  > If the kernel has sched_setaffinity() or some other way of binding a process
 >  > to a given CPU (as numbered by the kernel, which may or may not be related
 >  > to any physical CPU numbers), then this will do it: execute CPUID on each
 >  > CPU and check the initial APIC ID field. If you find one that's non-zero,
 >  > then HT is enabled.
 > 
 > That's a horrible way of reimplementing /dev/cpu/x/cpuid  8-)

True. I forgot about that interface since I personally never configure or use it.

So this answers the original question. Just access /dev/cpu/$CPU/cpuid
for all (accessible) values of $CPU.

Re: detecting hyperthreading in linux 2.4.19

[Dave Jones]

On Thu, Jan 09, 2003 at 08:02:33PM +0000, Jason Lunz wrote:
 > Is there a way for a userspace program running on linux 2.4.19 to tell
 > the difference between a single hyperthreaded xeon P4 with HT enabled
 > and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
 > for the two cases are indistinguishable.

Check out x86info[1]. It should do the right thing in both cases.
It counts siblings, and also checks the BIOS MP table.

		Dave

[1] http://www.codemonkey.org.uk/x86info/
		
-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

RE: detecting hyperthreading in linux 2.4.19

[Kamble, Nitin A]

Hi James,
I have posted a patch for this on LKML. Please have look at:
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0301.0/1886.html
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0301.0/1887.html

It should solve the problem you are trying to solve.

Nitin

> -----Original Message-----
> From: Jason Lunz [mailto:lunz@falooley.org]
> Sent: Thursday, January 09, 2003 1:57 PM
> To: linux-kernel@vger.kernel.org
> Subject: Re: detecting hyperthreading in linux 2.4.19
> 
> jamesclv@us.ibm.com said:
> > I don't know of any way to do this in userland.  The whole point is
> > that the sibling processors are supposed to look like real ones.
> 
> That's unfortunately not always true. I'm writing a program that will
> run on a system that will be doing high-load routing. Testing has
shown
> that we get better performance when binding each NIC's interrupts to a
> separate physical processor using /proc/irq/*/smp_affinity (especially
> when all the interrupts would hit the first CPU, another problem i've
> yet to address). That only works for real processors, though, not
> HT siblings.
> 
> I'm writing a program to run on machines of unknown (by me)
> configuration, that will spread out the NIC interrupts appropriately.
> So userspace needs to know the difference, at least until interrupts
can
> be automatically distributed by the kernel in a satisfactory way.
> 
> Jason

RE: detecting hyperthreading in linux 2.4.19

[Pallipadi, Venkatesh]

Systems running the latest RH, SuSE releases, with standard smp kernel
should see some entries in /proc/cpuinfo, which gives information about
HT configuration on the system. The latest 2.5 kernel also has similar
information in /proc/cpuinfo.
The entries look something like:
Physical processor ID   : 0
Number of siblings      : 2

Unfortunately, 2.4.19 doesn't have such information. One possible
workaround is to check big enough 'dmesg' and look for cpu_sibling_map.
That tells you the mapping of logical processors to physical package. Or
you can have a init script like this (attached) in place, to run at boot
time, so that you have a better chance of finding these particular
messages in log (without overflowing).

HTH,
Venkatesh

-------------------
#!/bin/sh
# checkht kernel RC file.
#
# chkconfig: 35 98 98
# description: check for hyperthreaded CPUs

HASHT=`/bin/dmesg | /bin/grep cpu_sibling_map`

if [ -n "$HASHT" ]; then
        /bin/echo "Machine is running HT"
        /bin/echo 1 > /etc/hyperthreaded
else
        /bin/echo "Machine isn't running HT"
        /bin/echo 0 > /etc/hyperthreaded
Fi
--------------


> -----Original Message-----
> From: Jason Lunz [mailto:lunz@falooley.org] 
> Sent: Thursday, January 09, 2003 12:03 PM
> To: linux-kernel@vger.kernel.org
> Subject: detecting hyperthreading in linux 2.4.19
> 
> 
> Is there a way for a userspace program running on linux 2.4.19 to tell
> the difference between a single hyperthreaded xeon P4 with HT enabled
> and a dual hyperthreaded xeon P4 with HT disabled? The /proc/cpuinfos
> for the two cases are indistinguishable.
> 
> Jason
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

RE: detecting hyperthreading in linux 2.4.19

[Pallipadi, Venkatesh]

> -----Original Message-----
> From: Mikael Pettersson [mailto:mikpe@csd.uu.se]
> 
> My performance monitoring counters driver uses this approach 
> in kernel-space
> using smp_call_function(). I don't use the siblings tables 
> because they suck :-)
> [I don't think they distinguish between logical CPUs #0 and 
> #1, and they aren't
> exported to modules. The CPUID check is simple and portable 
> across kernel versions.]

I believe it is better to use a OS interface to find out HT, rather than 
using CPUID. The reason being, it is possible to have HT disabled, in OS,
even after processor and the BIOS supports it. 
1) Consider the case when processor and BIOS supports HT, but linux
was booted with "noht" boot option (now I am not sure whether that option is 
there in 2.4.19. But is is certainly there in some other kernels).
2) What about some other kernel which is totally ignorant about HT, and 
doesn't initialize logical processor (kernel which looks at MPS in place
of ACPI)
I think, in both these cases cpuid can still say HT is present.

I know that sibling table is not exported. But I couldn't get your other
comment about sibling table "they distinguish between logical CPUs #0 and #1:"
Can you elaborate..

Thanks,
-Venkatesh

RE: detecting hyperthreading in linux 2.4.19

[Mikael Pettersson]

Pallipadi, Venkatesh writes:
 > 
 > > -----Original Message-----
 > > From: Mikael Pettersson [mailto:mikpe@csd.uu.se]
 > > 
 > > My performance monitoring counters driver uses this approach 
 > > in kernel-space
 > > using smp_call_function(). I don't use the siblings tables 
 > > because they suck :-)
 > > [I don't think they distinguish between logical CPUs #0 and 
 > > #1, and they aren't
 > > exported to modules. The CPUID check is simple and portable 
 > > across kernel versions.]
 > 
 > I believe it is better to use a OS interface to find out HT, rather than 
 > using CPUID. The reason being, it is possible to have HT disabled, in OS,
 > even after processor and the BIOS supports it. 
 > 1) Consider the case when processor and BIOS supports HT, but linux
 > was booted with "noht" boot option (now I am not sure whether that option is 
 > there in 2.4.19. But is is certainly there in some other kernels).
 > 2) What about some other kernel which is totally ignorant about HT, and 
 > doesn't initialize logical processor (kernel which looks at MPS in place
 > of ACPI)
 > I think, in both these cases cpuid can still say HT is present.

With smp_call_function() I execute code on exactly those CPUs the kernel
knows about. If the physical processors support HT but the kernel can't
run on the non-#0 logical CPUs (UP kernel or "noht"), then smp_call_function()
won't reach those logical CPUs #1 and everything's fine: my test will say
"not HT" which is exactly what I want in this case.

The point about checking the "local APIC physical ID" in CPUID 1, EBX,
instead of the field in the local APIC is _exactly_ because the CPUID
data is assigned by HW and read-only, while the local APIC field can be
changed. I don't want to depend on possibly-broken MP or ACPI tables or
have to know about the kernel's physical <--> virtual CPU # mapping de jour.

 > I know that sibling table is not exported. But I couldn't get your other
 > comment about sibling table "they distinguish between logical CPUs #0 and #1:"
 > Can you elaborate..

The sibling table tells you whether two linux-numbered CPUs are siblings,
but it doesn't tell which one is logical CPU #0 and which one is #1.
That distinction is important in some cases.