FW: sched_setaffinity not working with kernel 2.6.32.15

FW: sched_setaffinity not working with kernel 2.6.32.15

[Αλέξανδρος Παπαδογιαννάκης]

[-- Attachment #1: Type: text/plain, Size: 3062 bytes --]


From: psxlover@hotmail.com
To: mingo@elte.hu; peterz@infradead.org
Subject:
 sched_setaffinity not working with kernel 2.6.32.15
Date: Sun, 20 
Jun 2010 17:14:12 +0300






.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}



On kernel 2.6.32.15 after I execute sched_setaffinity even
though 
sched_getaffinity returns the right cpus the threads
of my program 
are running on different cpus.

  First time I came across the bug
 was on a PC on my unive-
rsity which had just been updated to Debian
 'squeeze'. I was
using a library calles hwloc to get information 
about the to-
pology of the system and bind threads to cpus. The 
binding
while it wasn't returning any error wasn't working (I used 
top
to check which cpus where being used). After contacting 
with 
the developers of hwloc they suggested that the pro-
blem is either 
in the kernel or in glibc.
  I made a virtual machine on my pc using 
the latest Debian
'squeeze' and the problem was there. So I tried an 
older
kernel I found in the Synaptic Package Manager and my pro-
gram
 was working fine. So the problem was in the kernel. I
e-mailed hwloc
 mailing list again and he suggested I tried
the vanilla kernels and 
use git bisect to find where the problem
started.
  With 2.6.34 
and 2.6.33.5 I had no problem but with 2.6.32.15
sched_setaffinity 
wasn't working. I used git bisect and in the 
end I got this:


c6fc81afa2d7ef2f775e48672693d8a0a8a7325d
 is the first bad commit
commit 
c6fc81afa2d7ef2f775e48672693d8a0a8a7325d
Author: John Wright 
<john.wright@hp.com>
Date:   Tue Apr 13 16:55:37 2010 -0600

   
 sched: Fix a race between ttwu() and migrate_task()
    
    
Based on commit e2912009fb7b715728311b0d8fe327a1432b3f79 upstream, but
   
 done differently as this issue is not present in .33 or .34 kernels due
   
 to rework in this area.
    
    If a task is in the TASK_WAITING
 state, then try_to_wake_up() is working
    on it, and it will place
 it on the correct cpu.
    
    This commit ensures that neither 
migrate_task() nor __migrate_task()
    calls set_task_cpu(p) while p
 is in the TASK_WAKING state.  Otherwise,
    there could be two 
concurrent calls to set_task_cpu(p), resulting in
    the task's 
cfs_rq being inconsistent with its cpu.
    
    Signed-off-by: 
John Wright <john.wright@hp.com>
    Cc: Ingo Molnar 
<mingo@elte.hu>
    Cc: Peter Zijlstra 
<peterz@infradead.org>
    Signed-off-by: Greg Kroah-Hartman 
<gregkh@suse.de>

:040000 040000 
a9d18950d8edddb761c0266706f671f0e9a006fe 
2c3a7d7d5e616ecc276b4e93f4b6e5162a9382c8 M    kernel


  I've 
attached a simple program that binds a few threads on
the first cpu. 
With 2.6.32.15 when I run it I can see with 
System Monitor that more
 than one cpu is being used by the
threads.


--------------------------
Alexandros
 Papadogiannakis 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with Microsoft's powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test.c --]
[-- Type: text/x-csrc, Size: 2148 bytes --]


#ifndef _GNU_SOURCE
#define _GNU_SOURCE 1
#endif

#include <stdio.h>
//#include <sched.h>
//#include <hwloc.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/syscall.h>

#define NR_CPUS 8
#define NR_THREADS 5

pid_t tid[NR_THREADS];

pthread_barrier_t init_start, barrier2;

void print_affinity(pthread_t pid, int threadno) {
	int i;
	cpu_set_t set;
	
	CPU_ZERO(&set);
	
	sched_getaffinity( pid, sizeof(set), &set);  

	
	for( i = 0; i < NR_CPUS; i++) {
		if (CPU_ISSET(i, &set))
			printf("Cpu #%d is in %d thread's affinity\n", i, threadno);
	}
}

void set_schedaffinity(pthread_t pid, int cpu) {
	cpu_set_t set;
	
	CPU_ZERO(&set);
	CPU_SET(cpu, &set);
	
	sched_setaffinity(pid, sizeof(set), &set);
}

void set_threadaffinity(pthread_t pid, int cpu) {
	cpu_set_t set;
	
	CPU_ZERO(&set);
	CPU_SET(cpu, &set);
	
	pthread_setaffinity_np(pid, sizeof(set), &set);
}

void * threadCode(void * threadNo) {
	long int thread = (long int) threadNo;
	
	//Set current thread id
	//tid[thread] = (pid_t) syscall (SYS_gettid);
	
	//Wait for all the threads to set their thread id
	//pthread_barrier_wait(&init_start);
	
	//Wait for main to set the affinity
	pthread_barrier_wait(&barrier2);
	//sleep(1);
	
	//Print affinity
	print_affinity(0,thread);
	
	//Do something so that we can see cpu being used
	while (1);

	pthread_exit(0);
}



int main(int argc , char *argv[]) {
	long int i;
	pthread_t pid[NR_THREADS];
	
	// Initialize barriers
	if(pthread_barrier_init(&init_start, NULL, NR_THREADS)) {
		printf("Could not create a barrier\n");
		exit(1);
	}
	if(pthread_barrier_init(&barrier2, NULL, NR_THREADS)) {
		printf("Could not create a barrier\n");
		exit(1);
	}
	
	//Create threads
	for (i = 0; i < NR_THREADS; i++)
		pthread_create(&pid[i], NULL, threadCode, (void *) i);
	
	//Wait for threads to set their thread id
	//pthread_barrier_wait(&init_start);
	
	//Set affinity
	for (i = 0; i < NR_THREADS; i++)
		//set_schedaffinity(tid[i],1);
		set_threadaffinity(pid[i],0);
	
	//Tell the threads to continue after setting affinity
	//pthread_barrier_wait(&barrier2);
	
	getchar(); 
	
    return 0;
}

Re: FW: sched_setaffinity not working with kernel 2.6.32.15

[Andrew Morton]

On Sun, 20 Jun 2010 22:10:18 +0300
____________________ ______________________________  <psxlover@hotmail.com> wrote:

> 
> From: psxlover@hotmail.com
> To: mingo@elte.hu; peterz@infradead.org
> Subject:
>  sched_setaffinity not working with kernel 2.6.32.15
> Date: Sun, 20 
> Jun 2010 17:14:12 +0300

I assume from this that you initially sent the email privately to
Peter&Ingo, then forwarded it to the list without cc'ing Peter&Ingo. 
So if they're already responded to the inital email, I'm wasting my
time.  Ho hum.

> ...

That's a great bug report - fully bisected and it includes a testcase. 
Thanks.

However I think your testcase is buggy.  You have main() racing against
print_affinity().  If a sub-thread runs print_affinity() before main()
has run set_threadaffinity(), print_affinity() will obviously print
main()'s affinity.

Between 2.6.18 and 2.6.35-rc3 the timing changed a bit - in the later
kernel the sub-threads are running before main() is able to change
their affinity.  In the earlier kernel the sub-threads run first.

In both kernels, all CPU consumption is on CPU 0 as desired.

Re-enabling the sleep(1) in threadCode() fixes the print_affinity()
output, confirming that it's a userspace race.

Re: FW: sched_setaffinity not working with kernel 2.6.32.15

[Peter Zijlstra]

On Tue, 2010-06-22 at 17:16 -0700, Andrew Morton wrote:
> On Sun, 20 Jun 2010 22:10:18 +0300
> ____________________ ______________________________  <psxlover@hotmail.com> wrote:
> 
> > 
> > From: psxlover@hotmail.com
> > To: mingo@elte.hu; peterz@infradead.org
> > Subject:
> >  sched_setaffinity not working with kernel 2.6.32.15
> > Date: Sun, 20 
> > Jun 2010 17:14:12 +0300
> 
> I assume from this that you initially sent the email privately to
> Peter&Ingo, then forwarded it to the list without cc'ing Peter&Ingo. 
> So if they're already responded to the inital email, I'm wasting my
> time.  Ho hum.

My private copy seems to have gone missing..

> > ...
> 
> That's a great bug report - fully bisected and it includes a testcase. 
> Thanks.
> 
> However I think your testcase is buggy.  You have main() racing against
> print_affinity().  If a sub-thread runs print_affinity() before main()
> has run set_threadaffinity(), print_affinity() will obviously print
> main()'s affinity.
> 
> Between 2.6.18 and 2.6.35-rc3 the timing changed a bit - in the later
> kernel the sub-threads are running before main() is able to change
> their affinity.  In the earlier kernel the sub-threads run first.
> 
> In both kernels, all CPU consumption is on CPU 0 as desired.
> 
> Re-enabling the sleep(1) in threadCode() fixes the print_affinity()
> output, confirming that it's a userspace race.

Andrew is right, your program is buggy. 

RE: sched_setaffinity not working with kernel 2.6.32.15

[Αλέξανδρος Παπαδογιαννάκης]

Thanks for the reply. I first send a msg to Peter and Ingo and then someone
suggested I should also send it to the kernel mailing list so I forwarded 
it. Don't worry I didn't get any reply yet.

About the testcase, seems like I accidentaly commented:
    //pthread_barrier_wait(&barrier2);
That's in the main() and it was there so that the threads won't call
print_affinity() before main calls set_threadaffinity(). 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969

RE: sched_setaffinity not working with kernel 2.6.32.15

[Αλέξανδρος Παπαδογιαννάκης]

And:
if(pthread_barrier_init(&barrier2, NULL, NR_THREADS)) {
should be:
if(pthread_barrier_init(&barrier2, NULL, NR_THREADS+1)) {
 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969

RE: sched_setaffinity not working with kernel 2.6.32.15

[Αλέξανδρος Παπαδογιαννάκης]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]


Here's the correct testcase

    
            

            
 		 	   		  
_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
https://signup.live.com/signup.aspx?id=60969

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: sched_check.c --]
[-- Type: text/x-csrc, Size: 2273 bytes --]


#ifndef _GNU_SOURCE
#define _GNU_SOURCE 1
#endif

#include <stdio.h>
//#include <sched.h>
//#include <hwloc.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/syscall.h>

#define NR_CPUS 8
#define NR_THREADS 5
#define USE_THREADAFFINITY 1

pid_t tid[NR_THREADS];

pthread_barrier_t init_start, barrier2;

void print_affinity(pthread_t pid, int threadno) {
	int i;
	cpu_set_t set;
	
	CPU_ZERO(&set);
	
	sched_getaffinity( pid, sizeof(set), &set);  

	
	for( i = 0; i < NR_CPUS; i++) {
		if (CPU_ISSET(i, &set))
			printf("Cpu #%d is in %d thread's affinity\n", i, threadno);
	}
}

void set_schedaffinity(pthread_t pid, int cpu) {
	cpu_set_t set;
	
	CPU_ZERO(&set);
	CPU_SET(cpu, &set);
	
	sched_setaffinity(pid, sizeof(set), &set);
}

void set_threadaffinity(pthread_t pid, int cpu) {
	cpu_set_t set;
	
	CPU_ZERO(&set);
	CPU_SET(cpu, &set);
	
	pthread_setaffinity_np(pid, sizeof(set), &set);
}

void * threadCode(void * threadNo) {
	long int thread = (long int) threadNo;
	
	if (!USE_THREADAFFINITY) {
		//Set current thread id
		tid[thread] = (pid_t) syscall (SYS_gettid);
	
		//Wait for all the threads to set their thread id
		pthread_barrier_wait(&init_start);
	}
	
	//Wait for main to set the affinity
	pthread_barrier_wait(&barrier2);
	//sleep(1);
	
	//Print affinity
	print_affinity(0,thread);
	
	//Do something so that we can see cpu being used
	while (1);

	pthread_exit(0);
}



int main(int argc , char *argv[]) {
	long int i;
	pthread_t pid[NR_THREADS];
	
	// Initialize barriers
	if(pthread_barrier_init(&init_start, NULL, NR_THREADS+1)) {
		printf("Could not create a barrier\n");
		exit(1);
	}
	if(pthread_barrier_init(&barrier2, NULL, NR_THREADS+1)) {
		printf("Could not create a barrier\n");
		exit(1);
	}
	
	//Create threads
	for (i = 0; i < NR_THREADS; i++)
		pthread_create(&pid[i], NULL, threadCode, (void *) i);
	
	//Wait for threads to set their thread id
	if (!USE_THREADAFFINITY)
		pthread_barrier_wait(&init_start);
	
	//Set affinity
	for (i = 0; i < NR_THREADS; i++) {
		if (USE_THREADAFFINITY)
			set_threadaffinity(pid[i],0);
		else
			set_schedaffinity(tid[i],0);
	}
	
	//Tell the threads to continue after setting affinity
	pthread_barrier_wait(&barrier2);
	
	getchar(); 
	
    return 0;
}