Re: CFS Bandwidth Control - Test results of cgroups tasks pinned vs unpinnede

[Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>]

* Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> [2011-09-14 00:05:02]:

> * Peter Zijlstra <a.p.zijlstra@chello.nl> [2011-09-13 20:30:46]:
> 
> > On Tue, 2011-09-13 at 23:58 +0530, Srivatsa Vaddagiri wrote:
> > > * Peter Zijlstra <a.p.zijlstra@chello.nl> [2011-09-13 20:19:55]:
> > > 
> > > > On Tue, 2011-09-13 at 23:24 +0530, Srivatsa Vaddagiri wrote:
> > > > > We saw considerably high migration count on latest tip compared to
> > > > > previous kernels. Kamalesh, can you please post the migration count
> > > > > data? 
> > > > 
> > > > Hrmm, yes this looks horrid.. even without cgroup crap, something's funny.
> > > 
> > > Yes ..we could visualize that very much in top o/p .. A task's cpu would keep 
> > > changing *every* screen refresh (refreshed every 0.5 sec that too!).
> > > 
> > > We didn't see that with older kernels ..Kamalesh is planning to do a
> > > git bisect and see which commit lead to this "mad" hopping ..
> > 
> > Awesome, thanks! Btw, what is 'older'? 3.0?
> 
> We went back all the way upto 2.6.32! I think 2.6.38 and 2.6.39 were
> pretty stable ..I don't have the migration count data with me readily. I
> will let Kamalesh post that info soon.

Test Setup :
-----------
Machine is 2 socket Quad Core Intel (x5570) box. The lb.sh
script was run in a loop to execute 5 times after the box
was bought up with the kernel.  

lb.sh script spawns 2x number of CPU hogs, where x is
number of CPUs on the system. The script collects the
se.nr_migration before/after 60 seconds sleep and subtracts
the after_se.nr_migration - before_se.nr_migrations for 
all the spawned hogs.
----------------+-------+-------+-------+-------+-------+
Kernel		| Run 1	| Run 2	| Run 3	| Run 4	| Run 5	|
----------------+-------+-------+-------+-------+-------+
2.6.33		| 9604	| 101	| 66	| 2543	| 3488	|
----------------+-------+-------+-------+-------+-------+
2.6.34		| 28469	| 1514	| 1602	| 185	| 139	|	
----------------+-------+-------+-------+-------+-------+
2.6.35		| 1052	| 12	| 4	| 11	| 6	|
----------------+-------+-------+-------+-------+-------+
2.6.36		| 1253	| 53	| 78	| 76	| 50	|
----------------+-------+-------+-------+-------+-------+
2.6.37		| 262	| 36	| 48	| 61	| 43	|
----------------+-------+-------+-------+-------+-------+
2.6.38		| 1551	| 48	| 62	| 47	| 50	|
----------------+-------+-------+-------+-------+-------+
2.6.39		| 3784	| 457	| 722	| 3209	| 1037	|
----------------+-------+-------+-------+-------+-------+
3.0		| 933	| 608	| 658	| 1424	| 1415	|
----------------+-------+-------+-------+-------+-------+
3.1.0-rc4-tip	|	|	|	|	|	|
(e467f18f945)	| 1672	| 1643	| 1316	| 1577	| 61	|
----------------+-------+-------+-------+-------+-------+

lb.sh
------
#!/bin/bash

rm -rf test*
rm -rf t*

ITERATIONS=60			# No of Iterations to capture the details
NUM_CPUS=$(cat /proc/cpuinfo |grep -i proces|wc -l)

NUM_HOGS=$((NUM_CPUS * 2))	# No of hogs threads to invoke

echo "System has $NUM_CPUS cpus..... Spawing $NUM_HOGS cpu hogs ... for $ITERATIONS seconds.."
if [ ! -e while1.c ]
then
	cat >> while1.c << EOF
	int
	main (int argc, char **argv)
	{
		while(1);
		return (0);
	}
EOF
fi

for i in $(seq 1 $NUM_HOGS)
do
	gcc -o while$i while1.c
	if [ $? -ne 0 ]
	then
		echo "Looks like gcc is not present ... aborting"
		exit
	fi
done

for i in $(seq 1 $NUM_HOGS)
do
	./while$i &
	pids[$i]=$!
	pids_old[$i]=`cat /proc/$!/sched |grep -i nr_migr|grep -iv cold|cut -d ":" -f2|sed 's/  //g'`
done

sleep $ITERATIONS

j=1
old_nr_migrations=0
new_nr_migrations=0
echo  -e  "       \t New \t Old"
for i in $(seq 1 $NUM_HOGS)
do
	a=`echo ${pids[i]}`	
	new=`cat /proc/$a/sched |grep -i nr_migr|grep -iv cold|cut -d ":" -f2|sed 's/  //g'`
	old=`echo ${pids_old[i]}`
	old_nr_migrations=$((old_nr_migrations + old))
	c=$(($new - $old))
	new_nr_migrations=$((new_nr_migrations + c))
	echo -e "while$i\t[$new]\t[$old]\t"
done
echo "*******************************************"
echo -e "      $new_nr_migrations\t$old_nr_migrations"
echo "*******************************************"
pkill -9 while

exit