* unfair scheduling with tbb application observed, could it be a kernel issue?
@ 2012-08-06 14:04 Pedro Larroy
2012-08-06 14:25 ` Mike Galbraith
0 siblings, 1 reply; 6+ messages in thread
From: Pedro Larroy @ 2012-08-06 14:04 UTC (permalink / raw)
To: linux-kernel
Hi
I think we are observing unfair scheduling of processes that use intel
TBB thread scheduler, as we have several processes with nice of 19 and
ioniced idle, and somehow the process with nice 0 should be getting
more than 1000% cpu
Any ideas? Kernel is 3.0.0.-17-generic on unbutu 11.10.
Avg[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||99.6%]
Tasks: 331; 81 running
Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||||||36386/96868MB]
Load average: 76.99 46.08 34.32
Swp[|||||||||
534/4099MB] Time: 15:51:51
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
10597 disco 39 19 6772M 6366M 14384 R 636. 6.6 37:56.63
/map_store/developers/disco/12_08_06-13_53_1344254009-j3UGmL/mmcc -X
178 -c /map_store/de
11629 visciano 20 0 6106M 5504M 18668 R 585. 5.7 39:02.15
build/release/mmcc -D --compact -f 8 -h bfmapcomp03.europe.nokia.com
-d 1521 -s LDMTEST -
10235 disco 39 19 4566M 4190M 12512 R 197. 4.3 25:13.58
/map_store/developers/disco/12_08_06-13_53_1344254009-j3UGmL/mmcc -X
183 -c /map_store/de
11599 disco 39 19 4935M 4644M 12572 R 188. 4.8 15:44.48
/map_store/developers/disco/12_08_06-13_53_1344254009-j3UGmL/mmcc -X
176 -c /map_store/de
11996 disco 39 19 4407M 4164M 12580 R 103. 4.3 12:08.25
/map_store/developers/disco/12_08_06-13_53_1344254009-j3UGmL/mmcc -X
174 -c /map_store/de
12630 disco 39 19 1804M 1589M 12248 R 101. 1.6 4:31.51
/map_store/developers/disco/12_08_06-13_53_1344254009-j3UGmL/mmcc -X
172 -c /map_store/de
Another example, the processes at 100% not being throttled at all when
having more processes waiting with higher priority:
Tasks: 559 total, 37 running, 522 sleeping, 0 stopped, 0 zombie
Cpu(s): 67.8%us, 16.0%sy, 13.0%ni, 1.7%id, 0.6%wa, 0.0%hi, 1.0%si, 0.0%st
Mem: 98998032k total, 97444688k used, 1553344k free, 53772k buffers
Swap: 4198316k total, 704860k used, 3493456k free, 73270776k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14373 disco 39 19 8734m 6.9g 12m R 107 7.3 36:09.72 mmcc
15293 disco 39 19 3174m 1.4g 12m R 101 1.5 19:33.79 mmcc
20341 disco 39 19 2735m 1.1g 12m R 101 1.1 8:40.38 mmcc
18241 disco 39 19 3040m 1.3g 11m R 100 1.4 14:27.91 mmcc
15204 disco 39 19 5418m 3.7g 12m R 99 3.9 20:53.89 mmcc
24901 larroy 20 0 327m 296m 4276 R 88 0.3 0:04.14 cc1plus
24942 larroy 20 0 193m 159m 4008 R 87 0.2 0:01.47 cc1plus
24862 larroy 20 0 417m 388m 7992 R 84 0.4 0:07.02 cc1plus
24959 larroy 20 0 184m 153m 4008 R 80 0.2 0:01.32 cc1plus
24935 larroy 20 0 254m 222m 4024 R 77 0.2 0:02.44 cc1plus
24919 larroy 20 0 336m 301m 4036 R 76 0.3 0:03.61 cc1plus
24972 larroy 20 0 43160 15m 2332 R 76 0.0 0:00.95 cc1plus
24918 larroy 20 0 287m 255m 4024 R 70 0.3 0:02.99 cc1plus
24962 larroy 20 0 44872 17m 2332 R 69 0.0 0:01.23 cc1plus
24976 larroy 20 0 41212 14m 2332 R 66 0.0 0:00.67 cc1plus
24501 larroy 20 0 687m 657m 8044 R 64 0.7 0:22.97 cc1plus
24933 larroy 20 0 211m 177m 4008 R 62 0.2 0:01.79 cc1plus
24899 larroy 20 0 327m 296m 4276 R 57 0.3 0:04.25 cc1plus
This is 3.2.0-26-generic on ubuntu 12.04
Regards.
Pedro.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unfair scheduling with tbb application observed, could it be a kernel issue?
2012-08-06 14:04 unfair scheduling with tbb application observed, could it be a kernel issue? Pedro Larroy
@ 2012-08-06 14:25 ` Mike Galbraith
2012-08-06 14:28 ` Pedro Larroy
0 siblings, 1 reply; 6+ messages in thread
From: Mike Galbraith @ 2012-08-06 14:25 UTC (permalink / raw)
To: Pedro Larroy; +Cc: linux-kernel
On Mon, 2012-08-06 at 16:04 +0200, Pedro Larroy wrote:
> Hi
>
> I think we are observing unfair scheduling of processes that use intel
> TBB thread scheduler, as we have several processes with nice of 19 and
> ioniced idle, and somehow the process with nice 0 should be getting
> more than 1000% cpu
..
> Tasks: 559 total, 37 running, 522 sleeping, 0 stopped, 0 zombie
> Cpu(s): 67.8%us, 16.0%sy, 13.0%ni, 1.7%id, 0.6%wa, 0.0%hi, 1.0%si, 0.0%st
> Mem: 98998032k total, 97444688k used, 1553344k free, 53772k buffers
> Swap: 4198316k total, 704860k used, 3493456k free, 73270776k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 14373 disco 39 19 8734m 6.9g 12m R 107 7.3 36:09.72 mmcc
> 15293 disco 39 19 3174m 1.4g 12m R 101 1.5 19:33.79 mmcc
> 20341 disco 39 19 2735m 1.1g 12m R 101 1.1 8:40.38 mmcc
> 18241 disco 39 19 3040m 1.3g 11m R 100 1.4 14:27.91 mmcc
> 15204 disco 39 19 5418m 3.7g 12m R 99 3.9 20:53.89 mmcc
> 24901 larroy 20 0 327m 296m 4276 R 88 0.3 0:04.14 cc1plus
> 24942 larroy 20 0 193m 159m 4008 R 87 0.2 0:01.47 cc1plus
> 24862 larroy 20 0 417m 388m 7992 R 84 0.4 0:07.02 cc1plus
> 24959 larroy 20 0 184m 153m 4008 R 80 0.2 0:01.32 cc1plus
> 24935 larroy 20 0 254m 222m 4024 R 77 0.2 0:02.44 cc1plus
> 24919 larroy 20 0 336m 301m 4036 R 76 0.3 0:03.61 cc1plus
> 24972 larroy 20 0 43160 15m 2332 R 76 0.0 0:00.95 cc1plus
> 24918 larroy 20 0 287m 255m 4024 R 70 0.3 0:02.99 cc1plus
> 24962 larroy 20 0 44872 17m 2332 R 69 0.0 0:01.23 cc1plus
> 24976 larroy 20 0 41212 14m 2332 R 66 0.0 0:00.67 cc1plus
> 24501 larroy 20 0 687m 657m 8044 R 64 0.7 0:22.97 cc1plus
> 24933 larroy 20 0 211m 177m 4008 R 62 0.2 0:01.79 cc1plus
> 24899 larroy 20 0 327m 296m 4276 R 57 0.3 0:04.25 cc1plus
Are tasks running in per user cgroups or such? If so, you'd need to
adjust group shares.
-Mike
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unfair scheduling with tbb application observed, could it be a kernel issue?
2012-08-06 14:25 ` Mike Galbraith
@ 2012-08-06 14:28 ` Pedro Larroy
2012-08-06 14:38 ` Mike Galbraith
0 siblings, 1 reply; 6+ messages in thread
From: Pedro Larroy @ 2012-08-06 14:28 UTC (permalink / raw)
To: Mike Galbraith; +Cc: linux-kernel
I didn't enable cgroups explicitly, and don't have userspace tools for
that installed.
Thanks.
Pedro.
On Mon, Aug 6, 2012 at 4:25 PM, Mike Galbraith <efault@gmx.de> wrote:
> On Mon, 2012-08-06 at 16:04 +0200, Pedro Larroy wrote:
>> Hi
>>
>> I think we are observing unfair scheduling of processes that use intel
>> TBB thread scheduler, as we have several processes with nice of 19 and
>> ioniced idle, and somehow the process with nice 0 should be getting
>> more than 1000% cpu
> ..
>> Tasks: 559 total, 37 running, 522 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 67.8%us, 16.0%sy, 13.0%ni, 1.7%id, 0.6%wa, 0.0%hi, 1.0%si, 0.0%st
>> Mem: 98998032k total, 97444688k used, 1553344k free, 53772k buffers
>> Swap: 4198316k total, 704860k used, 3493456k free, 73270776k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 14373 disco 39 19 8734m 6.9g 12m R 107 7.3 36:09.72 mmcc
>> 15293 disco 39 19 3174m 1.4g 12m R 101 1.5 19:33.79 mmcc
>> 20341 disco 39 19 2735m 1.1g 12m R 101 1.1 8:40.38 mmcc
>> 18241 disco 39 19 3040m 1.3g 11m R 100 1.4 14:27.91 mmcc
>> 15204 disco 39 19 5418m 3.7g 12m R 99 3.9 20:53.89 mmcc
>> 24901 larroy 20 0 327m 296m 4276 R 88 0.3 0:04.14 cc1plus
>> 24942 larroy 20 0 193m 159m 4008 R 87 0.2 0:01.47 cc1plus
>> 24862 larroy 20 0 417m 388m 7992 R 84 0.4 0:07.02 cc1plus
>> 24959 larroy 20 0 184m 153m 4008 R 80 0.2 0:01.32 cc1plus
>> 24935 larroy 20 0 254m 222m 4024 R 77 0.2 0:02.44 cc1plus
>> 24919 larroy 20 0 336m 301m 4036 R 76 0.3 0:03.61 cc1plus
>> 24972 larroy 20 0 43160 15m 2332 R 76 0.0 0:00.95 cc1plus
>> 24918 larroy 20 0 287m 255m 4024 R 70 0.3 0:02.99 cc1plus
>> 24962 larroy 20 0 44872 17m 2332 R 69 0.0 0:01.23 cc1plus
>> 24976 larroy 20 0 41212 14m 2332 R 66 0.0 0:00.67 cc1plus
>> 24501 larroy 20 0 687m 657m 8044 R 64 0.7 0:22.97 cc1plus
>> 24933 larroy 20 0 211m 177m 4008 R 62 0.2 0:01.79 cc1plus
>> 24899 larroy 20 0 327m 296m 4276 R 57 0.3 0:04.25 cc1plus
>
> Are tasks running in per user cgroups or such? If so, you'd need to
> adjust group shares.
>
> -Mike
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unfair scheduling with tbb application observed, could it be a kernel issue?
2012-08-06 14:28 ` Pedro Larroy
@ 2012-08-06 14:38 ` Mike Galbraith
2012-08-06 14:46 ` Mike Galbraith
0 siblings, 1 reply; 6+ messages in thread
From: Mike Galbraith @ 2012-08-06 14:38 UTC (permalink / raw)
To: Pedro Larroy; +Cc: linux-kernel
On Mon, 2012-08-06 at 16:28 +0200, Pedro Larroy wrote:
> I didn't enable cgroups explicitly, and don't have userspace tools for
> that installed.
You don't userspace tools. cat /proc/<pid>/cgroup will show the group.
-Mike
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unfair scheduling with tbb application observed, could it be a kernel issue?
2012-08-06 14:38 ` Mike Galbraith
@ 2012-08-06 14:46 ` Mike Galbraith
2012-08-06 15:14 ` Mike Galbraith
0 siblings, 1 reply; 6+ messages in thread
From: Mike Galbraith @ 2012-08-06 14:46 UTC (permalink / raw)
To: Pedro Larroy; +Cc: linux-kernel
On Mon, 2012-08-06 at 16:38 +0200, Mike Galbraith wrote:
> On Mon, 2012-08-06 at 16:28 +0200, Pedro Larroy wrote:
> > I didn't enable cgroups explicitly, and don't have userspace tools for
> > that installed.
>
> You don't userspace tools. cat /proc/<pid>/cgroup will show the group.
^need
Did you mean "didn't explicitly enable" as in..
marge:~ # zcat /proc/config.gz|grep CONFIG_CGROUP_SCHED
# CONFIG_CGROUP_SCHED is not set
-Mike
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: unfair scheduling with tbb application observed, could it be a kernel issue?
2012-08-06 14:46 ` Mike Galbraith
@ 2012-08-06 15:14 ` Mike Galbraith
0 siblings, 0 replies; 6+ messages in thread
From: Mike Galbraith @ 2012-08-06 15:14 UTC (permalink / raw)
To: Pedro Larroy; +Cc: linux-kernel
Ok, with the sched_debug info you sent offline, booting with noautogroup
appended to the kernel command line should fix it up for you. You've
got autogroup enabled, which creates automagic per session task groups.
-Mike
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-08-06 15:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-06 14:04 unfair scheduling with tbb application observed, could it be a kernel issue? Pedro Larroy
2012-08-06 14:25 ` Mike Galbraith
2012-08-06 14:28 ` Pedro Larroy
2012-08-06 14:38 ` Mike Galbraith
2012-08-06 14:46 ` Mike Galbraith
2012-08-06 15:14 ` Mike Galbraith
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox