find_busiest_group using lots of CPU

Jens Axboe
2009-09-30T08:18:09Z

Hi,

I stuffed a few more SSDs into my text box. Running a simple workload
that just does streaming reads from 10 processes (throughput is around
2.2GB/sec), find_busiest_group() is using > 10% of the CPU time. This is
a 64 thread box.

The top two profile entries are:

10.86% fio [kernel] [k] find_busiest_group
|
|--99.91%-- thread_return
| io_schedule
| sys_io_getevents
| system_call_fastpath
| 0x7f4b50b61604
| |
| --100.00%-- td_io_getevents
| io_u_queued_complete
| thread_main
| run_threads
| main
| __libc_start_main
--0.09%-- [...]

5.78% fio [kernel] [k] cpumask_next_and
|
|--67.21%-- thread_return
| io_schedule
| sys_io_getevents
| system_call_fastpath
| 0x7f4b50b61604
| |
| --100.00%-- td_io_getevents
| io_u_queued_complete
| thread_main
| run_threads
| main
| __libc_start_main
|
--32.79%-- find_busiest_group
thread_return
io_schedule
sys_io_getevents
system_call_fastpath
0x7f4b50b61604
|
--100.00%-- td_io_getevents
io_u_queued_complete
thread_main
run_threads
main
__libc_start_main

This is with SCHED_DEBUG=y and SCHEDSTATS=y enabled, I just tried with
both disabled but that yields the same result (well actually worse, 22%
spent in there. dunno if that's normal "fluctuation"). GROUP_SCHED is
not set. This seems way excessive!

--
Jens Axboe