* More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
@ 2005-02-23 18:07 Lee Revell
2005-02-23 19:16 ` Hugh Dickins
2005-02-23 19:52 ` Lee Revell
0 siblings, 2 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 18:07 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 407 bytes --]
Ingo,
Did something change recently in the VM that made copy_pte_range and
clear_page_range a lot more expensive? I noticed a reference in the
"Page Table Iterators" thread to excessive overhead introduced by
aggressive page freeing. That sure looks like what is going on in
trace2. trace1 and trace3 look like big fork latencies associated with
copy_pte_range.
This is all with PREEMPT_DESKTOP.
Lee
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace1.txt --]
[-- Type: text/plain; name=trace1.txt; charset=ANSI_X3.4-1968, Size: 3844 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 323 µs, #22/22, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) valgrind.bin 15934 0 9 00000002 00000000 [0032064022903959] 0.000ms (+3534259.157ms): <676c6176> (<646e6972>)
(T1/#2) valgrind.bin 15934 0 9 00000002 00000002 [0032064022904377] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) valgrind.bin 15934 0 9 00000000 00000003 [0032064022904934] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) valgrind-15934 0dn.2 2µs!< (1)
(T1/#5) valgrind.bin 15934 0 2 00000001 00000005 [0032064023088406] 0.307ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (copy_pte_range+0xb7/0x1c0 <c0142ad7>)
(T1/#6) valgrind.bin 15934 0 2 00000001 00000006 [0032064023088847] 0.308ms (+0.000ms): __cond_resched_raw_spinlock+0x8/0x50 <c0111398> (copy_pte_range+0xa7/0x1c0 <c0142ac7>)
(T1/#7) valgrind.bin 15934 0 2 00000000 00000007 [0032064023089269] 0.308ms (+0.001ms): __cond_resched+0x9/0x70 <c0111329> (__cond_resched_raw_spinlock+0x3d/0x50 <c01113cd>)
(T1/#8) valgrind.bin 15934 0 3 00000000 00000008 [0032064023089903] 0.309ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (__cond_resched+0x45/0x70 <c0111365>)
(T1/#9) valgrind.bin 15934 0 3 00000000 00000009 [0032064023090288] 0.310ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#10) valgrind.bin 15934 0 3 00000001 0000000a [0032064023090866] 0.311ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#11) valgrind.bin 15934 0 3 00000002 0000000b [0032064023091857] 0.313ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#12) valgrind.bin 15934 0 3 00000002 0000000c [0032064023092280] 0.313ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#13) valgrind.bin 15934 0 3 00000002 0000000d [0032064023092690] 0.314ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#14) valgrind.bin 15934 0 3 00000002 0000000e [0032064023093035] 0.315ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#15) [ => valgrind.bin ] 0.316ms (+0.001ms)
(T1/#16) <...> 2 0 1 00000002 00000010 [0032064023094748] 0.317ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#17) <...>-2 0d..2 319µs : __schedule+0x2ea/0x630 <c027cc6a> <valgrind-15934> (7d 69):
(T1/#18) <...> 2 0 1 00000002 00000012 [0032064023095766] 0.319ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#19) <...> 2 0 1 00000001 00000013 [0032064023096192] 0.320ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#20) <...>-2 0d..1 320µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#21) <...> 2 0 1 00000001 00000015 [0032064023097511] 0.322ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: trace2.txt --]
[-- Type: text/plain; name=trace2.txt; charset=ANSI_X3.4-1968, Size: 49744 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 361 µs, #289/289, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) evolution 16047 0 9 00000002 00000000 [0032772517080312] 0.000ms (+3483891.566ms): <6c6f7665> (<6f697475>)
(T1/#2) evolution 16047 0 9 00000002 00000002 [0032772517080803] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) evolution 16047 0 9 00000000 00000003 [0032772517081392] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) evolutio-16047 0dn.2 2µs < (1)
(T1/#5) evolution 16047 0 2 00000002 00000005 [0032772517082355] 0.003ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#6) evolution 16047 0 2 00000002 00000006 [0032772517082977] 0.004ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#7) evolution 16047 0 2 00000002 00000007 [0032772517083332] 0.005ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#8) evolution 16047 0 2 00000002 00000008 [0032772517083589] 0.005ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#9) evolution 16047 0 2 00000002 00000009 [0032772517083937] 0.006ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#10) evolution 16047 0 2 00000002 0000000a [0032772517084750] 0.007ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#11) evolution 16047 0 2 00000002 0000000b [0032772517085209] 0.008ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#12) evolution 16047 0 2 00000002 0000000c [0032772517085614] 0.008ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#13) evolution 16047 0 2 00000002 0000000d [0032772517085996] 0.009ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#14) evolution 16047 0 2 00000002 0000000e [0032772517086537] 0.010ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#15) evolution 16047 0 2 00000002 0000000f [0032772517086869] 0.010ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#16) evolution 16047 0 2 00000002 00000010 [0032772517087171] 0.011ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#17) evolution 16047 0 2 00000002 00000011 [0032772517087540] 0.012ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#18) evolution 16047 0 2 00000002 00000012 [0032772517088332] 0.013ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#19) evolution 16047 0 2 00000002 00000013 [0032772517088800] 0.014ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#20) evolution 16047 0 2 00000002 00000014 [0032772517089160] 0.014ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#21) evolution 16047 0 2 00000002 00000015 [0032772517089620] 0.015ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#22) evolution 16047 0 2 00000002 00000016 [0032772517090190] 0.016ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#23) evolution 16047 0 2 00000002 00000017 [0032772517090529] 0.017ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#24) evolution 16047 0 2 00000002 00000018 [0032772517090829] 0.017ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#25) evolution 16047 0 2 00000002 00000019 [0032772517091189] 0.018ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#26) evolution 16047 0 3 00000003 0000001a [0032772517091761] 0.019ms (+0.001ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#27) evolution 16047 0 3 00000004 0000001b [0032772517092489] 0.020ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#28) evolution 16047 0 3 00000004 0000001c [0032772517092884] 0.020ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#29) evolution 16047 0 3 00000004 0000001d [0032772517093678] 0.022ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#30) evolution 16047 0 3 00000004 0000001e [0032772517094096] 0.022ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#31) evolution 16047 0 3 00000004 0000001f [0032772517094674] 0.023ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#32) evolution 16047 0 3 00000004 00000020 [0032772517095137] 0.024ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#33) evolution 16047 0 3 00000004 00000021 [0032772517095623] 0.025ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#34) evolution 16047 0 3 00000004 00000022 [0032772517096307] 0.026ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#35) evolution 16047 0 3 00000004 00000023 [0032772517096766] 0.027ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#36) evolution 16047 0 3 00000004 00000024 [0032772517097495] 0.028ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#37) evolution 16047 0 3 00000004 00000025 [0032772517097968] 0.029ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#38) evolution 16047 0 3 00000004 00000026 [0032772517098688] 0.030ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#39) evolution 16047 0 3 00000004 00000027 [0032772517099138] 0.031ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#40) evolution 16047 0 3 00000004 00000028 [0032772517099961] 0.032ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#41) evolution 16047 0 3 00000004 00000029 [0032772517100452] 0.033ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#42) evolution 16047 0 3 00000004 0000002a [0032772517101122] 0.034ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#43) evolution 16047 0 3 00000004 0000002b [0032772517101568] 0.035ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#44) evolution 16047 0 3 00000004 0000002c [0032772517102283] 0.036ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#45) evolution 16047 0 3 00000004 0000002d [0032772517102729] 0.037ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#46) evolution 16047 0 3 00000004 0000002e [0032772517103453] 0.038ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#47) evolution 16047 0 3 00000004 0000002f [0032772517103899] 0.039ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#48) evolution 16047 0 3 00000004 00000030 [0032772517104601] 0.040ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#49) evolution 16047 0 3 00000004 00000031 [0032772517105060] 0.041ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#50) evolution 16047 0 3 00000004 00000032 [0032772517105618] 0.042ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#51) evolution 16047 0 3 00000004 00000033 [0032772517106081] 0.042ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#52) evolution 16047 0 3 00000004 00000034 [0032772517106581] 0.043ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#53) evolution 16047 0 3 00000004 00000035 [0032772517107359] 0.045ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#54) evolution 16047 0 3 00000004 00000036 [0032772517107818] 0.045ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#55) evolution 16047 0 3 00000004 00000037 [0032772517108498] 0.046ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#56) evolution 16047 0 3 00000004 00000038 [0032772517108948] 0.047ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#57) evolution 16047 0 3 00000004 00000039 [0032772517109609] 0.048ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#58) evolution 16047 0 3 00000004 0000003a [0032772517110055] 0.049ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#59) evolution 16047 0 3 00000004 0000003b [0032772517110779] 0.050ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#60) evolution 16047 0 3 00000004 0000003c [0032772517111229] 0.051ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#61) evolution 16047 0 3 00000003 0000003d [0032772517111877] 0.052ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#62) evolution 16047 0 2 00000002 0000003e [0032772517112404] 0.053ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#63) evolution 16047 0 2 00000002 0000003f [0032772517113047] 0.054ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#64) evolution 16047 0 2 00000002 00000040 [0032772517113493] 0.055ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#65) evolution 16047 0 2 00000002 00000041 [0032772517113952] 0.056ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#66) evolution 16047 0 2 00000002 00000042 [0032772517114564] 0.057ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#67) evolution 16047 0 2 00000002 00000043 [0032772517114937] 0.057ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#68) evolution 16047 0 2 00000002 00000044 [0032772517115266] 0.058ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#69) evolution 16047 0 2 00000002 00000045 [0032772517115612] 0.058ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#70) evolution 16047 0 2 00000002 00000046 [0032772517116422] 0.060ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#71) evolution 16047 0 2 00000002 00000047 [0032772517116886] 0.060ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#72) evolution 16047 0 2 00000002 00000048 [0032772517117322] 0.061ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#73) evolution 16047 0 2 00000002 00000049 [0032772517117813] 0.062ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#74) evolution 16047 0 2 00000002 0000004a [0032772517118416] 0.063ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#75) evolution 16047 0 2 00000002 0000004b [0032772517118785] 0.064ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#76) evolution 16047 0 2 00000002 0000004c [0032772517119113] 0.064ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#77) evolution 16047 0 2 00000002 0000004d [0032772517119469] 0.065ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#78) evolution 16047 0 2 00000002 0000004e [0032772517120247] 0.066ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#79) evolution 16047 0 2 00000002 0000004f [0032772517120715] 0.067ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#80) evolution 16047 0 2 00000002 00000050 [0032772517121156] 0.068ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#81) evolution 16047 0 2 00000002 00000051 [0032772517121606] 0.068ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#82) evolution 16047 0 2 00000002 00000052 [0032772517122214] 0.069ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#83) evolution 16047 0 2 00000002 00000053 [0032772517122578] 0.070ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#84) evolution 16047 0 2 00000002 00000054 [0032772517122911] 0.070ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#85) evolution 16047 0 2 00000002 00000055 [0032772517123271] 0.071ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#86) evolution 16047 0 2 00000002 00000056 [0032772517124081] 0.072ms (+0.037ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#87) evolution 16047 0 2 00000002 00000057 [0032772517146379] 0.110ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#88) evolution 16047 0 2 00000002 00000058 [0032772517146815] 0.110ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#89) evolution 16047 0 2 00000002 00000059 [0032772517147234] 0.111ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#90) evolution 16047 0 2 00000002 0000005a [0032772517147832] 0.112ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#91) evolution 16047 0 2 00000002 0000005b [0032772517148201] 0.113ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#92) evolution 16047 0 2 00000002 0000005c [0032772517148530] 0.113ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#93) evolution 16047 0 2 00000002 0000005d [0032772517148885] 0.114ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#94) evolution 16047 0 2 00000002 0000005e [0032772517149677] 0.115ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#95) evolution 16047 0 2 00000002 0000005f [0032772517150150] 0.116ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#96) evolution 16047 0 2 00000002 00000060 [0032772517150595] 0.117ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#97) evolution 16047 0 2 00000002 00000061 [0032772517151041] 0.117ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#98) evolution 16047 0 2 00000002 00000062 [0032772517151653] 0.118ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#99) evolution 16047 0 2 00000002 00000063 [0032772517152017] 0.119ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#100) evolution 16047 0 2 00000002 00000064 [0032772517152359] 0.120ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#101) evolution 16047 0 2 00000002 00000065 [0032772517152719] 0.120ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#102) evolution 16047 0 2 00000002 00000066 [0032772517153507] 0.121ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#103) evolution 16047 0 2 00000002 00000067 [0032772517153975] 0.122ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#104) evolution 16047 0 2 00000002 00000068 [0032772517154407] 0.123ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#105) evolution 16047 0 2 00000002 00000069 [0032772517154879] 0.124ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#106) evolution 16047 0 2 00000002 0000006a [0032772517155482] 0.125ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#107) evolution 16047 0 2 00000002 0000006b [0032772517155842] 0.125ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#108) evolution 16047 0 2 00000002 0000006c [0032772517156171] 0.126ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#109) evolution 16047 0 2 00000002 0000006d [0032772517156517] 0.127ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#110) evolution 16047 0 2 00000002 0000006e [0032772517157291] 0.128ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#111) evolution 16047 0 2 00000002 0000006f [0032772517158029] 0.129ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#112) evolution 16047 0 2 00000002 00000070 [0032772517158466] 0.130ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#113) evolution 16047 0 2 00000002 00000071 [0032772517158929] 0.131ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#114) evolution 16047 0 2 00000002 00000072 [0032772517159532] 0.132ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#115) evolution 16047 0 2 00000002 00000073 [0032772517159910] 0.132ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#116) evolution 16047 0 2 00000002 00000074 [0032772517160243] 0.133ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#117) evolution 16047 0 2 00000002 00000075 [0032772517160590] 0.133ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#118) evolution 16047 0 2 00000002 00000076 [0032772517161377] 0.135ms (+0.075ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#119) evolution 16047 0 2 00000002 00000077 [0032772517206701] 0.210ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#120) evolution 16047 0 2 00000002 00000078 [0032772517207133] 0.211ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#121) evolution 16047 0 2 00000002 00000079 [0032772517207574] 0.212ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#122) evolution 16047 0 2 00000002 0000007a [0032772517208173] 0.213ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#123) evolution 16047 0 2 00000002 0000007b [0032772517208537] 0.213ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#124) evolution 16047 0 2 00000002 0000007c [0032772517208866] 0.214ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#125) evolution 16047 0 2 00000002 0000007d [0032772517209221] 0.214ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#126) evolution 16047 0 2 00000002 0000007e [0032772517209995] 0.216ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#127) evolution 16047 0 2 00000002 0000007f [0032772517210567] 0.217ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#128) evolution 16047 0 2 00000002 00000080 [0032772517211012] 0.217ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#129) evolution 16047 0 2 00000002 00000081 [0032772517211462] 0.218ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#130) evolution 16047 0 2 00000002 00000082 [0032772517212065] 0.219ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#131) evolution 16047 0 2 00000002 00000083 [0032772517212430] 0.220ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#132) evolution 16047 0 2 00000002 00000084 [0032772517212763] 0.220ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#133) evolution 16047 0 2 00000002 00000085 [0032772517213114] 0.221ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#134) evolution 16047 0 2 00000002 00000086 [0032772517213892] 0.222ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#135) evolution 16047 0 2 00000002 00000087 [0032772517214365] 0.223ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#136) evolution 16047 0 2 00000002 00000088 [0032772517214833] 0.224ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#137) evolution 16047 0 2 00000002 00000089 [0032772517215328] 0.225ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#138) evolution 16047 0 2 00000002 0000008a [0032772517215944] 0.226ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#139) evolution 16047 0 2 00000002 0000008b [0032772517216304] 0.226ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#140) evolution 16047 0 2 00000002 0000008c [0032772517216633] 0.227ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#141) evolution 16047 0 2 00000002 0000008d [0032772517216988] 0.227ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#142) evolution 16047 0 2 00000002 0000008e [0032772517217767] 0.229ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#143) evolution 16047 0 2 00000002 0000008f [0032772517218428] 0.230ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#144) evolution 16047 0 2 00000002 00000090 [0032772517218860] 0.230ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#145) evolution 16047 0 2 00000002 00000091 [0032772517219279] 0.231ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#146) evolution 16047 0 2 00000002 00000092 [0032772517219882] 0.232ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#147) evolution 16047 0 2 00000002 00000093 [0032772517220246] 0.233ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#148) evolution 16047 0 2 00000002 00000094 [0032772517220606] 0.233ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#149) evolution 16047 0 2 00000002 00000095 [0032772517221027] 0.234ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#150) evolution 16047 0 2 00000002 00000096 [0032772517221808] 0.235ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#151) evolution 16047 0 2 00000002 00000097 [0032772517222276] 0.236ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#152) evolution 16047 0 2 00000002 00000098 [0032772517222721] 0.237ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#153) evolution 16047 0 2 00000002 00000099 [0032772517223176] 0.238ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#154) evolution 16047 0 2 00000002 0000009a [0032772517223779] 0.239ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#155) evolution 16047 0 2 00000002 0000009b [0032772517224143] 0.239ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#156) evolution 16047 0 2 00000002 0000009c [0032772517224476] 0.240ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#157) evolution 16047 0 2 00000002 0000009d [0032772517224823] 0.240ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#158) evolution 16047 0 2 00000002 0000009e [0032772517225606] 0.242ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#159) evolution 16047 0 2 00000002 0000009f [0032772517226227] 0.243ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#160) evolution 16047 0 2 00000002 000000a0 [0032772517226668] 0.243ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#161) evolution 16047 0 2 00000002 000000a1 [0032772517227113] 0.244ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#162) evolution 16047 0 2 00000002 000000a2 [0032772517227712] 0.245ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#163) evolution 16047 0 2 00000002 000000a3 [0032772517228072] 0.246ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#164) evolution 16047 0 2 00000002 000000a4 [0032772517228400] 0.246ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#165) evolution 16047 0 2 00000002 000000a5 [0032772517228751] 0.247ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#166) evolution 16047 0 2 00000002 000000a6 [0032772517229539] 0.248ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#167) evolution 16047 0 2 00000002 000000a7 [0032772517230209] 0.249ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#168) evolution 16047 0 2 00000002 000000a8 [0032772517230646] 0.250ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#169) evolution 16047 0 2 00000002 000000a9 [0032772517231064] 0.251ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#170) evolution 16047 0 2 00000002 000000aa [0032772517231667] 0.252ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#171) evolution 16047 0 2 00000002 000000ab [0032772517232041] 0.252ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#172) evolution 16047 0 2 00000002 000000ac [0032772517232374] 0.253ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#173) evolution 16047 0 2 00000002 000000ad [0032772517232734] 0.254ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#174) evolution 16047 0 2 00000002 000000ae [0032772517233521] 0.255ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#175) evolution 16047 0 2 00000002 000000af [0032772517234102] 0.256ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#176) evolution 16047 0 2 00000002 000000b0 [0032772517234534] 0.257ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#177) evolution 16047 0 2 00000002 000000b1 [0032772517235087] 0.257ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#178) evolution 16047 0 2 00000002 000000b2 [0032772517235695] 0.258ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#179) evolution 16047 0 2 00000002 000000b3 [0032772517236064] 0.259ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#180) evolution 16047 0 2 00000002 000000b4 [0032772517236392] 0.260ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#181) evolution 16047 0 2 00000002 000000b5 [0032772517236739] 0.260ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#182) evolution 16047 0 2 00000002 000000b6 [0032772517237517] 0.262ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#183) evolution 16047 0 2 00000002 000000b7 [0032772517238093] 0.262ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#184) evolution 16047 0 2 00000002 000000b8 [0032772517238543] 0.263ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#185) evolution 16047 0 2 00000002 000000b9 [0032772517238998] 0.264ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#186) evolution 16047 0 2 00000002 000000ba [0032772517239614] 0.265ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#187) evolution 16047 0 2 00000002 000000bb [0032772517239988] 0.266ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#188) evolution 16047 0 2 00000002 000000bc [0032772517240330] 0.266ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#189) evolution 16047 0 2 00000002 000000bd [0032772517240690] 0.267ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#190) evolution 16047 0 3 00000003 000000be [0032772517241338] 0.268ms (+0.001ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#191) evolution 16047 0 3 00000004 000000bf [0032772517242121] 0.269ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#192) evolution 16047 0 3 00000004 000000c0 [0032772517242566] 0.270ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#193) evolution 16047 0 3 00000004 000000c1 [0032772517243291] 0.271ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#194) evolution 16047 0 3 00000004 000000c2 [0032772517243736] 0.272ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#195) evolution 16047 0 3 00000004 000000c3 [0032772517244461] 0.273ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#196) evolution 16047 0 3 00000004 000000c4 [0032772517244915] 0.274ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#197) evolution 16047 0 3 00000004 000000c5 [0032772517245653] 0.275ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#198) evolution 16047 0 3 00000004 000000c6 [0032772517246099] 0.276ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#199) evolution 16047 0 3 00000004 000000c7 [0032772517246738] 0.277ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#200) evolution 16047 0 3 00000004 000000c8 [0032772517247188] 0.278ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#201) evolution 16047 0 3 00000004 000000c9 [0032772517247939] 0.279ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#202) evolution 16047 0 3 00000004 000000ca [0032772517248398] 0.280ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#203) evolution 16047 0 3 00000004 000000cb [0032772517249163] 0.281ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#204) evolution 16047 0 3 00000004 000000cc [0032772517249622] 0.282ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#205) evolution 16047 0 3 00000004 000000cd [0032772517250338] 0.283ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#206) evolution 16047 0 3 00000004 000000ce [0032772517250788] 0.284ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#207) evolution 16047 0 3 00000004 000000cf [0032772517251499] 0.285ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#208) evolution 16047 0 3 00000004 000000d0 [0032772517251944] 0.286ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#209) evolution 16047 0 3 00000004 000000d1 [0032772517252574] 0.287ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#210) evolution 16047 0 3 00000004 000000d2 [0032772517253020] 0.287ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#211) evolution 16047 0 3 00000004 000000d3 [0032772517253735] 0.289ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#212) evolution 16047 0 3 00000004 000000d4 [0032772517254190] 0.289ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#213) evolution 16047 0 3 00000004 000000d5 [0032772517254923] 0.291ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#214) evolution 16047 0 3 00000004 000000d6 [0032772517255373] 0.291ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#215) evolution 16047 0 3 00000004 000000d7 [0032772517256089] 0.292ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#216) evolution 16047 0 3 00000004 000000d8 [0032772517256539] 0.293ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#217) evolution 16047 0 3 00000004 000000d9 [0032772517257218] 0.294ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#218) evolution 16047 0 3 00000004 000000da [0032772517257700] 0.295ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#219) evolution 16047 0 3 00000004 000000db [0032772517258402] 0.296ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#220) evolution 16047 0 3 00000004 000000dc [0032772517258852] 0.297ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#221) evolution 16047 0 3 00000004 000000dd [0032772517259648] 0.298ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#222) evolution 16047 0 3 00000004 000000de [0032772517260098] 0.299ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#223) evolution 16047 0 3 00000003 000000df [0032772517260737] 0.300ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#224) evolution 16047 0 2 00000002 000000e0 [0032772517261268] 0.301ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#225) evolution 16047 0 2 00000002 000000e1 [0032772517261732] 0.302ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#226) evolution 16047 0 2 00000002 000000e2 [0032772517262177] 0.303ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#227) evolution 16047 0 2 00000002 000000e3 [0032772517262627] 0.303ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#228) evolution 16047 0 2 00000002 000000e4 [0032772517263226] 0.304ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#229) evolution 16047 0 2 00000002 000000e5 [0032772517263595] 0.305ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#230) evolution 16047 0 2 00000002 000000e6 [0032772517263977] 0.306ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#231) evolution 16047 0 2 00000002 000000e7 [0032772517264328] 0.306ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#232) evolution 16047 0 2 00000002 000000e8 [0032772517265129] 0.308ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#233) evolution 16047 0 2 00000002 000000e9 [0032772517265606] 0.308ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#234) evolution 16047 0 2 00000002 000000ea [0032772517266056] 0.309ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#235) evolution 16047 0 2 00000002 000000eb [0032772517266547] 0.310ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#236) evolution 16047 0 2 00000002 000000ec [0032772517267150] 0.311ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#237) evolution 16047 0 2 00000002 000000ed [0032772517267514] 0.312ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#238) evolution 16047 0 2 00000002 000000ee [0032772517267852] 0.312ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#239) evolution 16047 0 2 00000002 000000ef [0032772517268207] 0.313ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#240) evolution 16047 0 2 00000002 000000f0 [0032772517268995] 0.314ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#241) evolution 16047 0 2 00000002 000000f1 [0032772517269458] 0.315ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#242) evolution 16047 0 2 00000002 000000f2 [0032772517269890] 0.315ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#243) evolution 16047 0 2 00000002 000000f3 [0032772517270331] 0.316ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#244) evolution 16047 0 2 00000002 000000f4 [0032772517270930] 0.317ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#245) evolution 16047 0 2 00000002 000000f5 [0032772517271290] 0.318ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#246) evolution 16047 0 2 00000002 000000f6 [0032772517271618] 0.318ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#247) evolution 16047 0 2 00000002 000000f7 [0032772517271965] 0.319ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#248) evolution 16047 0 2 00000002 000000f8 [0032772517272739] 0.320ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#249) evolution 16047 0 2 00000002 000000f9 [0032772517273216] 0.321ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#250) evolution 16047 0 2 00000002 000000fa [0032772517273670] 0.322ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#251) evolution 16047 0 2 00000002 000000fb [0032772517274111] 0.322ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#252) evolution 16047 0 2 00000002 000000fc [0032772517274723] 0.324ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#253) evolution 16047 0 2 00000002 000000fd [0032772517275097] 0.324ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#254) evolution 16047 0 2 00000002 000000fe [0032772517275439] 0.325ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#255) evolution 16047 0 2 00000002 000000ff [0032772517275790] 0.325ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#256) evolution 16047 0 2 00000002 00000100 [0032772517276568] 0.327ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#257) evolution 16047 0 2 00000002 00000101 [0032772517277041] 0.327ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#258) evolution 16047 0 2 00000002 00000102 [0032772517277486] 0.328ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#259) evolution 16047 0 2 00000002 00000103 [0032772517277923] 0.329ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#260) evolution 16047 0 2 00000002 00000104 [0032772517278530] 0.330ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#261) evolution 16047 0 2 00000002 00000105 [0032772517278899] 0.330ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#262) evolution 16047 0 2 00000002 00000106 [0032772517279327] 0.331ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#263) evolution 16047 0 2 00000002 00000107 [0032772517279673] 0.332ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#264) evolution 16047 0 2 00000002 00000108 [0032772517280456] 0.333ms (+0.005ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#265) evolution 16047 0 2 00000002 00000109 [0032772517283989] 0.339ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#266) evolution 16047 0 2 00000002 0000010a [0032772517284425] 0.340ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#267) evolution 16047 0 2 00000002 0000010b [0032772517284871] 0.340ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#268) evolution 16047 0 2 00000002 0000010c [0032772517285474] 0.341ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#269) evolution 16047 0 2 00000002 0000010d [0032772517285838] 0.342ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#270) evolution 16047 0 2 00000002 0000010e [0032772517286171] 0.343ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#271) evolution 16047 0 2 00000002 0000010f [0032772517286531] 0.343ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#272) evolution 16047 0 2 00000002 00000110 [0032772517287314] 0.345ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#273) evolution 16047 0 2 00000001 00000111 [0032772517288385] 0.346ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (exit_mmap+0x164/0x1b0 <c01480a4>)
(T1/#274) evolution 16047 0 2 00000000 00000112 [0032772517288876] 0.347ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (exit_mmap+0x15d/0x1b0 <c014809d>)
(T1/#275) evolution 16047 0 3 00000000 00000113 [0032772517289389] 0.348ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (preempt_schedule+0x4f/0x70 <c027d10f>)
(T1/#276) evolution 16047 0 3 00000000 00000114 [0032772517289753] 0.349ms (+0.001ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#277) evolution 16047 0 3 00000001 00000115 [0032772517290390] 0.350ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#278) evolution 16047 0 3 00000002 00000116 [0032772517291265] 0.351ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#279) evolution 16047 0 3 00000002 00000117 [0032772517291697] 0.352ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#280) evolution 16047 0 3 00000002 00000118 [0032772517292107] 0.352ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#281) evolution 16047 0 3 00000002 00000119 [0032772517292503] 0.353ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#282) [ => evolution ] 0.354ms (+0.001ms)
(T1/#283) <...> 2 0 1 00000002 0000011b [0032772517293976] 0.356ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#284) <...>-2 0d..2 357µs : __schedule+0x2ea/0x630 <c027cc6a> <evolutio-16047> (77 69):
(T1/#285) <...> 2 0 1 00000002 0000011d [0032772517294976] 0.357ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#286) <...> 2 0 1 00000001 0000011e [0032772517295421] 0.358ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#287) <...>-2 0d..1 359µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#288) <...> 2 0 1 00000001 00000120 [0032772517296501] 0.360ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: trace3.txt --]
[-- Type: text/plain; name=trace3.txt; charset=ANSI_X3.4-1968, Size: 3840 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 327 µs, #22/22, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) dpkg 16229 0 9 00000002 00000000 [0034184390920878] 0.000ms (+3535970.095ms): <676b7064> (<00746500>)
(T1/#2) dpkg 16229 0 9 00000002 00000002 [0034184390921351] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) dpkg 16229 0 9 00000000 00000003 [0034184390921935] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) dpkg-16229 0dn.2 2µs!< (1)
(T1/#5) dpkg 16229 0 2 00000001 00000005 [0034184391106965] 0.310ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (copy_pte_range+0xb7/0x1c0 <c0142ad7>)
(T1/#6) dpkg 16229 0 2 00000001 00000006 [0034184391107405] 0.310ms (+0.000ms): __cond_resched_raw_spinlock+0x8/0x50 <c0111398> (copy_pte_range+0xa7/0x1c0 <c0142ac7>)
(T1/#7) dpkg 16229 0 2 00000000 00000007 [0034184391107918] 0.311ms (+0.001ms): __cond_resched+0x9/0x70 <c0111329> (__cond_resched_raw_spinlock+0x3d/0x50 <c01113cd>)
(T1/#8) dpkg 16229 0 3 00000000 00000008 [0034184391108602] 0.312ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (__cond_resched+0x45/0x70 <c0111365>)
(T1/#9) dpkg 16229 0 3 00000000 00000009 [0034184391109088] 0.313ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#10) dpkg 16229 0 3 00000001 0000000a [0034184391109573] 0.314ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#11) dpkg 16229 0 3 00000002 0000000b [0034184391110546] 0.316ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#12) dpkg 16229 0 3 00000002 0000000c [0034184391110969] 0.316ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#13) dpkg 16229 0 3 00000002 0000000d [0034184391111383] 0.317ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#14) dpkg 16229 0 3 00000002 0000000e [0034184391111775] 0.318ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#15) [ => dpkg ] 0.319ms (+0.001ms)
(T1/#16) <...> 2 0 1 00000002 00000010 [0034184391113480] 0.321ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#17) <...>-2 0d..2 322µs : __schedule+0x2ea/0x630 <c027cc6a> <dpkg-16229> (74 69):
(T1/#18) <...> 2 0 1 00000002 00000012 [0034184391114743] 0.323ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#19) <...> 2 0 1 00000001 00000013 [0034184391115178] 0.323ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#20) <...>-2 0d..1 324µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#21) <...> 2 0 1 00000001 00000015 [0034184391116385] 0.325ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
@ 2005-02-23 19:16 ` Hugh Dickins
2005-02-23 19:36 ` Lee Revell
2005-02-23 19:52 ` Lee Revell
1 sibling, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 19:16 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 23 Feb 2005, Lee Revell wrote:
>
> Did something change recently in the VM that made copy_pte_range and
> clear_page_range a lot more expensive? I noticed a reference in the
> "Page Table Iterators" thread to excessive overhead introduced by
> aggressive page freeing. That sure looks like what is going on in
> trace2. trace1 and trace3 look like big fork latencies associated with
> copy_pte_range.
I'm just about to test this patch below: please give it a try: thanks...
Ingo's patch to reduce scheduling latencies, by checking for lockbreak
in copy_page_range, was in the -VP and -mm patchsets some months ago;
but got preempted by the 4level rework, and not reinstated since.
Restore it now in copy_pte_range - which mercifully makes it easier.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
--- 2.6.11-rc4-bk9/mm/memory.c 2005-02-21 11:32:19.000000000 +0000
+++ linux/mm/memory.c 2005-02-23 18:35:28.000000000 +0000
@@ -328,6 +328,7 @@ static int copy_pte_range(struct mm_stru
pte_t *s, *d;
unsigned long vm_flags = vma->vm_flags;
+again:
d = dst_pte = pte_alloc_map(dst_mm, dst_pmd, addr);
if (!dst_pte)
return -ENOMEM;
@@ -338,11 +339,22 @@ static int copy_pte_range(struct mm_stru
if (pte_none(*s))
continue;
copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+ /*
+ * We are holding two locks at this point - either of them
+ * could generate latencies in another task on another CPU.
+ */
+ if (need_resched() ||
+ need_lockbreak(&src_mm->page_table_lock) ||
+ need_lockbreak(&dst_mm->page_table_lock))
+ break;
}
pte_unmap_nested(src_pte);
pte_unmap(dst_pte);
spin_unlock(&src_mm->page_table_lock);
+
cond_resched_lock(&dst_mm->page_table_lock);
+ if (addr < end)
+ goto again;
return 0;
}
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 19:16 ` Hugh Dickins
@ 2005-02-23 19:36 ` Lee Revell
2005-02-23 20:06 ` Hugh Dickins
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 19:36 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]
On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> >
> > Did something change recently in the VM that made copy_pte_range and
> > clear_page_range a lot more expensive? I noticed a reference in the
> > "Page Table Iterators" thread to excessive overhead introduced by
> > aggressive page freeing. That sure looks like what is going on in
> > trace2. trace1 and trace3 look like big fork latencies associated with
> > copy_pte_range.
>
> I'm just about to test this patch below: please give it a try: thanks...
>
> Ingo's patch to reduce scheduling latencies, by checking for lockbreak
> in copy_page_range, was in the -VP and -mm patchsets some months ago;
> but got preempted by the 4level rework, and not reinstated since.
> Restore it now in copy_pte_range - which mercifully makes it easier.
Aha, that explains why all the latency regressions involve the VM
subsystem.
Thanks, your patch fixes the copy_pte_range latency. Now zap_pte_range,
which Ingo also fixed a few months ago, is the worst offender. Can this
fix be easily ported too?
Lee
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace5.txt --]
[-- Type: text/plain; name=trace5.txt; charset=ANSI_X3.4-1968, Size: 12790 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 197 µs, #74/74, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) dmesg 3249 0 9 00000002 00000000 [0000163455166655] 0.000ms (+886841.090ms): <73656d64> (<61700067>)
(T1/#2) dmesg 3249 0 9 00000002 00000002 [0000163455167105] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) dmesg 3249 0 9 00000000 00000003 [0000163455167685] 0.001ms (+0.001ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) dmesg-3249 0dn.2 2µs < (1)
(T1/#5) dmesg 3249 0 2 00000002 00000005 [0000163455168677] 0.003ms (+0.000ms): page_remove_rmap+0x8/0x40 <c0149f78> (zap_pte_range+0x13a/0x250 <c0142f4a>)
(T1/#6) dmesg 3249 0 2 00000002 00000006 [0000163455168975] 0.003ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (page_remove_rmap+0x2c/0x40 <c0149f9c>)
(T1/#7) dmesg 3249 0 2 00000002 00000007 [0000163455169390] 0.004ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#8) dmesg 3249 0 2 00000002 00000008 [0000163455169687] 0.005ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#9) dmesg 3249 0 2 00000002 00000009 [0000163455170314] 0.006ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#10) dmesg 3249 0 2 00000002 0000000a [0000163455170636] 0.006ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#11) dmesg 3249 0 2 00000002 0000000b [0000163455170944] 0.007ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#12) dmesg 3249 0 2 00000002 0000000c [0000163455171314] 0.007ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#13) dmesg 3249 0 2 00000002 0000000d [0000163455172090] 0.009ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#14) dmesg 3249 0 2 00000002 0000000e [0000163455172432] 0.009ms (+0.000ms): set_page_dirty+0x8/0x60 <c013b428> (zap_pte_range+0x168/0x250 <c0142f78>)
(T1/#15) dmesg 3249 0 2 00000002 0000000f [0000163455172774] 0.010ms (+0.000ms): page_remove_rmap+0x8/0x40 <c0149f78> (zap_pte_range+0x13a/0x250 <c0142f4a>)
(T1/#16) dmesg 3249 0 2 00000002 00000010 [0000163455173040] 0.010ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (page_remove_rmap+0x2c/0x40 <c0149f9c>)
(T1/#17) dmesg 3249 0 2 00000002 00000011 [0000163455173421] 0.011ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#18) dmesg 3249 0 2 00000002 00000012 [0000163455173704] 0.011ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#19) dmesg 3249 0 2 00000002 00000013 [0000163455174397] 0.012ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#20) dmesg 3249 0 2 00000002 00000014 [0000163455174687] 0.013ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#21) dmesg 3249 0 2 00000002 00000015 [0000163455175101] 0.014ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#22) dmesg 3249 0 2 00000002 00000016 [0000163455175436] 0.014ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#23) dmesg 3249 0 2 00000002 00000017 [0000163455176191] 0.015ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#24) dmesg 3249 0 2 00000002 00000018 [0000163455176892] 0.017ms (+0.007ms): clear_page_range+0xe/0x1d0 <c01426fe> (exit_mmap+0xa3/0x1b0 <c0148013>)
(T1/#25) dmesg 3249 0 2 00000002 00000019 [0000163455181331] 0.024ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#26) dmesg 3249 0 2 00000002 0000001a [0000163455181781] 0.025ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#27) dmesg 3249 0 2 00000002 0000001b [0000163455182166] 0.025ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#28) dmesg 3249 0 2 00000002 0000001c [0000163455182746] 0.026ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#29) dmesg 3249 0 2 00000002 0000001d [0000163455183036] 0.027ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#30) dmesg 3249 0 2 00000002 0000001e [0000163455183293] 0.027ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#31) dmesg 3249 0 2 00000002 0000001f [0000163455183557] 0.028ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#32) dmesg 3249 0 2 00000002 00000020 [0000163455184330] 0.029ms (+0.042ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#33) dmesg 3249 0 2 00000002 00000021 [0000163455210040] 0.072ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#34) dmesg 3249 0 2 00000002 00000022 [0000163455210487] 0.072ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#35) dmesg 3249 0 2 00000002 00000023 [0000163455210867] 0.073ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#36) dmesg 3249 0 2 00000002 00000024 [0000163455211481] 0.074ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#37) dmesg 3249 0 2 00000002 00000025 [0000163455211846] 0.075ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#38) dmesg 3249 0 2 00000002 00000026 [0000163455212103] 0.075ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#39) dmesg 3249 0 2 00000002 00000027 [0000163455212367] 0.076ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#40) dmesg 3249 0 2 00000002 00000028 [0000163455213164] 0.077ms (+0.088ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#41) dmesg 3249 0 2 00000002 00000029 [0000163455266176] 0.165ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#42) dmesg 3249 0 2 00000002 0000002a [0000163455266536] 0.166ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#43) dmesg 3249 0 2 00000002 0000002b [0000163455267095] 0.167ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#44) dmesg 3249 0 2 00000002 0000002c [0000163455267667] 0.168ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#45) dmesg 3249 0 2 00000002 0000002d [0000163455268033] 0.168ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#46) dmesg 3249 0 2 00000002 0000002e [0000163455268375] 0.169ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#47) dmesg 3249 0 2 00000002 0000002f [0000163455268717] 0.169ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#48) dmesg 3249 0 2 00000002 00000030 [0000163455269434] 0.171ms (+0.006ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#49) dmesg 3249 0 2 00000002 00000031 [0000163455273343] 0.177ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#50) dmesg 3249 0 2 00000002 00000032 [0000163455273779] 0.178ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#51) dmesg 3249 0 2 00000002 00000033 [0000163455274128] 0.178ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#52) dmesg 3249 0 2 00000002 00000034 [0000163455274724] 0.179ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#53) dmesg 3249 0 2 00000002 00000035 [0000163455275084] 0.180ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#54) dmesg 3249 0 2 00000002 00000036 [0000163455275413] 0.180ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#55) dmesg 3249 0 2 00000002 00000037 [0000163455275677] 0.181ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#56) dmesg 3249 0 2 00000002 00000038 [0000163455276470] 0.182ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#57) dmesg 3249 0 2 00000002 00000039 [0000163455277321] 0.184ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x190/0x1b0 <c0148100>)
(T1/#58) dmesg 3249 0 2 00000001 0000003a [0000163455277715] 0.184ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x164/0x1b0 <c01480d4>)
(T1/#59) dmesg 3249 0 2 00000000 0000003b [0000163455278192] 0.185ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x15d/0x1b0 <c01480cd>)
(T1/#60) dmesg 3249 0 3 00000000 0000003c [0000163455278638] 0.186ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#61) dmesg 3249 0 3 00000000 0000003d [0000163455278977] 0.186ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#62) dmesg 3249 0 3 00000001 0000003e [0000163455279418] 0.187ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#63) dmesg 3249 0 3 00000002 0000003f [0000163455280259] 0.189ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#64) dmesg 3249 0 3 00000002 00000040 [0000163455280596] 0.189ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#65) dmesg 3249 0 3 00000002 00000041 [0000163455280988] 0.190ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#66) dmesg 3249 0 3 00000002 00000042 [0000163455281297] 0.190ms (+0.000ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#67) [ => dmesg ] 0.191ms (+0.000ms)
(T1/#68) <...> 2 0 1 00000002 00000044 [0000163455282330] 0.192ms (+0.000ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#69) <...>-2 0d..2 193µs : __schedule+0x2ea/0x630 <c027cc9a> <dmesg-3249> (75 69):
(T1/#70) <...> 2 0 1 00000002 00000046 [0000163455283124] 0.193ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#71) <...> 2 0 1 00000001 00000047 [0000163455283506] 0.194ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#72) <...>-2 0d..1 194µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#73) <...> 2 0 1 00000001 00000049 [0000163455284627] 0.196ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
2005-02-23 19:16 ` Hugh Dickins
@ 2005-02-23 19:52 ` Lee Revell
1 sibling, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 19:52 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel, Andrew Morton
[-- Attachment #1: Type: text/plain, Size: 163 bytes --]
On Wed, 2005-02-23 at 13:07 -0500, Lee Revell wrote:
> This is all with PREEMPT_DESKTOP.
>
Here is another new one, this time in the ext3 reservation code.
Lee
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace6.txt --]
[-- Type: text/plain; name=trace6.txt; charset=ANSI_X3.4-1968, Size: 63651 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 296 µs, #339/339, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) dbench 3399 0 9 00000002 00000000 [0000784436927163] 0.000ms (+876876.851ms): <6e656264> (<61006863>)
(T1/#2) dbench 3399 0 9 00000002 00000002 [0000784436927557] 0.000ms (+0.001ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) dbench 3399 0 9 00000000 00000003 [0000784436928168] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) dbench-3399 0dn.2 2µs < (1)
(T1/#5) dbench 3399 0 2 00000002 00000005 [0000784436929084] 0.003ms (+0.003ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#6) dbench 3399 0 2 00000002 00000006 [0000784436931285] 0.006ms (+0.001ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#7) dbench 3399 0 2 00000002 00000007 [0000784436931964] 0.007ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#8) dbench 3399 0 2 00000002 00000008 [0000784436932467] 0.008ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#9) dbench 3399 0 2 00000002 00000009 [0000784436932849] 0.009ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#10) dbench 3399 0 2 00000002 0000000a [0000784436933339] 0.010ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#11) dbench 3399 0 2 00000002 0000000b [0000784436933798] 0.011ms (+0.005ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#12) dbench 3399 0 2 00000002 0000000c [0000784436937044] 0.016ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#13) dbench 3399 0 2 00000002 0000000d [0000784436937638] 0.017ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#14) dbench 3399 0 2 00000002 0000000e [0000784436938093] 0.018ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#15) dbench 3399 0 2 00000002 0000000f [0000784436938460] 0.018ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#16) dbench 3399 0 2 00000002 00000010 [0000784436938946] 0.019ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#17) dbench 3399 0 2 00000002 00000011 [0000784436939518] 0.020ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#18) dbench 3399 0 2 00000002 00000012 [0000784436939929] 0.021ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#19) dbench 3399 0 2 00000002 00000013 [0000784436940424] 0.022ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#20) dbench 3399 0 2 00000002 00000014 [0000784436940865] 0.022ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#21) dbench 3399 0 2 00000002 00000015 [0000784436941335] 0.023ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#22) dbench 3399 0 2 00000002 00000016 [0000784436941819] 0.024ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#23) dbench 3399 0 2 00000002 00000017 [0000784436942273] 0.025ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#24) dbench 3399 0 2 00000002 00000018 [0000784436942674] 0.025ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#25) dbench 3399 0 2 00000002 00000019 [0000784436943164] 0.026ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#26) dbench 3399 0 2 00000002 0000001a [0000784436943610] 0.027ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#27) dbench 3399 0 2 00000002 0000001b [0000784436944010] 0.028ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#28) dbench 3399 0 2 00000002 0000001c [0000784436944501] 0.028ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#29) dbench 3399 0 2 00000002 0000001d [0000784436945572] 0.030ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#30) dbench 3399 0 2 00000002 0000001e [0000784436946139] 0.031ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#31) dbench 3399 0 2 00000002 0000001f [0000784436946629] 0.032ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#32) dbench 3399 0 2 00000002 00000020 [0000784436947084] 0.033ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#33) dbench 3399 0 2 00000002 00000021 [0000784436947556] 0.033ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#34) dbench 3399 0 2 00000002 00000022 [0000784436948042] 0.034ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#35) dbench 3399 0 2 00000002 00000023 [0000784436948501] 0.035ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#36) dbench 3399 0 2 00000002 00000024 [0000784436948897] 0.036ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#37) dbench 3399 0 2 00000002 00000025 [0000784436949383] 0.036ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#38) dbench 3399 0 2 00000002 00000026 [0000784436949824] 0.037ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#39) dbench 3399 0 2 00000002 00000027 [0000784436950229] 0.038ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#40) dbench 3399 0 2 00000002 00000028 [0000784436950715] 0.039ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#41) dbench 3399 0 2 00000002 00000029 [0000784436951593] 0.040ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#42) dbench 3399 0 2 00000002 0000002a [0000784436952106] 0.041ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#43) dbench 3399 0 2 00000002 0000002b [0000784436952596] 0.042ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#44) dbench 3399 0 2 00000002 0000002c [0000784436953114] 0.043ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#45) dbench 3399 0 2 00000002 0000002d [0000784436953528] 0.043ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#46) dbench 3399 0 2 00000002 0000002e [0000784436954018] 0.044ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#47) dbench 3399 0 2 00000002 0000002f [0000784436954540] 0.045ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#48) dbench 3399 0 2 00000002 00000030 [0000784436955197] 0.046ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#49) dbench 3399 0 2 00000002 00000031 [0000784436955755] 0.047ms (+0.004ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#50) dbench 3399 0 2 00000002 00000032 [0000784436958356] 0.051ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#51) dbench 3399 0 2 00000002 00000033 [0000784436958860] 0.052ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#52) dbench 3399 0 2 00000002 00000034 [0000784436959346] 0.053ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#53) dbench 3399 0 2 00000002 00000035 [0000784436959801] 0.054ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#54) dbench 3399 0 2 00000002 00000036 [0000784436960201] 0.054ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#55) dbench 3399 0 2 00000002 00000037 [0000784436960705] 0.055ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#56) dbench 3399 0 2 00000002 00000038 [0000784436961223] 0.056ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#57) dbench 3399 0 2 00000002 00000039 [0000784436961637] 0.057ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#58) dbench 3399 0 2 00000002 0000003a [0000784436962127] 0.058ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#59) dbench 3399 0 2 00000002 0000003b [0000784436962717] 0.059ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#60) dbench 3399 0 2 00000002 0000003c [0000784436963131] 0.059ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#61) dbench 3399 0 2 00000002 0000003d [0000784436963621] 0.060ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#62) dbench 3399 0 2 00000002 0000003e [0000784436964062] 0.061ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#63) dbench 3399 0 2 00000002 0000003f [0000784436964539] 0.062ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#64) dbench 3399 0 2 00000002 00000040 [0000784436965025] 0.062ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#65) dbench 3399 0 2 00000002 00000041 [0000784436965642] 0.064ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#66) dbench 3399 0 2 00000002 00000042 [0000784436966636] 0.065ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#67) dbench 3399 0 2 00000002 00000043 [0000784436967181] 0.066ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#68) dbench 3399 0 2 00000002 00000044 [0000784436967716] 0.067ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#69) dbench 3399 0 2 00000002 00000045 [0000784436968688] 0.069ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#70) dbench 3399 0 2 00000002 00000046 [0000784436969251] 0.070ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#71) dbench 3399 0 2 00000002 00000047 [0000784436969840] 0.071ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#72) dbench 3399 0 2 00000002 00000048 [0000784436970263] 0.071ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#73) dbench 3399 0 2 00000002 00000049 [0000784436970754] 0.072ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#74) dbench 3399 0 2 00000002 0000004a [0000784436971276] 0.073ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#75) dbench 3399 0 2 00000002 0000004b [0000784436971838] 0.074ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#76) dbench 3399 0 2 00000002 0000004c [0000784436972329] 0.075ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#77) dbench 3399 0 2 00000002 0000004d [0000784436973184] 0.076ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#78) dbench 3399 0 2 00000002 0000004e [0000784436973742] 0.077ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#79) dbench 3399 0 2 00000002 0000004f [0000784436974237] 0.078ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#80) dbench 3399 0 2 00000002 00000050 [0000784436974691] 0.079ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#81) dbench 3399 0 2 00000002 00000051 [0000784436975092] 0.079ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#82) dbench 3399 0 2 00000002 00000052 [0000784436975578] 0.080ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#83) dbench 3399 0 2 00000002 00000053 [0000784436976019] 0.081ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#84) dbench 3399 0 2 00000002 00000054 [0000784436976415] 0.081ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#85) dbench 3399 0 2 00000002 00000055 [0000784436976901] 0.082ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#86) dbench 3399 0 2 00000002 00000056 [0000784436977423] 0.083ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#87) dbench 3399 0 2 00000002 00000057 [0000784436977837] 0.084ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#88) dbench 3399 0 2 00000002 00000058 [0000784436978327] 0.085ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#89) dbench 3399 0 2 00000002 00000059 [0000784436978845] 0.085ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#90) dbench 3399 0 2 00000002 0000005a [0000784436979412] 0.086ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#91) dbench 3399 0 2 00000002 0000005b [0000784436979902] 0.087ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#92) dbench 3399 0 2 00000002 0000005c [0000784436980361] 0.088ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#93) dbench 3399 0 2 00000002 0000005d [0000784436980762] 0.089ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#94) dbench 3399 0 2 00000002 0000005e [0000784436981252] 0.089ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#95) dbench 3399 0 2 00000002 0000005f [0000784436981698] 0.090ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#96) dbench 3399 0 2 00000002 00000060 [0000784436982098] 0.091ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#97) dbench 3399 0 2 00000002 00000061 [0000784436982584] 0.092ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#98) dbench 3399 0 2 00000002 00000062 [0000784436983021] 0.092ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#99) dbench 3399 0 2 00000002 00000063 [0000784436983417] 0.093ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#100) dbench 3399 0 2 00000002 00000064 [0000784436983916] 0.094ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#101) dbench 3399 0 2 00000002 00000065 [0000784436984357] 0.095ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#102) dbench 3399 0 2 00000002 00000066 [0000784436984839] 0.095ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#103) dbench 3399 0 2 00000002 00000067 [0000784436985392] 0.096ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#104) dbench 3399 0 2 00000002 00000068 [0000784436986108] 0.098ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#105) dbench 3399 0 2 00000002 00000069 [0000784436986972] 0.099ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#106) dbench 3399 0 2 00000002 0000006a [0000784436987570] 0.100ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#107) dbench 3399 0 2 00000002 0000006b [0000784436988083] 0.101ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#108) dbench 3399 0 2 00000002 0000006c [0000784436988497] 0.102ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#109) dbench 3399 0 2 00000002 0000006d [0000784436988997] 0.102ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#110) dbench 3399 0 2 00000002 0000006e [0000784436989442] 0.103ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#111) dbench 3399 0 2 00000002 0000006f [0000784436990144] 0.104ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#112) dbench 3399 0 2 00000002 00000070 [0000784436990702] 0.105ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#113) dbench 3399 0 2 00000002 00000071 [0000784436991157] 0.106ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#114) dbench 3399 0 2 00000002 00000072 [0000784436991634] 0.107ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#115) dbench 3399 0 2 00000002 00000073 [0000784436992187] 0.108ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#116) dbench 3399 0 2 00000002 00000074 [0000784436992646] 0.108ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#117) dbench 3399 0 2 00000002 00000075 [0000784436993218] 0.109ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#118) dbench 3399 0 2 00000002 00000076 [0000784436993713] 0.110ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#119) dbench 3399 0 2 00000002 00000077 [0000784436994167] 0.111ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#120) dbench 3399 0 2 00000002 00000078 [0000784436994563] 0.112ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#121) dbench 3399 0 2 00000002 00000079 [0000784436995081] 0.113ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#122) dbench 3399 0 2 00000002 0000007a [0000784436995679] 0.114ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#123) dbench 3399 0 2 00000002 0000007b [0000784436996102] 0.114ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#124) dbench 3399 0 2 00000002 0000007c [0000784436996593] 0.115ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#125) dbench 3399 0 2 00000002 0000007d [0000784436997038] 0.116ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#126) dbench 3399 0 2 00000002 0000007e [0000784436997524] 0.117ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#127) dbench 3399 0 2 00000002 0000007f [0000784436998015] 0.117ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#128) dbench 3399 0 2 00000002 00000080 [0000784436998532] 0.118ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#129) dbench 3399 0 2 00000002 00000081 [0000784436999126] 0.119ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#130) dbench 3399 0 2 00000002 00000082 [0000784436999689] 0.120ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#131) dbench 3399 0 2 00000002 00000083 [0000784437000193] 0.121ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#132) dbench 3399 0 2 00000002 00000084 [0000784437000625] 0.122ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#133) dbench 3399 0 2 00000002 00000085 [0000784437001111] 0.123ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#134) dbench 3399 0 2 00000002 00000086 [0000784437001822] 0.124ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#135) dbench 3399 0 2 00000002 00000087 [0000784437002375] 0.125ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#136) dbench 3399 0 2 00000002 00000088 [0000784437002861] 0.125ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#137) dbench 3399 0 2 00000002 00000089 [0000784437003320] 0.126ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#138) dbench 3399 0 2 00000002 0000008a [0000784437003788] 0.127ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#139) dbench 3399 0 2 00000002 0000008b [0000784437004279] 0.128ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#140) dbench 3399 0 2 00000002 0000008c [0000784437004859] 0.129ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#141) dbench 3399 0 2 00000002 0000008d [0000784437005444] 0.130ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#142) dbench 3399 0 2 00000002 0000008e [0000784437005935] 0.131ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#143) dbench 3399 0 2 00000002 0000008f [0000784437006398] 0.131ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#144) dbench 3399 0 2 00000002 00000090 [0000784437006799] 0.132ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#145) dbench 3399 0 2 00000002 00000091 [0000784437007289] 0.133ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#146) dbench 3399 0 2 00000002 00000092 [0000784437008005] 0.134ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#147) dbench 3399 0 2 00000002 00000093 [0000784437008545] 0.135ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#148) dbench 3399 0 2 00000002 00000094 [0000784437009031] 0.136ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#149) dbench 3399 0 2 00000002 00000095 [0000784437009546] 0.137ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#150) dbench 3399 0 2 00000002 00000096 [0000784437009944] 0.137ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#151) dbench 3399 0 2 00000002 00000097 [0000784437010435] 0.138ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#152) dbench 3399 0 2 00000002 00000098 [0000784437010871] 0.139ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#153) dbench 3399 0 2 00000002 00000099 [0000784437011461] 0.140ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#154) dbench 3399 0 2 00000002 0000009a [0000784437012014] 0.141ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#155) dbench 3399 0 2 00000002 0000009b [0000784437012464] 0.141ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#156) dbench 3399 0 2 00000002 0000009c [0000784437012874] 0.142ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#157) dbench 3399 0 2 00000002 0000009d [0000784437013364] 0.143ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#158) dbench 3399 0 2 00000002 0000009e [0000784437013823] 0.144ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#159) dbench 3399 0 2 00000002 0000009f [0000784437014224] 0.144ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#160) dbench 3399 0 2 00000002 000000a0 [0000784437014714] 0.145ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#161) dbench 3399 0 2 00000002 000000a1 [0000784437015160] 0.146ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#162) dbench 3399 0 2 00000002 000000a2 [0000784437015637] 0.147ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#163) dbench 3399 0 2 00000002 000000a3 [0000784437016127] 0.148ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#164) dbench 3399 0 2 00000002 000000a4 [0000784437016609] 0.148ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#165) dbench 3399 0 2 00000002 000000a5 [0000784437017032] 0.149ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#166) dbench 3399 0 2 00000002 000000a6 [0000784437017603] 0.150ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#167) dbench 3399 0 2 00000002 000000a7 [0000784437018040] 0.151ms (+0.002ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#168) dbench 3399 0 2 00000002 000000a8 [0000784437019696] 0.153ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#169) dbench 3399 0 2 00000002 000000a9 [0000784437020249] 0.154ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#170) dbench 3399 0 2 00000002 000000aa [0000784437020834] 0.155ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#171) dbench 3399 0 2 00000002 000000ab [0000784437021392] 0.156ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#172) dbench 3399 0 2 00000002 000000ac [0000784437021883] 0.157ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#173) dbench 3399 0 2 00000002 000000ad [0000784437022342] 0.158ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#174) dbench 3399 0 2 00000002 000000ae [0000784437022828] 0.159ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#175) dbench 3399 0 2 00000002 000000af [0000784437023327] 0.160ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#176) dbench 3399 0 2 00000002 000000b0 [0000784437023786] 0.160ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#177) dbench 3399 0 2 00000002 000000b1 [0000784437024187] 0.161ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#178) dbench 3399 0 2 00000002 000000b2 [0000784437024677] 0.162ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#179) dbench 3399 0 2 00000002 000000b3 [0000784437025118] 0.162ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#180) dbench 3399 0 2 00000002 000000b4 [0000784437025694] 0.163ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#181) dbench 3399 0 2 00000002 000000b5 [0000784437026198] 0.164ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#182) dbench 3399 0 2 00000002 000000b6 [0000784437026653] 0.165ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#183) dbench 3399 0 2 00000002 000000b7 [0000784437027121] 0.166ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#184) dbench 3399 0 2 00000002 000000b8 [0000784437027683] 0.167ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#185) dbench 3399 0 2 00000002 000000b9 [0000784437028142] 0.168ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#186) dbench 3399 0 2 00000002 000000ba [0000784437028538] 0.168ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#187) dbench 3399 0 2 00000002 000000bb [0000784437029024] 0.169ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#188) dbench 3399 0 2 00000002 000000bc [0000784437029461] 0.170ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#189) dbench 3399 0 2 00000002 000000bd [0000784437029857] 0.170ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#190) dbench 3399 0 2 00000002 000000be [0000784437030343] 0.171ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#191) dbench 3399 0 2 00000002 000000bf [0000784437030869] 0.172ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#192) dbench 3399 0 2 00000002 000000c0 [0000784437031337] 0.173ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#193) dbench 3399 0 2 00000002 000000c1 [0000784437031828] 0.174ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#194) dbench 3399 0 2 00000002 000000c2 [0000784437032395] 0.175ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#195) dbench 3399 0 2 00000002 000000c3 [0000784437033169] 0.176ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#196) dbench 3399 0 2 00000002 000000c4 [0000784437033722] 0.177ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#197) dbench 3399 0 2 00000002 000000c5 [0000784437034496] 0.178ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#198) dbench 3399 0 2 00000002 000000c6 [0000784437035036] 0.179ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#199) dbench 3399 0 2 00000002 000000c7 [0000784437035522] 0.180ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#200) dbench 3399 0 2 00000002 000000c8 [0000784437035963] 0.181ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#201) dbench 3399 0 2 00000002 000000c9 [0000784437036436] 0.181ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#202) dbench 3399 0 2 00000002 000000ca [0000784437037007] 0.182ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#203) dbench 3399 0 2 00000002 000000cb [0000784437037462] 0.183ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#204) dbench 3399 0 2 00000002 000000cc [0000784437037939] 0.184ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#205) dbench 3399 0 2 00000002 000000cd [0000784437038425] 0.185ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#206) dbench 3399 0 2 00000002 000000ce [0000784437038996] 0.186ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#207) dbench 3399 0 2 00000002 000000cf [0000784437039563] 0.187ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#208) dbench 3399 0 2 00000002 000000d0 [0000784437040054] 0.187ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#209) dbench 3399 0 2 00000002 000000d1 [0000784437040535] 0.188ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#210) dbench 3399 0 2 00000002 000000d2 [0000784437041017] 0.189ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#211) dbench 3399 0 2 00000002 000000d3 [0000784437041507] 0.190ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#212) dbench 3399 0 2 00000002 000000d4 [0000784437041971] 0.191ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#213) dbench 3399 0 2 00000002 000000d5 [0000784437042443] 0.191ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#214) dbench 3399 0 2 00000002 000000d6 [0000784437042934] 0.192ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#215) dbench 3399 0 2 00000002 000000d7 [0000784437043591] 0.193ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#216) dbench 3399 0 2 00000002 000000d8 [0000784437044005] 0.194ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#217) dbench 3399 0 2 00000002 000000d9 [0000784437044491] 0.195ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#218) dbench 3399 0 2 00000002 000000da [0000784437045004] 0.196ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#219) dbench 3399 0 2 00000002 000000db [0000784437045566] 0.197ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#220) dbench 3399 0 2 00000002 000000dc [0000784437046061] 0.197ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#221) dbench 3399 0 2 00000002 000000dd [0000784437046574] 0.198ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#222) dbench 3399 0 2 00000002 000000de [0000784437046984] 0.199ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#223) dbench 3399 0 2 00000002 000000df [0000784437047470] 0.200ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#224) dbench 3399 0 2 00000002 000000e0 [0000784437047983] 0.201ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#225) dbench 3399 0 2 00000002 000000e1 [0000784437048572] 0.202ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#226) dbench 3399 0 2 00000002 000000e2 [0000784437049130] 0.202ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#227) dbench 3399 0 2 00000002 000000e3 [0000784437049639] 0.203ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#228) dbench 3399 0 2 00000002 000000e4 [0000784437050053] 0.204ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#229) dbench 3399 0 2 00000002 000000e5 [0000784437050543] 0.205ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#230) dbench 3399 0 2 00000002 000000e6 [0000784437051061] 0.206ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#231) dbench 3399 0 2 00000002 000000e7 [0000784437051484] 0.206ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#232) dbench 3399 0 2 00000002 000000e8 [0000784437051974] 0.207ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#233) dbench 3399 0 2 00000002 000000e9 [0000784437052492] 0.208ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#234) dbench 3399 0 2 00000002 000000ea [0000784437053108] 0.209ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#235) dbench 3399 0 2 00000002 000000eb [0000784437053662] 0.210ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#236) dbench 3399 0 2 00000002 000000ec [0000784437054166] 0.211ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#237) dbench 3399 0 2 00000002 000000ed [0000784437054607] 0.212ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#238) dbench 3399 0 2 00000002 000000ee [0000784437055093] 0.212ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#239) dbench 3399 0 2 00000002 000000ef [0000784437055529] 0.213ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#240) dbench 3399 0 2 00000002 000000f0 [0000784437055925] 0.214ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#241) dbench 3399 0 2 00000002 000000f1 [0000784437056420] 0.215ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#242) dbench 3399 0 2 00000002 000000f2 [0000784437056933] 0.215ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#243) dbench 3399 0 2 00000002 000000f3 [0000784437057347] 0.216ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#244) dbench 3399 0 2 00000002 000000f4 [0000784437057838] 0.217ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#245) dbench 3399 0 2 00000002 000000f5 [0000784437058355] 0.218ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#246) dbench 3399 0 2 00000002 000000f6 [0000784437058769] 0.218ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#247) dbench 3399 0 2 00000002 000000f7 [0000784437059260] 0.219ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#248) dbench 3399 0 2 00000002 000000f8 [0000784437059701] 0.220ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#249) dbench 3399 0 2 00000002 000000f9 [0000784437060101] 0.221ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#250) dbench 3399 0 2 00000002 000000fa [0000784437060592] 0.222ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#251) dbench 3399 0 2 00000002 000000fb [0000784437061186] 0.223ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#252) dbench 3399 0 2 00000002 000000fc [0000784437061613] 0.223ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#253) dbench 3399 0 2 00000002 000000fd [0000784437062104] 0.224ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#254) dbench 3399 0 2 00000002 000000fe [0000784437062545] 0.225ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#255) dbench 3399 0 2 00000002 000000ff [0000784437062941] 0.225ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#256) dbench 3399 0 2 00000002 00000100 [0000784437063463] 0.226ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#257) dbench 3399 0 2 00000002 00000101 [0000784437063980] 0.227ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#258) dbench 3399 0 2 00000002 00000102 [0000784437064484] 0.228ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#259) dbench 3399 0 2 00000002 00000103 [0000784437064970] 0.229ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#260) dbench 3399 0 2 00000002 00000104 [0000784437065411] 0.230ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#261) dbench 3399 0 2 00000002 00000105 [0000784437065843] 0.230ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#262) dbench 3399 0 2 00000002 00000106 [0000784437066334] 0.231ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#263) dbench 3399 0 2 00000002 00000107 [0000784437066856] 0.232ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#264) dbench 3399 0 2 00000002 00000108 [0000784437067283] 0.233ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#265) dbench 3399 0 2 00000002 00000109 [0000784437067774] 0.233ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#266) dbench 3399 0 2 00000002 0000010a [0000784437068296] 0.234ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#267) dbench 3399 0 2 00000002 0000010b [0000784437068737] 0.235ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#268) dbench 3399 0 2 00000002 0000010c [0000784437069227] 0.236ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#269) dbench 3399 0 2 00000002 0000010d [0000784437069745] 0.237ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#270) dbench 3399 0 2 00000002 0000010e [0000784437070181] 0.237ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#271) dbench 3399 0 2 00000002 0000010f [0000784437070681] 0.238ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#272) dbench 3399 0 2 00000002 00000110 [0000784437071144] 0.239ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#273) dbench 3399 0 2 00000002 00000111 [0000784437071540] 0.240ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#274) dbench 3399 0 2 00000002 00000112 [0000784437072026] 0.241ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#275) dbench 3399 0 2 00000002 00000113 [0000784437072548] 0.241ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#276) dbench 3399 0 2 00000002 00000114 [0000784437073124] 0.242ms (+0.001ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#277) dbench 3399 0 2 00000002 00000115 [0000784437073828] 0.244ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#278) dbench 3399 0 2 00000002 00000116 [0000784437074281] 0.244ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#279) dbench 3399 0 2 00000002 00000117 [0000784437074758] 0.245ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#280) dbench 3399 0 2 00000002 00000118 [0000784437075248] 0.246ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#281) dbench 3399 0 2 00000002 00000119 [0000784437075716] 0.247ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#282) dbench 3399 0 2 00000002 0000011a [0000784437076130] 0.247ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#283) dbench 3399 0 2 00000002 0000011b [0000784437076621] 0.248ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#284) dbench 3399 0 2 00000002 0000011c [0000784437077066] 0.249ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#285) dbench 3399 0 2 00000002 0000011d [0000784437077467] 0.250ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#286) dbench 3399 0 2 00000002 0000011e [0000784437077957] 0.250ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#287) dbench 3399 0 2 00000002 0000011f [0000784437078403] 0.251ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#288) dbench 3399 0 2 00000002 00000120 [0000784437078803] 0.252ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#289) dbench 3399 0 2 00000002 00000121 [0000784437079294] 0.253ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#290) dbench 3399 0 2 00000002 00000122 [0000784437079735] 0.253ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#291) dbench 3399 0 2 00000002 00000123 [0000784437080315] 0.254ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#292) dbench 3399 0 2 00000002 00000124 [0000784437080878] 0.255ms (+0.000ms): find_next_reservable_window+0xb/0x60 <c01b58bb> (alloc_new_reservation+0xd2/0x1c0 <c01b59e2>)
(T1/#293) dbench 3399 0 2 00000002 00000125 [0000784437081243] 0.256ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#294) dbench 3399 0 2 00000002 00000126 [0000784437081652] 0.257ms (+0.001ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#295) dbench 3399 0 2 00000002 00000127 [0000784437082466] 0.258ms (+0.000ms): bitmap_search_next_usable_block+0xc/0xd0 <c01b551c> (alloc_new_reservation+0xf7/0x1c0 <c01b5a07>)
(T1/#296) dbench 3399 0 2 00000002 00000128 [0000784437082822] 0.259ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#297) dbench 3399 0 2 00000002 00000129 [0000784437083312] 0.259ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#298) dbench 3399 0 2 00000002 0000012a [0000784437083906] 0.260ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#299) dbench 3399 0 2 00000002 0000012b [0000784437084424] 0.261ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#300) dbench 3399 0 2 00000002 0000012c [0000784437084896] 0.262ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#301) dbench 3399 0 2 00000002 0000012d [0000784437085396] 0.263ms (+0.000ms): find_next_reservable_window+0xb/0x60 <c01b58bb> (alloc_new_reservation+0xd2/0x1c0 <c01b59e2>)
(T1/#302) dbench 3399 0 2 00000002 0000012e [0000784437085756] 0.263ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#303) dbench 3399 0 2 00000002 0000012f [0000784437086143] 0.264ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#304) dbench 3399 0 2 00000002 00000130 [0000784437086642] 0.265ms (+0.000ms): bitmap_search_next_usable_block+0xc/0xd0 <c01b551c> (alloc_new_reservation+0xf7/0x1c0 <c01b5a07>)
(T1/#305) dbench 3399 0 2 00000002 00000131 [0000784437087002] 0.265ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#306) dbench 3399 0 2 00000002 00000132 [0000784437087416] 0.266ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#307) dbench 3399 0 2 00000002 00000133 [0000784437087916] 0.267ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#308) dbench 3399 0 2 00000002 00000134 [0000784437088361] 0.268ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#309) dbench 3399 0 2 00000002 00000135 [0000784437088838] 0.269ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#310) dbench 3399 0 2 00000002 00000136 [0000784437089324] 0.269ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#311) dbench 3399 0 2 00000002 00000137 [0000784437089801] 0.270ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#312) dbench 3399 0 2 00000002 00000138 [0000784437090233] 0.271ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#313) dbench 3399 0 2 00000002 00000139 [0000784437090719] 0.272ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#314) dbench 3399 0 2 00000002 0000013a [0000784437091160] 0.272ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#315) dbench 3399 0 2 00000002 0000013b [0000784437091556] 0.273ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#316) dbench 3399 0 2 00000002 0000013c [0000784437092042] 0.274ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#317) dbench 3399 0 2 00000002 0000013d [0000784437092483] 0.275ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#318) dbench 3399 0 2 00000002 0000013e [0000784437092884] 0.275ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#319) dbench 3399 0 2 00000002 0000013f [0000784437093374] 0.276ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#320) dbench 3399 0 2 00000002 00000140 [0000784437093820] 0.277ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#321) dbench 3399 0 2 00000002 00000141 [0000784437094229] 0.277ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#322) dbench 3399 0 2 00000002 00000142 [0000784437094787] 0.278ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#323) dbench 3399 0 2 00000001 00000143 [0000784437095705] 0.280ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (ext3_try_to_allocate_with_rsv+0x28f/0x330 <c01b5d5f>)
(T1/#324) dbench 3399 0 2 00000000 00000144 [0000784437096304] 0.281ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (ext3_try_to_allocate_with_rsv+0x285/0x330 <c01b5d55>)
(T1/#325) dbench 3399 0 3 00000000 00000145 [0000784437096839] 0.282ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#326) dbench 3399 0 3 00000000 00000146 [0000784437097231] 0.282ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#327) dbench 3399 0 3 00000001 00000147 [0000784437097667] 0.283ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#328) dbench 3399 0 3 00000002 00000148 [0000784437098608] 0.285ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#329) dbench 3399 0 3 00000002 00000149 [0000784437099166] 0.286ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#330) dbench 3399 0 3 00000002 0000014a [0000784437099611] 0.286ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#331) dbench 3399 0 3 00000002 0000014b [0000784437100012] 0.287ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#332) [ => dbench ] 0.288ms (+0.001ms)
(T1/#333) <...> 2 0 1 00000002 0000014d [0000784437101515] 0.290ms (+0.002ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#334) <...>-2 0d..2 292µs : __schedule+0x2ea/0x630 <c027cc9a> <dbench-3399> (76 69):
(T1/#335) <...> 2 0 1 00000002 0000014f [0000784437103484] 0.293ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#336) <...> 2 0 1 00000001 00000150 [0000784437103916] 0.294ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#337) <...>-2 0d..1 294µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#338) <...> 2 0 1 00000001 00000152 [0000784437105041] 0.295ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 19:36 ` Lee Revell
@ 2005-02-23 20:06 ` Hugh Dickins
2005-02-23 20:10 ` Lee Revell
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 20:06 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 23 Feb 2005, Lee Revell wrote:
> On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> >
> > I'm just about to test this patch below: please give it a try: thanks...
I'm very sorry, there's two things wrong with that version: _must_
increment addr before breaking out, and better to check after pte_none
too (we can question whether it might be checking too often, but this
replicates what Ingo was doing). Please replace by new patch below,
which I'm now running through lmbench.
> Aha, that explains why all the latency regressions involve the VM
> subsystem.
>
> Thanks, your patch fixes the copy_pte_range latency.
Great, if the previous patch fixed that latency then this new one
will too, no need to report on that; but please get rid of the old
patch before it leaks too many of your pages.
> Now zap_pte_range,
> which Ingo also fixed a few months ago, is the worst offender. Can this
> fix be easily ported too?
That surprises me: all the zap_pte_range latency fixes I know of
are in 2.6.11-rc, perhaps Ingo knows of something missing there?
Hugh
Ingo's patch to reduce scheduling latencies, by checking for lockbreak
in copy_page_range, was in the -VP and -mm patchsets some months ago;
but got preempted by the 4level rework, and not reinstated since.
Restore it now in copy_pte_range - which mercifully makes it easier.
Signed-off-by: Hugh Dickins <hugh@veritas.com>
--- 2.6.11-rc4-bk/mm/memory.c 2005-02-21 11:32:19.000000000 +0000
+++ linux/mm/memory.c 2005-02-23 19:46:40.000000000 +0000
@@ -328,21 +328,33 @@ static int copy_pte_range(struct mm_stru
pte_t *s, *d;
unsigned long vm_flags = vma->vm_flags;
+again:
d = dst_pte = pte_alloc_map(dst_mm, dst_pmd, addr);
if (!dst_pte)
return -ENOMEM;
spin_lock(&src_mm->page_table_lock);
s = src_pte = pte_offset_map_nested(src_pmd, addr);
- for (; addr < end; addr += PAGE_SIZE, s++, d++) {
- if (pte_none(*s))
- continue;
- copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+ for (; addr < end; s++, d++) {
+ if (!pte_none(*s))
+ copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+ addr += PAGE_SIZE;
+ /*
+ * We are holding two locks at this point - either of them
+ * could generate latencies in another task on another CPU.
+ */
+ if (need_resched() ||
+ need_lockbreak(&src_mm->page_table_lock) ||
+ need_lockbreak(&dst_mm->page_table_lock))
+ break;
}
pte_unmap_nested(src_pte);
pte_unmap(dst_pte);
spin_unlock(&src_mm->page_table_lock);
+
cond_resched_lock(&dst_mm->page_table_lock);
+ if (addr < end)
+ goto again;
return 0;
}
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 20:06 ` Hugh Dickins
@ 2005-02-23 20:10 ` Lee Revell
2005-02-23 20:30 ` Lee Revell
2005-02-23 20:53 ` Hugh Dickins
2 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 20:10 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 2005-02-23 at 20:06 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> > >
> > > I'm just about to test this patch below: please give it a try: thanks...
>
> I'm very sorry, there's two things wrong with that version: _must_
> increment addr before breaking out, and better to check after pte_none
> too (we can question whether it might be checking too often, but this
> replicates what Ingo was doing). Please replace by new patch below,
> which I'm now running through lmbench.
OK, I will report any interesting results with the new patch.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 20:06 ` Hugh Dickins
2005-02-23 20:10 ` Lee Revell
@ 2005-02-23 20:30 ` Lee Revell
2005-02-23 21:03 ` Hugh Dickins
2005-02-23 20:53 ` Hugh Dickins
2 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 20:30 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 349 bytes --]
On Wed, 2005-02-23 at 20:06 +0000, Hugh Dickins wrote:
> >
> > Thanks, your patch fixes the copy_pte_range latency.
>
> Great, if the previous patch fixed that latency then this new one
> will too, no need to report on that; but please get rid of the old
> patch before it leaks too many of your pages.
clear_page_range is also problematic.
Lee
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace7.txt --]
[-- Type: text/plain; name=trace7.txt; charset=ANSI_X3.4-1968, Size: 49938 bytes --]
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 313 µs, #291/291, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) cc1plus 3510 0 9 00000002 00000000 [0000188528469818] 0.000ms (+811152.283ms): <70316363> (<0073756c>)
(T1/#2) cc1plus 3510 0 9 00000002 00000002 [0000188528470164] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3) cc1plus 3510 0 9 00000000 00000003 [0000188528470660] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) cc1plus-3510 0dn.2 2µs+< (1)
(T1/#5) cc1plus 3510 0 2 00000002 00000005 [0000188528521949] 0.086ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#6) cc1plus 3510 0 2 00000002 00000006 [0000188528522356] 0.087ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#7) cc1plus 3510 0 2 00000002 00000007 [0000188528522777] 0.088ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#8) cc1plus 3510 0 2 00000002 00000008 [0000188528523382] 0.089ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#9) cc1plus 3510 0 2 00000002 00000009 [0000188528523695] 0.089ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#10) cc1plus 3510 0 2 00000002 0000000a [0000188528523952] 0.090ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#11) cc1plus 3510 0 2 00000002 0000000b [0000188528524216] 0.090ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#12) cc1plus 3510 0 2 00000002 0000000c [0000188528524975] 0.091ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#13) cc1plus 3510 0 2 00000002 0000000d [0000188528525518] 0.092ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#14) cc1plus 3510 0 2 00000002 0000000e [0000188528525878] 0.093ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#15) cc1plus 3510 0 2 00000002 0000000f [0000188528526264] 0.093ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#16) cc1plus 3510 0 2 00000002 00000010 [0000188528526791] 0.094ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#17) cc1plus 3510 0 2 00000002 00000011 [0000188528527169] 0.095ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#18) cc1plus 3510 0 2 00000002 00000012 [0000188528527426] 0.095ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#19) cc1plus 3510 0 2 00000002 00000013 [0000188528527898] 0.096ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#20) cc1plus 3510 0 2 00000002 00000014 [0000188528528616] 0.097ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#21) cc1plus 3510 0 2 00000002 00000015 [0000188528529086] 0.098ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#22) cc1plus 3510 0 2 00000002 00000016 [0000188528529446] 0.099ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#23) cc1plus 3510 0 2 00000002 00000017 [0000188528529833] 0.099ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#24) cc1plus 3510 0 2 00000002 00000018 [0000188528530360] 0.100ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#25) cc1plus 3510 0 2 00000002 00000019 [0000188528530665] 0.101ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#26) cc1plus 3510 0 2 00000002 0000001a [0000188528530922] 0.101ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#27) cc1plus 3510 0 2 00000002 0000001b [0000188528531186] 0.102ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#28) cc1plus 3510 0 2 00000002 0000001c [0000188528531904] 0.103ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#29) cc1plus 3510 0 2 00000002 0000001d [0000188528532309] 0.103ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#30) cc1plus 3510 0 2 00000002 0000001e [0000188528532669] 0.104ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#31) cc1plus 3510 0 2 00000002 0000001f [0000188528533055] 0.105ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#32) cc1plus 3510 0 2 00000002 00000020 [0000188528533582] 0.106ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#33) cc1plus 3510 0 2 00000002 00000021 [0000188528533887] 0.106ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#34) cc1plus 3510 0 2 00000002 00000022 [0000188528534144] 0.107ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#35) cc1plus 3510 0 2 00000002 00000023 [0000188528534408] 0.107ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#36) cc1plus 3510 0 2 00000002 00000024 [0000188528535126] 0.108ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#37) cc1plus 3510 0 2 00000002 00000025 [0000188528535531] 0.109ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#38) cc1plus 3510 0 2 00000002 00000026 [0000188528535891] 0.109ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#39) cc1plus 3510 0 2 00000002 00000027 [0000188528536277] 0.110ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#40) cc1plus 3510 0 2 00000002 00000028 [0000188528536804] 0.111ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#41) cc1plus 3510 0 2 00000002 00000029 [0000188528537109] 0.111ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#42) cc1plus 3510 0 2 00000002 0000002a [0000188528537366] 0.112ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#43) cc1plus 3510 0 2 00000002 0000002b [0000188528537630] 0.112ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#44) cc1plus 3510 0 3 00000003 0000002c [0000188528538238] 0.113ms (+0.000ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#45) cc1plus 3510 0 3 00000004 0000002d [0000188528538772] 0.114ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#46) cc1plus 3510 0 3 00000004 0000002e [0000188528539128] 0.115ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#47) cc1plus 3510 0 3 00000004 0000002f [0000188528539654] 0.116ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#48) cc1plus 3510 0 3 00000004 00000030 [0000188528540010] 0.116ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#49) cc1plus 3510 0 3 00000004 00000031 [0000188528540515] 0.117ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#50) cc1plus 3510 0 3 00000004 00000032 [0000188528540871] 0.118ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#51) cc1plus 3510 0 3 00000004 00000033 [0000188528541384] 0.119ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#52) cc1plus 3510 0 3 00000004 00000034 [0000188528541740] 0.119ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#53) cc1plus 3510 0 3 00000004 00000035 [0000188528542243] 0.120ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#54) cc1plus 3510 0 3 00000004 00000036 [0000188528542690] 0.121ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#55) cc1plus 3510 0 3 00000004 00000037 [0000188528543212] 0.122ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#56) cc1plus 3510 0 3 00000004 00000038 [0000188528543568] 0.122ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#57) cc1plus 3510 0 3 00000004 00000039 [0000188528544045] 0.123ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#58) cc1plus 3510 0 3 00000004 0000003a [0000188528544401] 0.124ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#59) cc1plus 3510 0 3 00000004 0000003b [0000188528544806] 0.124ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#60) cc1plus 3510 0 3 00000004 0000003c [0000188528545162] 0.125ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#61) cc1plus 3510 0 3 00000004 0000003d [0000188528545567] 0.126ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#62) cc1plus 3510 0 3 00000004 0000003e [0000188528545923] 0.126ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#63) cc1plus 3510 0 3 00000004 0000003f [0000188528546353] 0.127ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#64) cc1plus 3510 0 3 00000004 00000040 [0000188528546709] 0.127ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#65) cc1plus 3510 0 3 00000004 00000041 [0000188528547211] 0.128ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#66) cc1plus 3510 0 3 00000004 00000042 [0000188528547567] 0.129ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#67) cc1plus 3510 0 3 00000004 00000043 [0000188528548041] 0.130ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#68) cc1plus 3510 0 3 00000004 00000044 [0000188528548397] 0.130ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#69) cc1plus 3510 0 3 00000004 00000045 [0000188528548916] 0.131ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#70) cc1plus 3510 0 3 00000004 00000046 [0000188528549272] 0.132ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#71) cc1plus 3510 0 3 00000004 00000047 [0000188528549688] 0.132ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#72) cc1plus 3510 0 3 00000004 00000048 [0000188528550044] 0.133ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#73) cc1plus 3510 0 3 00000004 00000049 [0000188528550449] 0.134ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#74) cc1plus 3510 0 3 00000004 0000004a [0000188528550805] 0.134ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#75) cc1plus 3510 0 3 00000004 0000004b [0000188528551286] 0.135ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#76) cc1plus 3510 0 3 00000004 0000004c [0000188528551642] 0.136ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#77) cc1plus 3510 0 3 00000003 0000004d [0000188528552147] 0.136ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#78) cc1plus 3510 0 2 00000002 0000004e [0000188528552606] 0.137ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#79) cc1plus 3510 0 2 00000002 0000004f [0000188528553011] 0.138ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#80) cc1plus 3510 0 2 00000002 00000050 [0000188528553386] 0.139ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#81) cc1plus 3510 0 2 00000002 00000051 [0000188528553768] 0.139ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#82) cc1plus 3510 0 2 00000002 00000052 [0000188528554309] 0.140ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#83) cc1plus 3510 0 2 00000002 00000053 [0000188528554599] 0.141ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#84) cc1plus 3510 0 2 00000002 00000054 [0000188528554856] 0.141ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#85) cc1plus 3510 0 2 00000002 00000055 [0000188528555120] 0.141ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#86) cc1plus 3510 0 2 00000002 00000056 [0000188528555860] 0.143ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#87) cc1plus 3510 0 2 00000002 00000057 [0000188528556332] 0.143ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#88) cc1plus 3510 0 2 00000002 00000058 [0000188528556707] 0.144ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#89) cc1plus 3510 0 2 00000002 00000059 [0000188528557089] 0.145ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#90) cc1plus 3510 0 2 00000002 0000005a [0000188528557616] 0.146ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#91) cc1plus 3510 0 2 00000002 0000005b [0000188528557921] 0.146ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#92) cc1plus 3510 0 2 00000002 0000005c [0000188528558178] 0.147ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#93) cc1plus 3510 0 2 00000002 0000005d [0000188528558442] 0.147ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#94) cc1plus 3510 0 2 00000002 0000005e [0000188528559160] 0.148ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#95) cc1plus 3510 0 2 00000002 0000005f [0000188528559565] 0.149ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#96) cc1plus 3510 0 2 00000002 00000060 [0000188528559925] 0.149ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#97) cc1plus 3510 0 2 00000002 00000061 [0000188528560311] 0.150ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#98) cc1plus 3510 0 2 00000002 00000062 [0000188528560838] 0.151ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#99) cc1plus 3510 0 2 00000002 00000063 [0000188528561143] 0.151ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#100) cc1plus 3510 0 2 00000002 00000064 [0000188528561400] 0.152ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#101) cc1plus 3510 0 2 00000002 00000065 [0000188528561664] 0.152ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#102) cc1plus 3510 0 2 00000002 00000066 [0000188528562382] 0.154ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#103) cc1plus 3510 0 2 00000002 00000067 [0000188528562787] 0.154ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#104) cc1plus 3510 0 2 00000002 00000068 [0000188528563147] 0.155ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#105) cc1plus 3510 0 2 00000002 00000069 [0000188528563533] 0.155ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#106) cc1plus 3510 0 2 00000002 0000006a [0000188528564060] 0.156ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#107) cc1plus 3510 0 2 00000002 0000006b [0000188528564365] 0.157ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#108) cc1plus 3510 0 2 00000002 0000006c [0000188528564622] 0.157ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#109) cc1plus 3510 0 2 00000002 0000006d [0000188528564886] 0.158ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#110) cc1plus 3510 0 2 00000002 0000006e [0000188528565604] 0.159ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#111) cc1plus 3510 0 2 00000002 0000006f [0000188528566009] 0.160ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#112) cc1plus 3510 0 2 00000002 00000070 [0000188528566369] 0.160ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#113) cc1plus 3510 0 2 00000002 00000071 [0000188528566724] 0.161ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#114) cc1plus 3510 0 2 00000002 00000072 [0000188528567251] 0.162ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#115) cc1plus 3510 0 2 00000002 00000073 [0000188528567556] 0.162ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#116) cc1plus 3510 0 2 00000002 00000074 [0000188528567813] 0.163ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#117) cc1plus 3510 0 2 00000002 00000075 [0000188528568077] 0.163ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#118) cc1plus 3510 0 2 00000002 00000076 [0000188528568795] 0.164ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#119) cc1plus 3510 0 2 00000002 00000077 [0000188528569200] 0.165ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#120) cc1plus 3510 0 2 00000002 00000078 [0000188528569560] 0.165ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#121) cc1plus 3510 0 2 00000002 00000079 [0000188528569946] 0.166ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#122) cc1plus 3510 0 2 00000002 0000007a [0000188528570473] 0.167ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#123) cc1plus 3510 0 2 00000002 0000007b [0000188528570778] 0.167ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#124) cc1plus 3510 0 2 00000002 0000007c [0000188528571035] 0.168ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#125) cc1plus 3510 0 2 00000002 0000007d [0000188528571299] 0.168ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#126) cc1plus 3510 0 2 00000002 0000007e [0000188528572017] 0.170ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#127) cc1plus 3510 0 2 00000002 0000007f [0000188528572422] 0.170ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#128) cc1plus 3510 0 2 00000002 00000080 [0000188528572782] 0.171ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#129) cc1plus 3510 0 2 00000002 00000081 [0000188528573218] 0.172ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#130) cc1plus 3510 0 2 00000002 00000082 [0000188528573745] 0.172ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#131) cc1plus 3510 0 2 00000002 00000083 [0000188528574122] 0.173ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#132) cc1plus 3510 0 2 00000002 00000084 [0000188528574379] 0.173ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#133) cc1plus 3510 0 2 00000002 00000085 [0000188528574643] 0.174ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#134) cc1plus 3510 0 2 00000002 00000086 [0000188528575361] 0.175ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#135) cc1plus 3510 0 2 00000002 00000087 [0000188528575810] 0.176ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#136) cc1plus 3510 0 2 00000002 00000088 [0000188528576170] 0.176ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#137) cc1plus 3510 0 2 00000002 00000089 [0000188528576525] 0.177ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#138) cc1plus 3510 0 2 00000002 0000008a [0000188528577148] 0.178ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#139) cc1plus 3510 0 2 00000002 0000008b [0000188528577530] 0.179ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#140) cc1plus 3510 0 2 00000002 0000008c [0000188528577872] 0.179ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#141) cc1plus 3510 0 2 00000002 0000008d [0000188528578219] 0.180ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#142) cc1plus 3510 0 2 00000002 0000008e [0000188528579020] 0.181ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#143) cc1plus 3510 0 2 00000002 0000008f [0000188528579510] 0.182ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#144) cc1plus 3510 0 2 00000002 00000090 [0000188528579951] 0.183ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#145) cc1plus 3510 0 2 00000002 00000091 [0000188528580473] 0.184ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#146) cc1plus 3510 0 2 00000002 00000092 [0000188528581085] 0.185ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#147) cc1plus 3510 0 2 00000002 00000093 [0000188528581472] 0.185ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#148) cc1plus 3510 0 2 00000002 00000094 [0000188528581810] 0.186ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#149) cc1plus 3510 0 2 00000002 00000095 [0000188528582226] 0.187ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#150) cc1plus 3510 0 2 00000002 00000096 [0000188528583025] 0.188ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#151) cc1plus 3510 0 2 00000002 00000097 [0000188528583515] 0.189ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#152) cc1plus 3510 0 2 00000002 00000098 [0000188528583961] 0.189ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#153) cc1plus 3510 0 2 00000002 00000099 [0000188528584424] 0.190ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#154) cc1plus 3510 0 2 00000002 0000009a [0000188528585027] 0.191ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#155) cc1plus 3510 0 2 00000002 0000009b [0000188528585419] 0.192ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#156) cc1plus 3510 0 2 00000002 0000009c [0000188528585761] 0.192ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#157) cc1plus 3510 0 2 00000002 0000009d [0000188528586112] 0.193ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#158) cc1plus 3510 0 2 00000002 0000009e [0000188528586913] 0.194ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#159) cc1plus 3510 0 2 00000002 0000009f [0000188528587403] 0.195ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#160) cc1plus 3510 0 2 00000002 000000a0 [0000188528587849] 0.196ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#161) cc1plus 3510 0 2 00000002 000000a1 [0000188528588281] 0.197ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#162) cc1plus 3510 0 2 00000002 000000a2 [0000188528588888] 0.198ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#163) cc1plus 3510 0 2 00000002 000000a3 [0000188528589275] 0.198ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#164) cc1plus 3510 0 2 00000002 000000a4 [0000188528589613] 0.199ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#165) cc1plus 3510 0 2 00000002 000000a5 [0000188528589964] 0.199ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#166) cc1plus 3510 0 2 00000002 000000a6 [0000188528590760] 0.201ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#167) cc1plus 3510 0 2 00000002 000000a7 [0000188528591246] 0.202ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#168) cc1plus 3510 0 2 00000002 000000a8 [0000188528591687] 0.202ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#169) cc1plus 3510 0 2 00000002 000000a9 [0000188528592146] 0.203ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#170) cc1plus 3510 0 2 00000002 000000aa [0000188528592749] 0.204ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#171) cc1plus 3510 0 2 00000002 000000ab [0000188528593141] 0.205ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#172) cc1plus 3510 0 2 00000002 000000ac [0000188528593474] 0.205ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#173) cc1plus 3510 0 2 00000002 000000ad [0000188528593816] 0.206ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#174) cc1plus 3510 0 2 00000002 000000ae [0000188528594608] 0.207ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#175) cc1plus 3510 0 2 00000002 000000af [0000188528595112] 0.208ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#176) cc1plus 3510 0 2 00000002 000000b0 [0000188528595548] 0.209ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#177) cc1plus 3510 0 2 00000002 000000b1 [0000188528595976] 0.209ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#178) cc1plus 3510 0 2 00000002 000000b2 [0000188528596588] 0.210ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#179) cc1plus 3510 0 2 00000002 000000b3 [0000188528596970] 0.211ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#180) cc1plus 3510 0 2 00000002 000000b4 [0000188528597308] 0.212ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#181) cc1plus 3510 0 2 00000002 000000b5 [0000188528597645] 0.212ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#182) cc1plus 3510 0 2 00000002 000000b6 [0000188528598433] 0.214ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#183) cc1plus 3510 0 2 00000002 000000b7 [0000188528598923] 0.214ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#184) cc1plus 3510 0 2 00000002 000000b8 [0000188528599355] 0.215ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#185) cc1plus 3510 0 2 00000002 000000b9 [0000188528599769] 0.216ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#186) cc1plus 3510 0 2 00000002 000000ba [0000188528600368] 0.217ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#187) cc1plus 3510 0 2 00000002 000000bb [0000188528600755] 0.217ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#188) cc1plus 3510 0 2 00000002 000000bc [0000188528601083] 0.218ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#189) cc1plus 3510 0 2 00000002 000000bd [0000188528601416] 0.218ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#190) cc1plus 3510 0 2 00000002 000000be [0000188528602217] 0.220ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#191) cc1plus 3510 0 2 00000002 000000bf [0000188528602699] 0.221ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#192) cc1plus 3510 0 2 00000002 000000c0 [0000188528603135] 0.221ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#193) cc1plus 3510 0 2 00000002 000000c1 [0000188528603500] 0.222ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#194) cc1plus 3510 0 2 00000002 000000c2 [0000188528604103] 0.223ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#195) cc1plus 3510 0 2 00000002 000000c3 [0000188528604494] 0.224ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#196) cc1plus 3510 0 2 00000002 000000c4 [0000188528604836] 0.224ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#197) cc1plus 3510 0 2 00000002 000000c5 [0000188528605183] 0.225ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#198) cc1plus 3510 0 2 00000002 000000c6 [0000188528605984] 0.226ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#199) cc1plus 3510 0 2 00000002 000000c7 [0000188528606461] 0.227ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#200) cc1plus 3510 0 2 00000002 000000c8 [0000188528606902] 0.228ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#201) cc1plus 3510 0 2 00000002 000000c9 [0000188528607253] 0.228ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#202) cc1plus 3510 0 2 00000002 000000ca [0000188528607856] 0.229ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#203) cc1plus 3510 0 2 00000002 000000cb [0000188528608243] 0.230ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#204) cc1plus 3510 0 2 00000002 000000cc [0000188528608571] 0.230ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#205) cc1plus 3510 0 2 00000002 000000cd [0000188528608904] 0.231ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#206) cc1plus 3510 0 3 00000003 000000ce [0000188528609579] 0.232ms (+0.000ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#207) cc1plus 3510 0 3 00000004 000000cf [0000188528610178] 0.233ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#208) cc1plus 3510 0 3 00000004 000000d0 [0000188528610610] 0.234ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#209) cc1plus 3510 0 3 00000004 000000d1 [0000188528611181] 0.235ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#210) cc1plus 3510 0 3 00000004 000000d2 [0000188528611613] 0.235ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#211) cc1plus 3510 0 3 00000004 000000d3 [0000188528612144] 0.236ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#212) cc1plus 3510 0 3 00000004 000000d4 [0000188528612585] 0.237ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#213) cc1plus 3510 0 3 00000004 000000d5 [0000188528613085] 0.238ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#214) cc1plus 3510 0 3 00000004 000000d6 [0000188528613535] 0.239ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#215) cc1plus 3510 0 3 00000004 000000d7 [0000188528614120] 0.240ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#216) cc1plus 3510 0 3 00000004 000000d8 [0000188528614556] 0.240ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#217) cc1plus 3510 0 3 00000004 000000d9 [0000188528615132] 0.241ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#218) cc1plus 3510 0 3 00000004 000000da [0000188528615560] 0.242ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#219) cc1plus 3510 0 3 00000004 000000db [0000188528616145] 0.243ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#220) cc1plus 3510 0 3 00000004 000000dc [0000188528616577] 0.244ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#221) cc1plus 3510 0 3 00000004 000000dd [0000188528617135] 0.245ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#222) cc1plus 3510 0 3 00000004 000000de [0000188528617571] 0.245ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#223) cc1plus 3510 0 3 00000004 000000df [0000188528618143] 0.246ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#224) cc1plus 3510 0 3 00000004 000000e0 [0000188528618579] 0.247ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#225) cc1plus 3510 0 3 00000004 000000e1 [0000188528619137] 0.248ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#226) cc1plus 3510 0 3 00000004 000000e2 [0000188528619578] 0.249ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#227) cc1plus 3510 0 3 00000004 000000e3 [0000188528620150] 0.250ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#228) cc1plus 3510 0 3 00000004 000000e4 [0000188528620586] 0.250ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#229) cc1plus 3510 0 3 00000004 000000e5 [0000188528621140] 0.251ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#230) cc1plus 3510 0 3 00000004 000000e6 [0000188528621581] 0.252ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#231) cc1plus 3510 0 3 00000004 000000e7 [0000188528622134] 0.253ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#232) cc1plus 3510 0 3 00000004 000000e8 [0000188528622566] 0.254ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#233) cc1plus 3510 0 3 00000004 000000e9 [0000188528623174] 0.255ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#234) cc1plus 3510 0 3 00000004 000000ea [0000188528623615] 0.255ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#235) cc1plus 3510 0 3 00000004 000000eb [0000188528624267] 0.256ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#236) cc1plus 3510 0 3 00000004 000000ec [0000188528624704] 0.257ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#237) cc1plus 3510 0 3 00000004 000000ed [0000188528625289] 0.258ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#238) cc1plus 3510 0 3 00000004 000000ee [0000188528625833] 0.259ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#239) cc1plus 3510 0 3 00000004 000000ef [0000188528626481] 0.260ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#240) cc1plus 3510 0 3 00000004 000000f0 [0000188528627134] 0.261ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#241) cc1plus 3510 0 3 00000004 000000f1 [0000188528627822] 0.262ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#242) cc1plus 3510 0 3 00000004 000000f2 [0000188528628524] 0.264ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#243) cc1plus 3510 0 3 00000004 000000f3 [0000188528629226] 0.265ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#244) cc1plus 3510 0 3 00000004 000000f4 [0000188528629919] 0.266ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#245) cc1plus 3510 0 3 00000004 000000f5 [0000188528630558] 0.267ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#246) cc1plus 3510 0 3 00000004 000000f6 [0000188528630999] 0.268ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#247) cc1plus 3510 0 3 00000004 000000f7 [0000188528631715] 0.269ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#248) cc1plus 3510 0 3 00000004 000000f8 [0000188528632241] 0.270ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#249) cc1plus 3510 0 3 00000004 000000f9 [0000188528632759] 0.271ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#250) cc1plus 3510 0 3 00000004 000000fa [0000188528633393] 0.272ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#251) cc1plus 3510 0 3 00000004 000000fb [0000188528634154] 0.273ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#252) cc1plus 3510 0 3 00000004 000000fc [0000188528634703] 0.274ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#253) cc1plus 3510 0 3 00000004 000000fd [0000188528635297] 0.275ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#254) cc1plus 3510 0 3 00000004 000000fe [0000188528635963] 0.276ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#255) cc1plus 3510 0 3 00000004 000000ff [0000188528636507] 0.277ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#256) cc1plus 3510 0 3 00000003 00000100 [0000188528637160] 0.278ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#257) cc1plus 3510 0 2 00000002 00000101 [0000188528637700] 0.279ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#258) cc1plus 3510 0 2 00000002 00000102 [0000188528638181] 0.280ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#259) cc1plus 3510 0 2 00000002 00000103 [0000188528638640] 0.280ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#260) cc1plus 3510 0 2 00000002 00000104 [0000188528639081] 0.281ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#261) cc1plus 3510 0 2 00000002 00000105 [0000188528639693] 0.282ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#262) cc1plus 3510 0 2 00000002 00000106 [0000188528640053] 0.283ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#263) cc1plus 3510 0 2 00000002 00000107 [0000188528640382] 0.283ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#264) cc1plus 3510 0 2 00000002 00000108 [0000188528640719] 0.284ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#265) cc1plus 3510 0 2 00000002 00000109 [0000188528641547] 0.285ms (+0.006ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#266) cc1plus 3510 0 2 00000002 0000010a [0000188528645458] 0.292ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#267) cc1plus 3510 0 2 00000002 0000010b [0000188528645908] 0.292ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#268) cc1plus 3510 0 2 00000002 0000010c [0000188528646353] 0.293ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#269) cc1plus 3510 0 2 00000002 0000010d [0000188528646956] 0.294ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#270) cc1plus 3510 0 2 00000002 0000010e [0000188528647348] 0.295ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#271) cc1plus 3510 0 2 00000002 0000010f [0000188528647681] 0.295ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#272) cc1plus 3510 0 2 00000002 00000110 [0000188528648032] 0.296ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#273) cc1plus 3510 0 2 00000002 00000111 [0000188528648824] 0.297ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#274) cc1plus 3510 0 2 00000002 00000112 [0000188528649746] 0.299ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x190/0x1b0 <c0148100>)
(T1/#275) cc1plus 3510 0 2 00000001 00000113 [0000188528650210] 0.300ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x164/0x1b0 <c01480d4>)
(T1/#276) cc1plus 3510 0 2 00000000 00000114 [0000188528650714] 0.300ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x15d/0x1b0 <c01480cd>)
(T1/#277) cc1plus 3510 0 3 00000000 00000115 [0000188528651386] 0.302ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#278) cc1plus 3510 0 3 00000000 00000116 [0000188528651843] 0.302ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#279) cc1plus 3510 0 3 00000001 00000117 [0000188528652284] 0.303ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#280) cc1plus 3510 0 3 00000002 00000118 [0000188528653099] 0.304ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#281) cc1plus 3510 0 3 00000002 00000119 [0000188528653508] 0.305ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#282) cc1plus 3510 0 3 00000002 0000011a [0000188528653895] 0.306ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#283) cc1plus 3510 0 3 00000002 0000011b [0000188528654291] 0.306ms (+0.000ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#284) [ => cc1plus ] 0.307ms (+0.000ms)
(T1/#285) <...> 2 0 1 00000002 0000011d [0000188528655354] 0.308ms (+0.000ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#286) <...>-2 0d..2 309µs : __schedule+0x2ea/0x630 <c027cc9a> <cc1plus-3510> (7d 69):
(T1/#287) <...> 2 0 1 00000002 0000011f [0000188528656152] 0.310ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#288) <...> 2 0 1 00000001 00000120 [0000188528656584] 0.310ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#289) <...>-2 0d..1 311µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0):
(T1/#290) <...> 2 0 1 00000001 00000122 [0000188528657710] 0.312ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)
vim:ft=help
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 20:06 ` Hugh Dickins
2005-02-23 20:10 ` Lee Revell
2005-02-23 20:30 ` Lee Revell
@ 2005-02-23 20:53 ` Hugh Dickins
2005-02-23 22:13 ` Lee Revell
2 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 20:53 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 23 Feb 2005, Hugh Dickins wrote:
> Please replace by new patch below, which I'm now running through lmbench.
That second patch seems fine, and I see no lmbench regression from it.
Hugh
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 20:30 ` Lee Revell
@ 2005-02-23 21:03 ` Hugh Dickins
2005-02-23 22:14 ` Lee Revell
2005-02-23 23:27 ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
0 siblings, 2 replies; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 21:03 UTC (permalink / raw)
To: Lee Revell; +Cc: Nick Piggin, Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 23 Feb 2005, Lee Revell wrote:
> > >
> > > Thanks, your patch fixes the copy_pte_range latency.
>
> clear_page_range is also problematic.
Yes, I saw that from your other traces too. I know there are plans
to improve clear_page_range during 2.6.12, but I didn't realize that
it had become very much worse than its antecedent clear_page_tables,
and I don't see missing latency fixes for that. Nick's the expert.
Hugh
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 20:53 ` Hugh Dickins
@ 2005-02-23 22:13 ` Lee Revell
2005-02-24 4:56 ` Hugh Dickins
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 22:13 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > Please replace by new patch below, which I'm now running through lmbench.
>
> That second patch seems fine, and I see no lmbench regression from it.
Should go into 2.6.11, right?
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 21:03 ` Hugh Dickins
@ 2005-02-23 22:14 ` Lee Revell
2005-02-23 23:52 ` PPC RT Patch john cooper
2005-02-23 23:27 ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
1 sibling, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 22:14 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Nick Piggin, Ingo Molnar, Andrew Morton, linux-kernel
On Wed, 2005-02-23 at 21:03 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > > >
> > > > Thanks, your patch fixes the copy_pte_range latency.
> >
> > clear_page_range is also problematic.
>
> Yes, I saw that from your other traces too.
Heh, sorry, that one was a dupe... I should know to give the files
better names.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 21:03 ` Hugh Dickins
2005-02-23 22:14 ` Lee Revell
@ 2005-02-23 23:27 ` Nick Piggin
2005-02-24 1:03 ` Lee Revell
1 sibling, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-23 23:27 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Lee Revell, Ingo Molnar, Andrew Morton, linux-kernel
Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
>
>>>>Thanks, your patch fixes the copy_pte_range latency.
>>
>>clear_page_range is also problematic.
>
>
> Yes, I saw that from your other traces too. I know there are plans
> to improve clear_page_range during 2.6.12, but I didn't realize that
> it had become very much worse than its antecedent clear_page_tables,
> and I don't see missing latency fixes for that. Nick's the expert.
>
I wouldn't have thought it should have become worse, latency
wise. What is actually happening is that the lower level freeing
functions are being called more often. But this should result in
the work being spread out more, if anything. Rather than in the
old system things would tend to be batched up into bigger chunks
(typically at exit() time).
If you are using i386 with 2-level page tables (no highmem), then
the behaviour should be more or less identical. Odd.
Nick
^ permalink raw reply [flat|nested] 26+ messages in thread
* PPC RT Patch..
2005-02-23 22:14 ` Lee Revell
@ 2005-02-23 23:52 ` john cooper
2005-02-24 4:20 ` Frank Rowand
0 siblings, 1 reply; 26+ messages in thread
From: john cooper @ 2005-02-23 23:52 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel, john cooper, Saksena, Manas
[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]
Ingo,
We've had a PPC port of your RT work underway with
a focus on trace instrumentation. This is based upon
realtime-preempt-2.6.11-rc2-V0.7.37-02. A diff is
attached.
To the extent possible the tracing facilities are the
same as your x86 work. In the process a few PPC/gcc
issues needed to be resolved. There is also a bug fix
contained for tlb_gather_mmu() which was causing debug
assertions to be generated in a path which attempted to
sleep with a non-zero preempt count.
This does build and function when SMP is configured,
though we have not yet verified it on other than a
uniprocessor. As a simplifying assumption, testing has
thus far concentrated on the following modes:
PREEMPT_NONE
- verify baseline regression
PREEMPT_RT && !PREEMPT_SMP
- typical for an embedded RT PPC application
PREEMPT_RT && PREEMPT_SMP
- kicks in live locking code which otherwise receives no
coverage. This is functionally equivalent to the above
config on a single CPU target thus no MP dynamic testing
is achieved. Still quite useful IMHO.
The target used for development/testing is an Ampro EnCore PP1
which sports a 300Mhz MPC8245. For testing this boots with NFS
as root. An mp3 decode at nice --20 is launched which requires
just under 20% of the CPU to maintain an uninterrupted audio
decode and output. To this a series of "du -s /" are launched
to soak up excess CPU bandwidth. Perhaps not rigorous but a
fair sanity check and load for the purpose at hand.
Under these conditions maximum scheduling latencies are seen in
the 120-150us range. Note no attempt has yet been made to
optimize arch specific paths and full trace instrumentation has
been enabled.
I've written some logging code to help find problems such as
the tlb issue above. As it has not been made general I've
removed it from this patch. At some point I'll likely revisit
this.
Comments/suggestions welcome.
-john
--
john.cooper@timesys.com
[-- Attachment #2: realtime-preempt-2.6.11-rc2-V0.7.37-02-ppc --]
[-- Type: text/plain, Size: 39151 bytes --]
--- ./arch/ppc/syslib/i8259.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/syslib/i8259.c 2005-02-10 12:57:45.000000000 -0500
@@ -10,7 +10,7 @@ unsigned char cached_8259[2] = { 0xff, 0
#define cached_A1 (cached_8259[0])
#define cached_21 (cached_8259[1])
-static DEFINE_SPINLOCK(i8259_lock);
+static DEFINE_RAW_SPINLOCK(i8259_lock);
int i8259_pic_irq_offset;
=================================================================
--- ./arch/ppc/syslib/ocp.c.ORG 2004-12-24 16:35:23.000000000 -0500
+++ ./arch/ppc/syslib/ocp.c 2005-02-23 16:51:04.009548560 -0500
@@ -49,7 +49,6 @@
#include <asm/io.h>
#include <asm/ocp.h>
#include <asm/errno.h>
-#include <asm/rwsem.h>
#include <asm/semaphore.h>
//#define DBG(x) printk x
=================================================================
--- ./arch/ppc/kernel/Makefile.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/Makefile 2005-02-18 20:34:07.000000000 -0500
@@ -13,7 +13,7 @@ extra-y += vmlinux.lds
obj-y := entry.o traps.o irq.o idle.o time.o misc.o \
process.o signal.o ptrace.o align.o \
- semaphore.o syscalls.o setup.o \
+ syscalls.o setup.o \
cputable.o ppc_htab.o perfmon.o
obj-$(CONFIG_6xx) += l2cr.o cpu_setup_6xx.o
obj-$(CONFIG_POWER4) += cpu_setup_power4.o
@@ -26,6 +26,9 @@ obj-$(CONFIG_TAU) += temp.o
obj-$(CONFIG_ALTIVEC) += vecemu.o vector.o
obj-$(CONFIG_FSL_BOOKE) += perfmon_fsl_booke.o
+obj-$(CONFIG_ASM_SEMAPHORES) += semaphore.o
+obj-$(CONFIG_MCOUNT) += mcount.o
+
ifndef CONFIG_MATH_EMULATION
obj-$(CONFIG_8xx) += softemu8xx.o
endif
=================================================================
--- ./arch/ppc/kernel/signal.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/signal.c 2005-02-09 19:16:25.000000000 -0500
@@ -704,6 +704,13 @@ int do_signal(sigset_t *oldset, struct p
unsigned long frame, newsp;
int signr, ret;
+#ifdef CONFIG_PREEMPT_RT
+ /*
+ * Fully-preemptible kernel does not need interrupts disabled:
+ */
+ local_irq_enable();
+ preempt_check_resched();
+#endif
if (!oldset)
oldset = ¤t->blocked;
=================================================================
--- ./arch/ppc/kernel/time.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/time.c 2005-02-19 10:46:17.000000000 -0500
@@ -91,7 +91,7 @@ extern unsigned long wall_jiffies;
static long time_offset;
-DEFINE_SPINLOCK(rtc_lock);
+DEFINE_RAW_SPINLOCK(rtc_lock);
EXPORT_SYMBOL(rtc_lock);
@@ -109,7 +109,7 @@ static inline int tb_delta(unsigned *jif
}
#ifdef CONFIG_SMP
-unsigned long profile_pc(struct pt_regs *regs)
+unsigned long notrace profile_pc(struct pt_regs *regs)
{
unsigned long pc = instruction_pointer(regs);
@@ -126,7 +126,7 @@ EXPORT_SYMBOL(profile_pc);
* with interrupts disabled.
* We set it up to overflow again in 1/HZ seconds.
*/
-void timer_interrupt(struct pt_regs * regs)
+void notrace timer_interrupt(struct pt_regs * regs)
{
int next_dec;
unsigned long cpu = smp_processor_id();
=================================================================
--- ./arch/ppc/kernel/traps.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/traps.c 2005-02-09 18:59:40.000000000 -0500
@@ -72,7 +72,7 @@ void (*debugger_fault_handler)(struct pt
* Trap & Exception support
*/
-DEFINE_SPINLOCK(die_lock);
+DEFINE_RAW_SPINLOCK(die_lock);
void die(const char * str, struct pt_regs * fp, long err)
{
@@ -111,6 +111,10 @@ void _exception(int signr, struct pt_reg
debugger(regs);
die("Exception in kernel mode", regs, signr);
}
+#ifdef CONFIG_PREEMPT_RT
+ local_irq_enable();
+ preempt_check_resched();
+#endif
info.si_signo = signr;
info.si_errno = 0;
info.si_code = code;
=================================================================
--- ./arch/ppc/kernel/ppc_ksyms.c.ORG 2004-12-24 16:35:28.000000000 -0500
+++ ./arch/ppc/kernel/ppc_ksyms.c 2005-02-08 12:00:56.000000000 -0500
@@ -294,9 +294,11 @@ EXPORT_SYMBOL(console_drivers);
EXPORT_SYMBOL(xmon);
EXPORT_SYMBOL(xmon_printf);
#endif
+#ifdef CONFIG_ASM_SEMAPHORES
EXPORT_SYMBOL(__up);
EXPORT_SYMBOL(__down);
EXPORT_SYMBOL(__down_interruptible);
+#endif
#if defined(CONFIG_KGDB) || defined(CONFIG_XMON)
extern void (*debugger)(struct pt_regs *regs);
=================================================================
--- ./arch/ppc/kernel/entry.S.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/entry.S 2005-02-23 16:30:55.924205448 -0500
@@ -8,6 +8,7 @@
* rewritten by Paul Mackerras.
* Copyright (C) 1996 Paul Mackerras.
* MPC8xx modifications Copyright (C) 1997 Dan Malek (dmalek@jlc.net).
+ * RT_PREEMPT support (C) Timesys Corp. <john.cooper@timesys.com>
*
* This file contains the system call entry code, context switch
* code, and exception/interrupt return code for PowerPC.
@@ -44,6 +45,48 @@
#define LOAD_MSR_KERNEL(r, x) li r,(x)
#endif
+#if defined(CONFIG_LATENCY_TRACE) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING) \
+ || defined(CONFIG_CRITICAL_TIMING)
+#define TFSIZE 64 /* frame SHDB multiple of 16 bytes */
+#define TFR12 48 /* TODO: prune this down -- safe but overkill */
+#define TFR11 44
+#define TFR10 40
+#define TFR9 36
+#define TFR8 32
+#define TFR7 28
+#define TFR6 24
+#define TFR5 20
+#define TFR4 16
+#define TFR3 12
+#define TFR0 8
+#define PUSHFRAME() \
+ stwu r1, -TFSIZE(r1); \
+ stw r12, TFR12(r1); \
+ stw r11, TFR11(r1); \
+ stw r10, TFR10(r1); \
+ stw r9, TFR9(r1); \
+ stw r8, TFR8(r1); \
+ stw r7, TFR7(r1); \
+ stw r6, TFR6(r1); \
+ stw r5, TFR5(r1); \
+ stw r4, TFR4(r1); \
+ stw r3, TFR3(r1); \
+ stw r0, TFR0(r1)
+#define POPFRAME() \
+ lwz r12, TFR12(r1); \
+ lwz r11, TFR11(r1); \
+ lwz r10, TFR10(r1); \
+ lwz r9, TFR9(r1); \
+ lwz r8, TFR8(r1); \
+ lwz r7, TFR7(r1); \
+ lwz r6, TFR6(r1); \
+ lwz r5, TFR5(r1); \
+ lwz r4, TFR4(r1); \
+ lwz r3, TFR3(r1); \
+ lwz r0, TFR0(r1); \
+ addi r1, r1, TFSIZE
+#endif
+
#ifdef CONFIG_BOOKE
#define COR r8 /* Critical Offset Register (COR) */
#define BOOKE_LOAD_COR lis COR,crit_save@ha
@@ -197,6 +240,20 @@ _GLOBAL(DoSyscall)
lwz r11,_CCR(r1) /* Clear SO bit in CR */
rlwinm r11,r11,0,4,2
stw r11,_CCR(r1)
+#ifdef CONFIG_LATENCY_TRACE
+ lwz r3,GPR0(r1)
+ lwz r4,GPR3(r1)
+ lwz r5,GPR4(r1)
+ lwz r6,GPR5(r1)
+ bl sys_call
+ lwz r0,GPR0(r1)
+ lwz r3,GPR3(r1)
+ lwz r4,GPR4(r1)
+ lwz r5,GPR5(r1)
+ lwz r6,GPR6(r1)
+ lwz r7,GPR7(r1)
+ lwz r8,GPR8(r1)
+#endif
#ifdef SHOW_SYSCALLS
bl do_show_syscall
#endif /* SHOW_SYSCALLS */
@@ -250,6 +307,21 @@ syscall_exit_cont:
andis. r10,r0,DBCR0_IC@h
bnel- load_dbcr0
#endif
+#if defined(CONFIG_LATENCY_TRACE) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING) \
+ || defined(CONFIG_CRITICAL_TIMING)
+ PUSHFRAME();
+#ifdef CONFIG_CRITICAL_TIMING
+ bl touch_critical_timing
+#endif
+#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING
+ bl trace_irqs_on
+#endif
+#ifdef CONFIG_LATENCY_TRACE
+ lwz r3, RESULT+TFSIZE(r1)
+ bl sys_ret
+#endif
+ POPFRAME();
+#endif
stwcx. r0,0,r1 /* to clear the reservation */
lwz r4,_LINK(r1)
lwz r5,_CCR(r1)
@@ -614,32 +686,38 @@ restore_user:
/* N.B. the only way to get here is from the beq following ret_from_except. */
resume_kernel:
+ lis r9,kernel_preemption@ha
+ lwz r9,kernel_preemption@l(r9)
+ cmpwi 0,r9,0
+ beq restore
/* check current_thread_info->preempt_count */
rlwinm r9,r1,0,0,18
lwz r0,TI_PREEMPT(r9)
cmpwi 0,r0,0 /* if non-zero, just restore regs and return */
bne restore
+check_resched:
lwz r0,TI_FLAGS(r9)
andi. r0,r0,_TIF_NEED_RESCHED
beq+ restore
andi. r0,r3,MSR_EE /* interrupts off? */
beq restore /* don't schedule if so */
-1: lis r0,PREEMPT_ACTIVE@h
- stw r0,TI_PREEMPT(r9)
+1:
ori r10,r10,MSR_EE
- SYNC
- MTMSRD(r10) /* hard-enable interrupts */
- bl schedule
LOAD_MSR_KERNEL(r10,MSR_KERNEL)
SYNC
MTMSRD(r10) /* disable interrupts */
+ bl preempt_schedule_irq
rlwinm r9,r1,0,0,18
li r0,0
stw r0,TI_PREEMPT(r9)
+#if 0
lwz r3,TI_FLAGS(r9)
andi. r0,r3,_TIF_NEED_RESCHED
bne- 1b
#else
+ b check_resched
+#endif
+#else
resume_kernel:
#endif /* CONFIG_PREEMPT */
=================================================================
--- ./arch/ppc/kernel/process.c.ORG 2004-12-24 16:34:45.000000000 -0500
+++ ./arch/ppc/kernel/process.c 2005-02-09 18:43:14.000000000 -0500
@@ -360,6 +360,7 @@ void show_regs(struct pt_regs * regs)
print_symbol("%s\n", regs->nip);
printk("LR [%08lx] ", regs->link);
print_symbol("%s\n", regs->link);
+ printk("preempt: %08x\n", preempt_count());
#endif
show_stack(current, (unsigned long *) regs->gpr[1]);
}
=================================================================
--- ./arch/ppc/kernel/mcount.S.ORG 2005-02-18 19:51:33.000000000 -0500
+++ ./arch/ppc/kernel/mcount.S 2005-02-23 14:25:18.780025528 -0500
@@ -0,0 +1,86 @@
+/*
+ * linux/arch/ppc/kernel/mcount.S
+ *
+ * Copyright (C) 2005 TimeSys Corporation, john.cooper@timesys.com
+ */
+
+#include <asm/ppc_asm.h>
+
+/*
+ * stack frame in effect when calling __mcount():
+ *
+ * 52: RA to caller
+ * 48: caller chain
+ * 44: [alignment pad]
+ * 40: saved LR to prolog/target function
+ * 36: r10
+ * 32: r9
+ * 28: r8
+ * 24: r7
+ * 20: r6
+ * 16: r5
+ * 12: r4
+ * 8: r3
+ * 4: LR save for __mcount() use
+ * 0: frame chain pointer / current r1
+ */
+
+/* preamble present in each traced function:
+ *
+ * .data
+ * .align 2
+ * 0:
+ * .long 0
+ * .previous
+ * mflr r0
+ * lis r11, 0b@ha
+ * stw r0, 4(r1)
+ * addi r0, r11, 0b@l
+ * bl _mcount
+ */
+
+ .text
+.globl _mcount
+_mcount:
+ lis r11,mcount_enabled@ha
+ lwz r11,mcount_enabled@l(r11)
+ cmpwi 0,r11,0
+ beq disabled
+ stwu r1,-48(r1) /* local frame */
+ stw r3, 8(r1)
+ stw r4, 12(r1)
+ stw r5, 16(r1)
+ stw r6, 20(r1)
+ mflr r4 /* RA to function prolog */
+ stw r7, 24(r1)
+ stw r8, 28(r1)
+ stw r9, 32(r1)
+ stw r10,36(r1)
+ stw r4, 40(r1)
+ bl __mcount /* __void mcount(void) */
+ nop
+ lwz r0, 40(r1)
+ lwz r3, 8(r1)
+ mtctr r0
+ lwz r4, 12(r1)
+ lwz r5, 16(r1)
+ lwz r6, 20(r1)
+ lwz r0, 52(r1) /* RA to function caller */
+ lwz r7, 24(r1)
+ lwz r8, 28(r1)
+ mtlr r0
+ lwz r9, 32(r1)
+ lwz r10,36(r1)
+ addi r1,r1,48 /* toss frame */
+ bctr /* return to target function */
+
+/* the function preamble modified LR in getting here so we need
+ * to restore its LR and return to the preamble otherwise
+ */
+disabled:
+ mflr r12 /* return address to preamble */
+ lwz r11, 4(r1)
+ mtlr r11 /* restore LR modified by preamble */
+ mtctr r12
+ bctr
+
=================================================================
--- ./arch/ppc/kernel/smp.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/smp.c 2005-02-22 10:21:39.000000000 -0500
@@ -163,7 +163,7 @@ void smp_send_stop(void)
* static memory requirements. It also looks cleaner.
* Stolen from the i386 version.
*/
-static DEFINE_SPINLOCK(call_lock);
+static DEFINE_RAW_SPINLOCK(call_lock);
static struct call_data_struct {
void (*func) (void *info);
@@ -397,5 +397,15 @@ int __cpu_up(unsigned int cpu)
void smp_cpus_done(unsigned int max_cpus)
{
- smp_ops->setup_cpu(0);
+ if (smp_ops)
+ smp_ops->setup_cpu(0);
+}
+
+/* this function sends a 'reschedule' IPI to all other CPUs.
+ * This is used when RT tasks are starving and other CPUs
+ * might be able to run them
+ */
+void smp_send_reschedule_allbutself(void)
+{
+ smp_message_pass(MSG_ALL_BUT_SELF, PPC_MSG_RESCHEDULE, 0, 0);
}
=================================================================
--- ./arch/ppc/kernel/irq.c.ORG 2004-12-24 16:35:24.000000000 -0500
+++ ./arch/ppc/kernel/irq.c 2005-02-19 21:43:53.000000000 -0500
@@ -135,10 +135,11 @@ skip:
return 0;
}
-void do_IRQ(struct pt_regs *regs)
+void notrace do_IRQ(struct pt_regs *regs)
{
int irq, first = 1;
irq_enter();
+ trace_special(regs->nip, irq, 0);
/*
* Every platform is required to implement ppc_md.get_irq.
=================================================================
--- ./arch/ppc/kernel/idle.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/idle.c 2005-02-23 13:02:34.000000000 -0500
@@ -39,6 +39,7 @@ void default_idle(void)
powersave = ppc_md.power_save;
if (!need_resched()) {
+ stop_critical_timing();
if (powersave != NULL)
powersave();
else {
=================================================================
--- ./arch/ppc/mm/fault.c.ORG 2004-12-24 16:34:29.000000000 -0500
+++ ./arch/ppc/mm/fault.c 2005-02-19 21:32:02.000000000 -0500
@@ -92,7 +92,7 @@ static int store_updates_sp(struct pt_re
* the error_code parameter is ESR for a data fault, 0 for an instruction
* fault.
*/
-int do_page_fault(struct pt_regs *regs, unsigned long address,
+int notrace do_page_fault(struct pt_regs *regs, unsigned long address,
unsigned long error_code)
{
struct vm_area_struct * vma;
@@ -104,6 +104,7 @@ int do_page_fault(struct pt_regs *regs,
#else
int is_write = 0;
+ trace_special(regs->nip, error_code, address);
/*
* Fortunately the bit assignments in SRR1 for an instruction
* fault and DSISR for a data fault are mostly the same for the
=================================================================
--- ./arch/ppc/mm/init.c.ORG 2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/mm/init.c 2005-02-21 13:26:10.000000000 -0500
@@ -56,7 +56,7 @@
#endif
#define MAX_LOW_MEM CONFIG_LOWMEM_SIZE
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU_LOCKED(struct mmu_gather, mmu_gathers);
unsigned long total_memory;
unsigned long total_lowmem;
=================================================================
--- ./arch/ppc/boot/common/util.S.ORG 2004-12-24 16:35:49.000000000 -0500
+++ ./arch/ppc/boot/common/util.S 2005-02-23 14:27:14.577421648 -0500
@@ -289,5 +289,15 @@ _GLOBAL(flush_data_cache)
bdnz 00b
10: blr
- .previous
+#ifdef CONFIG_MCOUNT
+/* innocuous _mcount for boot header
+ */
+_GLOBAL(_mcount)
+ mflr r12 /* return address to preamble */
+ lwz r11, 4(r1)
+ mtlr r11 /* restore LR modified by preamble */
+ mtctr r12
+ bctr
+#endif
+ .previous
=================================================================
--- ./arch/ppc/platforms/encpp1_time.c.ORG 2005-02-02 20:42:55.000000000 -0500
+++ ./arch/ppc/platforms/encpp1_time.c 2005-02-09 16:35:10.000000000 -0500
@@ -155,9 +155,9 @@ int pp1_set_rtc_time (unsigned long nowt
{
unsigned char save_control, save_freq_select;
struct rtc_time tm;
- extern spinlock_t rtc_lock;
+ extern raw_spinlock_t rtc_lock;
- spin_lock (&rtc_lock);
+ __raw_spin_lock (&rtc_lock);
to_tm (nowtime, &tm);
called ();
@@ -202,7 +202,7 @@ int pp1_set_rtc_time (unsigned long nowt
if ((time_state == TIME_ERROR) || (time_state == TIME_BAD))
time_state = TIME_OK;
- spin_unlock (&rtc_lock);
+ __raw_spin_unlock (&rtc_lock);
return 0;
}
=================================================================
--- ./arch/ppc/lib/locks.c.ORG 2004-12-24 16:34:26.000000000 -0500
+++ ./arch/ppc/lib/locks.c 2005-02-07 19:21:14.000000000 -0500
@@ -8,6 +8,7 @@
#include <linux/sched.h>
#include <linux/spinlock.h>
#include <linux/module.h>
+#include <linux/rt_lock.h>
#include <asm/ppc_asm.h>
#include <asm/smp.h>
@@ -43,7 +44,7 @@ static inline unsigned long __spin_trylo
return ret;
}
-void _raw_spin_lock(spinlock_t *lock)
+void __raw_spin_lock(raw_spinlock_t *lock)
{
int cpu = smp_processor_id();
unsigned int stuck = INIT_STUCK;
@@ -63,9 +64,9 @@ void _raw_spin_lock(spinlock_t *lock)
lock->owner_pc = (unsigned long)__builtin_return_address(0);
lock->owner_cpu = cpu;
}
-EXPORT_SYMBOL(_raw_spin_lock);
+EXPORT_SYMBOL(__raw_spin_lock);
-int _raw_spin_trylock(spinlock_t *lock)
+int __raw_spin_trylock(raw_spinlock_t *lock)
{
if (__spin_trylock(&lock->lock))
return 0;
@@ -73,9 +74,9 @@ int _raw_spin_trylock(spinlock_t *lock)
lock->owner_pc = (unsigned long)__builtin_return_address(0);
return 1;
}
-EXPORT_SYMBOL(_raw_spin_trylock);
+EXPORT_SYMBOL(__raw_spin_trylock);
-void _raw_spin_unlock(spinlock_t *lp)
+void __raw_spin_unlock(raw_spinlock_t *lp)
{
if ( !lp->lock )
printk("_spin_unlock(%p): no lock cpu %d curr PC %p %s/%d\n",
@@ -89,7 +90,7 @@ void _raw_spin_unlock(spinlock_t *lp)
wmb();
lp->lock = 0;
}
-EXPORT_SYMBOL(_raw_spin_unlock);
+EXPORT_SYMBOL(__raw_spin_unlock);
/*
@@ -97,7 +98,7 @@ EXPORT_SYMBOL(_raw_spin_unlock);
* with the high bit (sign) being the "write" bit.
* -- Cort
*/
-void _raw_read_lock(rwlock_t *rw)
+void __raw_read_lock(raw_rwlock_t *rw)
{
unsigned long stuck = INIT_STUCK;
int cpu = smp_processor_id();
@@ -123,9 +124,9 @@ again:
}
wmb();
}
-EXPORT_SYMBOL(_raw_read_lock);
+EXPORT_SYMBOL(__raw_read_lock);
-void _raw_read_unlock(rwlock_t *rw)
+void __raw_read_unlock(raw_rwlock_t *rw)
{
if ( rw->lock == 0 )
printk("_read_unlock(): %s/%d (nip %08lX) lock %lx\n",
@@ -134,9 +135,9 @@ void _raw_read_unlock(rwlock_t *rw)
wmb();
atomic_dec((atomic_t *) &(rw)->lock);
}
-EXPORT_SYMBOL(_raw_read_unlock);
+EXPORT_SYMBOL(__raw_read_unlock);
-void _raw_write_lock(rwlock_t *rw)
+void __raw_write_lock(raw_rwlock_t *rw)
{
unsigned long stuck = INIT_STUCK;
int cpu = smp_processor_id();
@@ -175,9 +176,9 @@ again:
}
wmb();
}
-EXPORT_SYMBOL(_raw_write_lock);
+EXPORT_SYMBOL(__raw_write_lock);
-int _raw_write_trylock(rwlock_t *rw)
+int __raw_write_trylock(raw_rwlock_t *rw)
{
if (test_and_set_bit(31, &(rw)->lock)) /* someone has a write lock */
return 0;
@@ -190,9 +191,9 @@ int _raw_write_trylock(rwlock_t *rw)
wmb();
return 1;
}
-EXPORT_SYMBOL(_raw_write_trylock);
+EXPORT_SYMBOL(__raw_write_trylock);
-void _raw_write_unlock(rwlock_t *rw)
+void __raw_write_unlock(raw_rwlock_t *rw)
{
if ( !(rw->lock & (1<<31)) )
printk("_write_lock(): %s/%d (nip %08lX) lock %lx\n",
@@ -201,6 +202,6 @@ void _raw_write_unlock(rwlock_t *rw)
wmb();
clear_bit(31,&(rw)->lock);
}
-EXPORT_SYMBOL(_raw_write_unlock);
+EXPORT_SYMBOL(__raw_write_unlock);
#endif
=================================================================
--- ./arch/ppc/lib/dec_and_lock.c.ORG 2004-12-24 16:35:27.000000000 -0500
+++ ./arch/ppc/lib/dec_and_lock.c 2005-02-07 21:25:52.000000000 -0500
@@ -19,7 +19,7 @@
*/
#ifndef ATOMIC_DEC_AND_LOCK
-int _atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock)
+int _atomic_dec_and_raw_spin_lock(atomic_t *atomic, raw_spinlock_t *lock)
{
int counter;
int newcount;
@@ -35,12 +35,12 @@ int _atomic_dec_and_lock(atomic_t *atomi
return 0;
}
- spin_lock(lock);
+ _raw_spin_lock(lock);
if (atomic_dec_and_test(atomic))
return 1;
- spin_unlock(lock);
+ _raw_spin_unlock(lock);
return 0;
}
-EXPORT_SYMBOL(_atomic_dec_and_lock);
+EXPORT_SYMBOL(_atomic_dec_and_raw_spin_lock);
#endif /* ATOMIC_DEC_AND_LOCK */
=================================================================
--- ./arch/ppc/Kconfig.ORG 2005-02-02 20:42:55.000000000 -0500
+++ ./arch/ppc/Kconfig 2005-02-08 19:54:51.000000000 -0500
@@ -15,13 +15,6 @@ config GENERIC_HARDIRQS
bool
default y
-config RWSEM_GENERIC_SPINLOCK
- bool
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y
-
config GENERIC_CALIBRATE_DELAY
bool
default y
@@ -866,15 +859,21 @@ config NR_CPUS
depends on SMP
default "4"
-config PREEMPT
- bool "Preemptible Kernel"
- help
- This option reduces the latency of the kernel when reacting to
- real-time or interactive events by allowing a low priority process to
- be preempted even if it is in kernel mode executing a system call.
+source "lib/Kconfig.RT"
- Say Y here if you are building a kernel for a desktop, embedded
- or real-time system. Say N if you are unsure.
+config RWSEM_GENERIC_SPINLOCK
+ bool
+ depends on !PREEMPT_RT
+ default y
+
+config ASM_SEMAPHORES
+ bool
+ depends on !PREEMPT_RT
+ default y
+
+config RWSEM_XCHGADD_ALGORITHM
+ bool
+ depends on !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT
config HIGHMEM
bool "High memory support"
=================================================================
--- ./include/asm-generic/tlb.h.ORG 2005-02-01 16:26:51.000000000 -0500
+++ ./include/asm-generic/tlb.h 2005-02-21 13:43:21.000000000 -0500
@@ -50,7 +50,8 @@ struct mmu_gather {
#define tlb_mm(tlb) ((tlb)->mm)
/* Users of the generic TLB shootdown code must declare this storage space. */
-DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
+
+DECLARE_PER_CPU_LOCKED(struct mmu_gather, mmu_gathers);
/* tlb_gather_mmu
* Return a pointer to an initialized struct mmu_gather.
@@ -58,7 +59,8 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_g
static inline struct mmu_gather *
tlb_gather_mmu(struct mm_struct *mm, unsigned int full_mm_flush)
{
- struct mmu_gather *tlb = &get_cpu_var(mmu_gathers);
+ struct mmu_gather *tlb = &get_cpu_var_locked(mmu_gathers,
+ _smp_processor_id());
tlb->mm = mm;
@@ -99,7 +101,7 @@ tlb_finish_mmu(struct mmu_gather *tlb, u
freed = rss;
mm->rss = rss - freed;
tlb_flush_mmu(tlb, start, end);
- put_cpu_var(mmu_gathers);
+ put_cpu_var_locked(mmu_gathers, _smp_processor_id());
/* keep the page table cache within bounds */
check_pgt_cache();
=================================================================
--- ./include/asm-generic/percpu.h.ORG 2005-02-01 16:26:51.000000000 -0500
+++ ./include/asm-generic/percpu.h 2005-02-21 13:11:53.000000000 -0500
@@ -53,6 +53,9 @@ do { \
#endif /* SMP */
#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#define DECLARE_PER_CPU_LOCKED(type, name) \
+ extern __typeof__(type) per_cpu__##name##_locked; \
+ extern spinlock_t per_cpu_lock__##name##_locked
#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
=================================================================
--- ./include/asm-ppc/rtc.h.ORG 2004-12-24 16:33:49.000000000 -0500
+++ ./include/asm-ppc/rtc.h 2005-02-23 14:30:06.705254240 -0500
@@ -24,6 +24,11 @@
#ifndef __ASM_RTC_H__
#define __ASM_RTC_H__
+#ifdef CONFIG_ENCPP1 /* Ampro work-around. Ugh. */
+#define cpu_mhz 300
+#define cpu_khz (cpu_mhz * 1000)
+#endif
+
#ifdef __KERNEL__
#include <linux/rtc.h>
=================================================================
--- ./include/asm-ppc/semaphore.h.ORG 2004-12-24 16:34:57.000000000 -0500
+++ ./include/asm-ppc/semaphore.h 2005-02-08 11:39:18.000000000 -0500
@@ -18,6 +18,13 @@
#include <asm/atomic.h>
#include <asm/system.h>
+
+#ifdef CONFIG_PREEMPT_RT
+
+#include <linux/rt_lock.h>
+
+#else
+
#include <linux/wait.h>
#include <linux/rwsem.h>
@@ -108,4 +115,8 @@ extern inline void up(struct semaphore *
#endif /* __KERNEL__ */
+extern int FASTCALL(sem_is_locked(struct semaphore *sem));
+
+#endif /* CONFIG_PREEMPT_RT */
+
#endif /* !(_PPC_SEMAPHORE_H) */
=================================================================
--- ./include/asm-ppc/spinlock.h.ORG 2005-02-01 16:26:45.000000000 -0500
+++ ./include/asm-ppc/spinlock.h 2005-02-22 09:50:38.000000000 -0500
@@ -7,17 +7,6 @@
* Simple spin lock operations.
*/
-typedef struct {
- volatile unsigned long lock;
-#ifdef CONFIG_DEBUG_SPINLOCK
- volatile unsigned long owner_pc;
- volatile unsigned long owner_cpu;
-#endif
-#ifdef CONFIG_PREEMPT
- unsigned int break_lock;
-#endif
-} spinlock_t;
-
#ifdef __KERNEL__
#ifdef CONFIG_DEBUG_SPINLOCK
#define SPINLOCK_DEBUG_INIT , 0, 0
@@ -25,16 +14,19 @@ typedef struct {
#define SPINLOCK_DEBUG_INIT /* */
#endif
-#define SPIN_LOCK_UNLOCKED (spinlock_t) { 0 SPINLOCK_DEBUG_INIT }
+#define __RAW_SPIN_LOCK_UNLOCKED { 0 SPINLOCK_DEBUG_INIT }
+#define RAW_SPIN_LOCK_UNLOCKED (raw_spinlock_t) __RAW_SPIN_LOCK_UNLOCKED
-#define spin_lock_init(x) do { *(x) = SPIN_LOCK_UNLOCKED; } while(0)
-#define spin_is_locked(x) ((x)->lock != 0)
-#define spin_unlock_wait(x) do { barrier(); } while(spin_is_locked(x))
-#define _raw_spin_lock_flags(lock, flags) _raw_spin_lock(lock)
+#define __raw_spin_lock_init(x) \
+ do { *(x) = RAW_SPIN_LOCK_UNLOCKED; } while(0)
+#define __raw_spin_is_locked(x) ((x)->lock != 0)
+#define __raw_spin_unlock_wait(x) \
+ do { barrier(); } while(__raw_spin_is_locked(x))
+#define __raw_spin_lock_flags(lock, flags) __raw_spin_lock(lock)
#ifndef CONFIG_DEBUG_SPINLOCK
-static inline void _raw_spin_lock(spinlock_t *lock)
+static inline void __raw_spin_lock(raw_spinlock_t *lock)
{
unsigned long tmp;
@@ -55,54 +47,37 @@ static inline void _raw_spin_lock(spinlo
: "cr0", "memory");
}
-static inline void _raw_spin_unlock(spinlock_t *lock)
+static inline void __raw_spin_unlock(raw_spinlock_t *lock)
{
__asm__ __volatile__("eieio # spin_unlock": : :"memory");
lock->lock = 0;
}
-#define _raw_spin_trylock(l) (!test_and_set_bit(0,&(l)->lock))
+#define __raw_spin_trylock(l) (!test_and_set_bit(0,&(l)->lock))
#else
-extern void _raw_spin_lock(spinlock_t *lock);
-extern void _raw_spin_unlock(spinlock_t *lock);
-extern int _raw_spin_trylock(spinlock_t *lock);
-
-#endif
+extern void __raw_spin_lock(raw_spinlock_t *lock);
+extern void __raw_spin_unlock(raw_spinlock_t *lock);
+extern int __raw_spin_trylock(raw_spinlock_t *lock);
-/*
- * Read-write spinlocks, allowing multiple readers
- * but only one writer.
- *
- * NOTE! it is quite common to have readers in interrupts
- * but no interrupt writers. For those circumstances we
- * can "mix" irq-safe locks - any writer needs to get a
- * irq-safe write-lock, but readers can get non-irqsafe
- * read-locks.
- */
-typedef struct {
- volatile unsigned long lock;
-#ifdef CONFIG_DEBUG_SPINLOCK
- volatile unsigned long owner_pc;
-#endif
-#ifdef CONFIG_PREEMPT
- unsigned int break_lock;
-#endif
-} rwlock_t;
+#endif /* CONFIG_DEBUG_SPINLOCK */
#ifdef CONFIG_DEBUG_SPINLOCK
-#define RWLOCK_DEBUG_INIT , 0
+#define RAW_RWLOCK_DEBUG_INIT , 0
#else
-#define RWLOCK_DEBUG_INIT /* */
+#define RAW_RWLOCK_DEBUG_INIT /* */
#endif
-#define RW_LOCK_UNLOCKED (rwlock_t) { 0 RWLOCK_DEBUG_INIT }
-#define rwlock_init(lp) do { *(lp) = RW_LOCK_UNLOCKED; } while(0)
+#define __RAW_RW_LOCK_UNLOCKED { 0 RAW_RWLOCK_DEBUG_INIT }
+#define RAW_RW_LOCK_UNLOCKED (raw_rwlock_t) __RAW_RW_LOCK_UNLOCKED
+#define __raw_rwlock_init(lp) do { *(lp) = RAW_RW_LOCK_UNLOCKED; } while(0)
+#define __raw_read_can_lock(lp) (0 <= (lp)->lock)
+#define __raw_write_can_lock(lp) (!(lp)->lock)
#ifndef CONFIG_DEBUG_SPINLOCK
-static __inline__ void _raw_read_lock(rwlock_t *rw)
+static __inline__ void __raw_read_lock(raw_rwlock_t *rw)
{
unsigned int tmp;
@@ -123,7 +98,7 @@ static __inline__ void _raw_read_lock(rw
: "cr0", "memory");
}
-static __inline__ void _raw_read_unlock(rwlock_t *rw)
+static __inline__ void __raw_read_unlock(raw_rwlock_t *rw)
{
unsigned int tmp;
@@ -139,7 +114,7 @@ static __inline__ void _raw_read_unlock(
: "cr0", "memory");
}
-static __inline__ int _raw_write_trylock(rwlock_t *rw)
+static __inline__ int __raw_write_trylock(raw_rwlock_t *rw)
{
unsigned int tmp;
@@ -159,7 +134,7 @@ static __inline__ int _raw_write_trylock
return tmp == 0;
}
-static __inline__ void _raw_write_lock(rwlock_t *rw)
+static __inline__ void __raw_write_lock(raw_rwlock_t *rw)
{
unsigned int tmp;
@@ -180,7 +155,7 @@ static __inline__ void _raw_write_lock(r
: "cr0", "memory");
}
-static __inline__ void _raw_write_unlock(rwlock_t *rw)
+static __inline__ void __raw_write_unlock(raw_rwlock_t *rw)
{
__asm__ __volatile__("eieio # write_unlock": : :"memory");
rw->lock = 0;
@@ -188,15 +163,15 @@ static __inline__ void _raw_write_unlock
#else
-extern void _raw_read_lock(rwlock_t *rw);
-extern void _raw_read_unlock(rwlock_t *rw);
-extern void _raw_write_lock(rwlock_t *rw);
-extern void _raw_write_unlock(rwlock_t *rw);
-extern int _raw_write_trylock(rwlock_t *rw);
+extern void __raw_read_lock(raw_rwlock_t *rw);
+extern void __raw_read_unlock(raw_rwlock_t *rw);
+extern void __raw_write_lock(raw_rwlock_t *rw);
+extern void __raw_write_unlock(raw_rwlock_t *rw);
+extern int __raw_write_trylock(raw_rwlock_t *rw);
#endif
-#define _raw_read_trylock(lock) generic_raw_read_trylock(lock)
+#define __raw_read_trylock(lock) generic_raw_read_trylock(lock)
#endif /* __ASM_SPINLOCK_H */
#endif /* __KERNEL__ */
=================================================================
--- ./include/asm-ppc/hw_irq.h.ORG 2004-12-24 16:35:15.000000000 -0500
+++ ./include/asm-ppc/hw_irq.h 2005-02-23 15:55:45.653015432 -0500
@@ -13,8 +13,17 @@ extern void timer_interrupt(struct pt_re
#define INLINE_IRQS
#define irqs_disabled() ((mfmsr() & MSR_EE) == 0)
+#define irqs_disabled_flags(f) (!((f) & MSR_EE))
-#ifdef INLINE_IRQS
+#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING
+ extern void notrace trace_irqs_off(void);
+ extern void notrace trace_irqs_on(void);
+#else
+# define trace_irqs_off() do { } while (0)
+# define trace_irqs_on() do { } while (0)
+#endif
+
+#if defined(INLINE_IRQS) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING)
static inline void local_irq_disable(void)
{
@@ -22,11 +31,14 @@ static inline void local_irq_disable(voi
msr = mfmsr();
mtmsr(msr & ~MSR_EE);
__asm__ __volatile__("": : :"memory");
+ trace_irqs_off();
}
static inline void local_irq_enable(void)
{
unsigned long msr;
+
+ trace_irqs_on();
__asm__ __volatile__("": : :"memory");
msr = mfmsr();
mtmsr(msr | MSR_EE);
@@ -39,11 +51,19 @@ static inline void local_irq_save_ptr(un
*flags = msr;
mtmsr(msr & ~MSR_EE);
__asm__ __volatile__("": : :"memory");
+ trace_irqs_off();
}
#define local_save_flags(flags) ((flags) = mfmsr())
#define local_irq_save(flags) local_irq_save_ptr(&flags)
-#define local_irq_restore(flags) mtmsr(flags)
+#define local_irq_restore(flags) \
+ do { \
+ if (irqs_disabled_flags(flags)) \
+ trace_irqs_off(); \
+ else \
+ trace_irqs_on(); \
+ mtmsr(flags); \
+ } while (0)
#else
=================================================================
--- ./include/asm-ppc/tlb.h.ORG 2004-12-24 16:34:58.000000000 -0500
+++ ./include/asm-ppc/tlb.h 2005-02-09 19:12:21.000000000 -0500
@@ -50,7 +50,11 @@ static inline void __tlb_remove_tlb_entr
#define tlb_flush(tlb) flush_tlb_mm((tlb)->mm)
/* Get the generic bits... */
+#ifdef CONFIG_PREEMPT_RT
+#include <asm-generic/tlb-simple.h>
+#else
#include <asm-generic/tlb.h>
+#endif
#endif /* CONFIG_PPC_STD_MMU */
=================================================================
--- ./include/asm-ppc/ocp.h.ORG 2004-12-24 16:34:26.000000000 -0500
+++ ./include/asm-ppc/ocp.h 2005-02-23 16:50:53.514144104 -0500
@@ -32,7 +32,6 @@
#include <asm/mmu.h>
#include <asm/ocp_ids.h>
-#include <asm/rwsem.h>
#include <asm/semaphore.h>
#ifdef CONFIG_PPC_OCP
=================================================================
--- ./include/linux/sched.h.ORG 2005-02-01 16:26:51.000000000 -0500
+++ ./include/linux/sched.h 2005-02-20 18:24:02.000000000 -0500
@@ -74,9 +74,18 @@ extern int debug_direct_keyboard;
#endif
#ifdef CONFIG_FRAME_POINTER
+#ifdef CONFIG_PPC
+# define __valid_ra(l) ((__builtin_frame_address(l) && \
+ *(unsigned long *)__builtin_frame_address(l)) ? \
+ (unsigned long)__builtin_return_address(l) : 0UL)
+# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
+# define CALLER_ADDR1 __valid_ra(1)
+# define CALLER_ADDR2 (__valid_ra(1) ? __valid_ra(2) : 0UL)
+#else
# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
# define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1))
# define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2))
+#endif
#else
# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
# define CALLER_ADDR1 0UL
@@ -84,9 +93,14 @@ extern int debug_direct_keyboard;
#endif
#ifdef CONFIG_MCOUNT
- extern void notrace mcount(void);
+#ifdef CONFIG_PPC
+# define ARCH_MCOUNT _mcount
+#else
+# define ARCH_MCOUNT mcount
+#endif
+ extern void notrace ARCH_MCOUNT(void);
#else
-# define mcount() do { } while (0)
+# define ARCH_MCOUNT() do { } while (0)
#endif
#ifdef CONFIG_LATENCY_TRACE
=================================================================
--- ./drivers/char/blocker.c.ORG 2005-02-01 16:26:48.000000000 -0500
+++ ./drivers/char/blocker.c 2005-02-02 17:16:24.000000000 -0500
@@ -4,6 +4,7 @@
#include <linux/fs.h>
#include <linux/miscdevice.h>
+#include <asm/time.h>
#define BLOCKER_MINOR 221
@@ -17,8 +18,18 @@ u64 notrace get_cpu_tick(void)
u64 tsc;
#ifdef ARCHARM
tsc = *oscr;
-#else
+#elif defined(CONFIG_X86)
__asm__ __volatile__("rdtsc" : "=A" (tsc));
+#elif defined(CONFIG_PPC)
+ unsigned long hi, lo;
+
+ do {
+ hi = get_tbu();
+ lo = get_tbl();
+ } while (get_tbu() != hi);
+ tsc = (u64)hi << 32 | lo;
+#else
+ #error Implement get_cpu_tick()
#endif
return tsc;
}
=================================================================
--- ./kernel/sched.c.ORG 2005-02-01 16:26:46.000000000 -0500
+++ ./kernel/sched.c 2005-02-23 14:37:51.325621208 -0500
@@ -1237,7 +1237,7 @@ int fastcall wake_up_process(task_t * p)
int ret = try_to_wake_up(p, TASK_STOPPED | TASK_TRACED |
TASK_RUNNING_MUTEX | TASK_INTERRUPTIBLE |
TASK_UNINTERRUPTIBLE, 0, 0);
- mcount();
+ ARCH_MCOUNT();
return ret;
}
@@ -1248,7 +1248,7 @@ int fastcall wake_up_process_mutex(task_
int ret = try_to_wake_up(p, TASK_STOPPED | TASK_TRACED |
TASK_RUNNING_MUTEX | TASK_INTERRUPTIBLE |
TASK_UNINTERRUPTIBLE, 0, 1);
- mcount();
+ ARCH_MCOUNT();
return ret;
}
@@ -1257,7 +1257,7 @@ EXPORT_SYMBOL(wake_up_process_mutex);
int fastcall wake_up_state(task_t *p, unsigned int state)
{
int ret = try_to_wake_up(p, state | TASK_RUNNING_MUTEX, 0, 0);
- mcount();
+ ARCH_MCOUNT();
return ret;
}
@@ -1502,7 +1502,11 @@ static void finish_task_switch(task_t *p
* schedule_tail - first thing a freshly forked thread must call.
* @prev: the thread we just switched away from.
*/
+#ifdef CONFIG_PPC
+asmlinkage notrace void schedule_tail(task_t *prev)
+#else
asmlinkage void schedule_tail(task_t *prev)
+#endif
__releases(rq->lock)
{
preempt_disable(); // TODO: move this to fork setup
=================================================================
--- ./kernel/latency.c.ORG 2005-02-01 16:26:46.000000000 -0500
+++ ./kernel/latency.c 2005-02-23 08:32:38.000000000 -0500
@@ -50,6 +50,12 @@ static __cacheline_aligned_in_smp struct
int wakeup_timing = 1;
#endif
+#ifdef NONASCII
+#define MU 'µ'
+#else
+#define MU 'u'
+#endif
+
#ifdef CONFIG_LATENCY_TIMING
/*
@@ -357,9 +363,9 @@ void notrace __trace(unsigned long eip,
___trace(TRACE_FN, eip, parent_eip, 0, 0, 0);
}
-extern void mcount(void);
+extern void ARCH_MCOUNT(void);
-EXPORT_SYMBOL(mcount);
+EXPORT_SYMBOL(ARCH_MCOUNT);
void notrace __mcount(void)
{
@@ -631,8 +637,8 @@ static void * notrace l_start(struct seq
if (!n) {
seq_printf(m, "preemption latency trace v1.1.4 on %s\n", UTS_RELEASE);
seq_puts(m, "--------------------------------------------------------------------\n");
- seq_printf(m, " latency: %lu µs, #%lu/%lu, CPU#%d | (M:%s VP:%d, KP:%d, SP:%d HP:%d #P:%d)\n",
- cycles_to_usecs(tr->saved_latency),
+ seq_printf(m, " latency: %lu %cs, #%lu/%lu, CPU#%d | (M:%s VP:%d, KP:%d, SP:%d HP:%d #P:%d)\n",
+ cycles_to_usecs(tr->saved_latency), MU,
entries, tr->trace_idx, out_tr.cpu,
#if defined(CONFIG_PREEMPT_NONE)
"server",
@@ -698,7 +704,7 @@ static void notrace l_stop(struct seq_fi
static void print_timestamp(struct seq_file *m, unsigned long abs_usecs,
unsigned long rel_usecs)
{
- seq_printf(m, " %4ldµs", abs_usecs);
+ seq_printf(m, " %4ld%cs", abs_usecs, MU);
if (rel_usecs > 100)
seq_puts(m, "!: ");
else if (rel_usecs > 1)
@@ -711,7 +717,7 @@ static void
print_timestamp_short(struct seq_file *m, unsigned long abs_usecs,
unsigned long rel_usecs)
{
- seq_printf(m, " %4ldµs", abs_usecs);
+ seq_printf(m, " %4ld%cs", abs_usecs, MU);
if (rel_usecs > 100)
seq_putc(m, '!');
else if (rel_usecs > 1)
@@ -1043,7 +1049,7 @@ static int setup_preempt_thresh(char *s)
get_option(&s, &thresh);
if (thresh > 0) {
preempt_thresh = usecs_to_cycles(thresh);
- printk("Preemption threshold = %u µs\n", thresh);
+ printk("Preemption threshold = %u %cs\n", thresh, MU);
}
return 1;
}
@@ -1091,18 +1097,18 @@ check_critical_timing(struct cpu_trace *
update_max_tr(tr);
if (preempt_thresh)
- printk("(%16s-%-5d|#%d): %lu µs critical section "
- "violates %lu µs threshold.\n"
+ printk("(%16s-%-5d|#%d): %lu %cs critical section "
+ "violates %lu %cs threshold.\n"
" => started at timestamp %lu: ",
current->comm, current->pid,
- _smp_processor_id(),
- latency, cycles_to_usecs(preempt_thresh), t0);
+ _smp_processor_id(), latency,
+ MU, cycles_to_usecs(preempt_thresh), MU, t0);
else
- printk("(%16s-%-5d|#%d): new %lu µs maximum-latency "
+ printk("(%16s-%-5d|#%d): new %lu %cs maximum-latency "
"critical section.\n => started at timestamp %lu: ",
current->comm, current->pid,
_smp_processor_id(),
- latency, t0);
+ latency, MU, t0);
print_symbol("<%s>\n", tr->critical_start);
printk(" => ended at timestamp %lu: ", t1);
@@ -1345,15 +1351,15 @@ check_wakeup_timing(struct cpu_trace *tr
update_max_tr(tr);
if (preempt_thresh)
- printk("(%16s-%-5d|#%d): %lu µs wakeup latency "
- "violates %lu µs threshold.\n",
+ printk("(%16s-%-5d|#%d): %lu %cs wakeup latency "
+ "violates %lu %cs threshold.\n",
current->comm, current->pid,
_smp_processor_id(), latency,
- cycles_to_usecs(preempt_thresh));
+ MU, cycles_to_usecs(preempt_thresh), MU);
else
- printk("(%16s-%-5d|#%d): new %lu µs maximum-latency "
+ printk("(%16s-%-5d|#%d): new %lu %cs maximum-latency "
"wakeup.\n", current->comm, current->pid,
- _smp_processor_id(), latency);
+ _smp_processor_id(), latency, MU);
max_sequence++;
@@ -1399,7 +1405,7 @@ void __trace_start_sched_wakeup(struct t
tr->preempt_timestamp = cycles();
tr->critical_start = CALLER_ADDR0;
trace_cmdline();
- mcount();
+ ARCH_MCOUNT();
out_unlock:
spin_unlock(&sch.trace_lock);
}
@@ -1489,7 +1495,7 @@ long user_trace_start(void)
tr->critical_sequence = max_sequence;
tr->preempt_timestamp = cycles();
trace_cmdline();
- mcount();
+ ARCH_MCOUNT();
preempt_enable();
up(&max_mutex);
@@ -1507,7 +1513,7 @@ long user_trace_stop(void)
return -EINVAL;
preempt_disable();
- mcount();
+ ARCH_MCOUNT();
if (wakeup_timing) {
spin_lock_irqsave(&sch.trace_lock, flags);
@@ -1538,15 +1544,15 @@ long user_trace_stop(void)
latency = cycles_to_usecs(delta);
if (preempt_thresh)
- printk("(%16s-%-5d|#%d): %lu µs user-latency "
- "violates %lu µs threshold.\n",
+ printk("(%16s-%-5d|#%d): %lu %cs user-latency "
+ "violates %lu %cs threshold.\n",
current->comm, current->pid,
- _smp_processor_id(), latency,
- cycles_to_usecs(preempt_thresh));
+ _smp_processor_id(), latency, MU,
+ cycles_to_usecs(preempt_thresh), MU);
else
- printk("(%16s-%-5d|#%d): new %lu µs user-latency.\n",
+ printk("(%16s-%-5d|#%d): new %lu %cs user-latency.\n",
current->comm, current->pid,
- _smp_processor_id(), latency);
+ _smp_processor_id(), latency, MU);
max_sequence++;
up(&max_mutex);
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 23:27 ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
@ 2005-02-24 1:03 ` Lee Revell
2005-02-24 1:29 ` Nick Piggin
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24 1:03 UTC (permalink / raw)
To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel
On Thu, 2005-02-24 at 10:27 +1100, Nick Piggin wrote:
> Hugh Dickins wrote:
> > On Wed, 23 Feb 2005, Lee Revell wrote:
> >
> >>>>Thanks, your patch fixes the copy_pte_range latency.
> >>
> >>clear_page_range is also problematic.
> >
> >
> > Yes, I saw that from your other traces too. I know there are plans
> > to improve clear_page_range during 2.6.12, but I didn't realize that
> > it had become very much worse than its antecedent clear_page_tables,
> > and I don't see missing latency fixes for that. Nick's the expert.
> >
>
> I wouldn't have thought it should have become worse, latency
> wise. What is actually happening is that the lower level freeing
> functions are being called more often. But this should result in
> the work being spread out more, if anything. Rather than in the
> old system things would tend to be batched up into bigger chunks
> (typically at exit() time).
>
> If you are using i386 with 2-level page tables (no highmem), then
> the behaviour should be more or less identical. Odd.
IIRC last time I really tested this a few months ago, the worst case
latency on that machine was about 150us. Currently its 422us from the
same clear_page_range code path.
On my Athlon XP the clear_page_range latency is not showing up at all,
and the worst delay so far is only 35us, most of which is the timer
interrupt IOW that machine is showing the best achievable latency (with
PREEMPT_DESKTOP). The machine seeing 422 us latencies in
clear_page_range is a 600Mhz C3, which is known to be a FSB limited
architecture.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 1:03 ` Lee Revell
@ 2005-02-24 1:29 ` Nick Piggin
2005-02-24 2:24 ` Lee Revell
0 siblings, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-24 1:29 UTC (permalink / raw)
To: Lee Revell; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel
Lee Revell wrote:
> On Thu, 2005-02-24 at 10:27 +1100, Nick Piggin wrote:
>>If you are using i386 with 2-level page tables (no highmem), then
>>the behaviour should be more or less identical. Odd.
>
>
> IIRC last time I really tested this a few months ago, the worst case
> latency on that machine was about 150us. Currently its 422us from the
> same clear_page_range code path.
>
> On my Athlon XP the clear_page_range latency is not showing up at all,
> and the worst delay so far is only 35us, most of which is the timer
> interrupt IOW that machine is showing the best achievable latency (with
> PREEMPT_DESKTOP). The machine seeing 422 us latencies in
> clear_page_range is a 600Mhz C3, which is known to be a FSB limited
> architecture.
>
Well it should be pretty trivial to add a break in there.
I don't think it can get into 2.6.11 at this point though,
so we'll revisit this for 2.6.12 if the clear_page_range
optimisations don't get anywhere.
Nick
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 1:29 ` Nick Piggin
@ 2005-02-24 2:24 ` Lee Revell
2005-02-24 2:41 ` Nick Piggin
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24 2:24 UTC (permalink / raw)
To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel
On Thu, 2005-02-24 at 12:29 +1100, Nick Piggin wrote:
> Lee Revell wrote:
> >
> > IIRC last time I really tested this a few months ago, the worst case
> > latency on that machine was about 150us. Currently its 422us from the
> > same clear_page_range code path.
> >
> Well it should be pretty trivial to add a break in there.
> I don't think it can get into 2.6.11 at this point though,
> so we'll revisit this for 2.6.12 if the clear_page_range
> optimisations don't get anywhere.
>
Agreed, it would be much better to optimize this away than just add a
scheduling point. It seems like we could do this lazily.
IMHO it's not critical that these latency fixes be merged until the VP
feature gets merged, until then people will be using Ingo's patches
anyway.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 2:24 ` Lee Revell
@ 2005-02-24 2:41 ` Nick Piggin
2005-02-24 3:03 ` Lee Revell
0 siblings, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-24 2:41 UTC (permalink / raw)
To: Lee Revell; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel
Lee Revell wrote:
> On Thu, 2005-02-24 at 12:29 +1100, Nick Piggin wrote:
>
>>Lee Revell wrote:
>>
>>>IIRC last time I really tested this a few months ago, the worst case
>>>latency on that machine was about 150us. Currently its 422us from the
>>>same clear_page_range code path.
>>>
>>
>>Well it should be pretty trivial to add a break in there.
>>I don't think it can get into 2.6.11 at this point though,
>>so we'll revisit this for 2.6.12 if the clear_page_range
>>optimisations don't get anywhere.
>>
>
>
> Agreed, it would be much better to optimize this away than just add a
> scheduling point. It seems like we could do this lazily.
>
Oh? What do you mean by lazy? IMO it is sort of implemented lazily now.
That is, we are too lazy to refcount page table pages in fastpaths, so
that pushes a lot of work to unmap time. Not necessarily a bad trade-off,
mind you. Just something I'm looking into.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 2:41 ` Nick Piggin
@ 2005-02-24 3:03 ` Lee Revell
0 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-24 3:03 UTC (permalink / raw)
To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel
On Thu, 2005-02-24 at 13:41 +1100, Nick Piggin wrote:
> Lee Revell wrote:
> >
> > Agreed, it would be much better to optimize this away than just add a
> > scheduling point. It seems like we could do this lazily.
> >
>
> Oh? What do you mean by lazy? IMO it is sort of implemented lazily now.
> That is, we are too lazy to refcount page table pages in fastpaths, so
> that pushes a lot of work to unmap time. Not necessarily a bad trade-off,
> mind you. Just something I'm looking into.
>
I guess I was thinking we could be even more lazy, and somehow defer it
until after unmap time (in lieu of memory pressure that is). Actually
that's kind of what a lock break would do.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: PPC RT Patch..
2005-02-23 23:52 ` PPC RT Patch john cooper
@ 2005-02-24 4:20 ` Frank Rowand
2005-02-24 13:56 ` john cooper
0 siblings, 1 reply; 26+ messages in thread
From: Frank Rowand @ 2005-02-24 4:20 UTC (permalink / raw)
To: john cooper; +Cc: Ingo Molnar, linux-kernel, Saksena, Manas
john cooper wrote:
> Ingo,
> We've had a PPC port of your RT work underway with
> a focus on trace instrumentation. This is based upon
> realtime-preempt-2.6.11-rc2-V0.7.37-02. A diff is
> attached.
>
> To the extent possible the tracing facilities are the
> same as your x86 work. In the process a few PPC/gcc
> issues needed to be resolved. There is also a bug fix
> contained for tlb_gather_mmu() which was causing debug
> assertions to be generated in a path which attempted to
> sleep with a non-zero preempt count.
Manish Lachwani mentioned to me that he faced the same issue
with the MIPS RT support and that when he discussed
it with Ingo that the solution was for include/asm-ppc/tlb.h
to include/asm-generic/tlb-simple.h when PREEMPT_RT is turned on.
The patch does this for the #ifdef CONFIG_PPC_STD_MMU case,
but not for the #else case. I don't know which case is used
for the Ampro board.
>
> This does build and function when SMP is configured,
> though we have not yet verified it on other than a
> uniprocessor. As a simplifying assumption, testing has
> thus far concentrated on the following modes:
>
> PREEMPT_NONE
> - verify baseline regression
>
> PREEMPT_RT && !PREEMPT_SMP
> - typical for an embedded RT PPC application
>
> PREEMPT_RT && PREEMPT_SMP
> - kicks in live locking code which otherwise receives no
> coverage. This is functionally equivalent to the above
> config on a single CPU target thus no MP dynamic testing
> is achieved. Still quite useful IMHO.
>
> The target used for development/testing is an Ampro EnCore PP1
> which sports a 300Mhz MPC8245. For testing this boots with NFS
> as root. An mp3 decode at nice --20 is launched which requires
> just under 20% of the CPU to maintain an uninterrupted audio
> decode and output. To this a series of "du -s /" are launched
> to soak up excess CPU bandwidth. Perhaps not rigorous but a
> fair sanity check and load for the purpose at hand.
>
> Under these conditions maximum scheduling latencies are seen in
> the 120-150us range. Note no attempt has yet been made to
> optimize arch specific paths and full trace instrumentation has
> been enabled.
>
> I've written some logging code to help find problems such as
> the tlb issue above. As it has not been made general I've
> removed it from this patch. At some point I'll likely revisit
> this.
>
> Comments/suggestions welcome.
I am glad to see the instrumentation and measurement related code
in your patch. (My patch of last week ("Frank's patch") is lacking
that code.)
Other differences between the two patches are:
arch/ppc/syslib/i8259.c
Frank neglected to convert i8259_lock to a raw spinlock.
arch/ppc/kernel/signal.c
John added an enable of irqs in do_signal() #ifdef CONFIG_PREEMPT_RT
arch/ppc/kernel/traps.c
John added an enable of irqs and preempt_check_resched() in _exception().
various files
Frank added the intrusive variable tb_to_us for use by cycles_to_usec()
and added an ugly #ifdef in cycles_to_usec().
John hard-coded cpu_khz for one specific board so that no change would
be needed in cycles_to_usec().
various files
John has the mmu_gather fix that is described above.
John's patch and Frank's patch are otherwise mostly the same, except for
the differences that result from being based on different kernel
versions. I am glad to see that because it means that two sets of
eyes have agreed.
Frank's patch may have missed some EXPORT_SYMBOL()s in arch/ppc/lib/locks.c.
I'll check those over again tomorrow.
> -john
-Frank
--
Frank Rowand <frank_rowand@mvista.com>
MontaVista Software, Inc
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-23 22:13 ` Lee Revell
@ 2005-02-24 4:56 ` Hugh Dickins
2005-02-24 6:32 ` Lee Revell
0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-24 4:56 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Wed, 23 Feb 2005, Lee Revell wrote:
> On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> > On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > > Please replace by new patch below, which I'm now running through lmbench.
> >
> > That second patch seems fine, and I see no lmbench regression from it.
>
> Should go into 2.6.11, right?
That's up to Andrew (and Linus).
I was thinking that way when I rushed you the patch. But given that
you have remaining unresolved latency issues nearby (zap_pte_range,
clear_page_range), and given the warning shot that I screwed up my
first attempt, I'd be inclined to say hold off.
It's a pity: for a while we were thinking 2.6.11 would be a big step
forward for mainline latency; but it now looks to me like these tests
have come too late in the cycle to be dealt with safely.
In other mail, you do expect people still to be using Ingo's patches,
so probably this patch should stick there (and in -mm) for now.
Hugh
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 4:56 ` Hugh Dickins
@ 2005-02-24 6:32 ` Lee Revell
2005-02-24 8:26 ` Hugh Dickins
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24 6:32 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> > > On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > > > Please replace by new patch below, which I'm now running through lmbench.
> > >
> > > That second patch seems fine, and I see no lmbench regression from it.
> >
> > Should go into 2.6.11, right?
>
> That's up to Andrew (and Linus).
>
> I was thinking that way when I rushed you the patch. But given that
> you have remaining unresolved latency issues nearby (zap_pte_range,
> clear_page_range), and given the warning shot that I screwed up my
> first attempt, I'd be inclined to say hold off.
>
> It's a pity: for a while we were thinking 2.6.11 would be a big step
> forward for mainline latency; but it now looks to me like these tests
> have come too late in the cycle to be dealt with safely.
>
> In other mail, you do expect people still to be using Ingo's patches,
> so probably this patch should stick there (and in -mm) for now.
Well all of these were fixed in the past so it may not be unreasonable
to fix them for 2.6.11.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 6:32 ` Lee Revell
@ 2005-02-24 8:26 ` Hugh Dickins
2005-02-25 3:30 ` Lee Revell
0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-24 8:26 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Thu, 24 Feb 2005, Lee Revell wrote:
> On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> >
> > In other mail, you do expect people still to be using Ingo's patches,
> > so probably this patch should stick there (and in -mm) for now.
>
> Well all of these were fixed in the past so it may not be unreasonable
> to fix them for 2.6.11.
If we'd got to it earlier, yes. But 2.6.11 looks to be just a day or
two away, and we've no idea why zap_pte_range or clear_page_range
would have reverted. Nor have we heard from Ingo yet.
Hugh
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: PPC RT Patch..
2005-02-24 4:20 ` Frank Rowand
@ 2005-02-24 13:56 ` john cooper
0 siblings, 0 replies; 26+ messages in thread
From: john cooper @ 2005-02-24 13:56 UTC (permalink / raw)
To: Frank Rowand; +Cc: Ingo Molnar, linux-kernel, Saksena, Manas, john cooper
Frank Rowand wrote:
> john cooper wrote:
>> ... There is also a bug fix
>> contained for tlb_gather_mmu() which was causing debug
>> assertions to be generated in a path which attempted to
>> sleep with a non-zero preempt count.
>
>
> Manish Lachwani mentioned to me that he faced the same issue
> with the MIPS RT support and that when he discussed
> it with Ingo that the solution was for include/asm-ppc/tlb.h
> to include/asm-generic/tlb-simple.h when PREEMPT_RT is turned on.
> The patch does this for the #ifdef CONFIG_PPC_STD_MMU case,
> but not for the #else case. I don't know which case is used
> for the Ampro board.
It appeared to me a generic issue though I believe a number
of solutions are possible. asm-generic/tlb.h:tlb_gather_mmu()
expands to linux/percpu.h:get_cpu_var() which does a
preempt_disable() and __get_cpu_var(). This caused the debug
assertion to kick when __page_cache_release() and to a lesser
extent activate_page() attempted to block on a mutex (though
other paths may well exist). My approach was to replace the
outer layer preempt_disable/enable calls with a mutex-spinlock.
The fix was fairly easy once it was known from where the
gratuitous call to preempt_disable() existed. I cobbled
together a logging mechanism which detected the problem. As it
wasn't very general I removed it from the patch. I didn't see
an alternate means of diagnosing such a scenario so I'll likely
address generalizing the code.
-john
--
john.cooper@timesys.com
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-24 8:26 ` Hugh Dickins
@ 2005-02-25 3:30 ` Lee Revell
2005-02-25 5:58 ` Hugh Dickins
0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-25 3:30 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> On Thu, 24 Feb 2005, Lee Revell wrote:
> > On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> > >
> > > In other mail, you do expect people still to be using Ingo's patches,
> > > so probably this patch should stick there (and in -mm) for now.
> >
> > Well all of these were fixed in the past so it may not be unreasonable
> > to fix them for 2.6.11.
>
> If we'd got to it earlier, yes. But 2.6.11 looks to be just a day or
> two away, and we've no idea why zap_pte_range or clear_page_range
> would have reverted. Nor have we heard from Ingo yet.
>
It's also not clear that the patch completely fixes the copy_pte_range
latency. This trace is from the Athlon XP.
Lee
preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
latency: 284 �s, #25/25, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
-----------------
| task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
(T1/#0) dpkg 9299 0 3 00000005 00000000 [0001017457529380] 0.000ms (+3633922.612ms): <676b7064> (<00746500>)
(T1/#2) dpkg 9299 0 3 00000005 00000002 [0001017457529620] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x9a/0xd0 <c012eaca> (try_to_wake_up+0x90/0x160 <c0110350>)
(T1/#3) dpkg 9299 0 3 00000004 00000003 [0001017457529825] 0.000ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (try_to_wake_up+0x90/0x160 <c0110350>)
(T3/#4) dpkg-9299 0dn.4 0�s : try_to_wake_up+0x118/0x160 <c01103d8> <<...>-2> (69 74):
(T1/#5) dpkg 9299 0 3 00000003 00000005 [0001017457530633] 0.000ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (try_to_wake_up+0xf2/0x160 <c01103b2>)
(T1/#6) dpkg 9299 0 3 00000003 00000006 [0001017457530809] 0.001ms (+0.000ms): wake_up_process+0x35/0x40 <c0110455> (do_softirq+0x3f/0x50 <c011aedf>)
(T6/#7) dpkg-9299 0dn.2 1�s!< (1)
(T1/#8) dpkg 9299 0 2 00000001 00000008 [0001017457898984] 0.276ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (copy_pte_range+0xbc/0x1b0 <c014573c>)
(T1/#9) dpkg 9299 0 2 00000001 00000009 [0001017457899172] 0.276ms (+0.000ms): __cond_resched_raw_spinlock+0xb/0x50 <c0111f9b> (copy_pte_range+0xad/0x1b0 <c014572d>)
(T1/#10) dpkg 9299 0 2 00000000 0000000a [0001017457899575] 0.277ms (+0.000ms): __cond_resched+0xe/0x70 <c0111f2e> (__cond_resched_raw_spinlock+0x35/0x50 <c0111fc5>)
(T1/#11) dpkg 9299 0 3 00000000 0000000b [0001017457900063] 0.277ms (+0.000ms): __schedule+0xe/0x680 <c028720e> (__cond_resched+0x4a/0x70 <c0111f6a>)
(T1/#12) dpkg 9299 0 3 00000000 0000000c [0001017457900379] 0.277ms (+0.000ms): profile_hit+0x9/0x50 <c0116449> (__schedule+0x43/0x680 <c0287243>)
(T1/#13) dpkg 9299 0 3 00000001 0000000d [0001017457900602] 0.277ms (+0.001ms): sched_clock+0x14/0x80 <c010cbb4> (__schedule+0x73/0x680 <c0287273>)
(T1/#14) dpkg 9299 0 3 00000002 0000000e [0001017457902490] 0.279ms (+0.000ms): dequeue_task+0x12/0x60 <c010ff32> (__schedule+0x1e0/0x680 <c02873e0>)
(T1/#15) dpkg 9299 0 3 00000002 0000000f [0001017457902687] 0.279ms (+0.000ms): recalc_task_prio+0xe/0x140 <c01100be> (__schedule+0x202/0x680 <c0287402>)
(T1/#16) dpkg 9299 0 3 00000002 00000010 [0001017457902848] 0.279ms (+0.000ms): effective_prio+0x8/0x60 <c0110058> (recalc_task_prio+0x88/0x140 <c0110138>)
(T1/#17) dpkg 9299 0 3 00000002 00000011 [0001017457902995] 0.279ms (+0.000ms): enqueue_task+0x11/0x80 <c010ff91> (__schedule+0x20e/0x680 <c028740e>)
(T4/#18) [ => dpkg ] 0.280ms (+0.000ms)
(T1/#19) <...> 2 0 1 00000002 00000013 [0001017457905091] 0.281ms (+0.000ms): __switch_to+0xe/0x190 <c010110e> (__schedule+0x306/0x680 <c0287506>)
(T3/#20) <...>-2 0d..2 281�s : __schedule+0x337/0x680 <c0287537> <dpkg-9299> (74 69):
(T1/#21) <...> 2 0 1 00000002 00000015 [0001017457906484] 0.282ms (+0.000ms): finish_task_switch+0x14/0xa0 <c0110844> (__schedule+0x33f/0x680 <c028753f>)
(T1/#22) <...> 2 0 1 00000001 00000016 [0001017457906713] 0.282ms (+0.000ms): trace_stop_sched_switched+0x11/0x180 <c012eb11> (finish_task_switch+0x51/0xa0 <c0110881>)
(T3/#23) <...>-2 0d..1 282�s : trace_stop_sched_switched+0x4c/0x180 <c012eb4c> <<...>-2> (69 0):
(T1/#24) <...> 2 0 1 00000001 00000018 [0001017457908107] 0.283ms (+0.000ms): trace_stop_sched_switched+0x11c/0x180 <c012ec1c> (finish_task_switch+0x51/0xa0 <c0110881>)
vim:ft=help
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-25 3:30 ` Lee Revell
@ 2005-02-25 5:58 ` Hugh Dickins
2005-02-25 15:02 ` Lee Revell
0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-25 5:58 UTC (permalink / raw)
To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Thu, 24 Feb 2005, Lee Revell wrote:
> On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> >
> > If we'd got to it earlier, yes. But 2.6.11 looks to be just a day or
> > two away, and we've no idea why zap_pte_range or clear_page_range
> > would have reverted. Nor have we heard from Ingo yet.
>
> It's also not clear that the patch completely fixes the copy_pte_range
> latency. This trace is from the Athlon XP.
Then we need Ingo to investigate and explain all these reversions.
I'm not _blaming_ Ingo for them, but I'm not familiar with his patches
nor with deciphering latency traces - he's the magician around here.
Hugh
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
2005-02-25 5:58 ` Hugh Dickins
@ 2005-02-25 15:02 ` Lee Revell
0 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-25 15:02 UTC (permalink / raw)
To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel
On Fri, 2005-02-25 at 05:58 +0000, Hugh Dickins wrote:
> On Thu, 24 Feb 2005, Lee Revell wrote:
> > On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> > >
> > > If we'd got to it earlier, yes. But 2.6.11 looks to be just a day or
> > > two away, and we've no idea why zap_pte_range or clear_page_range
> > > would have reverted. Nor have we heard from Ingo yet.
> >
> > It's also not clear that the patch completely fixes the copy_pte_range
> > latency. This trace is from the Athlon XP.
>
> Then we need Ingo to investigate and explain all these reversions.
> I'm not _blaming_ Ingo for them, but I'm not familiar with his patches
> nor with deciphering latency traces - he's the magician around here.
>
Yup. Oh well.
I'll try to compile a comprehensive list of these so we can fix them for
2.6.12.
Lee
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2005-02-25 15:04 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
2005-02-23 19:16 ` Hugh Dickins
2005-02-23 19:36 ` Lee Revell
2005-02-23 20:06 ` Hugh Dickins
2005-02-23 20:10 ` Lee Revell
2005-02-23 20:30 ` Lee Revell
2005-02-23 21:03 ` Hugh Dickins
2005-02-23 22:14 ` Lee Revell
2005-02-23 23:52 ` PPC RT Patch john cooper
2005-02-24 4:20 ` Frank Rowand
2005-02-24 13:56 ` john cooper
2005-02-23 23:27 ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
2005-02-24 1:03 ` Lee Revell
2005-02-24 1:29 ` Nick Piggin
2005-02-24 2:24 ` Lee Revell
2005-02-24 2:41 ` Nick Piggin
2005-02-24 3:03 ` Lee Revell
2005-02-23 20:53 ` Hugh Dickins
2005-02-23 22:13 ` Lee Revell
2005-02-24 4:56 ` Hugh Dickins
2005-02-24 6:32 ` Lee Revell
2005-02-24 8:26 ` Hugh Dickins
2005-02-25 3:30 ` Lee Revell
2005-02-25 5:58 ` Hugh Dickins
2005-02-25 15:02 ` Lee Revell
2005-02-23 19:52 ` Lee Revell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox