More latency regressions with 2.6.11-rc4-RT-V0.7.39-02

public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed

* More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
@ 2005-02-23 18:07 Lee Revell
  2005-02-23 19:16 ` Hugh Dickins
  2005-02-23 19:52 ` Lee Revell
  0 siblings, 2 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 18:07 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

Ingo,

Did something change recently in the VM that made copy_pte_range and
clear_page_range a lot more expensive?  I noticed a reference in the
"Page Table Iterators" thread to excessive overhead introduced by
aggressive page freeing.  That sure looks like what is going on in
trace2.  trace1 and trace3 look like big fork latencies associated with
copy_pte_range.

This is all with PREEMPT_DESKTOP.

Lee 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace1.txt --]
[-- Type: text/plain; name=trace1.txt; charset=ANSI_X3.4-1968, Size: 3844 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 323 µs, #22/22, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)     valgrind.bin 15934 0 9 00000002 00000000 [0032064022903959] 0.000ms (+3534259.157ms): <676c6176> (<646e6972>)
(T1/#2)     valgrind.bin 15934 0 9 00000002 00000002 [0032064022904377] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)     valgrind.bin 15934 0 9 00000000 00000003 [0032064022904934] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) valgrind-15934 0dn.2    2µs!< (1)
(T1/#5)     valgrind.bin 15934 0 2 00000001 00000005 [0032064023088406] 0.307ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (copy_pte_range+0xb7/0x1c0 <c0142ad7>)
(T1/#6)     valgrind.bin 15934 0 2 00000001 00000006 [0032064023088847] 0.308ms (+0.000ms): __cond_resched_raw_spinlock+0x8/0x50 <c0111398> (copy_pte_range+0xa7/0x1c0 <c0142ac7>)
(T1/#7)     valgrind.bin 15934 0 2 00000000 00000007 [0032064023089269] 0.308ms (+0.001ms): __cond_resched+0x9/0x70 <c0111329> (__cond_resched_raw_spinlock+0x3d/0x50 <c01113cd>)
(T1/#8)     valgrind.bin 15934 0 3 00000000 00000008 [0032064023089903] 0.309ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (__cond_resched+0x45/0x70 <c0111365>)
(T1/#9)     valgrind.bin 15934 0 3 00000000 00000009 [0032064023090288] 0.310ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#10)     valgrind.bin 15934 0 3 00000001 0000000a [0032064023090866] 0.311ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#11)     valgrind.bin 15934 0 3 00000002 0000000b [0032064023091857] 0.313ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#12)     valgrind.bin 15934 0 3 00000002 0000000c [0032064023092280] 0.313ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#13)     valgrind.bin 15934 0 3 00000002 0000000d [0032064023092690] 0.314ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#14)     valgrind.bin 15934 0 3 00000002 0000000e [0032064023093035] 0.315ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#15) [ =>     valgrind.bin ] 0.316ms (+0.001ms)
(T1/#16)            <...>     2 0 1 00000002 00000010 [0032064023094748] 0.317ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#17)    <...>-2     0d..2  319µs : __schedule+0x2ea/0x630 <c027cc6a> <valgrind-15934> (7d 69): 
(T1/#18)            <...>     2 0 1 00000002 00000012 [0032064023095766] 0.319ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#19)            <...>     2 0 1 00000001 00000013 [0032064023096192] 0.320ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#20)    <...>-2     0d..1  320µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#21)            <...>     2 0 1 00000001 00000015 [0032064023097511] 0.322ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: trace2.txt --]
[-- Type: text/plain; name=trace2.txt; charset=ANSI_X3.4-1968, Size: 49744 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 361 µs, #289/289, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)        evolution 16047 0 9 00000002 00000000 [0032772517080312] 0.000ms (+3483891.566ms): <6c6f7665> (<6f697475>)
(T1/#2)        evolution 16047 0 9 00000002 00000002 [0032772517080803] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)        evolution 16047 0 9 00000000 00000003 [0032772517081392] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4) evolutio-16047 0dn.2    2µs < (1)
(T1/#5)        evolution 16047 0 2 00000002 00000005 [0032772517082355] 0.003ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#6)        evolution 16047 0 2 00000002 00000006 [0032772517082977] 0.004ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#7)        evolution 16047 0 2 00000002 00000007 [0032772517083332] 0.005ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#8)        evolution 16047 0 2 00000002 00000008 [0032772517083589] 0.005ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#9)        evolution 16047 0 2 00000002 00000009 [0032772517083937] 0.006ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#10)        evolution 16047 0 2 00000002 0000000a [0032772517084750] 0.007ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#11)        evolution 16047 0 2 00000002 0000000b [0032772517085209] 0.008ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#12)        evolution 16047 0 2 00000002 0000000c [0032772517085614] 0.008ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#13)        evolution 16047 0 2 00000002 0000000d [0032772517085996] 0.009ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#14)        evolution 16047 0 2 00000002 0000000e [0032772517086537] 0.010ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#15)        evolution 16047 0 2 00000002 0000000f [0032772517086869] 0.010ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#16)        evolution 16047 0 2 00000002 00000010 [0032772517087171] 0.011ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#17)        evolution 16047 0 2 00000002 00000011 [0032772517087540] 0.012ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#18)        evolution 16047 0 2 00000002 00000012 [0032772517088332] 0.013ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#19)        evolution 16047 0 2 00000002 00000013 [0032772517088800] 0.014ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#20)        evolution 16047 0 2 00000002 00000014 [0032772517089160] 0.014ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#21)        evolution 16047 0 2 00000002 00000015 [0032772517089620] 0.015ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#22)        evolution 16047 0 2 00000002 00000016 [0032772517090190] 0.016ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#23)        evolution 16047 0 2 00000002 00000017 [0032772517090529] 0.017ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#24)        evolution 16047 0 2 00000002 00000018 [0032772517090829] 0.017ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#25)        evolution 16047 0 2 00000002 00000019 [0032772517091189] 0.018ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#26)        evolution 16047 0 3 00000003 0000001a [0032772517091761] 0.019ms (+0.001ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#27)        evolution 16047 0 3 00000004 0000001b [0032772517092489] 0.020ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#28)        evolution 16047 0 3 00000004 0000001c [0032772517092884] 0.020ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#29)        evolution 16047 0 3 00000004 0000001d [0032772517093678] 0.022ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#30)        evolution 16047 0 3 00000004 0000001e [0032772517094096] 0.022ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#31)        evolution 16047 0 3 00000004 0000001f [0032772517094674] 0.023ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#32)        evolution 16047 0 3 00000004 00000020 [0032772517095137] 0.024ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#33)        evolution 16047 0 3 00000004 00000021 [0032772517095623] 0.025ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#34)        evolution 16047 0 3 00000004 00000022 [0032772517096307] 0.026ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#35)        evolution 16047 0 3 00000004 00000023 [0032772517096766] 0.027ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#36)        evolution 16047 0 3 00000004 00000024 [0032772517097495] 0.028ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#37)        evolution 16047 0 3 00000004 00000025 [0032772517097968] 0.029ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#38)        evolution 16047 0 3 00000004 00000026 [0032772517098688] 0.030ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#39)        evolution 16047 0 3 00000004 00000027 [0032772517099138] 0.031ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#40)        evolution 16047 0 3 00000004 00000028 [0032772517099961] 0.032ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#41)        evolution 16047 0 3 00000004 00000029 [0032772517100452] 0.033ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#42)        evolution 16047 0 3 00000004 0000002a [0032772517101122] 0.034ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#43)        evolution 16047 0 3 00000004 0000002b [0032772517101568] 0.035ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#44)        evolution 16047 0 3 00000004 0000002c [0032772517102283] 0.036ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#45)        evolution 16047 0 3 00000004 0000002d [0032772517102729] 0.037ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#46)        evolution 16047 0 3 00000004 0000002e [0032772517103453] 0.038ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#47)        evolution 16047 0 3 00000004 0000002f [0032772517103899] 0.039ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#48)        evolution 16047 0 3 00000004 00000030 [0032772517104601] 0.040ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#49)        evolution 16047 0 3 00000004 00000031 [0032772517105060] 0.041ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#50)        evolution 16047 0 3 00000004 00000032 [0032772517105618] 0.042ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#51)        evolution 16047 0 3 00000004 00000033 [0032772517106081] 0.042ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#52)        evolution 16047 0 3 00000004 00000034 [0032772517106581] 0.043ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#53)        evolution 16047 0 3 00000004 00000035 [0032772517107359] 0.045ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#54)        evolution 16047 0 3 00000004 00000036 [0032772517107818] 0.045ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#55)        evolution 16047 0 3 00000004 00000037 [0032772517108498] 0.046ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#56)        evolution 16047 0 3 00000004 00000038 [0032772517108948] 0.047ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#57)        evolution 16047 0 3 00000004 00000039 [0032772517109609] 0.048ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#58)        evolution 16047 0 3 00000004 0000003a [0032772517110055] 0.049ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#59)        evolution 16047 0 3 00000004 0000003b [0032772517110779] 0.050ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#60)        evolution 16047 0 3 00000004 0000003c [0032772517111229] 0.051ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#61)        evolution 16047 0 3 00000003 0000003d [0032772517111877] 0.052ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#62)        evolution 16047 0 2 00000002 0000003e [0032772517112404] 0.053ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#63)        evolution 16047 0 2 00000002 0000003f [0032772517113047] 0.054ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#64)        evolution 16047 0 2 00000002 00000040 [0032772517113493] 0.055ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#65)        evolution 16047 0 2 00000002 00000041 [0032772517113952] 0.056ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#66)        evolution 16047 0 2 00000002 00000042 [0032772517114564] 0.057ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#67)        evolution 16047 0 2 00000002 00000043 [0032772517114937] 0.057ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#68)        evolution 16047 0 2 00000002 00000044 [0032772517115266] 0.058ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#69)        evolution 16047 0 2 00000002 00000045 [0032772517115612] 0.058ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#70)        evolution 16047 0 2 00000002 00000046 [0032772517116422] 0.060ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#71)        evolution 16047 0 2 00000002 00000047 [0032772517116886] 0.060ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#72)        evolution 16047 0 2 00000002 00000048 [0032772517117322] 0.061ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#73)        evolution 16047 0 2 00000002 00000049 [0032772517117813] 0.062ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#74)        evolution 16047 0 2 00000002 0000004a [0032772517118416] 0.063ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#75)        evolution 16047 0 2 00000002 0000004b [0032772517118785] 0.064ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#76)        evolution 16047 0 2 00000002 0000004c [0032772517119113] 0.064ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#77)        evolution 16047 0 2 00000002 0000004d [0032772517119469] 0.065ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#78)        evolution 16047 0 2 00000002 0000004e [0032772517120247] 0.066ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#79)        evolution 16047 0 2 00000002 0000004f [0032772517120715] 0.067ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#80)        evolution 16047 0 2 00000002 00000050 [0032772517121156] 0.068ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#81)        evolution 16047 0 2 00000002 00000051 [0032772517121606] 0.068ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#82)        evolution 16047 0 2 00000002 00000052 [0032772517122214] 0.069ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#83)        evolution 16047 0 2 00000002 00000053 [0032772517122578] 0.070ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#84)        evolution 16047 0 2 00000002 00000054 [0032772517122911] 0.070ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#85)        evolution 16047 0 2 00000002 00000055 [0032772517123271] 0.071ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#86)        evolution 16047 0 2 00000002 00000056 [0032772517124081] 0.072ms (+0.037ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#87)        evolution 16047 0 2 00000002 00000057 [0032772517146379] 0.110ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#88)        evolution 16047 0 2 00000002 00000058 [0032772517146815] 0.110ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#89)        evolution 16047 0 2 00000002 00000059 [0032772517147234] 0.111ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#90)        evolution 16047 0 2 00000002 0000005a [0032772517147832] 0.112ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#91)        evolution 16047 0 2 00000002 0000005b [0032772517148201] 0.113ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#92)        evolution 16047 0 2 00000002 0000005c [0032772517148530] 0.113ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#93)        evolution 16047 0 2 00000002 0000005d [0032772517148885] 0.114ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#94)        evolution 16047 0 2 00000002 0000005e [0032772517149677] 0.115ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#95)        evolution 16047 0 2 00000002 0000005f [0032772517150150] 0.116ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#96)        evolution 16047 0 2 00000002 00000060 [0032772517150595] 0.117ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#97)        evolution 16047 0 2 00000002 00000061 [0032772517151041] 0.117ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#98)        evolution 16047 0 2 00000002 00000062 [0032772517151653] 0.118ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#99)        evolution 16047 0 2 00000002 00000063 [0032772517152017] 0.119ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#100)        evolution 16047 0 2 00000002 00000064 [0032772517152359] 0.120ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#101)        evolution 16047 0 2 00000002 00000065 [0032772517152719] 0.120ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#102)        evolution 16047 0 2 00000002 00000066 [0032772517153507] 0.121ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#103)        evolution 16047 0 2 00000002 00000067 [0032772517153975] 0.122ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#104)        evolution 16047 0 2 00000002 00000068 [0032772517154407] 0.123ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#105)        evolution 16047 0 2 00000002 00000069 [0032772517154879] 0.124ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#106)        evolution 16047 0 2 00000002 0000006a [0032772517155482] 0.125ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#107)        evolution 16047 0 2 00000002 0000006b [0032772517155842] 0.125ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#108)        evolution 16047 0 2 00000002 0000006c [0032772517156171] 0.126ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#109)        evolution 16047 0 2 00000002 0000006d [0032772517156517] 0.127ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#110)        evolution 16047 0 2 00000002 0000006e [0032772517157291] 0.128ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#111)        evolution 16047 0 2 00000002 0000006f [0032772517158029] 0.129ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#112)        evolution 16047 0 2 00000002 00000070 [0032772517158466] 0.130ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#113)        evolution 16047 0 2 00000002 00000071 [0032772517158929] 0.131ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#114)        evolution 16047 0 2 00000002 00000072 [0032772517159532] 0.132ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#115)        evolution 16047 0 2 00000002 00000073 [0032772517159910] 0.132ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#116)        evolution 16047 0 2 00000002 00000074 [0032772517160243] 0.133ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#117)        evolution 16047 0 2 00000002 00000075 [0032772517160590] 0.133ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#118)        evolution 16047 0 2 00000002 00000076 [0032772517161377] 0.135ms (+0.075ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#119)        evolution 16047 0 2 00000002 00000077 [0032772517206701] 0.210ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#120)        evolution 16047 0 2 00000002 00000078 [0032772517207133] 0.211ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#121)        evolution 16047 0 2 00000002 00000079 [0032772517207574] 0.212ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#122)        evolution 16047 0 2 00000002 0000007a [0032772517208173] 0.213ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#123)        evolution 16047 0 2 00000002 0000007b [0032772517208537] 0.213ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#124)        evolution 16047 0 2 00000002 0000007c [0032772517208866] 0.214ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#125)        evolution 16047 0 2 00000002 0000007d [0032772517209221] 0.214ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#126)        evolution 16047 0 2 00000002 0000007e [0032772517209995] 0.216ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#127)        evolution 16047 0 2 00000002 0000007f [0032772517210567] 0.217ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#128)        evolution 16047 0 2 00000002 00000080 [0032772517211012] 0.217ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#129)        evolution 16047 0 2 00000002 00000081 [0032772517211462] 0.218ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#130)        evolution 16047 0 2 00000002 00000082 [0032772517212065] 0.219ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#131)        evolution 16047 0 2 00000002 00000083 [0032772517212430] 0.220ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#132)        evolution 16047 0 2 00000002 00000084 [0032772517212763] 0.220ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#133)        evolution 16047 0 2 00000002 00000085 [0032772517213114] 0.221ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#134)        evolution 16047 0 2 00000002 00000086 [0032772517213892] 0.222ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#135)        evolution 16047 0 2 00000002 00000087 [0032772517214365] 0.223ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#136)        evolution 16047 0 2 00000002 00000088 [0032772517214833] 0.224ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#137)        evolution 16047 0 2 00000002 00000089 [0032772517215328] 0.225ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#138)        evolution 16047 0 2 00000002 0000008a [0032772517215944] 0.226ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#139)        evolution 16047 0 2 00000002 0000008b [0032772517216304] 0.226ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#140)        evolution 16047 0 2 00000002 0000008c [0032772517216633] 0.227ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#141)        evolution 16047 0 2 00000002 0000008d [0032772517216988] 0.227ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#142)        evolution 16047 0 2 00000002 0000008e [0032772517217767] 0.229ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#143)        evolution 16047 0 2 00000002 0000008f [0032772517218428] 0.230ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#144)        evolution 16047 0 2 00000002 00000090 [0032772517218860] 0.230ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#145)        evolution 16047 0 2 00000002 00000091 [0032772517219279] 0.231ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#146)        evolution 16047 0 2 00000002 00000092 [0032772517219882] 0.232ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#147)        evolution 16047 0 2 00000002 00000093 [0032772517220246] 0.233ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#148)        evolution 16047 0 2 00000002 00000094 [0032772517220606] 0.233ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#149)        evolution 16047 0 2 00000002 00000095 [0032772517221027] 0.234ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#150)        evolution 16047 0 2 00000002 00000096 [0032772517221808] 0.235ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#151)        evolution 16047 0 2 00000002 00000097 [0032772517222276] 0.236ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#152)        evolution 16047 0 2 00000002 00000098 [0032772517222721] 0.237ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#153)        evolution 16047 0 2 00000002 00000099 [0032772517223176] 0.238ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#154)        evolution 16047 0 2 00000002 0000009a [0032772517223779] 0.239ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#155)        evolution 16047 0 2 00000002 0000009b [0032772517224143] 0.239ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#156)        evolution 16047 0 2 00000002 0000009c [0032772517224476] 0.240ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#157)        evolution 16047 0 2 00000002 0000009d [0032772517224823] 0.240ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#158)        evolution 16047 0 2 00000002 0000009e [0032772517225606] 0.242ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#159)        evolution 16047 0 2 00000002 0000009f [0032772517226227] 0.243ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#160)        evolution 16047 0 2 00000002 000000a0 [0032772517226668] 0.243ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#161)        evolution 16047 0 2 00000002 000000a1 [0032772517227113] 0.244ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#162)        evolution 16047 0 2 00000002 000000a2 [0032772517227712] 0.245ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#163)        evolution 16047 0 2 00000002 000000a3 [0032772517228072] 0.246ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#164)        evolution 16047 0 2 00000002 000000a4 [0032772517228400] 0.246ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#165)        evolution 16047 0 2 00000002 000000a5 [0032772517228751] 0.247ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#166)        evolution 16047 0 2 00000002 000000a6 [0032772517229539] 0.248ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#167)        evolution 16047 0 2 00000002 000000a7 [0032772517230209] 0.249ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#168)        evolution 16047 0 2 00000002 000000a8 [0032772517230646] 0.250ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#169)        evolution 16047 0 2 00000002 000000a9 [0032772517231064] 0.251ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#170)        evolution 16047 0 2 00000002 000000aa [0032772517231667] 0.252ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#171)        evolution 16047 0 2 00000002 000000ab [0032772517232041] 0.252ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#172)        evolution 16047 0 2 00000002 000000ac [0032772517232374] 0.253ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#173)        evolution 16047 0 2 00000002 000000ad [0032772517232734] 0.254ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#174)        evolution 16047 0 2 00000002 000000ae [0032772517233521] 0.255ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#175)        evolution 16047 0 2 00000002 000000af [0032772517234102] 0.256ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#176)        evolution 16047 0 2 00000002 000000b0 [0032772517234534] 0.257ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#177)        evolution 16047 0 2 00000002 000000b1 [0032772517235087] 0.257ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#178)        evolution 16047 0 2 00000002 000000b2 [0032772517235695] 0.258ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#179)        evolution 16047 0 2 00000002 000000b3 [0032772517236064] 0.259ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#180)        evolution 16047 0 2 00000002 000000b4 [0032772517236392] 0.260ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#181)        evolution 16047 0 2 00000002 000000b5 [0032772517236739] 0.260ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#182)        evolution 16047 0 2 00000002 000000b6 [0032772517237517] 0.262ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#183)        evolution 16047 0 2 00000002 000000b7 [0032772517238093] 0.262ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#184)        evolution 16047 0 2 00000002 000000b8 [0032772517238543] 0.263ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#185)        evolution 16047 0 2 00000002 000000b9 [0032772517238998] 0.264ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#186)        evolution 16047 0 2 00000002 000000ba [0032772517239614] 0.265ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#187)        evolution 16047 0 2 00000002 000000bb [0032772517239988] 0.266ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#188)        evolution 16047 0 2 00000002 000000bc [0032772517240330] 0.266ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#189)        evolution 16047 0 2 00000002 000000bd [0032772517240690] 0.267ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#190)        evolution 16047 0 3 00000003 000000be [0032772517241338] 0.268ms (+0.001ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#191)        evolution 16047 0 3 00000004 000000bf [0032772517242121] 0.269ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#192)        evolution 16047 0 3 00000004 000000c0 [0032772517242566] 0.270ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#193)        evolution 16047 0 3 00000004 000000c1 [0032772517243291] 0.271ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#194)        evolution 16047 0 3 00000004 000000c2 [0032772517243736] 0.272ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#195)        evolution 16047 0 3 00000004 000000c3 [0032772517244461] 0.273ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#196)        evolution 16047 0 3 00000004 000000c4 [0032772517244915] 0.274ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#197)        evolution 16047 0 3 00000004 000000c5 [0032772517245653] 0.275ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#198)        evolution 16047 0 3 00000004 000000c6 [0032772517246099] 0.276ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#199)        evolution 16047 0 3 00000004 000000c7 [0032772517246738] 0.277ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#200)        evolution 16047 0 3 00000004 000000c8 [0032772517247188] 0.278ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#201)        evolution 16047 0 3 00000004 000000c9 [0032772517247939] 0.279ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#202)        evolution 16047 0 3 00000004 000000ca [0032772517248398] 0.280ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#203)        evolution 16047 0 3 00000004 000000cb [0032772517249163] 0.281ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#204)        evolution 16047 0 3 00000004 000000cc [0032772517249622] 0.282ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#205)        evolution 16047 0 3 00000004 000000cd [0032772517250338] 0.283ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#206)        evolution 16047 0 3 00000004 000000ce [0032772517250788] 0.284ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#207)        evolution 16047 0 3 00000004 000000cf [0032772517251499] 0.285ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#208)        evolution 16047 0 3 00000004 000000d0 [0032772517251944] 0.286ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#209)        evolution 16047 0 3 00000004 000000d1 [0032772517252574] 0.287ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#210)        evolution 16047 0 3 00000004 000000d2 [0032772517253020] 0.287ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#211)        evolution 16047 0 3 00000004 000000d3 [0032772517253735] 0.289ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#212)        evolution 16047 0 3 00000004 000000d4 [0032772517254190] 0.289ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#213)        evolution 16047 0 3 00000004 000000d5 [0032772517254923] 0.291ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#214)        evolution 16047 0 3 00000004 000000d6 [0032772517255373] 0.291ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#215)        evolution 16047 0 3 00000004 000000d7 [0032772517256089] 0.292ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#216)        evolution 16047 0 3 00000004 000000d8 [0032772517256539] 0.293ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#217)        evolution 16047 0 3 00000004 000000d9 [0032772517257218] 0.294ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#218)        evolution 16047 0 3 00000004 000000da [0032772517257700] 0.295ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#219)        evolution 16047 0 3 00000004 000000db [0032772517258402] 0.296ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#220)        evolution 16047 0 3 00000004 000000dc [0032772517258852] 0.297ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#221)        evolution 16047 0 3 00000004 000000dd [0032772517259648] 0.298ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#222)        evolution 16047 0 3 00000004 000000de [0032772517260098] 0.299ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#223)        evolution 16047 0 3 00000003 000000df [0032772517260737] 0.300ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#224)        evolution 16047 0 2 00000002 000000e0 [0032772517261268] 0.301ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#225)        evolution 16047 0 2 00000002 000000e1 [0032772517261732] 0.302ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#226)        evolution 16047 0 2 00000002 000000e2 [0032772517262177] 0.303ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#227)        evolution 16047 0 2 00000002 000000e3 [0032772517262627] 0.303ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#228)        evolution 16047 0 2 00000002 000000e4 [0032772517263226] 0.304ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#229)        evolution 16047 0 2 00000002 000000e5 [0032772517263595] 0.305ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#230)        evolution 16047 0 2 00000002 000000e6 [0032772517263977] 0.306ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#231)        evolution 16047 0 2 00000002 000000e7 [0032772517264328] 0.306ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#232)        evolution 16047 0 2 00000002 000000e8 [0032772517265129] 0.308ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#233)        evolution 16047 0 2 00000002 000000e9 [0032772517265606] 0.308ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#234)        evolution 16047 0 2 00000002 000000ea [0032772517266056] 0.309ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#235)        evolution 16047 0 2 00000002 000000eb [0032772517266547] 0.310ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#236)        evolution 16047 0 2 00000002 000000ec [0032772517267150] 0.311ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#237)        evolution 16047 0 2 00000002 000000ed [0032772517267514] 0.312ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#238)        evolution 16047 0 2 00000002 000000ee [0032772517267852] 0.312ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#239)        evolution 16047 0 2 00000002 000000ef [0032772517268207] 0.313ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#240)        evolution 16047 0 2 00000002 000000f0 [0032772517268995] 0.314ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#241)        evolution 16047 0 2 00000002 000000f1 [0032772517269458] 0.315ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#242)        evolution 16047 0 2 00000002 000000f2 [0032772517269890] 0.315ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#243)        evolution 16047 0 2 00000002 000000f3 [0032772517270331] 0.316ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#244)        evolution 16047 0 2 00000002 000000f4 [0032772517270930] 0.317ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#245)        evolution 16047 0 2 00000002 000000f5 [0032772517271290] 0.318ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#246)        evolution 16047 0 2 00000002 000000f6 [0032772517271618] 0.318ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#247)        evolution 16047 0 2 00000002 000000f7 [0032772517271965] 0.319ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#248)        evolution 16047 0 2 00000002 000000f8 [0032772517272739] 0.320ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#249)        evolution 16047 0 2 00000002 000000f9 [0032772517273216] 0.321ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#250)        evolution 16047 0 2 00000002 000000fa [0032772517273670] 0.322ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#251)        evolution 16047 0 2 00000002 000000fb [0032772517274111] 0.322ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#252)        evolution 16047 0 2 00000002 000000fc [0032772517274723] 0.324ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#253)        evolution 16047 0 2 00000002 000000fd [0032772517275097] 0.324ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#254)        evolution 16047 0 2 00000002 000000fe [0032772517275439] 0.325ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#255)        evolution 16047 0 2 00000002 000000ff [0032772517275790] 0.325ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#256)        evolution 16047 0 2 00000002 00000100 [0032772517276568] 0.327ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#257)        evolution 16047 0 2 00000002 00000101 [0032772517277041] 0.327ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#258)        evolution 16047 0 2 00000002 00000102 [0032772517277486] 0.328ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#259)        evolution 16047 0 2 00000002 00000103 [0032772517277923] 0.329ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#260)        evolution 16047 0 2 00000002 00000104 [0032772517278530] 0.330ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#261)        evolution 16047 0 2 00000002 00000105 [0032772517278899] 0.330ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#262)        evolution 16047 0 2 00000002 00000106 [0032772517279327] 0.331ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#263)        evolution 16047 0 2 00000002 00000107 [0032772517279673] 0.332ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#264)        evolution 16047 0 2 00000002 00000108 [0032772517280456] 0.333ms (+0.005ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#265)        evolution 16047 0 2 00000002 00000109 [0032772517283989] 0.339ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#266)        evolution 16047 0 2 00000002 0000010a [0032772517284425] 0.340ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bb89> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#267)        evolution 16047 0 2 00000002 0000010b [0032772517284871] 0.340ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#268)        evolution 16047 0 2 00000002 0000010c [0032772517285474] 0.341ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#269)        evolution 16047 0 2 00000002 0000010d [0032772517285838] 0.342ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#270)        evolution 16047 0 2 00000002 0000010e [0032772517286171] 0.343ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#271)        evolution 16047 0 2 00000002 0000010f [0032772517286531] 0.343ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#272)        evolution 16047 0 2 00000002 00000110 [0032772517287314] 0.345ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0ca> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#273)        evolution 16047 0 2 00000001 00000111 [0032772517288385] 0.346ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (exit_mmap+0x164/0x1b0 <c01480a4>)
(T1/#274)        evolution 16047 0 2 00000000 00000112 [0032772517288876] 0.347ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (exit_mmap+0x15d/0x1b0 <c014809d>)
(T1/#275)        evolution 16047 0 3 00000000 00000113 [0032772517289389] 0.348ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (preempt_schedule+0x4f/0x70 <c027d10f>)
(T1/#276)        evolution 16047 0 3 00000000 00000114 [0032772517289753] 0.349ms (+0.001ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#277)        evolution 16047 0 3 00000001 00000115 [0032772517290390] 0.350ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#278)        evolution 16047 0 3 00000002 00000116 [0032772517291265] 0.351ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#279)        evolution 16047 0 3 00000002 00000117 [0032772517291697] 0.352ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#280)        evolution 16047 0 3 00000002 00000118 [0032772517292107] 0.352ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#281)        evolution 16047 0 3 00000002 00000119 [0032772517292503] 0.353ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#282) [ =>        evolution ] 0.354ms (+0.001ms)
(T1/#283)            <...>     2 0 1 00000002 0000011b [0032772517293976] 0.356ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#284)    <...>-2     0d..2  357µs : __schedule+0x2ea/0x630 <c027cc6a> <evolutio-16047> (77 69): 
(T1/#285)            <...>     2 0 1 00000002 0000011d [0032772517294976] 0.357ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#286)            <...>     2 0 1 00000001 0000011e [0032772517295421] 0.358ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#287)    <...>-2     0d..1  359µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#288)            <...>     2 0 1 00000001 00000120 [0032772517296501] 0.360ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: trace3.txt --]
[-- Type: text/plain; name=trace3.txt; charset=ANSI_X3.4-1968, Size: 3840 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 327 µs, #22/22, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)             dpkg 16229 0 9 00000002 00000000 [0034184390920878] 0.000ms (+3535970.095ms): <676b7064> (<00746500>)
(T1/#2)             dpkg 16229 0 9 00000002 00000002 [0034184390921351] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)             dpkg 16229 0 9 00000000 00000003 [0034184390921935] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4)     dpkg-16229 0dn.2    2µs!< (1)
(T1/#5)             dpkg 16229 0 2 00000001 00000005 [0034184391106965] 0.310ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0ca> (copy_pte_range+0xb7/0x1c0 <c0142ad7>)
(T1/#6)             dpkg 16229 0 2 00000001 00000006 [0034184391107405] 0.310ms (+0.000ms): __cond_resched_raw_spinlock+0x8/0x50 <c0111398> (copy_pte_range+0xa7/0x1c0 <c0142ac7>)
(T1/#7)             dpkg 16229 0 2 00000000 00000007 [0034184391107918] 0.311ms (+0.001ms): __cond_resched+0x9/0x70 <c0111329> (__cond_resched_raw_spinlock+0x3d/0x50 <c01113cd>)
(T1/#8)             dpkg 16229 0 3 00000000 00000008 [0034184391108602] 0.312ms (+0.000ms): __schedule+0xe/0x630 <c027c98e> (__cond_resched+0x45/0x70 <c0111365>)
(T1/#9)             dpkg 16229 0 3 00000000 00000009 [0034184391109088] 0.313ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ba>)
(T1/#10)             dpkg 16229 0 3 00000001 0000000a [0034184391109573] 0.314ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027c9e2>)
(T1/#11)             dpkg 16229 0 3 00000002 0000000b [0034184391110546] 0.316ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb2b>)
(T1/#12)             dpkg 16229 0 3 00000002 0000000c [0034184391110969] 0.316ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb45>)
(T1/#13)             dpkg 16229 0 3 00000002 0000000d [0034184391111383] 0.317ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#14)             dpkg 16229 0 3 00000002 0000000e [0034184391111775] 0.318ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb4c>)
(T4/#15) [ =>             dpkg ] 0.319ms (+0.001ms)
(T1/#16)            <...>     2 0 1 00000002 00000010 [0034184391113480] 0.321ms (+0.001ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc3d>)
(T3/#17)    <...>-2     0d..2  322µs : __schedule+0x2ea/0x630 <c027cc6a> <dpkg-16229> (74 69): 
(T1/#18)            <...>     2 0 1 00000002 00000012 [0034184391114743] 0.323ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cc76>)
(T1/#19)            <...>     2 0 1 00000001 00000013 [0034184391115178] 0.323ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#20)    <...>-2     0d..1  324µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#21)            <...>     2 0 1 00000001 00000015 [0034184391116385] 0.325ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
@ 2005-02-23 19:16 ` Hugh Dickins
  2005-02-23 19:36   ` Lee Revell
  2005-02-23 19:52 ` Lee Revell
  1 sibling, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 19:16 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 23 Feb 2005, Lee Revell wrote:
> 
> Did something change recently in the VM that made copy_pte_range and
> clear_page_range a lot more expensive?  I noticed a reference in the
> "Page Table Iterators" thread to excessive overhead introduced by
> aggressive page freeing.  That sure looks like what is going on in
> trace2.  trace1 and trace3 look like big fork latencies associated with
> copy_pte_range.

I'm just about to test this patch below: please give it a try: thanks...

Ingo's patch to reduce scheduling latencies, by checking for lockbreak
in copy_page_range, was in the -VP and -mm patchsets some months ago;
but got preempted by the 4level rework, and not reinstated since.
Restore it now in copy_pte_range - which mercifully makes it easier.

Signed-off-by: Hugh Dickins <hugh@veritas.com>

--- 2.6.11-rc4-bk9/mm/memory.c	2005-02-21 11:32:19.000000000 +0000
+++ linux/mm/memory.c	2005-02-23 18:35:28.000000000 +0000
@@ -328,6 +328,7 @@ static int copy_pte_range(struct mm_stru
 	pte_t *s, *d;
 	unsigned long vm_flags = vma->vm_flags;
 
+again:
 	d = dst_pte = pte_alloc_map(dst_mm, dst_pmd, addr);
 	if (!dst_pte)
 		return -ENOMEM;
@@ -338,11 +339,22 @@ static int copy_pte_range(struct mm_stru
 		if (pte_none(*s))
 			continue;
 		copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+		/*
+		 * We are holding two locks at this point - either of them
+		 * could generate latencies in another task on another CPU.
+		 */
+		if (need_resched() ||
+		    need_lockbreak(&src_mm->page_table_lock) ||
+		    need_lockbreak(&dst_mm->page_table_lock))
+			break;
 	}
 	pte_unmap_nested(src_pte);
 	pte_unmap(dst_pte);
 	spin_unlock(&src_mm->page_table_lock);
+
 	cond_resched_lock(&dst_mm->page_table_lock);
+	if (addr < end)
+		goto again;
 	return 0;
 }
 

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 19:16 ` Hugh Dickins
@ 2005-02-23 19:36   ` Lee Revell
  2005-02-23 20:06     ` Hugh Dickins
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 19:36 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > 
> > Did something change recently in the VM that made copy_pte_range and
> > clear_page_range a lot more expensive?  I noticed a reference in the
> > "Page Table Iterators" thread to excessive overhead introduced by
> > aggressive page freeing.  That sure looks like what is going on in
> > trace2.  trace1 and trace3 look like big fork latencies associated with
> > copy_pte_range.
> 
> I'm just about to test this patch below: please give it a try: thanks...
> 
> Ingo's patch to reduce scheduling latencies, by checking for lockbreak
> in copy_page_range, was in the -VP and -mm patchsets some months ago;
> but got preempted by the 4level rework, and not reinstated since.
> Restore it now in copy_pte_range - which mercifully makes it easier.

Aha, that explains why all the latency regressions involve the VM
subsystem. 

Thanks, your patch fixes the copy_pte_range latency.  Now zap_pte_range,
which Ingo also fixed a few months ago, is the worst offender.  Can this
fix be easily ported too?

Lee

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace5.txt --]
[-- Type: text/plain; name=trace5.txt; charset=ANSI_X3.4-1968, Size: 12790 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 197 µs, #74/74, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)            dmesg  3249 0 9 00000002 00000000 [0000163455166655] 0.000ms (+886841.090ms): <73656d64> (<61700067>)
(T1/#2)            dmesg  3249 0 9 00000002 00000002 [0000163455167105] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)            dmesg  3249 0 9 00000000 00000003 [0000163455167685] 0.001ms (+0.001ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4)    dmesg-3249  0dn.2    2µs < (1)
(T1/#5)            dmesg  3249 0 2 00000002 00000005 [0000163455168677] 0.003ms (+0.000ms): page_remove_rmap+0x8/0x40 <c0149f78> (zap_pte_range+0x13a/0x250 <c0142f4a>)
(T1/#6)            dmesg  3249 0 2 00000002 00000006 [0000163455168975] 0.003ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (page_remove_rmap+0x2c/0x40 <c0149f9c>)
(T1/#7)            dmesg  3249 0 2 00000002 00000007 [0000163455169390] 0.004ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#8)            dmesg  3249 0 2 00000002 00000008 [0000163455169687] 0.005ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#9)            dmesg  3249 0 2 00000002 00000009 [0000163455170314] 0.006ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#10)            dmesg  3249 0 2 00000002 0000000a [0000163455170636] 0.006ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#11)            dmesg  3249 0 2 00000002 0000000b [0000163455170944] 0.007ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#12)            dmesg  3249 0 2 00000002 0000000c [0000163455171314] 0.007ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#13)            dmesg  3249 0 2 00000002 0000000d [0000163455172090] 0.009ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#14)            dmesg  3249 0 2 00000002 0000000e [0000163455172432] 0.009ms (+0.000ms): set_page_dirty+0x8/0x60 <c013b428> (zap_pte_range+0x168/0x250 <c0142f78>)
(T1/#15)            dmesg  3249 0 2 00000002 0000000f [0000163455172774] 0.010ms (+0.000ms): page_remove_rmap+0x8/0x40 <c0149f78> (zap_pte_range+0x13a/0x250 <c0142f4a>)
(T1/#16)            dmesg  3249 0 2 00000002 00000010 [0000163455173040] 0.010ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (page_remove_rmap+0x2c/0x40 <c0149f9c>)
(T1/#17)            dmesg  3249 0 2 00000002 00000011 [0000163455173421] 0.011ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#18)            dmesg  3249 0 2 00000002 00000012 [0000163455173704] 0.011ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#19)            dmesg  3249 0 2 00000002 00000013 [0000163455174397] 0.012ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#20)            dmesg  3249 0 2 00000002 00000014 [0000163455174687] 0.013ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#21)            dmesg  3249 0 2 00000002 00000015 [0000163455175101] 0.014ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#22)            dmesg  3249 0 2 00000002 00000016 [0000163455175436] 0.014ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#23)            dmesg  3249 0 2 00000002 00000017 [0000163455176191] 0.015ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (zap_pte_range+0x14c/0x250 <c0142f5c>)
(T1/#24)            dmesg  3249 0 2 00000002 00000018 [0000163455176892] 0.017ms (+0.007ms): clear_page_range+0xe/0x1d0 <c01426fe> (exit_mmap+0xa3/0x1b0 <c0148013>)
(T1/#25)            dmesg  3249 0 2 00000002 00000019 [0000163455181331] 0.024ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#26)            dmesg  3249 0 2 00000002 0000001a [0000163455181781] 0.025ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#27)            dmesg  3249 0 2 00000002 0000001b [0000163455182166] 0.025ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#28)            dmesg  3249 0 2 00000002 0000001c [0000163455182746] 0.026ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#29)            dmesg  3249 0 2 00000002 0000001d [0000163455183036] 0.027ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#30)            dmesg  3249 0 2 00000002 0000001e [0000163455183293] 0.027ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#31)            dmesg  3249 0 2 00000002 0000001f [0000163455183557] 0.028ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#32)            dmesg  3249 0 2 00000002 00000020 [0000163455184330] 0.029ms (+0.042ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#33)            dmesg  3249 0 2 00000002 00000021 [0000163455210040] 0.072ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#34)            dmesg  3249 0 2 00000002 00000022 [0000163455210487] 0.072ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#35)            dmesg  3249 0 2 00000002 00000023 [0000163455210867] 0.073ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#36)            dmesg  3249 0 2 00000002 00000024 [0000163455211481] 0.074ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#37)            dmesg  3249 0 2 00000002 00000025 [0000163455211846] 0.075ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#38)            dmesg  3249 0 2 00000002 00000026 [0000163455212103] 0.075ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#39)            dmesg  3249 0 2 00000002 00000027 [0000163455212367] 0.076ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#40)            dmesg  3249 0 2 00000002 00000028 [0000163455213164] 0.077ms (+0.088ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#41)            dmesg  3249 0 2 00000002 00000029 [0000163455266176] 0.165ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#42)            dmesg  3249 0 2 00000002 0000002a [0000163455266536] 0.166ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#43)            dmesg  3249 0 2 00000002 0000002b [0000163455267095] 0.167ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#44)            dmesg  3249 0 2 00000002 0000002c [0000163455267667] 0.168ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#45)            dmesg  3249 0 2 00000002 0000002d [0000163455268033] 0.168ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#46)            dmesg  3249 0 2 00000002 0000002e [0000163455268375] 0.169ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#47)            dmesg  3249 0 2 00000002 0000002f [0000163455268717] 0.169ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#48)            dmesg  3249 0 2 00000002 00000030 [0000163455269434] 0.171ms (+0.006ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#49)            dmesg  3249 0 2 00000002 00000031 [0000163455273343] 0.177ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#50)            dmesg  3249 0 2 00000002 00000032 [0000163455273779] 0.178ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#51)            dmesg  3249 0 2 00000002 00000033 [0000163455274128] 0.178ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#52)            dmesg  3249 0 2 00000002 00000034 [0000163455274724] 0.179ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#53)            dmesg  3249 0 2 00000002 00000035 [0000163455275084] 0.180ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#54)            dmesg  3249 0 2 00000002 00000036 [0000163455275413] 0.180ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#55)            dmesg  3249 0 2 00000002 00000037 [0000163455275677] 0.181ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#56)            dmesg  3249 0 2 00000002 00000038 [0000163455276470] 0.182ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#57)            dmesg  3249 0 2 00000002 00000039 [0000163455277321] 0.184ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x190/0x1b0 <c0148100>)
(T1/#58)            dmesg  3249 0 2 00000001 0000003a [0000163455277715] 0.184ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x164/0x1b0 <c01480d4>)
(T1/#59)            dmesg  3249 0 2 00000000 0000003b [0000163455278192] 0.185ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x15d/0x1b0 <c01480cd>)
(T1/#60)            dmesg  3249 0 3 00000000 0000003c [0000163455278638] 0.186ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#61)            dmesg  3249 0 3 00000000 0000003d [0000163455278977] 0.186ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#62)            dmesg  3249 0 3 00000001 0000003e [0000163455279418] 0.187ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#63)            dmesg  3249 0 3 00000002 0000003f [0000163455280259] 0.189ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#64)            dmesg  3249 0 3 00000002 00000040 [0000163455280596] 0.189ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#65)            dmesg  3249 0 3 00000002 00000041 [0000163455280988] 0.190ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#66)            dmesg  3249 0 3 00000002 00000042 [0000163455281297] 0.190ms (+0.000ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#67) [ =>            dmesg ] 0.191ms (+0.000ms)
(T1/#68)            <...>     2 0 1 00000002 00000044 [0000163455282330] 0.192ms (+0.000ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#69)    <...>-2     0d..2  193µs : __schedule+0x2ea/0x630 <c027cc9a> <dmesg-3249> (75 69): 
(T1/#70)            <...>     2 0 1 00000002 00000046 [0000163455283124] 0.193ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#71)            <...>     2 0 1 00000001 00000047 [0000163455283506] 0.194ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#72)    <...>-2     0d..1  194µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#73)            <...>     2 0 1 00000001 00000049 [0000163455284627] 0.196ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 19:36   ` Lee Revell
@ 2005-02-23 20:06     ` Hugh Dickins
  2005-02-23 20:10       ` Lee Revell
                         ` (2 more replies)
  0 siblings, 3 replies; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 20:06 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 23 Feb 2005, Lee Revell wrote:
> On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> > 
> > I'm just about to test this patch below: please give it a try: thanks...

I'm very sorry, there's two things wrong with that version: _must_
increment addr before breaking out, and better to check after pte_none
too (we can question whether it might be checking too often, but this
replicates what Ingo was doing).  Please replace by new patch below,
which I'm now running through lmbench.

> Aha, that explains why all the latency regressions involve the VM
> subsystem. 
> 
> Thanks, your patch fixes the copy_pte_range latency.

Great, if the previous patch fixed that latency then this new one
will too, no need to report on that; but please get rid of the old
patch before it leaks too many of your pages.

> Now zap_pte_range,
> which Ingo also fixed a few months ago, is the worst offender.  Can this
> fix be easily ported too?

That surprises me: all the zap_pte_range latency fixes I know of
are in 2.6.11-rc, perhaps Ingo knows of something missing there?

Hugh

Ingo's patch to reduce scheduling latencies, by checking for lockbreak
in copy_page_range, was in the -VP and -mm patchsets some months ago;
but got preempted by the 4level rework, and not reinstated since.
Restore it now in copy_pte_range - which mercifully makes it easier.

Signed-off-by: Hugh Dickins <hugh@veritas.com>

--- 2.6.11-rc4-bk/mm/memory.c	2005-02-21 11:32:19.000000000 +0000
+++ linux/mm/memory.c	2005-02-23 19:46:40.000000000 +0000
@@ -328,21 +328,33 @@ static int copy_pte_range(struct mm_stru
 	pte_t *s, *d;
 	unsigned long vm_flags = vma->vm_flags;
 
+again:
 	d = dst_pte = pte_alloc_map(dst_mm, dst_pmd, addr);
 	if (!dst_pte)
 		return -ENOMEM;
 
 	spin_lock(&src_mm->page_table_lock);
 	s = src_pte = pte_offset_map_nested(src_pmd, addr);
-	for (; addr < end; addr += PAGE_SIZE, s++, d++) {
-		if (pte_none(*s))
-			continue;
-		copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+	for (; addr < end; s++, d++) {
+		if (!pte_none(*s))
+			copy_one_pte(dst_mm, src_mm, d, s, vm_flags, addr);
+		addr += PAGE_SIZE;
+		/*
+		 * We are holding two locks at this point - either of them
+		 * could generate latencies in another task on another CPU.
+		 */
+		if (need_resched() ||
+		    need_lockbreak(&src_mm->page_table_lock) ||
+		    need_lockbreak(&dst_mm->page_table_lock))
+			break;
 	}
 	pte_unmap_nested(src_pte);
 	pte_unmap(dst_pte);
 	spin_unlock(&src_mm->page_table_lock);
+
 	cond_resched_lock(&dst_mm->page_table_lock);
+	if (addr < end)
+		goto again;
 	return 0;
 }
 

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 20:06     ` Hugh Dickins
@ 2005-02-23 20:10       ` Lee Revell
  2005-02-23 20:30       ` Lee Revell
  2005-02-23 20:53       ` Hugh Dickins
  2 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 20:10 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 2005-02-23 at 20:06 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > On Wed, 2005-02-23 at 19:16 +0000, Hugh Dickins wrote:
> > > 
> > > I'm just about to test this patch below: please give it a try: thanks...
> 
> I'm very sorry, there's two things wrong with that version: _must_
> increment addr before breaking out, and better to check after pte_none
> too (we can question whether it might be checking too often, but this
> replicates what Ingo was doing).  Please replace by new patch below,
> which I'm now running through lmbench.

OK, I will report any interesting results with the new patch.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 20:06     ` Hugh Dickins
  2005-02-23 20:10       ` Lee Revell
@ 2005-02-23 20:30       ` Lee Revell
  2005-02-23 21:03         ` Hugh Dickins
  2005-02-23 20:53       ` Hugh Dickins
  2 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 20:30 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Wed, 2005-02-23 at 20:06 +0000, Hugh Dickins wrote:
> > 
> > Thanks, your patch fixes the copy_pte_range latency.
> 
> Great, if the previous patch fixed that latency then this new one
> will too, no need to report on that; but please get rid of the old
> patch before it leaks too many of your pages.

clear_page_range is also problematic.

Lee

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace7.txt --]
[-- Type: text/plain; name=trace7.txt; charset=ANSI_X3.4-1968, Size: 49938 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 313 µs, #291/291, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)          cc1plus  3510 0 9 00000002 00000000 [0000188528469818] 0.000ms (+811152.283ms): <70316363> (<0073756c>)
(T1/#2)          cc1plus  3510 0 9 00000002 00000002 [0000188528470164] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)          cc1plus  3510 0 9 00000000 00000003 [0000188528470660] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4)  cc1plus-3510  0dn.2    2µs+< (1)
(T1/#5)          cc1plus  3510 0 2 00000002 00000005 [0000188528521949] 0.086ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#6)          cc1plus  3510 0 2 00000002 00000006 [0000188528522356] 0.087ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#7)          cc1plus  3510 0 2 00000002 00000007 [0000188528522777] 0.088ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#8)          cc1plus  3510 0 2 00000002 00000008 [0000188528523382] 0.089ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#9)          cc1plus  3510 0 2 00000002 00000009 [0000188528523695] 0.089ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#10)          cc1plus  3510 0 2 00000002 0000000a [0000188528523952] 0.090ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#11)          cc1plus  3510 0 2 00000002 0000000b [0000188528524216] 0.090ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#12)          cc1plus  3510 0 2 00000002 0000000c [0000188528524975] 0.091ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#13)          cc1plus  3510 0 2 00000002 0000000d [0000188528525518] 0.092ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#14)          cc1plus  3510 0 2 00000002 0000000e [0000188528525878] 0.093ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#15)          cc1plus  3510 0 2 00000002 0000000f [0000188528526264] 0.093ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#16)          cc1plus  3510 0 2 00000002 00000010 [0000188528526791] 0.094ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#17)          cc1plus  3510 0 2 00000002 00000011 [0000188528527169] 0.095ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#18)          cc1plus  3510 0 2 00000002 00000012 [0000188528527426] 0.095ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#19)          cc1plus  3510 0 2 00000002 00000013 [0000188528527898] 0.096ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#20)          cc1plus  3510 0 2 00000002 00000014 [0000188528528616] 0.097ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#21)          cc1plus  3510 0 2 00000002 00000015 [0000188528529086] 0.098ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#22)          cc1plus  3510 0 2 00000002 00000016 [0000188528529446] 0.099ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#23)          cc1plus  3510 0 2 00000002 00000017 [0000188528529833] 0.099ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#24)          cc1plus  3510 0 2 00000002 00000018 [0000188528530360] 0.100ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#25)          cc1plus  3510 0 2 00000002 00000019 [0000188528530665] 0.101ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#26)          cc1plus  3510 0 2 00000002 0000001a [0000188528530922] 0.101ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#27)          cc1plus  3510 0 2 00000002 0000001b [0000188528531186] 0.102ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#28)          cc1plus  3510 0 2 00000002 0000001c [0000188528531904] 0.103ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#29)          cc1plus  3510 0 2 00000002 0000001d [0000188528532309] 0.103ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#30)          cc1plus  3510 0 2 00000002 0000001e [0000188528532669] 0.104ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#31)          cc1plus  3510 0 2 00000002 0000001f [0000188528533055] 0.105ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#32)          cc1plus  3510 0 2 00000002 00000020 [0000188528533582] 0.106ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#33)          cc1plus  3510 0 2 00000002 00000021 [0000188528533887] 0.106ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#34)          cc1plus  3510 0 2 00000002 00000022 [0000188528534144] 0.107ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#35)          cc1plus  3510 0 2 00000002 00000023 [0000188528534408] 0.107ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#36)          cc1plus  3510 0 2 00000002 00000024 [0000188528535126] 0.108ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#37)          cc1plus  3510 0 2 00000002 00000025 [0000188528535531] 0.109ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#38)          cc1plus  3510 0 2 00000002 00000026 [0000188528535891] 0.109ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#39)          cc1plus  3510 0 2 00000002 00000027 [0000188528536277] 0.110ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#40)          cc1plus  3510 0 2 00000002 00000028 [0000188528536804] 0.111ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#41)          cc1plus  3510 0 2 00000002 00000029 [0000188528537109] 0.111ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#42)          cc1plus  3510 0 2 00000002 0000002a [0000188528537366] 0.112ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#43)          cc1plus  3510 0 2 00000002 0000002b [0000188528537630] 0.112ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#44)          cc1plus  3510 0 3 00000003 0000002c [0000188528538238] 0.113ms (+0.000ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#45)          cc1plus  3510 0 3 00000004 0000002d [0000188528538772] 0.114ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#46)          cc1plus  3510 0 3 00000004 0000002e [0000188528539128] 0.115ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#47)          cc1plus  3510 0 3 00000004 0000002f [0000188528539654] 0.116ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#48)          cc1plus  3510 0 3 00000004 00000030 [0000188528540010] 0.116ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#49)          cc1plus  3510 0 3 00000004 00000031 [0000188528540515] 0.117ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#50)          cc1plus  3510 0 3 00000004 00000032 [0000188528540871] 0.118ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#51)          cc1plus  3510 0 3 00000004 00000033 [0000188528541384] 0.119ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#52)          cc1plus  3510 0 3 00000004 00000034 [0000188528541740] 0.119ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#53)          cc1plus  3510 0 3 00000004 00000035 [0000188528542243] 0.120ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#54)          cc1plus  3510 0 3 00000004 00000036 [0000188528542690] 0.121ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#55)          cc1plus  3510 0 3 00000004 00000037 [0000188528543212] 0.122ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#56)          cc1plus  3510 0 3 00000004 00000038 [0000188528543568] 0.122ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#57)          cc1plus  3510 0 3 00000004 00000039 [0000188528544045] 0.123ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#58)          cc1plus  3510 0 3 00000004 0000003a [0000188528544401] 0.124ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#59)          cc1plus  3510 0 3 00000004 0000003b [0000188528544806] 0.124ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#60)          cc1plus  3510 0 3 00000004 0000003c [0000188528545162] 0.125ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#61)          cc1plus  3510 0 3 00000004 0000003d [0000188528545567] 0.126ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#62)          cc1plus  3510 0 3 00000004 0000003e [0000188528545923] 0.126ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#63)          cc1plus  3510 0 3 00000004 0000003f [0000188528546353] 0.127ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#64)          cc1plus  3510 0 3 00000004 00000040 [0000188528546709] 0.127ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#65)          cc1plus  3510 0 3 00000004 00000041 [0000188528547211] 0.128ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#66)          cc1plus  3510 0 3 00000004 00000042 [0000188528547567] 0.129ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#67)          cc1plus  3510 0 3 00000004 00000043 [0000188528548041] 0.130ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#68)          cc1plus  3510 0 3 00000004 00000044 [0000188528548397] 0.130ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#69)          cc1plus  3510 0 3 00000004 00000045 [0000188528548916] 0.131ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#70)          cc1plus  3510 0 3 00000004 00000046 [0000188528549272] 0.132ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#71)          cc1plus  3510 0 3 00000004 00000047 [0000188528549688] 0.132ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#72)          cc1plus  3510 0 3 00000004 00000048 [0000188528550044] 0.133ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#73)          cc1plus  3510 0 3 00000004 00000049 [0000188528550449] 0.134ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#74)          cc1plus  3510 0 3 00000004 0000004a [0000188528550805] 0.134ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#75)          cc1plus  3510 0 3 00000004 0000004b [0000188528551286] 0.135ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#76)          cc1plus  3510 0 3 00000004 0000004c [0000188528551642] 0.136ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#77)          cc1plus  3510 0 3 00000003 0000004d [0000188528552147] 0.136ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#78)          cc1plus  3510 0 2 00000002 0000004e [0000188528552606] 0.137ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#79)          cc1plus  3510 0 2 00000002 0000004f [0000188528553011] 0.138ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#80)          cc1plus  3510 0 2 00000002 00000050 [0000188528553386] 0.139ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#81)          cc1plus  3510 0 2 00000002 00000051 [0000188528553768] 0.139ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#82)          cc1plus  3510 0 2 00000002 00000052 [0000188528554309] 0.140ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#83)          cc1plus  3510 0 2 00000002 00000053 [0000188528554599] 0.141ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#84)          cc1plus  3510 0 2 00000002 00000054 [0000188528554856] 0.141ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#85)          cc1plus  3510 0 2 00000002 00000055 [0000188528555120] 0.141ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#86)          cc1plus  3510 0 2 00000002 00000056 [0000188528555860] 0.143ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#87)          cc1plus  3510 0 2 00000002 00000057 [0000188528556332] 0.143ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#88)          cc1plus  3510 0 2 00000002 00000058 [0000188528556707] 0.144ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#89)          cc1plus  3510 0 2 00000002 00000059 [0000188528557089] 0.145ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#90)          cc1plus  3510 0 2 00000002 0000005a [0000188528557616] 0.146ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#91)          cc1plus  3510 0 2 00000002 0000005b [0000188528557921] 0.146ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#92)          cc1plus  3510 0 2 00000002 0000005c [0000188528558178] 0.147ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#93)          cc1plus  3510 0 2 00000002 0000005d [0000188528558442] 0.147ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#94)          cc1plus  3510 0 2 00000002 0000005e [0000188528559160] 0.148ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#95)          cc1plus  3510 0 2 00000002 0000005f [0000188528559565] 0.149ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#96)          cc1plus  3510 0 2 00000002 00000060 [0000188528559925] 0.149ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#97)          cc1plus  3510 0 2 00000002 00000061 [0000188528560311] 0.150ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#98)          cc1plus  3510 0 2 00000002 00000062 [0000188528560838] 0.151ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#99)          cc1plus  3510 0 2 00000002 00000063 [0000188528561143] 0.151ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#100)          cc1plus  3510 0 2 00000002 00000064 [0000188528561400] 0.152ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#101)          cc1plus  3510 0 2 00000002 00000065 [0000188528561664] 0.152ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#102)          cc1plus  3510 0 2 00000002 00000066 [0000188528562382] 0.154ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#103)          cc1plus  3510 0 2 00000002 00000067 [0000188528562787] 0.154ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#104)          cc1plus  3510 0 2 00000002 00000068 [0000188528563147] 0.155ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#105)          cc1plus  3510 0 2 00000002 00000069 [0000188528563533] 0.155ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#106)          cc1plus  3510 0 2 00000002 0000006a [0000188528564060] 0.156ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#107)          cc1plus  3510 0 2 00000002 0000006b [0000188528564365] 0.157ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#108)          cc1plus  3510 0 2 00000002 0000006c [0000188528564622] 0.157ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#109)          cc1plus  3510 0 2 00000002 0000006d [0000188528564886] 0.158ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#110)          cc1plus  3510 0 2 00000002 0000006e [0000188528565604] 0.159ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#111)          cc1plus  3510 0 2 00000002 0000006f [0000188528566009] 0.160ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#112)          cc1plus  3510 0 2 00000002 00000070 [0000188528566369] 0.160ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#113)          cc1plus  3510 0 2 00000002 00000071 [0000188528566724] 0.161ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#114)          cc1plus  3510 0 2 00000002 00000072 [0000188528567251] 0.162ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#115)          cc1plus  3510 0 2 00000002 00000073 [0000188528567556] 0.162ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#116)          cc1plus  3510 0 2 00000002 00000074 [0000188528567813] 0.163ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#117)          cc1plus  3510 0 2 00000002 00000075 [0000188528568077] 0.163ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#118)          cc1plus  3510 0 2 00000002 00000076 [0000188528568795] 0.164ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#119)          cc1plus  3510 0 2 00000002 00000077 [0000188528569200] 0.165ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#120)          cc1plus  3510 0 2 00000002 00000078 [0000188528569560] 0.165ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#121)          cc1plus  3510 0 2 00000002 00000079 [0000188528569946] 0.166ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#122)          cc1plus  3510 0 2 00000002 0000007a [0000188528570473] 0.167ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#123)          cc1plus  3510 0 2 00000002 0000007b [0000188528570778] 0.167ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#124)          cc1plus  3510 0 2 00000002 0000007c [0000188528571035] 0.168ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#125)          cc1plus  3510 0 2 00000002 0000007d [0000188528571299] 0.168ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#126)          cc1plus  3510 0 2 00000002 0000007e [0000188528572017] 0.170ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#127)          cc1plus  3510 0 2 00000002 0000007f [0000188528572422] 0.170ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#128)          cc1plus  3510 0 2 00000002 00000080 [0000188528572782] 0.171ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#129)          cc1plus  3510 0 2 00000002 00000081 [0000188528573218] 0.172ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#130)          cc1plus  3510 0 2 00000002 00000082 [0000188528573745] 0.172ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#131)          cc1plus  3510 0 2 00000002 00000083 [0000188528574122] 0.173ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#132)          cc1plus  3510 0 2 00000002 00000084 [0000188528574379] 0.173ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#133)          cc1plus  3510 0 2 00000002 00000085 [0000188528574643] 0.174ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#134)          cc1plus  3510 0 2 00000002 00000086 [0000188528575361] 0.175ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#135)          cc1plus  3510 0 2 00000002 00000087 [0000188528575810] 0.176ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#136)          cc1plus  3510 0 2 00000002 00000088 [0000188528576170] 0.176ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#137)          cc1plus  3510 0 2 00000002 00000089 [0000188528576525] 0.177ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#138)          cc1plus  3510 0 2 00000002 0000008a [0000188528577148] 0.178ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#139)          cc1plus  3510 0 2 00000002 0000008b [0000188528577530] 0.179ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#140)          cc1plus  3510 0 2 00000002 0000008c [0000188528577872] 0.179ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#141)          cc1plus  3510 0 2 00000002 0000008d [0000188528578219] 0.180ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#142)          cc1plus  3510 0 2 00000002 0000008e [0000188528579020] 0.181ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#143)          cc1plus  3510 0 2 00000002 0000008f [0000188528579510] 0.182ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#144)          cc1plus  3510 0 2 00000002 00000090 [0000188528579951] 0.183ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#145)          cc1plus  3510 0 2 00000002 00000091 [0000188528580473] 0.184ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#146)          cc1plus  3510 0 2 00000002 00000092 [0000188528581085] 0.185ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#147)          cc1plus  3510 0 2 00000002 00000093 [0000188528581472] 0.185ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#148)          cc1plus  3510 0 2 00000002 00000094 [0000188528581810] 0.186ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#149)          cc1plus  3510 0 2 00000002 00000095 [0000188528582226] 0.187ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#150)          cc1plus  3510 0 2 00000002 00000096 [0000188528583025] 0.188ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#151)          cc1plus  3510 0 2 00000002 00000097 [0000188528583515] 0.189ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#152)          cc1plus  3510 0 2 00000002 00000098 [0000188528583961] 0.189ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#153)          cc1plus  3510 0 2 00000002 00000099 [0000188528584424] 0.190ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#154)          cc1plus  3510 0 2 00000002 0000009a [0000188528585027] 0.191ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#155)          cc1plus  3510 0 2 00000002 0000009b [0000188528585419] 0.192ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#156)          cc1plus  3510 0 2 00000002 0000009c [0000188528585761] 0.192ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#157)          cc1plus  3510 0 2 00000002 0000009d [0000188528586112] 0.193ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#158)          cc1plus  3510 0 2 00000002 0000009e [0000188528586913] 0.194ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#159)          cc1plus  3510 0 2 00000002 0000009f [0000188528587403] 0.195ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#160)          cc1plus  3510 0 2 00000002 000000a0 [0000188528587849] 0.196ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#161)          cc1plus  3510 0 2 00000002 000000a1 [0000188528588281] 0.197ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#162)          cc1plus  3510 0 2 00000002 000000a2 [0000188528588888] 0.198ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#163)          cc1plus  3510 0 2 00000002 000000a3 [0000188528589275] 0.198ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#164)          cc1plus  3510 0 2 00000002 000000a4 [0000188528589613] 0.199ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#165)          cc1plus  3510 0 2 00000002 000000a5 [0000188528589964] 0.199ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#166)          cc1plus  3510 0 2 00000002 000000a6 [0000188528590760] 0.201ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#167)          cc1plus  3510 0 2 00000002 000000a7 [0000188528591246] 0.202ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#168)          cc1plus  3510 0 2 00000002 000000a8 [0000188528591687] 0.202ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#169)          cc1plus  3510 0 2 00000002 000000a9 [0000188528592146] 0.203ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#170)          cc1plus  3510 0 2 00000002 000000aa [0000188528592749] 0.204ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#171)          cc1plus  3510 0 2 00000002 000000ab [0000188528593141] 0.205ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#172)          cc1plus  3510 0 2 00000002 000000ac [0000188528593474] 0.205ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#173)          cc1plus  3510 0 2 00000002 000000ad [0000188528593816] 0.206ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#174)          cc1plus  3510 0 2 00000002 000000ae [0000188528594608] 0.207ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#175)          cc1plus  3510 0 2 00000002 000000af [0000188528595112] 0.208ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#176)          cc1plus  3510 0 2 00000002 000000b0 [0000188528595548] 0.209ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#177)          cc1plus  3510 0 2 00000002 000000b1 [0000188528595976] 0.209ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#178)          cc1plus  3510 0 2 00000002 000000b2 [0000188528596588] 0.210ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#179)          cc1plus  3510 0 2 00000002 000000b3 [0000188528596970] 0.211ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#180)          cc1plus  3510 0 2 00000002 000000b4 [0000188528597308] 0.212ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#181)          cc1plus  3510 0 2 00000002 000000b5 [0000188528597645] 0.212ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#182)          cc1plus  3510 0 2 00000002 000000b6 [0000188528598433] 0.214ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#183)          cc1plus  3510 0 2 00000002 000000b7 [0000188528598923] 0.214ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#184)          cc1plus  3510 0 2 00000002 000000b8 [0000188528599355] 0.215ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#185)          cc1plus  3510 0 2 00000002 000000b9 [0000188528599769] 0.216ms (+0.000ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#186)          cc1plus  3510 0 2 00000002 000000ba [0000188528600368] 0.217ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#187)          cc1plus  3510 0 2 00000002 000000bb [0000188528600755] 0.217ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#188)          cc1plus  3510 0 2 00000002 000000bc [0000188528601083] 0.218ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#189)          cc1plus  3510 0 2 00000002 000000bd [0000188528601416] 0.218ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#190)          cc1plus  3510 0 2 00000002 000000be [0000188528602217] 0.220ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#191)          cc1plus  3510 0 2 00000002 000000bf [0000188528602699] 0.221ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#192)          cc1plus  3510 0 2 00000002 000000c0 [0000188528603135] 0.221ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#193)          cc1plus  3510 0 2 00000002 000000c1 [0000188528603500] 0.222ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#194)          cc1plus  3510 0 2 00000002 000000c2 [0000188528604103] 0.223ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#195)          cc1plus  3510 0 2 00000002 000000c3 [0000188528604494] 0.224ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#196)          cc1plus  3510 0 2 00000002 000000c4 [0000188528604836] 0.224ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#197)          cc1plus  3510 0 2 00000002 000000c5 [0000188528605183] 0.225ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#198)          cc1plus  3510 0 2 00000002 000000c6 [0000188528605984] 0.226ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#199)          cc1plus  3510 0 2 00000002 000000c7 [0000188528606461] 0.227ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#200)          cc1plus  3510 0 2 00000002 000000c8 [0000188528606902] 0.228ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#201)          cc1plus  3510 0 2 00000002 000000c9 [0000188528607253] 0.228ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#202)          cc1plus  3510 0 2 00000002 000000ca [0000188528607856] 0.229ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#203)          cc1plus  3510 0 2 00000002 000000cb [0000188528608243] 0.230ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#204)          cc1plus  3510 0 2 00000002 000000cc [0000188528608571] 0.230ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#205)          cc1plus  3510 0 2 00000002 000000cd [0000188528608904] 0.231ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#206)          cc1plus  3510 0 3 00000003 000000ce [0000188528609579] 0.232ms (+0.000ms): free_pages_bulk+0xe/0x200 <c0138fde> (free_hot_cold_page+0x103/0x130 <c01395d3>)
(T1/#207)          cc1plus  3510 0 3 00000004 000000cf [0000188528610178] 0.233ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#208)          cc1plus  3510 0 3 00000004 000000d0 [0000188528610610] 0.234ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#209)          cc1plus  3510 0 3 00000004 000000d1 [0000188528611181] 0.235ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#210)          cc1plus  3510 0 3 00000004 000000d2 [0000188528611613] 0.235ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#211)          cc1plus  3510 0 3 00000004 000000d3 [0000188528612144] 0.236ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#212)          cc1plus  3510 0 3 00000004 000000d4 [0000188528612585] 0.237ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#213)          cc1plus  3510 0 3 00000004 000000d5 [0000188528613085] 0.238ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#214)          cc1plus  3510 0 3 00000004 000000d6 [0000188528613535] 0.239ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#215)          cc1plus  3510 0 3 00000004 000000d7 [0000188528614120] 0.240ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#216)          cc1plus  3510 0 3 00000004 000000d8 [0000188528614556] 0.240ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#217)          cc1plus  3510 0 3 00000004 000000d9 [0000188528615132] 0.241ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#218)          cc1plus  3510 0 3 00000004 000000da [0000188528615560] 0.242ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#219)          cc1plus  3510 0 3 00000004 000000db [0000188528616145] 0.243ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#220)          cc1plus  3510 0 3 00000004 000000dc [0000188528616577] 0.244ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#221)          cc1plus  3510 0 3 00000004 000000dd [0000188528617135] 0.245ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#222)          cc1plus  3510 0 3 00000004 000000de [0000188528617571] 0.245ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#223)          cc1plus  3510 0 3 00000004 000000df [0000188528618143] 0.246ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#224)          cc1plus  3510 0 3 00000004 000000e0 [0000188528618579] 0.247ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#225)          cc1plus  3510 0 3 00000004 000000e1 [0000188528619137] 0.248ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#226)          cc1plus  3510 0 3 00000004 000000e2 [0000188528619578] 0.249ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#227)          cc1plus  3510 0 3 00000004 000000e3 [0000188528620150] 0.250ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#228)          cc1plus  3510 0 3 00000004 000000e4 [0000188528620586] 0.250ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#229)          cc1plus  3510 0 3 00000004 000000e5 [0000188528621140] 0.251ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#230)          cc1plus  3510 0 3 00000004 000000e6 [0000188528621581] 0.252ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#231)          cc1plus  3510 0 3 00000004 000000e7 [0000188528622134] 0.253ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#232)          cc1plus  3510 0 3 00000004 000000e8 [0000188528622566] 0.254ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#233)          cc1plus  3510 0 3 00000004 000000e9 [0000188528623174] 0.255ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#234)          cc1plus  3510 0 3 00000004 000000ea [0000188528623615] 0.255ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#235)          cc1plus  3510 0 3 00000004 000000eb [0000188528624267] 0.256ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#236)          cc1plus  3510 0 3 00000004 000000ec [0000188528624704] 0.257ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#237)          cc1plus  3510 0 3 00000004 000000ed [0000188528625289] 0.258ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#238)          cc1plus  3510 0 3 00000004 000000ee [0000188528625833] 0.259ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#239)          cc1plus  3510 0 3 00000004 000000ef [0000188528626481] 0.260ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#240)          cc1plus  3510 0 3 00000004 000000f0 [0000188528627134] 0.261ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#241)          cc1plus  3510 0 3 00000004 000000f1 [0000188528627822] 0.262ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#242)          cc1plus  3510 0 3 00000004 000000f2 [0000188528628524] 0.264ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#243)          cc1plus  3510 0 3 00000004 000000f3 [0000188528629226] 0.265ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#244)          cc1plus  3510 0 3 00000004 000000f4 [0000188528629919] 0.266ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#245)          cc1plus  3510 0 3 00000004 000000f5 [0000188528630558] 0.267ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0xc0/0x200 <c0139090>)
(T1/#246)          cc1plus  3510 0 3 00000004 000000f6 [0000188528630999] 0.268ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#247)          cc1plus  3510 0 3 00000004 000000f7 [0000188528631715] 0.269ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#248)          cc1plus  3510 0 3 00000004 000000f8 [0000188528632241] 0.270ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#249)          cc1plus  3510 0 3 00000004 000000f9 [0000188528632759] 0.271ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#250)          cc1plus  3510 0 3 00000004 000000fa [0000188528633393] 0.272ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#251)          cc1plus  3510 0 3 00000004 000000fb [0000188528634154] 0.273ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#252)          cc1plus  3510 0 3 00000004 000000fc [0000188528634703] 0.274ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#253)          cc1plus  3510 0 3 00000004 000000fd [0000188528635297] 0.275ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#254)          cc1plus  3510 0 3 00000004 000000fe [0000188528635963] 0.276ms (+0.000ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#255)          cc1plus  3510 0 3 00000004 000000ff [0000188528636507] 0.277ms (+0.001ms): bad_range+0xb/0x60 <c0138eeb> (free_pages_bulk+0x114/0x200 <c01390e4>)
(T1/#256)          cc1plus  3510 0 3 00000003 00000100 [0000188528637160] 0.278ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (free_pages_bulk+0x1e5/0x200 <c01391b5>)
(T1/#257)          cc1plus  3510 0 2 00000002 00000101 [0000188528637700] 0.279ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#258)          cc1plus  3510 0 2 00000002 00000102 [0000188528638181] 0.280ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#259)          cc1plus  3510 0 2 00000002 00000103 [0000188528638640] 0.280ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#260)          cc1plus  3510 0 2 00000002 00000104 [0000188528639081] 0.281ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#261)          cc1plus  3510 0 2 00000002 00000105 [0000188528639693] 0.282ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#262)          cc1plus  3510 0 2 00000002 00000106 [0000188528640053] 0.283ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#263)          cc1plus  3510 0 2 00000002 00000107 [0000188528640382] 0.283ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#264)          cc1plus  3510 0 2 00000002 00000108 [0000188528640719] 0.284ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#265)          cc1plus  3510 0 2 00000002 00000109 [0000188528641547] 0.285ms (+0.006ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#266)          cc1plus  3510 0 2 00000002 0000010a [0000188528645458] 0.292ms (+0.000ms): __mod_page_state+0xa/0x30 <c013a09a> (clear_page_range+0x183/0x1d0 <c0142873>)
(T1/#267)          cc1plus  3510 0 2 00000002 0000010b [0000188528645908] 0.292ms (+0.000ms): free_page_and_swap_cache+0x9/0x70 <c014bbb9> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#268)          cc1plus  3510 0 2 00000002 0000010c [0000188528646353] 0.293ms (+0.001ms): __page_cache_release+0xb/0xc0 <c013ed4b> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#269)          cc1plus  3510 0 2 00000002 0000010d [0000188528646956] 0.294ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (__page_cache_release+0xad/0xc0 <c013eded>)
(T1/#270)          cc1plus  3510 0 2 00000002 0000010e [0000188528647348] 0.295ms (+0.000ms): free_hot_page+0x8/0x10 <c0139608> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#271)          cc1plus  3510 0 2 00000002 0000010f [0000188528647681] 0.295ms (+0.000ms): free_hot_cold_page+0xe/0x130 <c01394de> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#272)          cc1plus  3510 0 2 00000002 00000110 [0000188528648032] 0.296ms (+0.001ms): __mod_page_state+0xa/0x30 <c013a09a> (free_hot_cold_page+0x2a/0x130 <c01394fa>)
(T1/#273)          cc1plus  3510 0 2 00000002 00000111 [0000188528648824] 0.297ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (clear_page_range+0x1a0/0x1d0 <c0142890>)
(T1/#274)          cc1plus  3510 0 2 00000002 00000112 [0000188528649746] 0.299ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x190/0x1b0 <c0148100>)
(T1/#275)          cc1plus  3510 0 2 00000001 00000113 [0000188528650210] 0.300ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x164/0x1b0 <c01480d4>)
(T1/#276)          cc1plus  3510 0 2 00000000 00000114 [0000188528650714] 0.300ms (+0.001ms): preempt_schedule+0xa/0x70 <c027d0fa> (exit_mmap+0x15d/0x1b0 <c01480cd>)
(T1/#277)          cc1plus  3510 0 3 00000000 00000115 [0000188528651386] 0.302ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#278)          cc1plus  3510 0 3 00000000 00000116 [0000188528651843] 0.302ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#279)          cc1plus  3510 0 3 00000001 00000117 [0000188528652284] 0.303ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#280)          cc1plus  3510 0 3 00000002 00000118 [0000188528653099] 0.304ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#281)          cc1plus  3510 0 3 00000002 00000119 [0000188528653508] 0.305ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#282)          cc1plus  3510 0 3 00000002 0000011a [0000188528653895] 0.306ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#283)          cc1plus  3510 0 3 00000002 0000011b [0000188528654291] 0.306ms (+0.000ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#284) [ =>          cc1plus ] 0.307ms (+0.000ms)
(T1/#285)            <...>     2 0 1 00000002 0000011d [0000188528655354] 0.308ms (+0.000ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#286)    <...>-2     0d..2  309µs : __schedule+0x2ea/0x630 <c027cc9a> <cc1plus-3510> (7d 69): 
(T1/#287)            <...>     2 0 1 00000002 0000011f [0000188528656152] 0.310ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#288)            <...>     2 0 1 00000001 00000120 [0000188528656584] 0.310ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#289)    <...>-2     0d..1  311µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#290)            <...>     2 0 1 00000001 00000122 [0000188528657710] 0.312ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 20:30       ` Lee Revell
@ 2005-02-23 21:03         ` Hugh Dickins
  2005-02-23 22:14           ` Lee Revell
  2005-02-23 23:27           ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
  0 siblings, 2 replies; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 21:03 UTC (permalink / raw)
  To: Lee Revell; +Cc: Nick Piggin, Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 23 Feb 2005, Lee Revell wrote:
> > > 
> > > Thanks, your patch fixes the copy_pte_range latency.
> 
> clear_page_range is also problematic.

Yes, I saw that from your other traces too.  I know there are plans
to improve clear_page_range during 2.6.12, but I didn't realize that
it had become very much worse than its antecedent clear_page_tables,
and I don't see missing latency fixes for that.  Nick's the expert.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 21:03         ` Hugh Dickins
@ 2005-02-23 22:14           ` Lee Revell
  2005-02-23 23:52             ` PPC RT Patch john cooper
  2005-02-23 23:27           ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
  1 sibling, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 22:14 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Nick Piggin, Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 2005-02-23 at 21:03 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > > > 
> > > > Thanks, your patch fixes the copy_pte_range latency.
> > 
> > clear_page_range is also problematic.
> 
> Yes, I saw that from your other traces too.

Heh, sorry, that one was a dupe... I should know to give the files
better names.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* PPC RT Patch..
  2005-02-23 22:14           ` Lee Revell
@ 2005-02-23 23:52             ` john cooper
  2005-02-24  4:20               ` Frank Rowand
  0 siblings, 1 reply; 26+ messages in thread
From: john cooper @ 2005-02-23 23:52 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, john cooper, Saksena, Manas

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

Ingo,
     We've had a PPC port of your RT work underway with
a focus on trace instrumentation.  This is based upon
realtime-preempt-2.6.11-rc2-V0.7.37-02.  A diff is
attached.

To the extent possible the tracing facilities are the
same as your x86 work.  In the process a few PPC/gcc
issues needed to be resolved.  There is also a bug fix
contained for tlb_gather_mmu() which was causing debug
assertions to be generated in a path which attempted to
sleep with a non-zero preempt count.

This does build and function when SMP is configured,
though we have not yet verified it on other than a
uniprocessor.  As a simplifying assumption, testing has
thus far concentrated on the following modes:

PREEMPT_NONE
     - verify baseline regression

PREEMPT_RT && !PREEMPT_SMP
     - typical for an embedded RT PPC application

PREEMPT_RT && PREEMPT_SMP
     - kicks in live locking code which otherwise receives no
     coverage.  This is functionally equivalent to the above
     config on a single CPU target thus no MP dynamic testing
     is achieved.  Still quite useful IMHO.

The target used for development/testing is an Ampro EnCore PP1
which sports a 300Mhz MPC8245.  For testing this boots with NFS
as root.  An mp3 decode at nice --20 is launched which requires
just under 20% of the CPU to maintain an uninterrupted audio
decode and output.  To this a series of "du -s /" are launched
to soak up excess CPU bandwidth.  Perhaps not rigorous but a
fair sanity check and load for the purpose at hand.

Under these conditions maximum scheduling latencies are seen in
the 120-150us range.  Note no attempt has yet been made to
optimize arch specific paths and full trace instrumentation has
been enabled.

I've written some logging code to help find problems such as
the tlb issue above.  As it has not been made general I've
removed it from this patch.  At some point I'll likely revisit
this.

Comments/suggestions welcome.

-john


-- 
john.cooper@timesys.com

[-- Attachment #2: realtime-preempt-2.6.11-rc2-V0.7.37-02-ppc --]
[-- Type: text/plain, Size: 39151 bytes --]

--- ./arch/ppc/syslib/i8259.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/syslib/i8259.c	2005-02-10 12:57:45.000000000 -0500
@@ -10,7 +10,7 @@ unsigned char cached_8259[2] = { 0xff, 0
 #define cached_A1 (cached_8259[0])
 #define cached_21 (cached_8259[1])
 
-static DEFINE_SPINLOCK(i8259_lock);
+static DEFINE_RAW_SPINLOCK(i8259_lock);
 
 int i8259_pic_irq_offset;
 
=================================================================
--- ./arch/ppc/syslib/ocp.c.ORG	2004-12-24 16:35:23.000000000 -0500
+++ ./arch/ppc/syslib/ocp.c	2005-02-23 16:51:04.009548560 -0500
@@ -49,7 +49,6 @@
 #include <asm/io.h>
 #include <asm/ocp.h>
 #include <asm/errno.h>
-#include <asm/rwsem.h>
 #include <asm/semaphore.h>
 
 //#define DBG(x)	printk x
=================================================================
--- ./arch/ppc/kernel/Makefile.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/Makefile	2005-02-18 20:34:07.000000000 -0500
@@ -13,7 +13,7 @@ extra-y				+= vmlinux.lds
 
 obj-y				:= entry.o traps.o irq.o idle.o time.o misc.o \
 					process.o signal.o ptrace.o align.o \
-					semaphore.o syscalls.o setup.o \
+					syscalls.o setup.o \
 					cputable.o ppc_htab.o perfmon.o
 obj-$(CONFIG_6xx)		+= l2cr.o cpu_setup_6xx.o
 obj-$(CONFIG_POWER4)		+= cpu_setup_power4.o
@@ -26,6 +26,9 @@ obj-$(CONFIG_TAU)		+= temp.o
 obj-$(CONFIG_ALTIVEC)		+= vecemu.o vector.o
 obj-$(CONFIG_FSL_BOOKE)		+= perfmon_fsl_booke.o
 
+obj-$(CONFIG_ASM_SEMAPHORES)	+= semaphore.o
+obj-$(CONFIG_MCOUNT)		+= mcount.o
+
 ifndef CONFIG_MATH_EMULATION
 obj-$(CONFIG_8xx)		+= softemu8xx.o
 endif
=================================================================
--- ./arch/ppc/kernel/signal.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/signal.c	2005-02-09 19:16:25.000000000 -0500
@@ -704,6 +704,13 @@ int do_signal(sigset_t *oldset, struct p
 	unsigned long frame, newsp;
 	int signr, ret;
 
+#ifdef CONFIG_PREEMPT_RT
+	/*
+	 * Fully-preemptible kernel does not need interrupts disabled:
+	 */
+	local_irq_enable();
+	preempt_check_resched();
+#endif
 	if (!oldset)
 		oldset = &current->blocked;
 
=================================================================
--- ./arch/ppc/kernel/time.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/time.c	2005-02-19 10:46:17.000000000 -0500
@@ -91,7 +91,7 @@ extern unsigned long wall_jiffies;
 
 static long time_offset;
 
-DEFINE_SPINLOCK(rtc_lock);
+DEFINE_RAW_SPINLOCK(rtc_lock);
 
 EXPORT_SYMBOL(rtc_lock);
 
@@ -109,7 +109,7 @@ static inline int tb_delta(unsigned *jif
 }
 
 #ifdef CONFIG_SMP
-unsigned long profile_pc(struct pt_regs *regs)
+unsigned long notrace profile_pc(struct pt_regs *regs)
 {
 	unsigned long pc = instruction_pointer(regs);
 
@@ -126,7 +126,7 @@ EXPORT_SYMBOL(profile_pc);
  * with interrupts disabled.
  * We set it up to overflow again in 1/HZ seconds.
  */
-void timer_interrupt(struct pt_regs * regs)
+void notrace timer_interrupt(struct pt_regs * regs)
 {
 	int next_dec;
 	unsigned long cpu = smp_processor_id();
=================================================================
--- ./arch/ppc/kernel/traps.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/traps.c	2005-02-09 18:59:40.000000000 -0500
@@ -72,7 +72,7 @@ void (*debugger_fault_handler)(struct pt
  * Trap & Exception support
  */
 
-DEFINE_SPINLOCK(die_lock);
+DEFINE_RAW_SPINLOCK(die_lock);
 
 void die(const char * str, struct pt_regs * fp, long err)
 {
@@ -111,6 +111,10 @@ void _exception(int signr, struct pt_reg
 		debugger(regs);
 		die("Exception in kernel mode", regs, signr);
 	}
+#ifdef CONFIG_PREEMPT_RT
+	local_irq_enable();
+	preempt_check_resched();
+#endif
 	info.si_signo = signr;
 	info.si_errno = 0;
 	info.si_code = code;
=================================================================
--- ./arch/ppc/kernel/ppc_ksyms.c.ORG	2004-12-24 16:35:28.000000000 -0500
+++ ./arch/ppc/kernel/ppc_ksyms.c	2005-02-08 12:00:56.000000000 -0500
@@ -294,9 +294,11 @@ EXPORT_SYMBOL(console_drivers);
 EXPORT_SYMBOL(xmon);
 EXPORT_SYMBOL(xmon_printf);
 #endif
+#ifdef CONFIG_ASM_SEMAPHORES
 EXPORT_SYMBOL(__up);
 EXPORT_SYMBOL(__down);
 EXPORT_SYMBOL(__down_interruptible);
+#endif
 
 #if defined(CONFIG_KGDB) || defined(CONFIG_XMON)
 extern void (*debugger)(struct pt_regs *regs);
=================================================================
--- ./arch/ppc/kernel/entry.S.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/entry.S	2005-02-23 16:30:55.924205448 -0500
@@ -8,6 +8,7 @@
  *  rewritten by Paul Mackerras.
  *    Copyright (C) 1996 Paul Mackerras.
  *  MPC8xx modifications Copyright (C) 1997 Dan Malek (dmalek@jlc.net).
+ *  RT_PREEMPT support (C) Timesys Corp.  <john.cooper@timesys.com>
  *
  *  This file contains the system call entry code, context switch
  *  code, and exception/interrupt return code for PowerPC.
@@ -44,6 +45,48 @@
 #define LOAD_MSR_KERNEL(r, x)	li r,(x)
 #endif
 
+#if defined(CONFIG_LATENCY_TRACE) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING) \
+    || defined(CONFIG_CRITICAL_TIMING)
+#define TFSIZE 64	/* frame SHDB multiple of 16 bytes */
+#define TFR12 48	/* TODO: prune this down -- safe but overkill */
+#define TFR11 44
+#define TFR10 40
+#define TFR9 36
+#define TFR8 32
+#define TFR7 28
+#define TFR6 24
+#define TFR5 20
+#define TFR4 16
+#define TFR3 12
+#define TFR0 8
+#define PUSHFRAME() \
+	stwu	r1, -TFSIZE(r1); \
+	stw	r12, TFR12(r1); \
+	stw	r11, TFR11(r1); \
+	stw	r10, TFR10(r1); \
+	stw	r9, TFR9(r1); \
+	stw	r8, TFR8(r1); \
+	stw	r7, TFR7(r1); \
+	stw	r6, TFR6(r1); \
+	stw	r5, TFR5(r1); \
+	stw	r4, TFR4(r1); \
+	stw	r3, TFR3(r1); \
+	stw	r0, TFR0(r1)
+#define POPFRAME() \
+	lwz	r12, TFR12(r1); \
+	lwz	r11, TFR11(r1); \
+	lwz	r10, TFR10(r1); \
+	lwz	r9, TFR9(r1); \
+	lwz	r8, TFR8(r1); \
+	lwz	r7, TFR7(r1); \
+	lwz	r6, TFR6(r1); \
+	lwz	r5, TFR5(r1); \
+	lwz	r4, TFR4(r1); \
+	lwz	r3, TFR3(r1); \
+	lwz	r0, TFR0(r1); \
+	addi	r1, r1, TFSIZE
+#endif
+
 #ifdef CONFIG_BOOKE
 #define	COR	r8	/* Critical Offset Register (COR) */
 #define BOOKE_LOAD_COR	lis COR,crit_save@ha
@@ -197,6 +240,20 @@ _GLOBAL(DoSyscall)
 	lwz	r11,_CCR(r1)	/* Clear SO bit in CR */
 	rlwinm	r11,r11,0,4,2
 	stw	r11,_CCR(r1)
+#ifdef CONFIG_LATENCY_TRACE
+	lwz	r3,GPR0(r1)
+	lwz	r4,GPR3(r1)
+	lwz	r5,GPR4(r1)
+	lwz	r6,GPR5(r1)
+	bl	sys_call
+	lwz	r0,GPR0(r1)
+	lwz	r3,GPR3(r1)
+	lwz	r4,GPR4(r1)
+	lwz	r5,GPR5(r1)
+	lwz	r6,GPR6(r1)
+	lwz	r7,GPR7(r1)
+	lwz	r8,GPR8(r1)
+#endif
 #ifdef SHOW_SYSCALLS
 	bl	do_show_syscall
 #endif /* SHOW_SYSCALLS */
@@ -250,6 +307,21 @@ syscall_exit_cont:
 	andis.	r10,r0,DBCR0_IC@h
 	bnel-	load_dbcr0
 #endif
+#if defined(CONFIG_LATENCY_TRACE) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING) \
+    || defined(CONFIG_CRITICAL_TIMING)
+	PUSHFRAME();
+#ifdef CONFIG_CRITICAL_TIMING
+	bl	touch_critical_timing
+#endif
+#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING
+	bl	trace_irqs_on
+#endif
+#ifdef CONFIG_LATENCY_TRACE
+	lwz	r3, RESULT+TFSIZE(r1)
+	bl	sys_ret
+#endif
+	POPFRAME();
+#endif
 	stwcx.	r0,0,r1			/* to clear the reservation */
 	lwz	r4,_LINK(r1)
 	lwz	r5,_CCR(r1)
@@ -614,32 +686,38 @@ restore_user:
 
 /* N.B. the only way to get here is from the beq following ret_from_except. */
 resume_kernel:
+	lis	r9,kernel_preemption@ha
+	lwz	r9,kernel_preemption@l(r9)
+	cmpwi	0,r9,0
+	beq	restore
 	/* check current_thread_info->preempt_count */
 	rlwinm	r9,r1,0,0,18
 	lwz	r0,TI_PREEMPT(r9)
 	cmpwi	0,r0,0		/* if non-zero, just restore regs and return */
 	bne	restore
+check_resched:
 	lwz	r0,TI_FLAGS(r9)
 	andi.	r0,r0,_TIF_NEED_RESCHED
 	beq+	restore
 	andi.	r0,r3,MSR_EE	/* interrupts off? */
 	beq	restore		/* don't schedule if so */
-1:	lis	r0,PREEMPT_ACTIVE@h
-	stw	r0,TI_PREEMPT(r9)
+1:
 	ori	r10,r10,MSR_EE
-	SYNC
-	MTMSRD(r10)		/* hard-enable interrupts */
-	bl	schedule
 	LOAD_MSR_KERNEL(r10,MSR_KERNEL)
 	SYNC
 	MTMSRD(r10)		/* disable interrupts */
+	bl	preempt_schedule_irq
 	rlwinm	r9,r1,0,0,18
 	li	r0,0
 	stw	r0,TI_PREEMPT(r9)
+#if 0
 	lwz	r3,TI_FLAGS(r9)
 	andi.	r0,r3,_TIF_NEED_RESCHED
 	bne-	1b
 #else
+	b	check_resched
+#endif
+#else
 resume_kernel:
 #endif /* CONFIG_PREEMPT */
 
=================================================================
--- ./arch/ppc/kernel/process.c.ORG	2004-12-24 16:34:45.000000000 -0500
+++ ./arch/ppc/kernel/process.c	2005-02-09 18:43:14.000000000 -0500
@@ -360,6 +360,7 @@ void show_regs(struct pt_regs * regs)
 	print_symbol("%s\n", regs->nip);
 	printk("LR [%08lx] ", regs->link);
 	print_symbol("%s\n", regs->link);
+	printk("preempt: %08x\n", preempt_count());
 #endif
 	show_stack(current, (unsigned long *) regs->gpr[1]);
 }
=================================================================
--- ./arch/ppc/kernel/mcount.S.ORG	2005-02-18 19:51:33.000000000 -0500
+++ ./arch/ppc/kernel/mcount.S	2005-02-23 14:25:18.780025528 -0500
@@ -0,0 +1,86 @@
+/*
+ *  linux/arch/ppc/kernel/mcount.S
+ *
+ *  Copyright (C) 2005 TimeSys Corporation, john.cooper@timesys.com
+ */
+
+#include <asm/ppc_asm.h>
+
+/*
+ * stack frame in effect when calling __mcount():
+ *
+ *	52: RA to caller
+ *	48: caller chain
+ *	44: [alignment pad]
+ *	40: saved LR to prolog/target function
+ *	36: r10
+ *	32: r9
+ *	28: r8
+ *	24: r7
+ *	20: r6
+ *	16: r5
+ *	12: r4
+ *	 8: r3
+ *	 4: LR save for __mcount() use
+ *	 0: frame chain pointer / current r1
+ */
+
+/* preamble present in each traced function:
+ *
+ *	.data
+ *	.align	2
+ * 0:
+ *	.long	0
+ *	.previous
+ *	mflr	r0
+ *	lis	r11, 0b@ha
+ *	stw	r0, 4(r1)
+ *	addi	r0, r11, 0b@l
+ *	bl	_mcount
+ */
+
+	.text
+.globl _mcount
+_mcount:
+	lis	r11,mcount_enabled@ha
+	lwz	r11,mcount_enabled@l(r11)
+	cmpwi	0,r11,0
+	beq	disabled
+	stwu	r1,-48(r1)	/* local frame */
+	stw	r3, 8(r1)
+	stw	r4, 12(r1)
+	stw	r5, 16(r1)
+	stw	r6, 20(r1)
+	mflr	r4		/* RA to function prolog */
+	stw	r7, 24(r1)
+	stw	r8, 28(r1)
+	stw	r9, 32(r1)
+	stw	r10,36(r1)
+	stw	r4, 40(r1)
+	bl	__mcount	/* __void mcount(void) */
+	nop
+	lwz	r0, 40(r1)
+	lwz	r3, 8(r1)
+	mtctr	r0
+	lwz	r4, 12(r1)
+	lwz	r5, 16(r1)
+	lwz	r6, 20(r1)
+	lwz	r0, 52(r1)	/* RA to function caller */
+	lwz	r7, 24(r1)
+	lwz	r8, 28(r1)
+	mtlr	r0
+	lwz	r9, 32(r1)
+	lwz	r10,36(r1)
+	addi	r1,r1,48	/* toss frame */
+	bctr			/* return to target function */
+
+/* the function preamble modified LR in getting here so we need
+ * to restore its LR and return to the preamble otherwise
+ */
+disabled:
+        mflr    r12             /* return address to preamble */
+        lwz     r11, 4(r1)
+        mtlr    r11             /* restore LR modified by preamble */
+        mtctr   r12
+        bctr
+
=================================================================
--- ./arch/ppc/kernel/smp.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/smp.c	2005-02-22 10:21:39.000000000 -0500
@@ -163,7 +163,7 @@ void smp_send_stop(void)
  * static memory requirements. It also looks cleaner.
  * Stolen from the i386 version.
  */
-static DEFINE_SPINLOCK(call_lock);
+static DEFINE_RAW_SPINLOCK(call_lock);
 
 static struct call_data_struct {
 	void (*func) (void *info);
@@ -397,5 +397,15 @@ int __cpu_up(unsigned int cpu)
 
 void smp_cpus_done(unsigned int max_cpus)
 {
-	smp_ops->setup_cpu(0);
+	if (smp_ops)
+		smp_ops->setup_cpu(0);
+}
+
+/* this function sends a 'reschedule' IPI to all other CPUs.
+ * This is used when RT tasks are starving and other CPUs
+ * might be able to run them
+ */
+void smp_send_reschedule_allbutself(void)
+{
+	smp_message_pass(MSG_ALL_BUT_SELF, PPC_MSG_RESCHEDULE, 0, 0);
 }
=================================================================
--- ./arch/ppc/kernel/irq.c.ORG	2004-12-24 16:35:24.000000000 -0500
+++ ./arch/ppc/kernel/irq.c	2005-02-19 21:43:53.000000000 -0500
@@ -135,10 +135,11 @@ skip:
 	return 0;
 }
 
-void do_IRQ(struct pt_regs *regs)
+void notrace do_IRQ(struct pt_regs *regs)
 {
 	int irq, first = 1;
         irq_enter();
+	trace_special(regs->nip, irq, 0);
 
 	/*
 	 * Every platform is required to implement ppc_md.get_irq.
=================================================================
--- ./arch/ppc/kernel/idle.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/kernel/idle.c	2005-02-23 13:02:34.000000000 -0500
@@ -39,6 +39,7 @@ void default_idle(void)
 	powersave = ppc_md.power_save;
 
 	if (!need_resched()) {
+		stop_critical_timing();
 		if (powersave != NULL)
 			powersave();
 		else {
=================================================================
--- ./arch/ppc/mm/fault.c.ORG	2004-12-24 16:34:29.000000000 -0500
+++ ./arch/ppc/mm/fault.c	2005-02-19 21:32:02.000000000 -0500
@@ -92,7 +92,7 @@ static int store_updates_sp(struct pt_re
  * the error_code parameter is ESR for a data fault, 0 for an instruction
  * fault.
  */
-int do_page_fault(struct pt_regs *regs, unsigned long address,
+int notrace do_page_fault(struct pt_regs *regs, unsigned long address,
 		  unsigned long error_code)
 {
 	struct vm_area_struct * vma;
@@ -104,6 +104,7 @@ int do_page_fault(struct pt_regs *regs, 
 #else
 	int is_write = 0;
 
+	trace_special(regs->nip, error_code, address);
 	/*
 	 * Fortunately the bit assignments in SRR1 for an instruction
 	 * fault and DSISR for a data fault are mostly the same for the
=================================================================
--- ./arch/ppc/mm/init.c.ORG	2005-02-01 16:26:40.000000000 -0500
+++ ./arch/ppc/mm/init.c	2005-02-21 13:26:10.000000000 -0500
@@ -56,7 +56,7 @@
 #endif
 #define MAX_LOW_MEM	CONFIG_LOWMEM_SIZE
 
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU_LOCKED(struct mmu_gather, mmu_gathers);
 
 unsigned long total_memory;
 unsigned long total_lowmem;
=================================================================
--- ./arch/ppc/boot/common/util.S.ORG	2004-12-24 16:35:49.000000000 -0500
+++ ./arch/ppc/boot/common/util.S	2005-02-23 14:27:14.577421648 -0500
@@ -289,5 +289,15 @@ _GLOBAL(flush_data_cache)
 	bdnz	00b
 10:	blr
 
-	.previous
+#ifdef CONFIG_MCOUNT
+/* innocuous _mcount for boot header
+ */
+_GLOBAL(_mcount)
+	mflr	r12		/* return address to preamble */
+	lwz     r11, 4(r1)
+	mtlr    r11		/* restore LR modified by preamble */
+	mtctr	r12
+	bctr
+#endif
 
+	.previous
=================================================================
--- ./arch/ppc/platforms/encpp1_time.c.ORG	2005-02-02 20:42:55.000000000 -0500
+++ ./arch/ppc/platforms/encpp1_time.c	2005-02-09 16:35:10.000000000 -0500
@@ -155,9 +155,9 @@ int pp1_set_rtc_time (unsigned long nowt
 {
 	unsigned char save_control, save_freq_select;
 	struct rtc_time tm;
-	extern spinlock_t rtc_lock;
+	extern raw_spinlock_t rtc_lock;
 
-	spin_lock (&rtc_lock);
+	__raw_spin_lock (&rtc_lock);
 	to_tm (nowtime, &tm);
 
 	called ();
@@ -202,7 +202,7 @@ int pp1_set_rtc_time (unsigned long nowt
 
 	if ((time_state == TIME_ERROR) || (time_state == TIME_BAD))
 		time_state = TIME_OK;
-	spin_unlock (&rtc_lock);
+	__raw_spin_unlock (&rtc_lock);
 	return 0;
 }
 
=================================================================
--- ./arch/ppc/lib/locks.c.ORG	2004-12-24 16:34:26.000000000 -0500
+++ ./arch/ppc/lib/locks.c	2005-02-07 19:21:14.000000000 -0500
@@ -8,6 +8,7 @@
 #include <linux/sched.h>
 #include <linux/spinlock.h>
 #include <linux/module.h>
+#include <linux/rt_lock.h>
 #include <asm/ppc_asm.h>
 #include <asm/smp.h>
 
@@ -43,7 +44,7 @@ static inline unsigned long __spin_trylo
 	return ret;
 }
 
-void _raw_spin_lock(spinlock_t *lock)
+void __raw_spin_lock(raw_spinlock_t *lock)
 {
 	int cpu = smp_processor_id();
 	unsigned int stuck = INIT_STUCK;
@@ -63,9 +64,9 @@ void _raw_spin_lock(spinlock_t *lock)
 	lock->owner_pc = (unsigned long)__builtin_return_address(0);
 	lock->owner_cpu = cpu;
 }
-EXPORT_SYMBOL(_raw_spin_lock);
+EXPORT_SYMBOL(__raw_spin_lock);
 
-int _raw_spin_trylock(spinlock_t *lock)
+int __raw_spin_trylock(raw_spinlock_t *lock)
 {
 	if (__spin_trylock(&lock->lock))
 		return 0;
@@ -73,9 +74,9 @@ int _raw_spin_trylock(spinlock_t *lock)
 	lock->owner_pc = (unsigned long)__builtin_return_address(0);
 	return 1;
 }
-EXPORT_SYMBOL(_raw_spin_trylock);
+EXPORT_SYMBOL(__raw_spin_trylock);
 
-void _raw_spin_unlock(spinlock_t *lp)
+void __raw_spin_unlock(raw_spinlock_t *lp)
 {
   	if ( !lp->lock )
 		printk("_spin_unlock(%p): no lock cpu %d curr PC %p %s/%d\n",
@@ -89,7 +90,7 @@ void _raw_spin_unlock(spinlock_t *lp)
 	wmb();
 	lp->lock = 0;
 }
-EXPORT_SYMBOL(_raw_spin_unlock);
+EXPORT_SYMBOL(__raw_spin_unlock);
 
 
 /*
@@ -97,7 +98,7 @@ EXPORT_SYMBOL(_raw_spin_unlock);
  * with the high bit (sign) being the "write" bit.
  * -- Cort
  */
-void _raw_read_lock(rwlock_t *rw)
+void __raw_read_lock(raw_rwlock_t *rw)
 {
 	unsigned long stuck = INIT_STUCK;
 	int cpu = smp_processor_id();
@@ -123,9 +124,9 @@ again:
 	}
 	wmb();
 }
-EXPORT_SYMBOL(_raw_read_lock);
+EXPORT_SYMBOL(__raw_read_lock);
 
-void _raw_read_unlock(rwlock_t *rw)
+void __raw_read_unlock(raw_rwlock_t *rw)
 {
 	if ( rw->lock == 0 )
 		printk("_read_unlock(): %s/%d (nip %08lX) lock %lx\n",
@@ -134,9 +135,9 @@ void _raw_read_unlock(rwlock_t *rw)
 	wmb();
 	atomic_dec((atomic_t *) &(rw)->lock);
 }
-EXPORT_SYMBOL(_raw_read_unlock);
+EXPORT_SYMBOL(__raw_read_unlock);
 
-void _raw_write_lock(rwlock_t *rw)
+void __raw_write_lock(raw_rwlock_t *rw)
 {
 	unsigned long stuck = INIT_STUCK;
 	int cpu = smp_processor_id();
@@ -175,9 +176,9 @@ again:
 	}
 	wmb();
 }
-EXPORT_SYMBOL(_raw_write_lock);
+EXPORT_SYMBOL(__raw_write_lock);
 
-int _raw_write_trylock(rwlock_t *rw)
+int __raw_write_trylock(raw_rwlock_t *rw)
 {
 	if (test_and_set_bit(31, &(rw)->lock)) /* someone has a write lock */
 		return 0;
@@ -190,9 +191,9 @@ int _raw_write_trylock(rwlock_t *rw)
 	wmb();
 	return 1;
 }
-EXPORT_SYMBOL(_raw_write_trylock);
+EXPORT_SYMBOL(__raw_write_trylock);
 
-void _raw_write_unlock(rwlock_t *rw)
+void __raw_write_unlock(raw_rwlock_t *rw)
 {
 	if ( !(rw->lock & (1<<31)) )
 		printk("_write_lock(): %s/%d (nip %08lX) lock %lx\n",
@@ -201,6 +202,6 @@ void _raw_write_unlock(rwlock_t *rw)
 	wmb();
 	clear_bit(31,&(rw)->lock);
 }
-EXPORT_SYMBOL(_raw_write_unlock);
+EXPORT_SYMBOL(__raw_write_unlock);
 
 #endif
=================================================================
--- ./arch/ppc/lib/dec_and_lock.c.ORG	2004-12-24 16:35:27.000000000 -0500
+++ ./arch/ppc/lib/dec_and_lock.c	2005-02-07 21:25:52.000000000 -0500
@@ -19,7 +19,7 @@
  */
 
 #ifndef ATOMIC_DEC_AND_LOCK
-int _atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock)
+int _atomic_dec_and_raw_spin_lock(atomic_t *atomic, raw_spinlock_t *lock)
 {
 	int counter;
 	int newcount;
@@ -35,12 +35,12 @@ int _atomic_dec_and_lock(atomic_t *atomi
 			return 0;
 	}
 
-	spin_lock(lock);
+	_raw_spin_lock(lock);
 	if (atomic_dec_and_test(atomic))
 		return 1;
-	spin_unlock(lock);
+	_raw_spin_unlock(lock);
 	return 0;
 }
 
-EXPORT_SYMBOL(_atomic_dec_and_lock);
+EXPORT_SYMBOL(_atomic_dec_and_raw_spin_lock);
 #endif /* ATOMIC_DEC_AND_LOCK */
=================================================================
--- ./arch/ppc/Kconfig.ORG	2005-02-02 20:42:55.000000000 -0500
+++ ./arch/ppc/Kconfig	2005-02-08 19:54:51.000000000 -0500
@@ -15,13 +15,6 @@ config GENERIC_HARDIRQS
 	bool
 	default y
 
-config RWSEM_GENERIC_SPINLOCK
-	bool
-
-config RWSEM_XCHGADD_ALGORITHM
-	bool
-	default y
-
 config GENERIC_CALIBRATE_DELAY
 	bool
 	default y
@@ -866,15 +859,21 @@ config NR_CPUS
 	depends on SMP
 	default "4"
 
-config PREEMPT
-	bool "Preemptible Kernel"
-	help
-	  This option reduces the latency of the kernel when reacting to
-	  real-time or interactive events by allowing a low priority process to
-	  be preempted even if it is in kernel mode executing a system call.
+source "lib/Kconfig.RT"
 
-	  Say Y here if you are building a kernel for a desktop, embedded
-	  or real-time system.  Say N if you are unsure.
+config RWSEM_GENERIC_SPINLOCK
+	bool
+	depends on !PREEMPT_RT
+	default y
+
+config ASM_SEMAPHORES
+	bool
+	depends on !PREEMPT_RT
+	default y
+
+config RWSEM_XCHGADD_ALGORITHM
+	bool
+	depends on !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT
 
 config HIGHMEM
 	bool "High memory support"
=================================================================
--- ./include/asm-generic/tlb.h.ORG	2005-02-01 16:26:51.000000000 -0500
+++ ./include/asm-generic/tlb.h	2005-02-21 13:43:21.000000000 -0500
@@ -50,7 +50,8 @@ struct mmu_gather {
 #define tlb_mm(tlb) ((tlb)->mm)
 
 /* Users of the generic TLB shootdown code must declare this storage space. */
-DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
+
+DECLARE_PER_CPU_LOCKED(struct mmu_gather, mmu_gathers);
 
 /* tlb_gather_mmu
  *	Return a pointer to an initialized struct mmu_gather.
@@ -58,7 +59,8 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_g
 static inline struct mmu_gather *
 tlb_gather_mmu(struct mm_struct *mm, unsigned int full_mm_flush)
 {
-	struct mmu_gather *tlb = &get_cpu_var(mmu_gathers);
+	struct mmu_gather *tlb = &get_cpu_var_locked(mmu_gathers,
+		_smp_processor_id());
 
 	tlb->mm = mm;
 
@@ -99,7 +101,7 @@ tlb_finish_mmu(struct mmu_gather *tlb, u
 		freed = rss;
 	mm->rss = rss - freed;
 	tlb_flush_mmu(tlb, start, end);
-	put_cpu_var(mmu_gathers);
+	put_cpu_var_locked(mmu_gathers, _smp_processor_id());
 
 	/* keep the page table cache within bounds */
 	check_pgt_cache();
=================================================================
--- ./include/asm-generic/percpu.h.ORG	2005-02-01 16:26:51.000000000 -0500
+++ ./include/asm-generic/percpu.h	2005-02-21 13:11:53.000000000 -0500
@@ -53,6 +53,9 @@ do {								\
 #endif	/* SMP */
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#define DECLARE_PER_CPU_LOCKED(type, name) \
+	extern __typeof__(type) per_cpu__##name##_locked; \
+	extern spinlock_t per_cpu_lock__##name##_locked
 
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
=================================================================
--- ./include/asm-ppc/rtc.h.ORG	2004-12-24 16:33:49.000000000 -0500
+++ ./include/asm-ppc/rtc.h	2005-02-23 14:30:06.705254240 -0500
@@ -24,6 +24,11 @@
 #ifndef __ASM_RTC_H__
 #define __ASM_RTC_H__
 
+#ifdef CONFIG_ENCPP1	/* Ampro work-around.  Ugh. */
+#define cpu_mhz	300
+#define cpu_khz (cpu_mhz * 1000)
+#endif
+
 #ifdef __KERNEL__
 
 #include <linux/rtc.h>
=================================================================
--- ./include/asm-ppc/semaphore.h.ORG	2004-12-24 16:34:57.000000000 -0500
+++ ./include/asm-ppc/semaphore.h	2005-02-08 11:39:18.000000000 -0500
@@ -18,6 +18,13 @@
 
 #include <asm/atomic.h>
 #include <asm/system.h>
+
+#ifdef CONFIG_PREEMPT_RT
+
+#include <linux/rt_lock.h>
+
+#else
+
 #include <linux/wait.h>
 #include <linux/rwsem.h>
 
@@ -108,4 +115,8 @@ extern inline void up(struct semaphore *
 
 #endif /* __KERNEL__ */
 
+extern int FASTCALL(sem_is_locked(struct semaphore *sem));
+
+#endif /* CONFIG_PREEMPT_RT */
+
 #endif /* !(_PPC_SEMAPHORE_H) */
=================================================================
--- ./include/asm-ppc/spinlock.h.ORG	2005-02-01 16:26:45.000000000 -0500
+++ ./include/asm-ppc/spinlock.h	2005-02-22 09:50:38.000000000 -0500
@@ -7,17 +7,6 @@
  * Simple spin lock operations.
  */
 
-typedef struct {
-	volatile unsigned long lock;
-#ifdef CONFIG_DEBUG_SPINLOCK
-	volatile unsigned long owner_pc;
-	volatile unsigned long owner_cpu;
-#endif
-#ifdef CONFIG_PREEMPT
-	unsigned int break_lock;
-#endif
-} spinlock_t;
-
 #ifdef __KERNEL__
 #ifdef CONFIG_DEBUG_SPINLOCK
 #define SPINLOCK_DEBUG_INIT     , 0, 0
@@ -25,16 +14,19 @@ typedef struct {
 #define SPINLOCK_DEBUG_INIT     /* */
 #endif
 
-#define SPIN_LOCK_UNLOCKED	(spinlock_t) { 0 SPINLOCK_DEBUG_INIT }
+#define __RAW_SPIN_LOCK_UNLOCKED { 0 SPINLOCK_DEBUG_INIT }
+#define RAW_SPIN_LOCK_UNLOCKED	(raw_spinlock_t) __RAW_SPIN_LOCK_UNLOCKED
 
-#define spin_lock_init(x) 	do { *(x) = SPIN_LOCK_UNLOCKED; } while(0)
-#define spin_is_locked(x)	((x)->lock != 0)
-#define spin_unlock_wait(x)	do { barrier(); } while(spin_is_locked(x))
-#define _raw_spin_lock_flags(lock, flags) _raw_spin_lock(lock)
+#define __raw_spin_lock_init(x)  \
+	do { *(x) = RAW_SPIN_LOCK_UNLOCKED; } while(0)
+#define __raw_spin_is_locked(x)	((x)->lock != 0)
+#define __raw_spin_unlock_wait(x) \
+	do { barrier(); } while(__raw_spin_is_locked(x))
+#define __raw_spin_lock_flags(lock, flags) __raw_spin_lock(lock)
 
 #ifndef CONFIG_DEBUG_SPINLOCK
 
-static inline void _raw_spin_lock(spinlock_t *lock)
+static inline void __raw_spin_lock(raw_spinlock_t *lock)
 {
 	unsigned long tmp;
 
@@ -55,54 +47,37 @@ static inline void _raw_spin_lock(spinlo
 	: "cr0", "memory");
 }
 
-static inline void _raw_spin_unlock(spinlock_t *lock)
+static inline void __raw_spin_unlock(raw_spinlock_t *lock)
 {
 	__asm__ __volatile__("eieio		# spin_unlock": : :"memory");
 	lock->lock = 0;
 }
 
-#define _raw_spin_trylock(l) (!test_and_set_bit(0,&(l)->lock))
+#define __raw_spin_trylock(l) (!test_and_set_bit(0,&(l)->lock))
 
 #else
 
-extern void _raw_spin_lock(spinlock_t *lock);
-extern void _raw_spin_unlock(spinlock_t *lock);
-extern int _raw_spin_trylock(spinlock_t *lock);
-
-#endif
+extern void __raw_spin_lock(raw_spinlock_t *lock);
+extern void __raw_spin_unlock(raw_spinlock_t *lock);
+extern int __raw_spin_trylock(raw_spinlock_t *lock);
 
-/*
- * Read-write spinlocks, allowing multiple readers
- * but only one writer.
- *
- * NOTE! it is quite common to have readers in interrupts
- * but no interrupt writers. For those circumstances we
- * can "mix" irq-safe locks - any writer needs to get a
- * irq-safe write-lock, but readers can get non-irqsafe
- * read-locks.
- */
-typedef struct {
-	volatile unsigned long lock;
-#ifdef CONFIG_DEBUG_SPINLOCK
-	volatile unsigned long owner_pc;
-#endif
-#ifdef CONFIG_PREEMPT
-	unsigned int break_lock;
-#endif
-} rwlock_t;
+#endif	/* CONFIG_DEBUG_SPINLOCK */
 
 #ifdef CONFIG_DEBUG_SPINLOCK
-#define RWLOCK_DEBUG_INIT     , 0
+#define RAW_RWLOCK_DEBUG_INIT     , 0
 #else
-#define RWLOCK_DEBUG_INIT     /* */
+#define RAW_RWLOCK_DEBUG_INIT     /* */
 #endif
 
-#define RW_LOCK_UNLOCKED (rwlock_t) { 0 RWLOCK_DEBUG_INIT }
-#define rwlock_init(lp) do { *(lp) = RW_LOCK_UNLOCKED; } while(0)
+#define __RAW_RW_LOCK_UNLOCKED { 0 RAW_RWLOCK_DEBUG_INIT }
+#define RAW_RW_LOCK_UNLOCKED (raw_rwlock_t) __RAW_RW_LOCK_UNLOCKED
+#define __raw_rwlock_init(lp) do { *(lp) = RAW_RW_LOCK_UNLOCKED; } while(0)
+#define	__raw_read_can_lock(lp)	(0 <= (lp)->lock)
+#define	__raw_write_can_lock(lp)	(!(lp)->lock)
 
 #ifndef CONFIG_DEBUG_SPINLOCK
 
-static __inline__ void _raw_read_lock(rwlock_t *rw)
+static __inline__ void __raw_read_lock(raw_rwlock_t *rw)
 {
 	unsigned int tmp;
 
@@ -123,7 +98,7 @@ static __inline__ void _raw_read_lock(rw
 	: "cr0", "memory");
 }
 
-static __inline__ void _raw_read_unlock(rwlock_t *rw)
+static __inline__ void __raw_read_unlock(raw_rwlock_t *rw)
 {
 	unsigned int tmp;
 
@@ -139,7 +114,7 @@ static __inline__ void _raw_read_unlock(
 	: "cr0", "memory");
 }
 
-static __inline__ int _raw_write_trylock(rwlock_t *rw)
+static __inline__ int __raw_write_trylock(raw_rwlock_t *rw)
 {
 	unsigned int tmp;
 
@@ -159,7 +134,7 @@ static __inline__ int _raw_write_trylock
 	return tmp == 0;
 }
 
-static __inline__ void _raw_write_lock(rwlock_t *rw)
+static __inline__ void __raw_write_lock(raw_rwlock_t *rw)
 {
 	unsigned int tmp;
 
@@ -180,7 +155,7 @@ static __inline__ void _raw_write_lock(r
 	: "cr0", "memory");
 }
 
-static __inline__ void _raw_write_unlock(rwlock_t *rw)
+static __inline__ void __raw_write_unlock(raw_rwlock_t *rw)
 {
 	__asm__ __volatile__("eieio		# write_unlock": : :"memory");
 	rw->lock = 0;
@@ -188,15 +163,15 @@ static __inline__ void _raw_write_unlock
 
 #else
 
-extern void _raw_read_lock(rwlock_t *rw);
-extern void _raw_read_unlock(rwlock_t *rw);
-extern void _raw_write_lock(rwlock_t *rw);
-extern void _raw_write_unlock(rwlock_t *rw);
-extern int _raw_write_trylock(rwlock_t *rw);
+extern void __raw_read_lock(raw_rwlock_t *rw);
+extern void __raw_read_unlock(raw_rwlock_t *rw);
+extern void __raw_write_lock(raw_rwlock_t *rw);
+extern void __raw_write_unlock(raw_rwlock_t *rw);
+extern int __raw_write_trylock(raw_rwlock_t *rw);
 
 #endif
 
-#define _raw_read_trylock(lock) generic_raw_read_trylock(lock)
+#define __raw_read_trylock(lock) generic_raw_read_trylock(lock)
 
 #endif /* __ASM_SPINLOCK_H */
 #endif /* __KERNEL__ */
=================================================================
--- ./include/asm-ppc/hw_irq.h.ORG	2004-12-24 16:35:15.000000000 -0500
+++ ./include/asm-ppc/hw_irq.h	2005-02-23 15:55:45.653015432 -0500
@@ -13,8 +13,17 @@ extern void timer_interrupt(struct pt_re
 #define INLINE_IRQS
 
 #define irqs_disabled()	((mfmsr() & MSR_EE) == 0)
+#define irqs_disabled_flags(f)	(!((f) & MSR_EE))
 
-#ifdef INLINE_IRQS
+#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING
+  extern void notrace trace_irqs_off(void);
+  extern void notrace trace_irqs_on(void);
+#else
+# define trace_irqs_off()              do { } while (0)
+# define trace_irqs_on()               do { } while (0)
+#endif
+
+#if defined(INLINE_IRQS) || defined(CONFIG_CRITICAL_IRQSOFF_TIMING)
 
 static inline void local_irq_disable(void)
 {
@@ -22,11 +31,14 @@ static inline void local_irq_disable(voi
 	msr = mfmsr();
 	mtmsr(msr & ~MSR_EE);
 	__asm__ __volatile__("": : :"memory");
+	trace_irqs_off();
 }
 
 static inline void local_irq_enable(void)
 {
 	unsigned long msr;
+
+	trace_irqs_on();
 	__asm__ __volatile__("": : :"memory");
 	msr = mfmsr();
 	mtmsr(msr | MSR_EE);
@@ -39,11 +51,19 @@ static inline void local_irq_save_ptr(un
 	*flags = msr;
 	mtmsr(msr & ~MSR_EE);
 	__asm__ __volatile__("": : :"memory");
+	trace_irqs_off();
 }
 
 #define local_save_flags(flags)		((flags) = mfmsr())
 #define local_irq_save(flags)		local_irq_save_ptr(&flags)
-#define local_irq_restore(flags)	mtmsr(flags)
+#define local_irq_restore(flags) \
+	do { \
+		if (irqs_disabled_flags(flags)) \
+			trace_irqs_off(); \
+		else \
+			trace_irqs_on();  \
+		mtmsr(flags); \
+	} while (0)
 
 #else
 
=================================================================
--- ./include/asm-ppc/tlb.h.ORG	2004-12-24 16:34:58.000000000 -0500
+++ ./include/asm-ppc/tlb.h	2005-02-09 19:12:21.000000000 -0500
@@ -50,7 +50,11 @@ static inline void __tlb_remove_tlb_entr
 #define tlb_flush(tlb)			flush_tlb_mm((tlb)->mm)
 
 /* Get the generic bits... */
+#ifdef CONFIG_PREEMPT_RT
+#include <asm-generic/tlb-simple.h>
+#else
 #include <asm-generic/tlb.h>
+#endif
 
 #endif /* CONFIG_PPC_STD_MMU */
 
=================================================================
--- ./include/asm-ppc/ocp.h.ORG	2004-12-24 16:34:26.000000000 -0500
+++ ./include/asm-ppc/ocp.h	2005-02-23 16:50:53.514144104 -0500
@@ -32,7 +32,6 @@
 
 #include <asm/mmu.h>
 #include <asm/ocp_ids.h>
-#include <asm/rwsem.h>
 #include <asm/semaphore.h>
 
 #ifdef CONFIG_PPC_OCP
=================================================================
--- ./include/linux/sched.h.ORG	2005-02-01 16:26:51.000000000 -0500
+++ ./include/linux/sched.h	2005-02-20 18:24:02.000000000 -0500
@@ -74,9 +74,18 @@ extern int debug_direct_keyboard;
 #endif
 
 #ifdef CONFIG_FRAME_POINTER
+#ifdef CONFIG_PPC
+# define __valid_ra(l)	((__builtin_frame_address(l) && \
+	*(unsigned long *)__builtin_frame_address(l)) ? \
+	(unsigned long)__builtin_return_address(l) : 0UL)
+# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
+# define CALLER_ADDR1 __valid_ra(1)
+# define CALLER_ADDR2 (__valid_ra(1) ? __valid_ra(2) : 0UL)
+#else
 # define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
 # define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1))
 # define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2))
+#endif
 #else
 # define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
 # define CALLER_ADDR1 0UL
@@ -84,9 +93,14 @@ extern int debug_direct_keyboard;
 #endif
 
 #ifdef CONFIG_MCOUNT
-  extern void notrace mcount(void);
+#ifdef CONFIG_PPC
+# define ARCH_MCOUNT     _mcount
+#else
+# define ARCH_MCOUNT     mcount
+#endif
+  extern void notrace ARCH_MCOUNT(void);
 #else
-# define mcount() do { } while (0)
+# define ARCH_MCOUNT() do { } while (0)
 #endif
 
 #ifdef CONFIG_LATENCY_TRACE
=================================================================
--- ./drivers/char/blocker.c.ORG	2005-02-01 16:26:48.000000000 -0500
+++ ./drivers/char/blocker.c	2005-02-02 17:16:24.000000000 -0500
@@ -4,6 +4,7 @@
 
 #include <linux/fs.h>
 #include <linux/miscdevice.h>
+#include <asm/time.h>
 
 #define BLOCKER_MINOR		221
 
@@ -17,8 +18,18 @@ u64 notrace get_cpu_tick(void)
 	u64 tsc;
 #ifdef ARCHARM
 	tsc = *oscr;
-#else
+#elif defined(CONFIG_X86)
 	__asm__ __volatile__("rdtsc" : "=A" (tsc));
+#elif defined(CONFIG_PPC)
+	unsigned long hi, lo;
+
+	do {
+		hi = get_tbu();
+		lo = get_tbl();
+	} while (get_tbu() != hi);
+	tsc = (u64)hi << 32 | lo;
+#else
+	#error Implement get_cpu_tick()
 #endif
 	return tsc;
 }
=================================================================
--- ./kernel/sched.c.ORG	2005-02-01 16:26:46.000000000 -0500
+++ ./kernel/sched.c	2005-02-23 14:37:51.325621208 -0500
@@ -1237,7 +1237,7 @@ int fastcall wake_up_process(task_t * p)
 	int ret = try_to_wake_up(p, TASK_STOPPED | TASK_TRACED |
 				 TASK_RUNNING_MUTEX | TASK_INTERRUPTIBLE |
 				 TASK_UNINTERRUPTIBLE, 0, 0);
-	mcount();
+	ARCH_MCOUNT();
 	return ret;
 }
 
@@ -1248,7 +1248,7 @@ int fastcall wake_up_process_mutex(task_
 	int ret = try_to_wake_up(p, TASK_STOPPED | TASK_TRACED |
 				 TASK_RUNNING_MUTEX | TASK_INTERRUPTIBLE |
 				 TASK_UNINTERRUPTIBLE, 0, 1);
-	mcount();
+	ARCH_MCOUNT();
 	return ret;
 }
 
@@ -1257,7 +1257,7 @@ EXPORT_SYMBOL(wake_up_process_mutex);
 int fastcall wake_up_state(task_t *p, unsigned int state)
 {
 	int ret = try_to_wake_up(p, state | TASK_RUNNING_MUTEX, 0, 0);
-	mcount();
+	ARCH_MCOUNT();
 	return ret;
 }
 
@@ -1502,7 +1502,11 @@ static void finish_task_switch(task_t *p
  * schedule_tail - first thing a freshly forked thread must call.
  * @prev: the thread we just switched away from.
  */
+#ifdef CONFIG_PPC
+asmlinkage notrace void schedule_tail(task_t *prev)
+#else
 asmlinkage void schedule_tail(task_t *prev)
+#endif
 	__releases(rq->lock)
 {
 	preempt_disable(); // TODO: move this to fork setup
=================================================================
--- ./kernel/latency.c.ORG	2005-02-01 16:26:46.000000000 -0500
+++ ./kernel/latency.c	2005-02-23 08:32:38.000000000 -0500
@@ -50,6 +50,12 @@ static __cacheline_aligned_in_smp struct
 int wakeup_timing = 1;
 #endif
 
+#ifdef NONASCII
+#define MU 'µ'
+#else
+#define MU 'u'
+#endif
+
 #ifdef CONFIG_LATENCY_TIMING
 
 /*
@@ -357,9 +363,9 @@ void notrace __trace(unsigned long eip, 
 	___trace(TRACE_FN, eip, parent_eip, 0, 0, 0);
 }
 
-extern void mcount(void);
+extern void ARCH_MCOUNT(void);
 
-EXPORT_SYMBOL(mcount);
+EXPORT_SYMBOL(ARCH_MCOUNT);
 
 void notrace __mcount(void)
 {
@@ -631,8 +637,8 @@ static void * notrace l_start(struct seq
 	if (!n) {
 		seq_printf(m, "preemption latency trace v1.1.4 on %s\n", UTS_RELEASE);
 		seq_puts(m, "--------------------------------------------------------------------\n");
-		seq_printf(m, " latency: %lu µs, #%lu/%lu, CPU#%d | (M:%s VP:%d, KP:%d, SP:%d HP:%d #P:%d)\n",
-			cycles_to_usecs(tr->saved_latency),
+		seq_printf(m, " latency: %lu %cs, #%lu/%lu, CPU#%d | (M:%s VP:%d, KP:%d, SP:%d HP:%d #P:%d)\n",
+			cycles_to_usecs(tr->saved_latency), MU,
 			entries, tr->trace_idx, out_tr.cpu,
 #if defined(CONFIG_PREEMPT_NONE)
 			"server",
@@ -698,7 +704,7 @@ static void notrace l_stop(struct seq_fi
 static void print_timestamp(struct seq_file *m, unsigned long abs_usecs,
 						unsigned long rel_usecs)
 {
-	seq_printf(m, " %4ldµs", abs_usecs);
+	seq_printf(m, " %4ld%cs", abs_usecs, MU);
 	if (rel_usecs > 100)
 		seq_puts(m, "!: ");
 	else if (rel_usecs > 1)
@@ -711,7 +717,7 @@ static void
 print_timestamp_short(struct seq_file *m, unsigned long abs_usecs,
 			unsigned long rel_usecs)
 {
-	seq_printf(m, " %4ldµs", abs_usecs);
+	seq_printf(m, " %4ld%cs", abs_usecs, MU);
 	if (rel_usecs > 100)
 		seq_putc(m, '!');
 	else if (rel_usecs > 1)
@@ -1043,7 +1049,7 @@ static int setup_preempt_thresh(char *s)
 	get_option(&s, &thresh);
 	if (thresh > 0) {
 		preempt_thresh = usecs_to_cycles(thresh);
-		printk("Preemption threshold = %u µs\n", thresh);
+		printk("Preemption threshold = %u %cs\n", thresh, MU);
 	}
 	return 1;
 }
@@ -1091,18 +1097,18 @@ check_critical_timing(struct cpu_trace *
 	update_max_tr(tr);
 
 	if (preempt_thresh)
-		printk("(%16s-%-5d|#%d): %lu µs critical section "
-			"violates %lu µs threshold.\n"
+		printk("(%16s-%-5d|#%d): %lu %cs critical section "
+			"violates %lu %cs threshold.\n"
 			" => started at timestamp %lu: ",
 				current->comm, current->pid,
-				_smp_processor_id(),
-				latency, cycles_to_usecs(preempt_thresh), t0);
+				_smp_processor_id(), latency,
+				MU, cycles_to_usecs(preempt_thresh), MU, t0);
 	else
-		printk("(%16s-%-5d|#%d): new %lu µs maximum-latency "
+		printk("(%16s-%-5d|#%d): new %lu %cs maximum-latency "
 			"critical section.\n => started at timestamp %lu: ",
 				current->comm, current->pid,
 				_smp_processor_id(),
-				latency, t0);
+				latency, MU, t0);
 
 	print_symbol("<%s>\n", tr->critical_start);
 	printk(" =>   ended at timestamp %lu: ", t1);
@@ -1345,15 +1351,15 @@ check_wakeup_timing(struct cpu_trace *tr
 	update_max_tr(tr);
 
 	if (preempt_thresh)
-		printk("(%16s-%-5d|#%d): %lu µs wakeup latency "
-			"violates %lu µs threshold.\n",
+		printk("(%16s-%-5d|#%d): %lu %cs wakeup latency "
+			"violates %lu %cs threshold.\n",
 				current->comm, current->pid,
 				_smp_processor_id(), latency,
-				cycles_to_usecs(preempt_thresh));
+				MU, cycles_to_usecs(preempt_thresh), MU);
 	else
-		printk("(%16s-%-5d|#%d): new %lu µs maximum-latency "
+		printk("(%16s-%-5d|#%d): new %lu %cs maximum-latency "
 			"wakeup.\n", current->comm, current->pid,
-				_smp_processor_id(), latency);
+				_smp_processor_id(), latency, MU);
 
 	max_sequence++;
 
@@ -1399,7 +1405,7 @@ void __trace_start_sched_wakeup(struct t
 	tr->preempt_timestamp = cycles();
 	tr->critical_start = CALLER_ADDR0;
 	trace_cmdline();
-	mcount();
+	ARCH_MCOUNT();
 out_unlock:
 	spin_unlock(&sch.trace_lock);
 }
@@ -1489,7 +1495,7 @@ long user_trace_start(void)
 	tr->critical_sequence = max_sequence;
 	tr->preempt_timestamp = cycles();
 	trace_cmdline();
-	mcount();
+	ARCH_MCOUNT();
 	preempt_enable();
 
 	up(&max_mutex);
@@ -1507,7 +1513,7 @@ long user_trace_stop(void)
 		return -EINVAL;
 
 	preempt_disable();
-	mcount();
+	ARCH_MCOUNT();
 
 	if (wakeup_timing) {
 		spin_lock_irqsave(&sch.trace_lock, flags);
@@ -1538,15 +1544,15 @@ long user_trace_stop(void)
 		latency = cycles_to_usecs(delta);
 
 		if (preempt_thresh)
-			printk("(%16s-%-5d|#%d): %lu µs user-latency "
-				"violates %lu µs threshold.\n",
+			printk("(%16s-%-5d|#%d): %lu %cs user-latency "
+				"violates %lu %cs threshold.\n",
 					current->comm, current->pid,
-					_smp_processor_id(), latency,
-					cycles_to_usecs(preempt_thresh));
+					_smp_processor_id(), latency, MU,
+					cycles_to_usecs(preempt_thresh), MU);
 		else
-			printk("(%16s-%-5d|#%d): new %lu µs user-latency.\n",
+			printk("(%16s-%-5d|#%d): new %lu %cs user-latency.\n",
 				current->comm, current->pid,
-					_smp_processor_id(), latency);
+					_smp_processor_id(), latency, MU);
 
 		max_sequence++;
 		up(&max_mutex);

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: PPC RT Patch..
  2005-02-23 23:52             ` PPC RT Patch john cooper
@ 2005-02-24  4:20               ` Frank Rowand
  2005-02-24 13:56                 ` john cooper
  0 siblings, 1 reply; 26+ messages in thread
From: Frank Rowand @ 2005-02-24  4:20 UTC (permalink / raw)
  To: john cooper; +Cc: Ingo Molnar, linux-kernel, Saksena, Manas

john cooper wrote:
> Ingo,
>     We've had a PPC port of your RT work underway with
> a focus on trace instrumentation.  This is based upon
> realtime-preempt-2.6.11-rc2-V0.7.37-02.  A diff is
> attached.
> 
> To the extent possible the tracing facilities are the
> same as your x86 work.  In the process a few PPC/gcc
> issues needed to be resolved.  There is also a bug fix
> contained for tlb_gather_mmu() which was causing debug
> assertions to be generated in a path which attempted to
> sleep with a non-zero preempt count.

Manish Lachwani mentioned to me that he faced the same issue
with the MIPS RT support and that when he discussed
it with Ingo that the solution was for include/asm-ppc/tlb.h
to include/asm-generic/tlb-simple.h when PREEMPT_RT is turned on.
The patch does this for the #ifdef CONFIG_PPC_STD_MMU case,
but not for the #else case.  I don't know which case is used
for the Ampro board.


> 
> This does build and function when SMP is configured,
> though we have not yet verified it on other than a
> uniprocessor.  As a simplifying assumption, testing has
> thus far concentrated on the following modes:
> 
> PREEMPT_NONE
>     - verify baseline regression
> 
> PREEMPT_RT && !PREEMPT_SMP
>     - typical for an embedded RT PPC application
> 
> PREEMPT_RT && PREEMPT_SMP
>     - kicks in live locking code which otherwise receives no
>     coverage.  This is functionally equivalent to the above
>     config on a single CPU target thus no MP dynamic testing
>     is achieved.  Still quite useful IMHO.
> 
> The target used for development/testing is an Ampro EnCore PP1
> which sports a 300Mhz MPC8245.  For testing this boots with NFS
> as root.  An mp3 decode at nice --20 is launched which requires
> just under 20% of the CPU to maintain an uninterrupted audio
> decode and output.  To this a series of "du -s /" are launched
> to soak up excess CPU bandwidth.  Perhaps not rigorous but a
> fair sanity check and load for the purpose at hand.
> 
> Under these conditions maximum scheduling latencies are seen in
> the 120-150us range.  Note no attempt has yet been made to
> optimize arch specific paths and full trace instrumentation has
> been enabled.
> 
> I've written some logging code to help find problems such as
> the tlb issue above.  As it has not been made general I've
> removed it from this patch.  At some point I'll likely revisit
> this.
> 
> Comments/suggestions welcome.

I am glad to see the instrumentation and measurement related code
in your patch.  (My patch of last week ("Frank's patch") is lacking
that code.)

Other differences between the two patches are:

arch/ppc/syslib/i8259.c
    Frank neglected to convert i8259_lock to a raw spinlock.

arch/ppc/kernel/signal.c
    John added an enable of irqs in do_signal()  #ifdef CONFIG_PREEMPT_RT

arch/ppc/kernel/traps.c
    John added an enable of irqs and preempt_check_resched() in _exception().

various files
    Frank added the intrusive variable tb_to_us for use by cycles_to_usec()
    and added an ugly #ifdef in cycles_to_usec().
    John hard-coded cpu_khz for one specific board so that no change would
    be needed in cycles_to_usec().

various files
    John has the mmu_gather fix that is described above.

John's patch and Frank's patch are otherwise mostly the same, except for
the differences that result from being based on different kernel
versions.  I am glad to see that because it means that two sets of
eyes have agreed.

Frank's patch may have missed some EXPORT_SYMBOL()s in arch/ppc/lib/locks.c.
I'll check those over again tomorrow.


> -john


-Frank
-- 
Frank Rowand <frank_rowand@mvista.com>
MontaVista Software, Inc


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: PPC RT Patch..
  2005-02-24  4:20               ` Frank Rowand
@ 2005-02-24 13:56                 ` john cooper
  0 siblings, 0 replies; 26+ messages in thread
From: john cooper @ 2005-02-24 13:56 UTC (permalink / raw)
  To: Frank Rowand; +Cc: Ingo Molnar, linux-kernel, Saksena, Manas, john cooper

Frank Rowand wrote:
> john cooper wrote:
>> ... There is also a bug fix
>> contained for tlb_gather_mmu() which was causing debug
>> assertions to be generated in a path which attempted to
>> sleep with a non-zero preempt count.
> 
> 
> Manish Lachwani mentioned to me that he faced the same issue
> with the MIPS RT support and that when he discussed
> it with Ingo that the solution was for include/asm-ppc/tlb.h
> to include/asm-generic/tlb-simple.h when PREEMPT_RT is turned on.
> The patch does this for the #ifdef CONFIG_PPC_STD_MMU case,
> but not for the #else case.  I don't know which case is used
> for the Ampro board.

It appeared to me a generic issue though I believe a number
of solutions are possible.  asm-generic/tlb.h:tlb_gather_mmu()
expands to linux/percpu.h:get_cpu_var() which does a
preempt_disable() and __get_cpu_var().  This caused the debug
assertion to kick when __page_cache_release() and to a lesser
extent activate_page() attempted to block on a mutex (though
other paths may well exist).  My approach was to replace the
outer layer preempt_disable/enable calls with a mutex-spinlock.

The fix was fairly easy once it was known from where the
gratuitous call to preempt_disable() existed.  I cobbled
together a logging mechanism which detected the problem.  As it
wasn't very general I removed it from the patch.  I didn't see
an alternate means of diagnosing such a scenario so I'll likely
address generalizing the code.

-john

-- 
john.cooper@timesys.com

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 21:03         ` Hugh Dickins
  2005-02-23 22:14           ` Lee Revell
@ 2005-02-23 23:27           ` Nick Piggin
  2005-02-24  1:03             ` Lee Revell
  1 sibling, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-23 23:27 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Lee Revell, Ingo Molnar, Andrew Morton, linux-kernel

Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> 
>>>>Thanks, your patch fixes the copy_pte_range latency.
>>
>>clear_page_range is also problematic.
> 
> 
> Yes, I saw that from your other traces too.  I know there are plans
> to improve clear_page_range during 2.6.12, but I didn't realize that
> it had become very much worse than its antecedent clear_page_tables,
> and I don't see missing latency fixes for that.  Nick's the expert.
> 

I wouldn't have thought it should have become worse, latency
wise. What is actually happening is that the lower level freeing
functions are being called more often. But this should result in
the work being spread out more, if anything. Rather than in the
old system things would tend to be batched up into bigger chunks
(typically at exit() time).

If you are using i386 with 2-level page tables (no highmem), then
the behaviour should be more or less identical. Odd.

Nick

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 23:27           ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
@ 2005-02-24  1:03             ` Lee Revell
  2005-02-24  1:29               ` Nick Piggin
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24  1:03 UTC (permalink / raw)
  To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel

On Thu, 2005-02-24 at 10:27 +1100, Nick Piggin wrote: 
> Hugh Dickins wrote:
> > On Wed, 23 Feb 2005, Lee Revell wrote:
> > 
> >>>>Thanks, your patch fixes the copy_pte_range latency.
> >>
> >>clear_page_range is also problematic.
> > 
> > 
> > Yes, I saw that from your other traces too.  I know there are plans
> > to improve clear_page_range during 2.6.12, but I didn't realize that
> > it had become very much worse than its antecedent clear_page_tables,
> > and I don't see missing latency fixes for that.  Nick's the expert.
> > 
> 
> I wouldn't have thought it should have become worse, latency
> wise. What is actually happening is that the lower level freeing
> functions are being called more often. But this should result in
> the work being spread out more, if anything. Rather than in the
> old system things would tend to be batched up into bigger chunks
> (typically at exit() time).
> 
> If you are using i386 with 2-level page tables (no highmem), then
> the behaviour should be more or less identical. Odd.

IIRC last time I really tested this a few months ago, the worst case
latency on that machine was about 150us.  Currently its 422us from the
same clear_page_range code path.

On my Athlon XP the clear_page_range latency is not showing up at all,
and the worst delay so far is only 35us, most of which is the timer
interrupt IOW that machine is showing the best achievable latency (with
PREEMPT_DESKTOP).  The machine seeing 422 us latencies in
clear_page_range is a 600Mhz C3, which is known to be a FSB limited
architecture.

Lee 


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  1:03             ` Lee Revell
@ 2005-02-24  1:29               ` Nick Piggin
  2005-02-24  2:24                 ` Lee Revell
  0 siblings, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-24  1:29 UTC (permalink / raw)
  To: Lee Revell; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel

Lee Revell wrote:
> On Thu, 2005-02-24 at 10:27 +1100, Nick Piggin wrote: 

>>If you are using i386 with 2-level page tables (no highmem), then
>>the behaviour should be more or less identical. Odd.
> 
> 
> IIRC last time I really tested this a few months ago, the worst case
> latency on that machine was about 150us.  Currently its 422us from the
> same clear_page_range code path.
> 
> On my Athlon XP the clear_page_range latency is not showing up at all,
> and the worst delay so far is only 35us, most of which is the timer
> interrupt IOW that machine is showing the best achievable latency (with
> PREEMPT_DESKTOP).  The machine seeing 422 us latencies in
> clear_page_range is a 600Mhz C3, which is known to be a FSB limited
> architecture.
> 

Well it should be pretty trivial to add a break in there.
I don't think it can get into 2.6.11 at this point though,
so we'll revisit this for 2.6.12 if the clear_page_range
optimisations don't get anywhere.

Nick


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  1:29               ` Nick Piggin
@ 2005-02-24  2:24                 ` Lee Revell
  2005-02-24  2:41                   ` Nick Piggin
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24  2:24 UTC (permalink / raw)
  To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel

On Thu, 2005-02-24 at 12:29 +1100, Nick Piggin wrote:
> Lee Revell wrote:
> > 
> > IIRC last time I really tested this a few months ago, the worst case
> > latency on that machine was about 150us.  Currently its 422us from the
> > same clear_page_range code path.
> > 
> Well it should be pretty trivial to add a break in there.
> I don't think it can get into 2.6.11 at this point though,
> so we'll revisit this for 2.6.12 if the clear_page_range
> optimisations don't get anywhere.
> 

Agreed, it would be much better to optimize this away than just add a
scheduling point.  It seems like we could do this lazily.

IMHO it's not critical that these latency fixes be merged until the VP
feature gets merged, until then people will be using Ingo's patches
anyway.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  2:24                 ` Lee Revell
@ 2005-02-24  2:41                   ` Nick Piggin
  2005-02-24  3:03                     ` Lee Revell
  0 siblings, 1 reply; 26+ messages in thread
From: Nick Piggin @ 2005-02-24  2:41 UTC (permalink / raw)
  To: Lee Revell; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel

Lee Revell wrote:
> On Thu, 2005-02-24 at 12:29 +1100, Nick Piggin wrote:
> 
>>Lee Revell wrote:
>>
>>>IIRC last time I really tested this a few months ago, the worst case
>>>latency on that machine was about 150us.  Currently its 422us from the
>>>same clear_page_range code path.
>>>
>>
>>Well it should be pretty trivial to add a break in there.
>>I don't think it can get into 2.6.11 at this point though,
>>so we'll revisit this for 2.6.12 if the clear_page_range
>>optimisations don't get anywhere.
>>
> 
> 
> Agreed, it would be much better to optimize this away than just add a
> scheduling point.  It seems like we could do this lazily.
> 

Oh? What do you mean by lazy? IMO it is sort of implemented lazily now.
That is, we are too lazy to refcount page table pages in fastpaths, so
that pushes a lot of work to unmap time. Not necessarily a bad trade-off,
mind you. Just something I'm looking into.


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  2:41                   ` Nick Piggin
@ 2005-02-24  3:03                     ` Lee Revell
  0 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-24  3:03 UTC (permalink / raw)
  To: Nick Piggin; +Cc: Hugh Dickins, Ingo Molnar, Andrew Morton, linux-kernel

On Thu, 2005-02-24 at 13:41 +1100, Nick Piggin wrote:
> Lee Revell wrote:
> > 
> > Agreed, it would be much better to optimize this away than just add a
> > scheduling point.  It seems like we could do this lazily.
> > 
> 
> Oh? What do you mean by lazy? IMO it is sort of implemented lazily now.
> That is, we are too lazy to refcount page table pages in fastpaths, so
> that pushes a lot of work to unmap time. Not necessarily a bad trade-off,
> mind you. Just something I'm looking into.
> 

I guess I was thinking we could be even more lazy, and somehow defer it
until after unmap time (in lieu of memory pressure that is).  Actually
that's kind of what a lock break would do.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 20:06     ` Hugh Dickins
  2005-02-23 20:10       ` Lee Revell
  2005-02-23 20:30       ` Lee Revell
@ 2005-02-23 20:53       ` Hugh Dickins
  2005-02-23 22:13         ` Lee Revell
  2 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-23 20:53 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 23 Feb 2005, Hugh Dickins wrote:
> Please replace by new patch below, which I'm now running through lmbench.

That second patch seems fine, and I see no lmbench regression from it.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 20:53       ` Hugh Dickins
@ 2005-02-23 22:13         ` Lee Revell
  2005-02-24  4:56           ` Hugh Dickins
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-23 22:13 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Andrew Morton, linux-kernel

On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > Please replace by new patch below, which I'm now running through lmbench.
> 
> That second patch seems fine, and I see no lmbench regression from it.

Should go into 2.6.11, right?

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 22:13         ` Lee Revell
@ 2005-02-24  4:56           ` Hugh Dickins
  2005-02-24  6:32             ` Lee Revell
  0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-24  4:56 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Wed, 23 Feb 2005, Lee Revell wrote:
> On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> > On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > > Please replace by new patch below, which I'm now running through lmbench.
> > 
> > That second patch seems fine, and I see no lmbench regression from it.
> 
> Should go into 2.6.11, right?

That's up to Andrew (and Linus).

I was thinking that way when I rushed you the patch.  But given that
you have remaining unresolved latency issues nearby (zap_pte_range,
clear_page_range), and given the warning shot that I screwed up my
first attempt, I'd be inclined to say hold off.

It's a pity: for a while we were thinking 2.6.11 would be a big step
forward for mainline latency; but it now looks to me like these tests
have come too late in the cycle to be dealt with safely.

In other mail, you do expect people still to be using Ingo's patches,
so probably this patch should stick there (and in -mm) for now.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  4:56           ` Hugh Dickins
@ 2005-02-24  6:32             ` Lee Revell
  2005-02-24  8:26               ` Hugh Dickins
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-24  6:32 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> On Wed, 23 Feb 2005, Lee Revell wrote:
> > On Wed, 2005-02-23 at 20:53 +0000, Hugh Dickins wrote:
> > > On Wed, 23 Feb 2005, Hugh Dickins wrote:
> > > > Please replace by new patch below, which I'm now running through lmbench.
> > > 
> > > That second patch seems fine, and I see no lmbench regression from it.
> > 
> > Should go into 2.6.11, right?
> 
> That's up to Andrew (and Linus).
> 
> I was thinking that way when I rushed you the patch.  But given that
> you have remaining unresolved latency issues nearby (zap_pte_range,
> clear_page_range), and given the warning shot that I screwed up my
> first attempt, I'd be inclined to say hold off.
> 
> It's a pity: for a while we were thinking 2.6.11 would be a big step
> forward for mainline latency; but it now looks to me like these tests
> have come too late in the cycle to be dealt with safely.
> 
> In other mail, you do expect people still to be using Ingo's patches,
> so probably this patch should stick there (and in -mm) for now.

Well all of these were fixed in the past so it may not be unreasonable
to fix them for 2.6.11.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  6:32             ` Lee Revell
@ 2005-02-24  8:26               ` Hugh Dickins
  2005-02-25  3:30                 ` Lee Revell
  0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-24  8:26 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Thu, 24 Feb 2005, Lee Revell wrote:
> On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> > 
> > In other mail, you do expect people still to be using Ingo's patches,
> > so probably this patch should stick there (and in -mm) for now.
> 
> Well all of these were fixed in the past so it may not be unreasonable
> to fix them for 2.6.11.

If we'd got to it earlier, yes.  But 2.6.11 looks to be just a day or
two away, and we've no idea why zap_pte_range or clear_page_range
would have reverted.  Nor have we heard from Ingo yet.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-24  8:26               ` Hugh Dickins
@ 2005-02-25  3:30                 ` Lee Revell
  2005-02-25  5:58                   ` Hugh Dickins
  0 siblings, 1 reply; 26+ messages in thread
From: Lee Revell @ 2005-02-25  3:30 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> On Thu, 24 Feb 2005, Lee Revell wrote:
> > On Thu, 2005-02-24 at 04:56 +0000, Hugh Dickins wrote:
> > > 
> > > In other mail, you do expect people still to be using Ingo's patches,
> > > so probably this patch should stick there (and in -mm) for now.
> > 
> > Well all of these were fixed in the past so it may not be unreasonable
> > to fix them for 2.6.11.
> 
> If we'd got to it earlier, yes.  But 2.6.11 looks to be just a day or
> two away, and we've no idea why zap_pte_range or clear_page_range
> would have reverted.  Nor have we heard from Ingo yet.
> 

It's also not clear that the patch completely fixes the copy_pte_range
latency.  This trace is from the Athlon XP.

Lee

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 284 �s, #25/25, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)             dpkg  9299 0 3 00000005 00000000 [0001017457529380] 0.000ms (+3633922.612ms): <676b7064> (<00746500>)
(T1/#2)             dpkg  9299 0 3 00000005 00000002 [0001017457529620] 0.000ms (+0.000ms): __trace_start_sched_wakeup+0x9a/0xd0 <c012eaca> (try_to_wake_up+0x90/0x160 <c0110350>)
(T1/#3)             dpkg  9299 0 3 00000004 00000003 [0001017457529825] 0.000ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (try_to_wake_up+0x90/0x160 <c0110350>)
(T3/#4)     dpkg-9299  0dn.4    0�s : try_to_wake_up+0x118/0x160 <c01103d8> <<...>-2> (69 74): 
(T1/#5)             dpkg  9299 0 3 00000003 00000005 [0001017457530633] 0.000ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (try_to_wake_up+0xf2/0x160 <c01103b2>)
(T1/#6)             dpkg  9299 0 3 00000003 00000006 [0001017457530809] 0.001ms (+0.000ms): wake_up_process+0x35/0x40 <c0110455> (do_softirq+0x3f/0x50 <c011aedf>)
(T6/#7)     dpkg-9299  0dn.2    1�s!< (1)
(T1/#8)             dpkg  9299 0 2 00000001 00000008 [0001017457898984] 0.276ms (+0.000ms): preempt_schedule+0x11/0x80 <c02879d1> (copy_pte_range+0xbc/0x1b0 <c014573c>)
(T1/#9)             dpkg  9299 0 2 00000001 00000009 [0001017457899172] 0.276ms (+0.000ms): __cond_resched_raw_spinlock+0xb/0x50 <c0111f9b> (copy_pte_range+0xad/0x1b0 <c014572d>)
(T1/#10)             dpkg  9299 0 2 00000000 0000000a [0001017457899575] 0.277ms (+0.000ms): __cond_resched+0xe/0x70 <c0111f2e> (__cond_resched_raw_spinlock+0x35/0x50 <c0111fc5>)
(T1/#11)             dpkg  9299 0 3 00000000 0000000b [0001017457900063] 0.277ms (+0.000ms): __schedule+0xe/0x680 <c028720e> (__cond_resched+0x4a/0x70 <c0111f6a>)
(T1/#12)             dpkg  9299 0 3 00000000 0000000c [0001017457900379] 0.277ms (+0.000ms): profile_hit+0x9/0x50 <c0116449> (__schedule+0x43/0x680 <c0287243>)
(T1/#13)             dpkg  9299 0 3 00000001 0000000d [0001017457900602] 0.277ms (+0.001ms): sched_clock+0x14/0x80 <c010cbb4> (__schedule+0x73/0x680 <c0287273>)
(T1/#14)             dpkg  9299 0 3 00000002 0000000e [0001017457902490] 0.279ms (+0.000ms): dequeue_task+0x12/0x60 <c010ff32> (__schedule+0x1e0/0x680 <c02873e0>)
(T1/#15)             dpkg  9299 0 3 00000002 0000000f [0001017457902687] 0.279ms (+0.000ms): recalc_task_prio+0xe/0x140 <c01100be> (__schedule+0x202/0x680 <c0287402>)
(T1/#16)             dpkg  9299 0 3 00000002 00000010 [0001017457902848] 0.279ms (+0.000ms): effective_prio+0x8/0x60 <c0110058> (recalc_task_prio+0x88/0x140 <c0110138>)
(T1/#17)             dpkg  9299 0 3 00000002 00000011 [0001017457902995] 0.279ms (+0.000ms): enqueue_task+0x11/0x80 <c010ff91> (__schedule+0x20e/0x680 <c028740e>)
(T4/#18) [ =>             dpkg ] 0.280ms (+0.000ms)
(T1/#19)            <...>     2 0 1 00000002 00000013 [0001017457905091] 0.281ms (+0.000ms): __switch_to+0xe/0x190 <c010110e> (__schedule+0x306/0x680 <c0287506>)
(T3/#20)    <...>-2     0d..2  281�s : __schedule+0x337/0x680 <c0287537> <dpkg-9299> (74 69): 
(T1/#21)            <...>     2 0 1 00000002 00000015 [0001017457906484] 0.282ms (+0.000ms): finish_task_switch+0x14/0xa0 <c0110844> (__schedule+0x33f/0x680 <c028753f>)
(T1/#22)            <...>     2 0 1 00000001 00000016 [0001017457906713] 0.282ms (+0.000ms): trace_stop_sched_switched+0x11/0x180 <c012eb11> (finish_task_switch+0x51/0xa0 <c0110881>)
(T3/#23)    <...>-2     0d..1  282�s : trace_stop_sched_switched+0x4c/0x180 <c012eb4c> <<...>-2> (69 0): 
(T1/#24)            <...>     2 0 1 00000001 00000018 [0001017457908107] 0.283ms (+0.000ms): trace_stop_sched_switched+0x11c/0x180 <c012ec1c> (finish_task_switch+0x51/0xa0 <c0110881>)


vim:ft=help




^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-25  3:30                 ` Lee Revell
@ 2005-02-25  5:58                   ` Hugh Dickins
  2005-02-25 15:02                     ` Lee Revell
  0 siblings, 1 reply; 26+ messages in thread
From: Hugh Dickins @ 2005-02-25  5:58 UTC (permalink / raw)
  To: Lee Revell; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Thu, 24 Feb 2005, Lee Revell wrote:
> On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> > 
> > If we'd got to it earlier, yes.  But 2.6.11 looks to be just a day or
> > two away, and we've no idea why zap_pte_range or clear_page_range
> > would have reverted.  Nor have we heard from Ingo yet.
> 
> It's also not clear that the patch completely fixes the copy_pte_range
> latency.  This trace is from the Athlon XP.

Then we need Ingo to investigate and explain all these reversions.
I'm not _blaming_ Ingo for them, but I'm not familiar with his patches
nor with deciphering latency traces - he's the magician around here.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-25  5:58                   ` Hugh Dickins
@ 2005-02-25 15:02                     ` Lee Revell
  0 siblings, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-25 15:02 UTC (permalink / raw)
  To: Hugh Dickins; +Cc: Ingo Molnar, Nick Piggin, Andrew Morton, linux-kernel

On Fri, 2005-02-25 at 05:58 +0000, Hugh Dickins wrote:
> On Thu, 24 Feb 2005, Lee Revell wrote:
> > On Thu, 2005-02-24 at 08:26 +0000, Hugh Dickins wrote:
> > > 
> > > If we'd got to it earlier, yes.  But 2.6.11 looks to be just a day or
> > > two away, and we've no idea why zap_pte_range or clear_page_range
> > > would have reverted.  Nor have we heard from Ingo yet.
> > 
> > It's also not clear that the patch completely fixes the copy_pte_range
> > latency.  This trace is from the Athlon XP.
> 
> Then we need Ingo to investigate and explain all these reversions.
> I'm not _blaming_ Ingo for them, but I'm not familiar with his patches
> nor with deciphering latency traces - he's the magician around here.
> 

Yup.  Oh well.

I'll try to compile a comprehensive list of these so we can fix them for
2.6.12.

Lee


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: More latency regressions with 2.6.11-rc4-RT-V0.7.39-02
  2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
  2005-02-23 19:16 ` Hugh Dickins
@ 2005-02-23 19:52 ` Lee Revell
  1 sibling, 0 replies; 26+ messages in thread
From: Lee Revell @ 2005-02-23 19:52 UTC (permalink / raw)
  To: Ingo Molnar; +Cc: linux-kernel, Andrew Morton

[-- Attachment #1: Type: text/plain, Size: 163 bytes --]

On Wed, 2005-02-23 at 13:07 -0500, Lee Revell wrote:
> This is all with PREEMPT_DESKTOP.
> 

Here is another new one, this time in the ext3 reservation code.

Lee

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trace6.txt --]
[-- Type: text/plain; name=trace6.txt; charset=ANSI_X3.4-1968, Size: 63651 bytes --]

preemption latency trace v1.1.4 on 2.6.11-rc4-RT-V0.7.39-02
--------------------------------------------------------------------
 latency: 296 µs, #339/339, CPU#0 | (M:preempt VP:0, KP:1, SP:1 HP:1 #P:1)
    -----------------
    | task: ksoftirqd/0-2 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
(T1/#0)           dbench  3399 0 9 00000002 00000000 [0000784436927163] 0.000ms (+876876.851ms): <6e656264> (<61006863>)
(T1/#2)           dbench  3399 0 9 00000002 00000002 [0000784436927557] 0.000ms (+0.001ms): __trace_start_sched_wakeup+0x96/0xc0 <c012cbe6> (try_to_wake_up+0x81/0x150 <c010f911>)
(T1/#3)           dbench  3399 0 9 00000000 00000003 [0000784436928168] 0.001ms (+0.000ms): wake_up_process+0x1c/0x30 <c010f9fc> (do_softirq+0x4b/0x60 <c01042fb>)
(T6/#4)   dbench-3399  0dn.2    2µs < (1)
(T1/#5)           dbench  3399 0 2 00000002 00000005 [0000784436929084] 0.003ms (+0.003ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#6)           dbench  3399 0 2 00000002 00000006 [0000784436931285] 0.006ms (+0.001ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#7)           dbench  3399 0 2 00000002 00000007 [0000784436931964] 0.007ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#8)           dbench  3399 0 2 00000002 00000008 [0000784436932467] 0.008ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#9)           dbench  3399 0 2 00000002 00000009 [0000784436932849] 0.009ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#10)           dbench  3399 0 2 00000002 0000000a [0000784436933339] 0.010ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#11)           dbench  3399 0 2 00000002 0000000b [0000784436933798] 0.011ms (+0.005ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#12)           dbench  3399 0 2 00000002 0000000c [0000784436937044] 0.016ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#13)           dbench  3399 0 2 00000002 0000000d [0000784436937638] 0.017ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#14)           dbench  3399 0 2 00000002 0000000e [0000784436938093] 0.018ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#15)           dbench  3399 0 2 00000002 0000000f [0000784436938460] 0.018ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#16)           dbench  3399 0 2 00000002 00000010 [0000784436938946] 0.019ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#17)           dbench  3399 0 2 00000002 00000011 [0000784436939518] 0.020ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#18)           dbench  3399 0 2 00000002 00000012 [0000784436939929] 0.021ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#19)           dbench  3399 0 2 00000002 00000013 [0000784436940424] 0.022ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#20)           dbench  3399 0 2 00000002 00000014 [0000784436940865] 0.022ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#21)           dbench  3399 0 2 00000002 00000015 [0000784436941335] 0.023ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#22)           dbench  3399 0 2 00000002 00000016 [0000784436941819] 0.024ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#23)           dbench  3399 0 2 00000002 00000017 [0000784436942273] 0.025ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#24)           dbench  3399 0 2 00000002 00000018 [0000784436942674] 0.025ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#25)           dbench  3399 0 2 00000002 00000019 [0000784436943164] 0.026ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#26)           dbench  3399 0 2 00000002 0000001a [0000784436943610] 0.027ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#27)           dbench  3399 0 2 00000002 0000001b [0000784436944010] 0.028ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#28)           dbench  3399 0 2 00000002 0000001c [0000784436944501] 0.028ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#29)           dbench  3399 0 2 00000002 0000001d [0000784436945572] 0.030ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#30)           dbench  3399 0 2 00000002 0000001e [0000784436946139] 0.031ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#31)           dbench  3399 0 2 00000002 0000001f [0000784436946629] 0.032ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#32)           dbench  3399 0 2 00000002 00000020 [0000784436947084] 0.033ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#33)           dbench  3399 0 2 00000002 00000021 [0000784436947556] 0.033ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#34)           dbench  3399 0 2 00000002 00000022 [0000784436948042] 0.034ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#35)           dbench  3399 0 2 00000002 00000023 [0000784436948501] 0.035ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#36)           dbench  3399 0 2 00000002 00000024 [0000784436948897] 0.036ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#37)           dbench  3399 0 2 00000002 00000025 [0000784436949383] 0.036ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#38)           dbench  3399 0 2 00000002 00000026 [0000784436949824] 0.037ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#39)           dbench  3399 0 2 00000002 00000027 [0000784436950229] 0.038ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#40)           dbench  3399 0 2 00000002 00000028 [0000784436950715] 0.039ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#41)           dbench  3399 0 2 00000002 00000029 [0000784436951593] 0.040ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#42)           dbench  3399 0 2 00000002 0000002a [0000784436952106] 0.041ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#43)           dbench  3399 0 2 00000002 0000002b [0000784436952596] 0.042ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#44)           dbench  3399 0 2 00000002 0000002c [0000784436953114] 0.043ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#45)           dbench  3399 0 2 00000002 0000002d [0000784436953528] 0.043ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#46)           dbench  3399 0 2 00000002 0000002e [0000784436954018] 0.044ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#47)           dbench  3399 0 2 00000002 0000002f [0000784436954540] 0.045ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#48)           dbench  3399 0 2 00000002 00000030 [0000784436955197] 0.046ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#49)           dbench  3399 0 2 00000002 00000031 [0000784436955755] 0.047ms (+0.004ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#50)           dbench  3399 0 2 00000002 00000032 [0000784436958356] 0.051ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#51)           dbench  3399 0 2 00000002 00000033 [0000784436958860] 0.052ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#52)           dbench  3399 0 2 00000002 00000034 [0000784436959346] 0.053ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#53)           dbench  3399 0 2 00000002 00000035 [0000784436959801] 0.054ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#54)           dbench  3399 0 2 00000002 00000036 [0000784436960201] 0.054ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#55)           dbench  3399 0 2 00000002 00000037 [0000784436960705] 0.055ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#56)           dbench  3399 0 2 00000002 00000038 [0000784436961223] 0.056ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#57)           dbench  3399 0 2 00000002 00000039 [0000784436961637] 0.057ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#58)           dbench  3399 0 2 00000002 0000003a [0000784436962127] 0.058ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#59)           dbench  3399 0 2 00000002 0000003b [0000784436962717] 0.059ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#60)           dbench  3399 0 2 00000002 0000003c [0000784436963131] 0.059ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#61)           dbench  3399 0 2 00000002 0000003d [0000784436963621] 0.060ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#62)           dbench  3399 0 2 00000002 0000003e [0000784436964062] 0.061ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#63)           dbench  3399 0 2 00000002 0000003f [0000784436964539] 0.062ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#64)           dbench  3399 0 2 00000002 00000040 [0000784436965025] 0.062ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#65)           dbench  3399 0 2 00000002 00000041 [0000784436965642] 0.064ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#66)           dbench  3399 0 2 00000002 00000042 [0000784436966636] 0.065ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#67)           dbench  3399 0 2 00000002 00000043 [0000784436967181] 0.066ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#68)           dbench  3399 0 2 00000002 00000044 [0000784436967716] 0.067ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#69)           dbench  3399 0 2 00000002 00000045 [0000784436968688] 0.069ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#70)           dbench  3399 0 2 00000002 00000046 [0000784436969251] 0.070ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#71)           dbench  3399 0 2 00000002 00000047 [0000784436969840] 0.071ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#72)           dbench  3399 0 2 00000002 00000048 [0000784436970263] 0.071ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#73)           dbench  3399 0 2 00000002 00000049 [0000784436970754] 0.072ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#74)           dbench  3399 0 2 00000002 0000004a [0000784436971276] 0.073ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#75)           dbench  3399 0 2 00000002 0000004b [0000784436971838] 0.074ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#76)           dbench  3399 0 2 00000002 0000004c [0000784436972329] 0.075ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#77)           dbench  3399 0 2 00000002 0000004d [0000784436973184] 0.076ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#78)           dbench  3399 0 2 00000002 0000004e [0000784436973742] 0.077ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#79)           dbench  3399 0 2 00000002 0000004f [0000784436974237] 0.078ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#80)           dbench  3399 0 2 00000002 00000050 [0000784436974691] 0.079ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#81)           dbench  3399 0 2 00000002 00000051 [0000784436975092] 0.079ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#82)           dbench  3399 0 2 00000002 00000052 [0000784436975578] 0.080ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#83)           dbench  3399 0 2 00000002 00000053 [0000784436976019] 0.081ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#84)           dbench  3399 0 2 00000002 00000054 [0000784436976415] 0.081ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#85)           dbench  3399 0 2 00000002 00000055 [0000784436976901] 0.082ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#86)           dbench  3399 0 2 00000002 00000056 [0000784436977423] 0.083ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#87)           dbench  3399 0 2 00000002 00000057 [0000784436977837] 0.084ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#88)           dbench  3399 0 2 00000002 00000058 [0000784436978327] 0.085ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#89)           dbench  3399 0 2 00000002 00000059 [0000784436978845] 0.085ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#90)           dbench  3399 0 2 00000002 0000005a [0000784436979412] 0.086ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#91)           dbench  3399 0 2 00000002 0000005b [0000784436979902] 0.087ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#92)           dbench  3399 0 2 00000002 0000005c [0000784436980361] 0.088ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#93)           dbench  3399 0 2 00000002 0000005d [0000784436980762] 0.089ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#94)           dbench  3399 0 2 00000002 0000005e [0000784436981252] 0.089ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#95)           dbench  3399 0 2 00000002 0000005f [0000784436981698] 0.090ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#96)           dbench  3399 0 2 00000002 00000060 [0000784436982098] 0.091ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#97)           dbench  3399 0 2 00000002 00000061 [0000784436982584] 0.092ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#98)           dbench  3399 0 2 00000002 00000062 [0000784436983021] 0.092ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#99)           dbench  3399 0 2 00000002 00000063 [0000784436983417] 0.093ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#100)           dbench  3399 0 2 00000002 00000064 [0000784436983916] 0.094ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#101)           dbench  3399 0 2 00000002 00000065 [0000784436984357] 0.095ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#102)           dbench  3399 0 2 00000002 00000066 [0000784436984839] 0.095ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#103)           dbench  3399 0 2 00000002 00000067 [0000784436985392] 0.096ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#104)           dbench  3399 0 2 00000002 00000068 [0000784436986108] 0.098ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#105)           dbench  3399 0 2 00000002 00000069 [0000784436986972] 0.099ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#106)           dbench  3399 0 2 00000002 0000006a [0000784436987570] 0.100ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#107)           dbench  3399 0 2 00000002 0000006b [0000784436988083] 0.101ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#108)           dbench  3399 0 2 00000002 0000006c [0000784436988497] 0.102ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#109)           dbench  3399 0 2 00000002 0000006d [0000784436988997] 0.102ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#110)           dbench  3399 0 2 00000002 0000006e [0000784436989442] 0.103ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#111)           dbench  3399 0 2 00000002 0000006f [0000784436990144] 0.104ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#112)           dbench  3399 0 2 00000002 00000070 [0000784436990702] 0.105ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#113)           dbench  3399 0 2 00000002 00000071 [0000784436991157] 0.106ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#114)           dbench  3399 0 2 00000002 00000072 [0000784436991634] 0.107ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#115)           dbench  3399 0 2 00000002 00000073 [0000784436992187] 0.108ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#116)           dbench  3399 0 2 00000002 00000074 [0000784436992646] 0.108ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#117)           dbench  3399 0 2 00000002 00000075 [0000784436993218] 0.109ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#118)           dbench  3399 0 2 00000002 00000076 [0000784436993713] 0.110ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#119)           dbench  3399 0 2 00000002 00000077 [0000784436994167] 0.111ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#120)           dbench  3399 0 2 00000002 00000078 [0000784436994563] 0.112ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#121)           dbench  3399 0 2 00000002 00000079 [0000784436995081] 0.113ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#122)           dbench  3399 0 2 00000002 0000007a [0000784436995679] 0.114ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#123)           dbench  3399 0 2 00000002 0000007b [0000784436996102] 0.114ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#124)           dbench  3399 0 2 00000002 0000007c [0000784436996593] 0.115ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#125)           dbench  3399 0 2 00000002 0000007d [0000784436997038] 0.116ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#126)           dbench  3399 0 2 00000002 0000007e [0000784436997524] 0.117ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#127)           dbench  3399 0 2 00000002 0000007f [0000784436998015] 0.117ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#128)           dbench  3399 0 2 00000002 00000080 [0000784436998532] 0.118ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#129)           dbench  3399 0 2 00000002 00000081 [0000784436999126] 0.119ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#130)           dbench  3399 0 2 00000002 00000082 [0000784436999689] 0.120ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#131)           dbench  3399 0 2 00000002 00000083 [0000784437000193] 0.121ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#132)           dbench  3399 0 2 00000002 00000084 [0000784437000625] 0.122ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#133)           dbench  3399 0 2 00000002 00000085 [0000784437001111] 0.123ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#134)           dbench  3399 0 2 00000002 00000086 [0000784437001822] 0.124ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#135)           dbench  3399 0 2 00000002 00000087 [0000784437002375] 0.125ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#136)           dbench  3399 0 2 00000002 00000088 [0000784437002861] 0.125ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#137)           dbench  3399 0 2 00000002 00000089 [0000784437003320] 0.126ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#138)           dbench  3399 0 2 00000002 0000008a [0000784437003788] 0.127ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#139)           dbench  3399 0 2 00000002 0000008b [0000784437004279] 0.128ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#140)           dbench  3399 0 2 00000002 0000008c [0000784437004859] 0.129ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#141)           dbench  3399 0 2 00000002 0000008d [0000784437005444] 0.130ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#142)           dbench  3399 0 2 00000002 0000008e [0000784437005935] 0.131ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#143)           dbench  3399 0 2 00000002 0000008f [0000784437006398] 0.131ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#144)           dbench  3399 0 2 00000002 00000090 [0000784437006799] 0.132ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#145)           dbench  3399 0 2 00000002 00000091 [0000784437007289] 0.133ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#146)           dbench  3399 0 2 00000002 00000092 [0000784437008005] 0.134ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#147)           dbench  3399 0 2 00000002 00000093 [0000784437008545] 0.135ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#148)           dbench  3399 0 2 00000002 00000094 [0000784437009031] 0.136ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#149)           dbench  3399 0 2 00000002 00000095 [0000784437009546] 0.137ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#150)           dbench  3399 0 2 00000002 00000096 [0000784437009944] 0.137ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#151)           dbench  3399 0 2 00000002 00000097 [0000784437010435] 0.138ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#152)           dbench  3399 0 2 00000002 00000098 [0000784437010871] 0.139ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#153)           dbench  3399 0 2 00000002 00000099 [0000784437011461] 0.140ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#154)           dbench  3399 0 2 00000002 0000009a [0000784437012014] 0.141ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#155)           dbench  3399 0 2 00000002 0000009b [0000784437012464] 0.141ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#156)           dbench  3399 0 2 00000002 0000009c [0000784437012874] 0.142ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#157)           dbench  3399 0 2 00000002 0000009d [0000784437013364] 0.143ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#158)           dbench  3399 0 2 00000002 0000009e [0000784437013823] 0.144ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#159)           dbench  3399 0 2 00000002 0000009f [0000784437014224] 0.144ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#160)           dbench  3399 0 2 00000002 000000a0 [0000784437014714] 0.145ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#161)           dbench  3399 0 2 00000002 000000a1 [0000784437015160] 0.146ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#162)           dbench  3399 0 2 00000002 000000a2 [0000784437015637] 0.147ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#163)           dbench  3399 0 2 00000002 000000a3 [0000784437016127] 0.148ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#164)           dbench  3399 0 2 00000002 000000a4 [0000784437016609] 0.148ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#165)           dbench  3399 0 2 00000002 000000a5 [0000784437017032] 0.149ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#166)           dbench  3399 0 2 00000002 000000a6 [0000784437017603] 0.150ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#167)           dbench  3399 0 2 00000002 000000a7 [0000784437018040] 0.151ms (+0.002ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#168)           dbench  3399 0 2 00000002 000000a8 [0000784437019696] 0.153ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#169)           dbench  3399 0 2 00000002 000000a9 [0000784437020249] 0.154ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#170)           dbench  3399 0 2 00000002 000000aa [0000784437020834] 0.155ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#171)           dbench  3399 0 2 00000002 000000ab [0000784437021392] 0.156ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#172)           dbench  3399 0 2 00000002 000000ac [0000784437021883] 0.157ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#173)           dbench  3399 0 2 00000002 000000ad [0000784437022342] 0.158ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#174)           dbench  3399 0 2 00000002 000000ae [0000784437022828] 0.159ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#175)           dbench  3399 0 2 00000002 000000af [0000784437023327] 0.160ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#176)           dbench  3399 0 2 00000002 000000b0 [0000784437023786] 0.160ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#177)           dbench  3399 0 2 00000002 000000b1 [0000784437024187] 0.161ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#178)           dbench  3399 0 2 00000002 000000b2 [0000784437024677] 0.162ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#179)           dbench  3399 0 2 00000002 000000b3 [0000784437025118] 0.162ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#180)           dbench  3399 0 2 00000002 000000b4 [0000784437025694] 0.163ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#181)           dbench  3399 0 2 00000002 000000b5 [0000784437026198] 0.164ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#182)           dbench  3399 0 2 00000002 000000b6 [0000784437026653] 0.165ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#183)           dbench  3399 0 2 00000002 000000b7 [0000784437027121] 0.166ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#184)           dbench  3399 0 2 00000002 000000b8 [0000784437027683] 0.167ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#185)           dbench  3399 0 2 00000002 000000b9 [0000784437028142] 0.168ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#186)           dbench  3399 0 2 00000002 000000ba [0000784437028538] 0.168ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#187)           dbench  3399 0 2 00000002 000000bb [0000784437029024] 0.169ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#188)           dbench  3399 0 2 00000002 000000bc [0000784437029461] 0.170ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#189)           dbench  3399 0 2 00000002 000000bd [0000784437029857] 0.170ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#190)           dbench  3399 0 2 00000002 000000be [0000784437030343] 0.171ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#191)           dbench  3399 0 2 00000002 000000bf [0000784437030869] 0.172ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#192)           dbench  3399 0 2 00000002 000000c0 [0000784437031337] 0.173ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#193)           dbench  3399 0 2 00000002 000000c1 [0000784437031828] 0.174ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#194)           dbench  3399 0 2 00000002 000000c2 [0000784437032395] 0.175ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#195)           dbench  3399 0 2 00000002 000000c3 [0000784437033169] 0.176ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#196)           dbench  3399 0 2 00000002 000000c4 [0000784437033722] 0.177ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#197)           dbench  3399 0 2 00000002 000000c5 [0000784437034496] 0.178ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#198)           dbench  3399 0 2 00000002 000000c6 [0000784437035036] 0.179ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#199)           dbench  3399 0 2 00000002 000000c7 [0000784437035522] 0.180ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#200)           dbench  3399 0 2 00000002 000000c8 [0000784437035963] 0.181ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#201)           dbench  3399 0 2 00000002 000000c9 [0000784437036436] 0.181ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#202)           dbench  3399 0 2 00000002 000000ca [0000784437037007] 0.182ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#203)           dbench  3399 0 2 00000002 000000cb [0000784437037462] 0.183ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#204)           dbench  3399 0 2 00000002 000000cc [0000784437037939] 0.184ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#205)           dbench  3399 0 2 00000002 000000cd [0000784437038425] 0.185ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#206)           dbench  3399 0 2 00000002 000000ce [0000784437038996] 0.186ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#207)           dbench  3399 0 2 00000002 000000cf [0000784437039563] 0.187ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#208)           dbench  3399 0 2 00000002 000000d0 [0000784437040054] 0.187ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#209)           dbench  3399 0 2 00000002 000000d1 [0000784437040535] 0.188ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#210)           dbench  3399 0 2 00000002 000000d2 [0000784437041017] 0.189ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#211)           dbench  3399 0 2 00000002 000000d3 [0000784437041507] 0.190ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#212)           dbench  3399 0 2 00000002 000000d4 [0000784437041971] 0.191ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#213)           dbench  3399 0 2 00000002 000000d5 [0000784437042443] 0.191ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#214)           dbench  3399 0 2 00000002 000000d6 [0000784437042934] 0.192ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#215)           dbench  3399 0 2 00000002 000000d7 [0000784437043591] 0.193ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#216)           dbench  3399 0 2 00000002 000000d8 [0000784437044005] 0.194ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#217)           dbench  3399 0 2 00000002 000000d9 [0000784437044491] 0.195ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#218)           dbench  3399 0 2 00000002 000000da [0000784437045004] 0.196ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#219)           dbench  3399 0 2 00000002 000000db [0000784437045566] 0.197ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#220)           dbench  3399 0 2 00000002 000000dc [0000784437046061] 0.197ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#221)           dbench  3399 0 2 00000002 000000dd [0000784437046574] 0.198ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#222)           dbench  3399 0 2 00000002 000000de [0000784437046984] 0.199ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#223)           dbench  3399 0 2 00000002 000000df [0000784437047470] 0.200ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#224)           dbench  3399 0 2 00000002 000000e0 [0000784437047983] 0.201ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#225)           dbench  3399 0 2 00000002 000000e1 [0000784437048572] 0.202ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#226)           dbench  3399 0 2 00000002 000000e2 [0000784437049130] 0.202ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#227)           dbench  3399 0 2 00000002 000000e3 [0000784437049639] 0.203ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#228)           dbench  3399 0 2 00000002 000000e4 [0000784437050053] 0.204ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#229)           dbench  3399 0 2 00000002 000000e5 [0000784437050543] 0.205ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#230)           dbench  3399 0 2 00000002 000000e6 [0000784437051061] 0.206ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#231)           dbench  3399 0 2 00000002 000000e7 [0000784437051484] 0.206ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#232)           dbench  3399 0 2 00000002 000000e8 [0000784437051974] 0.207ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#233)           dbench  3399 0 2 00000002 000000e9 [0000784437052492] 0.208ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#234)           dbench  3399 0 2 00000002 000000ea [0000784437053108] 0.209ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#235)           dbench  3399 0 2 00000002 000000eb [0000784437053662] 0.210ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#236)           dbench  3399 0 2 00000002 000000ec [0000784437054166] 0.211ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#237)           dbench  3399 0 2 00000002 000000ed [0000784437054607] 0.212ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#238)           dbench  3399 0 2 00000002 000000ee [0000784437055093] 0.212ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#239)           dbench  3399 0 2 00000002 000000ef [0000784437055529] 0.213ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#240)           dbench  3399 0 2 00000002 000000f0 [0000784437055925] 0.214ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#241)           dbench  3399 0 2 00000002 000000f1 [0000784437056420] 0.215ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#242)           dbench  3399 0 2 00000002 000000f2 [0000784437056933] 0.215ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#243)           dbench  3399 0 2 00000002 000000f3 [0000784437057347] 0.216ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#244)           dbench  3399 0 2 00000002 000000f4 [0000784437057838] 0.217ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#245)           dbench  3399 0 2 00000002 000000f5 [0000784437058355] 0.218ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#246)           dbench  3399 0 2 00000002 000000f6 [0000784437058769] 0.218ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#247)           dbench  3399 0 2 00000002 000000f7 [0000784437059260] 0.219ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#248)           dbench  3399 0 2 00000002 000000f8 [0000784437059701] 0.220ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#249)           dbench  3399 0 2 00000002 000000f9 [0000784437060101] 0.221ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#250)           dbench  3399 0 2 00000002 000000fa [0000784437060592] 0.222ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#251)           dbench  3399 0 2 00000002 000000fb [0000784437061186] 0.223ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#252)           dbench  3399 0 2 00000002 000000fc [0000784437061613] 0.223ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#253)           dbench  3399 0 2 00000002 000000fd [0000784437062104] 0.224ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#254)           dbench  3399 0 2 00000002 000000fe [0000784437062545] 0.225ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#255)           dbench  3399 0 2 00000002 000000ff [0000784437062941] 0.225ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#256)           dbench  3399 0 2 00000002 00000100 [0000784437063463] 0.226ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#257)           dbench  3399 0 2 00000002 00000101 [0000784437063980] 0.227ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#258)           dbench  3399 0 2 00000002 00000102 [0000784437064484] 0.228ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#259)           dbench  3399 0 2 00000002 00000103 [0000784437064970] 0.229ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#260)           dbench  3399 0 2 00000002 00000104 [0000784437065411] 0.230ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#261)           dbench  3399 0 2 00000002 00000105 [0000784437065843] 0.230ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#262)           dbench  3399 0 2 00000002 00000106 [0000784437066334] 0.231ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#263)           dbench  3399 0 2 00000002 00000107 [0000784437066856] 0.232ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#264)           dbench  3399 0 2 00000002 00000108 [0000784437067283] 0.233ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#265)           dbench  3399 0 2 00000002 00000109 [0000784437067774] 0.233ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#266)           dbench  3399 0 2 00000002 0000010a [0000784437068296] 0.234ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#267)           dbench  3399 0 2 00000002 0000010b [0000784437068737] 0.235ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#268)           dbench  3399 0 2 00000002 0000010c [0000784437069227] 0.236ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#269)           dbench  3399 0 2 00000002 0000010d [0000784437069745] 0.237ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#270)           dbench  3399 0 2 00000002 0000010e [0000784437070181] 0.237ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#271)           dbench  3399 0 2 00000002 0000010f [0000784437070681] 0.238ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#272)           dbench  3399 0 2 00000002 00000110 [0000784437071144] 0.239ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#273)           dbench  3399 0 2 00000002 00000111 [0000784437071540] 0.240ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#274)           dbench  3399 0 2 00000002 00000112 [0000784437072026] 0.241ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#275)           dbench  3399 0 2 00000002 00000113 [0000784437072548] 0.241ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#276)           dbench  3399 0 2 00000002 00000114 [0000784437073124] 0.242ms (+0.001ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#277)           dbench  3399 0 2 00000002 00000115 [0000784437073828] 0.244ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#278)           dbench  3399 0 2 00000002 00000116 [0000784437074281] 0.244ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#279)           dbench  3399 0 2 00000002 00000117 [0000784437074758] 0.245ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#280)           dbench  3399 0 2 00000002 00000118 [0000784437075248] 0.246ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#281)           dbench  3399 0 2 00000002 00000119 [0000784437075716] 0.247ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#282)           dbench  3399 0 2 00000002 0000011a [0000784437076130] 0.247ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#283)           dbench  3399 0 2 00000002 0000011b [0000784437076621] 0.248ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#284)           dbench  3399 0 2 00000002 0000011c [0000784437077066] 0.249ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#285)           dbench  3399 0 2 00000002 0000011d [0000784437077467] 0.250ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#286)           dbench  3399 0 2 00000002 0000011e [0000784437077957] 0.250ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#287)           dbench  3399 0 2 00000002 0000011f [0000784437078403] 0.251ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#288)           dbench  3399 0 2 00000002 00000120 [0000784437078803] 0.252ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#289)           dbench  3399 0 2 00000002 00000121 [0000784437079294] 0.253ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#290)           dbench  3399 0 2 00000002 00000122 [0000784437079735] 0.253ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#291)           dbench  3399 0 2 00000002 00000123 [0000784437080315] 0.254ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#292)           dbench  3399 0 2 00000002 00000124 [0000784437080878] 0.255ms (+0.000ms): find_next_reservable_window+0xb/0x60 <c01b58bb> (alloc_new_reservation+0xd2/0x1c0 <c01b59e2>)
(T1/#293)           dbench  3399 0 2 00000002 00000125 [0000784437081243] 0.256ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#294)           dbench  3399 0 2 00000002 00000126 [0000784437081652] 0.257ms (+0.001ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#295)           dbench  3399 0 2 00000002 00000127 [0000784437082466] 0.258ms (+0.000ms): bitmap_search_next_usable_block+0xc/0xd0 <c01b551c> (alloc_new_reservation+0xf7/0x1c0 <c01b5a07>)
(T1/#296)           dbench  3399 0 2 00000002 00000128 [0000784437082822] 0.259ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#297)           dbench  3399 0 2 00000002 00000129 [0000784437083312] 0.259ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#298)           dbench  3399 0 2 00000002 0000012a [0000784437083906] 0.260ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#299)           dbench  3399 0 2 00000002 0000012b [0000784437084424] 0.261ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#300)           dbench  3399 0 2 00000002 0000012c [0000784437084896] 0.262ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#301)           dbench  3399 0 2 00000002 0000012d [0000784437085396] 0.263ms (+0.000ms): find_next_reservable_window+0xb/0x60 <c01b58bb> (alloc_new_reservation+0xd2/0x1c0 <c01b59e2>)
(T1/#302)           dbench  3399 0 2 00000002 0000012e [0000784437085756] 0.263ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#303)           dbench  3399 0 2 00000002 0000012f [0000784437086143] 0.264ms (+0.000ms): rb_next+0x8/0x60 <c01d9f88> (find_next_reservable_window+0x39/0x60 <c01b58e9>)
(T1/#304)           dbench  3399 0 2 00000002 00000130 [0000784437086642] 0.265ms (+0.000ms): bitmap_search_next_usable_block+0xc/0xd0 <c01b551c> (alloc_new_reservation+0xf7/0x1c0 <c01b5a07>)
(T1/#305)           dbench  3399 0 2 00000002 00000131 [0000784437087002] 0.265ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#306)           dbench  3399 0 2 00000002 00000132 [0000784437087416] 0.266ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#307)           dbench  3399 0 2 00000002 00000133 [0000784437087916] 0.267ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#308)           dbench  3399 0 2 00000002 00000134 [0000784437088361] 0.268ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#309)           dbench  3399 0 2 00000002 00000135 [0000784437088838] 0.269ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#310)           dbench  3399 0 2 00000002 00000136 [0000784437089324] 0.269ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#311)           dbench  3399 0 2 00000002 00000137 [0000784437089801] 0.270ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#312)           dbench  3399 0 2 00000002 00000138 [0000784437090233] 0.271ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#313)           dbench  3399 0 2 00000002 00000139 [0000784437090719] 0.272ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#314)           dbench  3399 0 2 00000002 0000013a [0000784437091160] 0.272ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#315)           dbench  3399 0 2 00000002 0000013b [0000784437091556] 0.273ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#316)           dbench  3399 0 2 00000002 0000013c [0000784437092042] 0.274ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#317)           dbench  3399 0 2 00000002 0000013d [0000784437092483] 0.275ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#318)           dbench  3399 0 2 00000002 0000013e [0000784437092884] 0.275ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#319)           dbench  3399 0 2 00000002 0000013f [0000784437093374] 0.276ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#320)           dbench  3399 0 2 00000002 00000140 [0000784437093820] 0.277ms (+0.000ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0x2b/0xd0 <c01b553b>)
(T1/#321)           dbench  3399 0 2 00000002 00000141 [0000784437094229] 0.277ms (+0.000ms): ext3_test_allocatable+0xa/0x90 <c01b548a> (bitmap_search_next_usable_block+0x41/0xd0 <c01b5551>)
(T1/#322)           dbench  3399 0 2 00000002 00000142 [0000784437094787] 0.278ms (+0.001ms): find_next_zero_bit+0xc/0xa8 <c01db78c> (bitmap_search_next_usable_block+0xb1/0xd0 <c01b55c1>)
(T1/#323)           dbench  3399 0 2 00000001 00000143 [0000784437095705] 0.280ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (ext3_try_to_allocate_with_rsv+0x28f/0x330 <c01b5d5f>)
(T1/#324)           dbench  3399 0 2 00000000 00000144 [0000784437096304] 0.281ms (+0.000ms): preempt_schedule+0xa/0x70 <c027d0fa> (ext3_try_to_allocate_with_rsv+0x285/0x330 <c01b5d55>)
(T1/#325)           dbench  3399 0 3 00000000 00000145 [0000784437096839] 0.282ms (+0.000ms): __schedule+0xe/0x630 <c027c9be> (preempt_schedule+0x4f/0x70 <c027d13f>)
(T1/#326)           dbench  3399 0 3 00000000 00000146 [0000784437097231] 0.282ms (+0.000ms): profile_hit+0x9/0x50 <c0115749> (__schedule+0x3a/0x630 <c027c9ea>)
(T1/#327)           dbench  3399 0 3 00000001 00000147 [0000784437097667] 0.283ms (+0.001ms): sched_clock+0xe/0xe0 <c010c3ae> (__schedule+0x62/0x630 <c027ca12>)
(T1/#328)           dbench  3399 0 3 00000002 00000148 [0000784437098608] 0.285ms (+0.000ms): dequeue_task+0xa/0x50 <c010f4ea> (__schedule+0x1ab/0x630 <c027cb5b>)
(T1/#329)           dbench  3399 0 3 00000002 00000149 [0000784437099166] 0.286ms (+0.000ms): recalc_task_prio+0xc/0x1a0 <c010f64c> (__schedule+0x1c5/0x630 <c027cb75>)
(T1/#330)           dbench  3399 0 3 00000002 0000014a [0000784437099611] 0.286ms (+0.000ms): effective_prio+0x8/0x50 <c010f5f8> (recalc_task_prio+0xa6/0x1a0 <c010f6e6>)
(T1/#331)           dbench  3399 0 3 00000002 0000014b [0000784437100012] 0.287ms (+0.001ms): enqueue_task+0xa/0x80 <c010f53a> (__schedule+0x1cc/0x630 <c027cb7c>)
(T4/#332) [ =>           dbench ] 0.288ms (+0.001ms)
(T1/#333)            <...>     2 0 1 00000002 0000014d [0000784437101515] 0.290ms (+0.002ms): __switch_to+0xb/0x1a0 <c0100f5b> (__schedule+0x2bd/0x630 <c027cc6d>)
(T3/#334)    <...>-2     0d..2  292µs : __schedule+0x2ea/0x630 <c027cc9a> <dbench-3399> (76 69): 
(T1/#335)            <...>     2 0 1 00000002 0000014f [0000784437103484] 0.293ms (+0.000ms): finish_task_switch+0xc/0x90 <c010fdec> (__schedule+0x2f6/0x630 <c027cca6>)
(T1/#336)            <...>     2 0 1 00000001 00000150 [0000784437103916] 0.294ms (+0.000ms): trace_stop_sched_switched+0xa/0x150 <c012cc1a> (finish_task_switch+0x43/0x90 <c010fe23>)
(T3/#337)    <...>-2     0d..1  294µs : trace_stop_sched_switched+0x42/0x150 <c012cc52> <<...>-2> (69 0): 
(T1/#338)            <...>     2 0 1 00000001 00000152 [0000784437105041] 0.295ms (+0.000ms): trace_stop_sched_switched+0xfe/0x150 <c012cd0e> (finish_task_switch+0x43/0x90 <c010fe23>)


vim:ft=help

^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, other threads:[~2005-02-25 15:04 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-02-23 18:07 More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Lee Revell
2005-02-23 19:16 ` Hugh Dickins
2005-02-23 19:36   ` Lee Revell
2005-02-23 20:06     ` Hugh Dickins
2005-02-23 20:10       ` Lee Revell
2005-02-23 20:30       ` Lee Revell
2005-02-23 21:03         ` Hugh Dickins
2005-02-23 22:14           ` Lee Revell
2005-02-23 23:52             ` PPC RT Patch john cooper
2005-02-24  4:20               ` Frank Rowand
2005-02-24 13:56                 ` john cooper
2005-02-23 23:27           ` More latency regressions with 2.6.11-rc4-RT-V0.7.39-02 Nick Piggin
2005-02-24  1:03             ` Lee Revell
2005-02-24  1:29               ` Nick Piggin
2005-02-24  2:24                 ` Lee Revell
2005-02-24  2:41                   ` Nick Piggin
2005-02-24  3:03                     ` Lee Revell
2005-02-23 20:53       ` Hugh Dickins
2005-02-23 22:13         ` Lee Revell
2005-02-24  4:56           ` Hugh Dickins
2005-02-24  6:32             ` Lee Revell
2005-02-24  8:26               ` Hugh Dickins
2005-02-25  3:30                 ` Lee Revell
2005-02-25  5:58                   ` Hugh Dickins
2005-02-25 15:02                     ` Lee Revell
2005-02-23 19:52 ` Lee Revell

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox