* oltp ~10% regression with 2.6.27-rc5 on stoakley machine
@ 2008-09-04 8:51 Lin Ming
2008-09-04 9:03 ` Peter Zijlstra
2008-09-20 21:38 ` Peter Zijlstra
0 siblings, 2 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-04 8:51 UTC (permalink / raw)
To: a.p.zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo
Comparing with 2.6.27-rc4, oltp has ~10% regression with 2.6.27-rc5 on
8-core stoakley machine.
Run oltp with 8 threads 120 seconds, vmstat shows much more idle time, about ~30%
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
10 0 0 7822824 42240 123740 0 0 312 47 442 1613 3 2 88 6 0
9 0 0 7822312 42240 123764 0 0 0 16 26691 232566 56 14 30 0 0
13 0 0 7821940 42240 123764 0 0 0 16 26661 228689 54 14 32 0 0
8 0 0 7821320 42240 123764 0 0 0 16 31508 263765 61 17 23 0 0
12 0 0 7820948 42240 123764 0 0 16 16 28666 242402 57 15 28 0 0
9 0 0 7820584 42240 123780 0 0 0 16 27107 230804 56 14 30 0 0
10 0 0 7819964 42240 123796 0 0 16 612 27599 244037 55 16 29 0 0
11 0 0 7819356 42240 123796 0 0 0 64 23540 209713 51 13 36 0 0
10 0 0 7819212 42240 123796 0 0 0 32 25674 224205 54 13 32 0 0
10 0 0 7818716 42240 123796 0 0 0 20 30106 257161 59 16 25 0 0
7 0 0 7818468 42240 123796 0 0 0 16 28356 241551 57 14 29 0 0
10 0 0 7818096 42240 123796 0 0 0 16 39174 273656 64 16 20 0 0
12 0 0 7817724 42240 123796 0 0 0 20 39688 276936 63 16 20 0 0
11 0 0 7817352 42240 123796 0 0 0 16 42543 285192 66 16 18 0 0
9 0 0 7817352 42240 123796 0 0 0 16 37083 259830 62 14 24 0 0
8 0 0 7817104 42240 123796 0 0 0 16 37450 259160 61 15 23 0 0
10 0 0 7816516 42240 123796 0 0 0 64 37425 261870 61 16 23 0 0
11 0 0 7815896 42240 123812 0 0 16 16 41558 279320 66 16 18 0 0
9 0 0 7815648 42240 123812 0 0 0 16 34017 235741 59 14 28 0 0
10 0 0 7815152 42240 123812 0 0 0 16 35642 248888 60 14 26 0 0
9 0 0 7814532 42240 123812 0 0 0 16 38517 263220 63 15 22 0 0
9 0 0 7814160 42240 123812 0 0 0 20 35965 246487 61 14 25 0 0
10 0 0 7814036 42240 123812 0 0 0 16 33852 236313 59 13 28 0 0
11 0 0 7813664 42240 123812 0 0 0 16 34958 244819 59 14 27 0 0
10 0 0 7813416 42240 123812 0 0 0 16 26106 202062 53 10 37 0 0
10 0 0 7812672 42240 123812 0 0 0 16 31174 222714 56 12 32 0 0
9 0 0 7812300 42240 123812 0 0 0 276 25089 196813 52 11 38 0 0
9 0 0 7812060 42240 123812 0 0 0 16 31877 228004 57 12 31 0 0
Bisect located below patch,
after reverted this patch the regression disappear.
commit 354879bb977e06695993435745f06a0f6d39ce2b
Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
Date: Mon Aug 25 17:15:34 2008 +0200
sched_clock: fix cpu_clock()
This patch fixes 3 issues:
a) it removes the dependency on jiffies, because jiffies are
incremented
by a single CPU, and the tick is not synchronized between CPUs.
Therefore
relying on it to calculate a window to clip whacky TSC values
doesn't work
as it can drift around.
So instead use [GTOD, GTOD+TICK_NSEC) as the window.
b) __update_sched_clock() did (roughly speaking):
delta = sched_clock() - scd->tick_raw;
clock += delta;
Which gives exponential growth, instead of linear.
c) allows the sched_clock_cpu() value to warp the u64 without
breaking.
the results are more reliable sched_clock() deltas:
-Lin Ming
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 8:51 oltp ~10% regression with 2.6.27-rc5 on stoakley machine Lin Ming
@ 2008-09-04 9:03 ` Peter Zijlstra
2008-09-04 10:52 ` Lin Ming
2008-09-05 1:26 ` Lin Ming
2008-09-20 21:38 ` Peter Zijlstra
1 sibling, 2 replies; 19+ messages in thread
From: Peter Zijlstra @ 2008-09-04 9:03 UTC (permalink / raw)
To: Lin Ming; +Cc: linux-kernel, Zhang, Yanmin, mingo
On Thu, 2008-09-04 at 16:51 +0800, Lin Ming wrote:
> Comparing with 2.6.27-rc4, oltp has ~10% regression with 2.6.27-rc5 on
> 8-core stoakley machine.
>
> Run oltp with 8 threads 120 seconds, vmstat shows much more idle time, about ~30%
>
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> 10 0 0 7822824 42240 123740 0 0 312 47 442 1613 3 2 88 6 0
> 9 0 0 7822312 42240 123764 0 0 0 16 26691 232566 56 14 30 0 0
> 13 0 0 7821940 42240 123764 0 0 0 16 26661 228689 54 14 32 0 0
> 8 0 0 7821320 42240 123764 0 0 0 16 31508 263765 61 17 23 0 0
> 12 0 0 7820948 42240 123764 0 0 16 16 28666 242402 57 15 28 0 0
> 9 0 0 7820584 42240 123780 0 0 0 16 27107 230804 56 14 30 0 0
> 10 0 0 7819964 42240 123796 0 0 16 612 27599 244037 55 16 29 0 0
> 11 0 0 7819356 42240 123796 0 0 0 64 23540 209713 51 13 36 0 0
> 10 0 0 7819212 42240 123796 0 0 0 32 25674 224205 54 13 32 0 0
> 10 0 0 7818716 42240 123796 0 0 0 20 30106 257161 59 16 25 0 0
> 7 0 0 7818468 42240 123796 0 0 0 16 28356 241551 57 14 29 0 0
> 10 0 0 7818096 42240 123796 0 0 0 16 39174 273656 64 16 20 0 0
> 12 0 0 7817724 42240 123796 0 0 0 20 39688 276936 63 16 20 0 0
> 11 0 0 7817352 42240 123796 0 0 0 16 42543 285192 66 16 18 0 0
> 9 0 0 7817352 42240 123796 0 0 0 16 37083 259830 62 14 24 0 0
> 8 0 0 7817104 42240 123796 0 0 0 16 37450 259160 61 15 23 0 0
> 10 0 0 7816516 42240 123796 0 0 0 64 37425 261870 61 16 23 0 0
> 11 0 0 7815896 42240 123812 0 0 16 16 41558 279320 66 16 18 0 0
> 9 0 0 7815648 42240 123812 0 0 0 16 34017 235741 59 14 28 0 0
> 10 0 0 7815152 42240 123812 0 0 0 16 35642 248888 60 14 26 0 0
> 9 0 0 7814532 42240 123812 0 0 0 16 38517 263220 63 15 22 0 0
> 9 0 0 7814160 42240 123812 0 0 0 20 35965 246487 61 14 25 0 0
> 10 0 0 7814036 42240 123812 0 0 0 16 33852 236313 59 13 28 0 0
> 11 0 0 7813664 42240 123812 0 0 0 16 34958 244819 59 14 27 0 0
> 10 0 0 7813416 42240 123812 0 0 0 16 26106 202062 53 10 37 0 0
> 10 0 0 7812672 42240 123812 0 0 0 16 31174 222714 56 12 32 0 0
> 9 0 0 7812300 42240 123812 0 0 0 276 25089 196813 52 11 38 0 0
> 9 0 0 7812060 42240 123812 0 0 0 16 31877 228004 57 12 31 0 0
>
>
>
> Bisect located below patch,
> after reverted this patch the regression disappear.
>
> commit 354879bb977e06695993435745f06a0f6d39ce2b
> Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Date: Mon Aug 25 17:15:34 2008 +0200
>
> sched_clock: fix cpu_clock()
>
> This patch fixes 3 issues:
>
> a) it removes the dependency on jiffies, because jiffies are
> incremented
> by a single CPU, and the tick is not synchronized between CPUs.
> Therefore
> relying on it to calculate a window to clip whacky TSC values
> doesn't work
> as it can drift around.
>
> So instead use [GTOD, GTOD+TICK_NSEC) as the window.
>
> b) __update_sched_clock() did (roughly speaking):
>
> delta = sched_clock() - scd->tick_raw;
> clock += delta;
>
> Which gives exponential growth, instead of linear.
>
> c) allows the sched_clock_cpu() value to warp the u64 without
> breaking.
>
> the results are more reliable sched_clock() deltas:
Thats bizarre... that just indicates the better clock, which should give
better (read fairer) scheduling hurts your workload.
Is there anything I can run to see if we can fix the scheduler perhaps?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 9:03 ` Peter Zijlstra
@ 2008-09-04 10:52 ` Lin Ming
2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
2008-09-05 1:26 ` Lin Ming
1 sibling, 2 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-04 10:52 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo
On Thu, 2008-09-04 at 11:03 +0200, Peter Zijlstra wrote:
> On Thu, 2008-09-04 at 16:51 +0800, Lin Ming wrote:
> > Comparing with 2.6.27-rc4, oltp has ~10% regression with 2.6.27-rc5 on
> > 8-core stoakley machine.
> >
> > Run oltp with 8 threads 120 seconds, vmstat shows much more idle time, about ~30%
> >
> > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
> > r b swpd free buff cache si so bi bo in cs us sy id wa st
> > 10 0 0 7822824 42240 123740 0 0 312 47 442 1613 3 2 88 6 0
> > 9 0 0 7822312 42240 123764 0 0 0 16 26691 232566 56 14 30 0 0
> > 13 0 0 7821940 42240 123764 0 0 0 16 26661 228689 54 14 32 0 0
> > 8 0 0 7821320 42240 123764 0 0 0 16 31508 263765 61 17 23 0 0
> > 12 0 0 7820948 42240 123764 0 0 16 16 28666 242402 57 15 28 0 0
> > 9 0 0 7820584 42240 123780 0 0 0 16 27107 230804 56 14 30 0 0
> > 10 0 0 7819964 42240 123796 0 0 16 612 27599 244037 55 16 29 0 0
> > 11 0 0 7819356 42240 123796 0 0 0 64 23540 209713 51 13 36 0 0
> > 10 0 0 7819212 42240 123796 0 0 0 32 25674 224205 54 13 32 0 0
> > 10 0 0 7818716 42240 123796 0 0 0 20 30106 257161 59 16 25 0 0
> > 7 0 0 7818468 42240 123796 0 0 0 16 28356 241551 57 14 29 0 0
> > 10 0 0 7818096 42240 123796 0 0 0 16 39174 273656 64 16 20 0 0
> > 12 0 0 7817724 42240 123796 0 0 0 20 39688 276936 63 16 20 0 0
> > 11 0 0 7817352 42240 123796 0 0 0 16 42543 285192 66 16 18 0 0
> > 9 0 0 7817352 42240 123796 0 0 0 16 37083 259830 62 14 24 0 0
> > 8 0 0 7817104 42240 123796 0 0 0 16 37450 259160 61 15 23 0 0
> > 10 0 0 7816516 42240 123796 0 0 0 64 37425 261870 61 16 23 0 0
> > 11 0 0 7815896 42240 123812 0 0 16 16 41558 279320 66 16 18 0 0
> > 9 0 0 7815648 42240 123812 0 0 0 16 34017 235741 59 14 28 0 0
> > 10 0 0 7815152 42240 123812 0 0 0 16 35642 248888 60 14 26 0 0
> > 9 0 0 7814532 42240 123812 0 0 0 16 38517 263220 63 15 22 0 0
> > 9 0 0 7814160 42240 123812 0 0 0 20 35965 246487 61 14 25 0 0
> > 10 0 0 7814036 42240 123812 0 0 0 16 33852 236313 59 13 28 0 0
> > 11 0 0 7813664 42240 123812 0 0 0 16 34958 244819 59 14 27 0 0
> > 10 0 0 7813416 42240 123812 0 0 0 16 26106 202062 53 10 37 0 0
> > 10 0 0 7812672 42240 123812 0 0 0 16 31174 222714 56 12 32 0 0
> > 9 0 0 7812300 42240 123812 0 0 0 276 25089 196813 52 11 38 0 0
> > 9 0 0 7812060 42240 123812 0 0 0 16 31877 228004 57 12 31 0 0
> >
> >
> >
> > Bisect located below patch,
> > after reverted this patch the regression disappear.
> >
> > commit 354879bb977e06695993435745f06a0f6d39ce2b
> > Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > Date: Mon Aug 25 17:15:34 2008 +0200
> >
> > sched_clock: fix cpu_clock()
> >
> > This patch fixes 3 issues:
> >
> > a) it removes the dependency on jiffies, because jiffies are
> > incremented
> > by a single CPU, and the tick is not synchronized between CPUs.
> > Therefore
> > relying on it to calculate a window to clip whacky TSC values
> > doesn't work
> > as it can drift around.
> >
> > So instead use [GTOD, GTOD+TICK_NSEC) as the window.
> >
> > b) __update_sched_clock() did (roughly speaking):
> >
> > delta = sched_clock() - scd->tick_raw;
> > clock += delta;
> >
> > Which gives exponential growth, instead of linear.
> >
> > c) allows the sched_clock_cpu() value to warp the u64 without
> > breaking.
> >
> > the results are more reliable sched_clock() deltas:
>
> Thats bizarre... that just indicates the better clock, which should give
> better (read fairer) scheduling hurts your workload.
>
> Is there anything I can run to see if we can fix the scheduler perhaps?
I observed schedstats of sysbench, there's more
"nr_failed_migrations_hot"
2.6.27-rc4: se.nr_failed_migrations_hot 11
2.6.27-rc5: se.nr_failed_migrations_hot 95
task migration failed because of task_hot, the system is un-balanced?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 10:52 ` Lin Ming
@ 2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 12:12 ` Lin Ming
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
1 sibling, 2 replies; 19+ messages in thread
From: Peter Zijlstra @ 2008-09-04 11:06 UTC (permalink / raw)
To: Lin Ming; +Cc: linux-kernel, Zhang, Yanmin, mingo, Gregory Haskins
On Thu, 2008-09-04 at 18:52 +0800, Lin Ming wrote:
> On Thu, 2008-09-04 at 11:03 +0200, Peter Zijlstra wrote:
> > On Thu, 2008-09-04 at 16:51 +0800, Lin Ming wrote:
> > > Comparing with 2.6.27-rc4, oltp has ~10% regression with 2.6.27-rc5 on
> > > 8-core stoakley machine.
> > >
> > > Run oltp with 8 threads 120 seconds, vmstat shows much more idle time, about ~30%
> > >
> > > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
> > > r b swpd free buff cache si so bi bo in cs us sy id wa st
> > > 10 0 0 7822824 42240 123740 0 0 312 47 442 1613 3 2 88 6 0
> > > 9 0 0 7822312 42240 123764 0 0 0 16 26691 232566 56 14 30 0 0
> > > 13 0 0 7821940 42240 123764 0 0 0 16 26661 228689 54 14 32 0 0
> > > 8 0 0 7821320 42240 123764 0 0 0 16 31508 263765 61 17 23 0 0
> > > 12 0 0 7820948 42240 123764 0 0 16 16 28666 242402 57 15 28 0 0
> > > 9 0 0 7820584 42240 123780 0 0 0 16 27107 230804 56 14 30 0 0
> > > 10 0 0 7819964 42240 123796 0 0 16 612 27599 244037 55 16 29 0 0
> > > 11 0 0 7819356 42240 123796 0 0 0 64 23540 209713 51 13 36 0 0
> > > 10 0 0 7819212 42240 123796 0 0 0 32 25674 224205 54 13 32 0 0
> > > 10 0 0 7818716 42240 123796 0 0 0 20 30106 257161 59 16 25 0 0
> > > 7 0 0 7818468 42240 123796 0 0 0 16 28356 241551 57 14 29 0 0
> > > 10 0 0 7818096 42240 123796 0 0 0 16 39174 273656 64 16 20 0 0
> > > 12 0 0 7817724 42240 123796 0 0 0 20 39688 276936 63 16 20 0 0
> > > 11 0 0 7817352 42240 123796 0 0 0 16 42543 285192 66 16 18 0 0
> > > 9 0 0 7817352 42240 123796 0 0 0 16 37083 259830 62 14 24 0 0
> > > 8 0 0 7817104 42240 123796 0 0 0 16 37450 259160 61 15 23 0 0
> > > 10 0 0 7816516 42240 123796 0 0 0 64 37425 261870 61 16 23 0 0
> > > 11 0 0 7815896 42240 123812 0 0 16 16 41558 279320 66 16 18 0 0
> > > 9 0 0 7815648 42240 123812 0 0 0 16 34017 235741 59 14 28 0 0
> > > 10 0 0 7815152 42240 123812 0 0 0 16 35642 248888 60 14 26 0 0
> > > 9 0 0 7814532 42240 123812 0 0 0 16 38517 263220 63 15 22 0 0
> > > 9 0 0 7814160 42240 123812 0 0 0 20 35965 246487 61 14 25 0 0
> > > 10 0 0 7814036 42240 123812 0 0 0 16 33852 236313 59 13 28 0 0
> > > 11 0 0 7813664 42240 123812 0 0 0 16 34958 244819 59 14 27 0 0
> > > 10 0 0 7813416 42240 123812 0 0 0 16 26106 202062 53 10 37 0 0
> > > 10 0 0 7812672 42240 123812 0 0 0 16 31174 222714 56 12 32 0 0
> > > 9 0 0 7812300 42240 123812 0 0 0 276 25089 196813 52 11 38 0 0
> > > 9 0 0 7812060 42240 123812 0 0 0 16 31877 228004 57 12 31 0 0
> > >
> > >
> > >
> > > Bisect located below patch,
> > > after reverted this patch the regression disappear.
> > >
> > > commit 354879bb977e06695993435745f06a0f6d39ce2b
> > > Author: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > > Date: Mon Aug 25 17:15:34 2008 +0200
> > >
> > > sched_clock: fix cpu_clock()
> > >
> > > This patch fixes 3 issues:
> > >
> > > a) it removes the dependency on jiffies, because jiffies are
> > > incremented
> > > by a single CPU, and the tick is not synchronized between CPUs.
> > > Therefore
> > > relying on it to calculate a window to clip whacky TSC values
> > > doesn't work
> > > as it can drift around.
> > >
> > > So instead use [GTOD, GTOD+TICK_NSEC) as the window.
> > >
> > > b) __update_sched_clock() did (roughly speaking):
> > >
> > > delta = sched_clock() - scd->tick_raw;
> > > clock += delta;
> > >
> > > Which gives exponential growth, instead of linear.
> > >
> > > c) allows the sched_clock_cpu() value to warp the u64 without
> > > breaking.
> > >
> > > the results are more reliable sched_clock() deltas:
> >
> > Thats bizarre... that just indicates the better clock, which should give
> > better (read fairer) scheduling hurts your workload.
> >
> > Is there anything I can run to see if we can fix the scheduler perhaps?
>
> I observed schedstats of sysbench, there's more
> "nr_failed_migrations_hot"
>
> 2.6.27-rc4: se.nr_failed_migrations_hot 11
> 2.6.27-rc5: se.nr_failed_migrations_hot 95
>
> task migration failed because of task_hot, the system is un-balanced?
Ah, that makes sense, a more accurate clock could indeed make more tasks
hot.
Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
Also, we used to have some auto-tuning in there, which dissapeared some
time ago, gregory brought it back to live recently, perhaps he likes to
share? :-)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 10:52 ` Lin Ming
2008-09-04 11:06 ` Peter Zijlstra
@ 2008-09-04 11:09 ` Ingo Molnar
2008-09-04 11:30 ` Lin Ming
1 sibling, 1 reply; 19+ messages in thread
From: Ingo Molnar @ 2008-09-04 11:09 UTC (permalink / raw)
To: Lin Ming; +Cc: Peter Zijlstra, linux-kernel, Zhang, Yanmin
* Lin Ming <ming.m.lin@intel.com> wrote:
> I observed schedstats of sysbench, there's more
> "nr_failed_migrations_hot"
>
> 2.6.27-rc4: se.nr_failed_migrations_hot 11
> 2.6.27-rc5: se.nr_failed_migrations_hot 95
>
> task migration failed because of task_hot, the system is un-balanced?
would be nice to get a representative (==steady state) scheduler trace
from the critical portions of that workload. See:
http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt
Ingo
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
@ 2008-09-04 11:30 ` Lin Ming
2008-09-04 11:35 ` Ingo Molnar
0 siblings, 1 reply; 19+ messages in thread
From: Lin Ming @ 2008-09-04 11:30 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Peter Zijlstra, linux-kernel, Zhang, Yanmin
[-- Attachment #1: Type: text/plain, Size: 8280 bytes --]
On Thu, 2008-09-04 at 13:09 +0200, Ingo Molnar wrote:
> * Lin Ming <ming.m.lin@intel.com> wrote:
>
> > I observed schedstats of sysbench, there's more
> > "nr_failed_migrations_hot"
> >
> > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> >
> > task migration failed because of task_hot, the system is un-balanced?
>
> would be nice to get a representative (==steady state) scheduler trace
> from the critical portions of that workload. See:
>
> http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt
>
> Ingo
A huge trace result file, see the attachment for more.
# tracer: sched_switch
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
mysqld-3799 [07] 151.411009: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411015: 3796:120:R + 3804:120:S
mysqld-3796 [00] 151.411018: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411020: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411020: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411028: 3799:120:R + 3807:120:S
mysqld-3796 [00] 151.411029: 3796:120:R + 3804:120:S
mysqld-3799 [07] 151.411030: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411031: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411032: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411049: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411052: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411053: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411055: 3796:120:R + 3804:120:S
sysbench-3807 [07] 151.411063: 3807:120:S ==> 3799:120:R
mysqld-3796 [00] 151.411069: 3796:120:S ==> 3804:120:R
mysqld-3799 [07] 151.411070: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411071: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411073: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411078: 3804:120:S ==> 3798:120:R
mysqld-3798 [00] 151.411093: 3798:120:R + 3806:120:S
mysqld-3799 [07] 151.411093: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411095: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411104: 3807:120:S ==> 3799:120:R
mysqld-3799 [07] 151.411114: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411115: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411116: 3807:120:S ==> 3799:120:R
mysqld-3798 [00] 151.411131: 3798:120:S ==> 3806:120:R
mysqld-3799 [07] 151.411140: 3799:120:R + 3807:120:S
sysbench-3806 [00] 151.411149: 3806:120:S ==> 3801:120:R
mysqld-3799 [07] 151.411149: 3799:120:S ==> 3807:120:R
mysqld-3801 [00] 151.411153: 3801:120:R + 3787:120:S
mysqld-3801 [00] 151.411154: 3801:120:R ==> 3787:120:R
tracer-3811 [02] 151.411195: 3811:120:R + 3812:120:R
tracer-3811 [02] 151.411198: 3811:120:R ==> 3812:120:R
sysbench-3807 [07] 151.411201: 3807:120:S ==> 0:140:R
mysqld-3787 [00] 151.411204: 0:140:R + 3797:120:S
mysqld-3787 [00] 151.411208: 3787:120:S ==> 3801:120:R
<idle>-0 [01] 151.411210: 0:140:R ==> 3797:120:R
mysqld-3801 [00] 151.411217: 3801:120:R + 3809:120:S
mysqld-3797 [01] 151.411225: 0:140:R + 3800:120:S
<idle>-0 [03] 151.411228: 0:140:R ==> 3800:120:R
mysqld-3797 [01] 151.411231: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.411257: 0:140:R + 3802:120:S
<idle>-0 [05] 151.411259: 0:140:R ==> 3802:120:R
mysqld-3800 [03] 151.411266: 3800:120:R + 3808:120:S
mysqld-3801 [00] 151.411267: 3801:120:S ==> 3809:120:R
mysqld-3802 [05] 151.411268: 0:140:R + 3791:120:S
<idle>-0 [07] 151.411270: 0:140:R ==> 3791:120:R
mysqld-3797 [01] 151.411271: 3797:120:S ==> 3805:120:R
sysbench-3809 [00] 151.411271: 3809:120:R + 3801:120:S
sysbench-3805 [01] 151.411274: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411275: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411277: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411277: 3809:120:R + 3796:120:S
sysbench-3810 [05] 151.411279: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.411282: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411283: 3791:120:R ==> 3803:120:R
sysbench-3803 [07] 151.411285: 3803:120:S ==> 3791:120:R
sysbench-3805 [01] 151.411289: 3805:120:S ==> 3797:120:R
sysbench-3809 [00] 151.411292: 3809:120:S ==> 3796:120:R
mysqld-3797 [01] 151.411292: 3797:120:R + 3805:120:S
mysqld-3796 [00] 151.411297: 3796:120:R + 3798:120:S
mysqld-3796 [00] 151.411300: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.411308: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411309: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411312: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411313: 3791:120:R ==> 3803:120:R
sysbench-3810 [05] 151.411323: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411326: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411332: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411333: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411333: 3791:120:R + 3803:120:S
sysbench-3810 [05] 151.411334: 3810:120:S ==> 3802:120:R
mysqld-3796 [00] 151.411341: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411344: 3804:120:R + 3796:120:S
sysbench-3804 [00] 151.411356: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.411357: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411358: 3802:120:R ==> 3810:120:R
mysqld-3796 [00] 151.411359: 3796:120:R + 3804:120:S
mysqld-3791 [07] 151.411363: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.411366: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411370: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411374: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411377: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.411378: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411394: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411396: 3804:120:R + 3796:120:S
mysqld-3797 [01] 151.411402: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.411405: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411406: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411407: 3804:120:S ==> 3796:120:R
sysbench-3810 [05] 151.411409: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.411410: 3791:120:S ==> 3803:120:R
mysqld-3796 [00] 151.411410: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411412: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411420: 3810:120:S ==> 3802:120:R
sysbench-3805 [01] 151.411422: 3805:120:S ==> 3797:120:R
mysqld-3802 [05] 151.411423: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.411424: 3803:120:S ==> 3791:120:R
mysqld-3797 [01] 151.411426: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.411427: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411443: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411445: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.411454: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411456: 3804:120:S ==> 3796:120:R
[-- Attachment #2: trace.txt --]
[-- Type: text/plain, Size: 121036 bytes --]
# tracer: sched_switch
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
mysqld-3799 [07] 151.411009: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411015: 3796:120:R + 3804:120:S
mysqld-3796 [00] 151.411018: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411020: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411020: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411028: 3799:120:R + 3807:120:S
mysqld-3796 [00] 151.411029: 3796:120:R + 3804:120:S
mysqld-3799 [07] 151.411030: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411031: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411032: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411049: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411052: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411053: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411055: 3796:120:R + 3804:120:S
sysbench-3807 [07] 151.411063: 3807:120:S ==> 3799:120:R
mysqld-3796 [00] 151.411069: 3796:120:S ==> 3804:120:R
mysqld-3799 [07] 151.411070: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411071: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411073: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411078: 3804:120:S ==> 3798:120:R
mysqld-3798 [00] 151.411093: 3798:120:R + 3806:120:S
mysqld-3799 [07] 151.411093: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411095: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411104: 3807:120:S ==> 3799:120:R
mysqld-3799 [07] 151.411114: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411115: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411116: 3807:120:S ==> 3799:120:R
mysqld-3798 [00] 151.411131: 3798:120:S ==> 3806:120:R
mysqld-3799 [07] 151.411140: 3799:120:R + 3807:120:S
sysbench-3806 [00] 151.411149: 3806:120:S ==> 3801:120:R
mysqld-3799 [07] 151.411149: 3799:120:S ==> 3807:120:R
mysqld-3801 [00] 151.411153: 3801:120:R + 3787:120:S
mysqld-3801 [00] 151.411154: 3801:120:R ==> 3787:120:R
tracer-3811 [02] 151.411195: 3811:120:R + 3812:120:R
tracer-3811 [02] 151.411198: 3811:120:R ==> 3812:120:R
sysbench-3807 [07] 151.411201: 3807:120:S ==> 0:140:R
mysqld-3787 [00] 151.411204: 0:140:R + 3797:120:S
mysqld-3787 [00] 151.411208: 3787:120:S ==> 3801:120:R
<idle>-0 [01] 151.411210: 0:140:R ==> 3797:120:R
mysqld-3801 [00] 151.411217: 3801:120:R + 3809:120:S
mysqld-3797 [01] 151.411225: 0:140:R + 3800:120:S
<idle>-0 [03] 151.411228: 0:140:R ==> 3800:120:R
mysqld-3797 [01] 151.411231: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.411257: 0:140:R + 3802:120:S
<idle>-0 [05] 151.411259: 0:140:R ==> 3802:120:R
mysqld-3800 [03] 151.411266: 3800:120:R + 3808:120:S
mysqld-3801 [00] 151.411267: 3801:120:S ==> 3809:120:R
mysqld-3802 [05] 151.411268: 0:140:R + 3791:120:S
<idle>-0 [07] 151.411270: 0:140:R ==> 3791:120:R
mysqld-3797 [01] 151.411271: 3797:120:S ==> 3805:120:R
sysbench-3809 [00] 151.411271: 3809:120:R + 3801:120:S
sysbench-3805 [01] 151.411274: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411275: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411277: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411277: 3809:120:R + 3796:120:S
sysbench-3810 [05] 151.411279: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.411282: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411283: 3791:120:R ==> 3803:120:R
sysbench-3803 [07] 151.411285: 3803:120:S ==> 3791:120:R
sysbench-3805 [01] 151.411289: 3805:120:S ==> 3797:120:R
sysbench-3809 [00] 151.411292: 3809:120:S ==> 3796:120:R
mysqld-3797 [01] 151.411292: 3797:120:R + 3805:120:S
mysqld-3796 [00] 151.411297: 3796:120:R + 3798:120:S
mysqld-3796 [00] 151.411300: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.411308: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411309: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411312: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411313: 3791:120:R ==> 3803:120:R
sysbench-3810 [05] 151.411323: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411326: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411332: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411333: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411333: 3791:120:R + 3803:120:S
sysbench-3810 [05] 151.411334: 3810:120:S ==> 3802:120:R
mysqld-3796 [00] 151.411341: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411344: 3804:120:R + 3796:120:S
sysbench-3804 [00] 151.411356: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.411357: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411358: 3802:120:R ==> 3810:120:R
mysqld-3796 [00] 151.411359: 3796:120:R + 3804:120:S
mysqld-3791 [07] 151.411363: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.411366: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411370: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411374: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411377: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.411378: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411394: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411396: 3804:120:R + 3796:120:S
mysqld-3797 [01] 151.411402: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.411405: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411406: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411407: 3804:120:S ==> 3796:120:R
sysbench-3810 [05] 151.411409: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.411410: 3791:120:S ==> 3803:120:R
mysqld-3796 [00] 151.411410: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411412: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411420: 3810:120:S ==> 3802:120:R
sysbench-3805 [01] 151.411422: 3805:120:S ==> 3797:120:R
mysqld-3802 [05] 151.411423: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.411424: 3803:120:S ==> 3791:120:R
mysqld-3797 [01] 151.411426: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.411427: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411443: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411445: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.411454: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411456: 3804:120:S ==> 3796:120:R
sysbench-3810 [05] 151.411456: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.411457: 3791:120:S ==> 3803:120:R
mysqld-3796 [00] 151.411460: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411460: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411467: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411469: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411470: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.411472: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411494: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411497: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.411500: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411504: 3808:120:R + 3800:120:S
mysqld-3802 [05] 151.411506: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.411506: 3791:120:S ==> 3803:120:R
sysbench-3804 [00] 151.411507: 3804:120:S ==> 3796:120:R
sysbench-3803 [07] 151.411509: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411509: 3810:120:R + 3802:120:S
mysqld-3796 [00] 151.411511: 3796:120:R + 3804:120:S
sysbench-3808 [03] 151.411516: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.411517: 3803:120:S ==> 3791:120:R
mysqld-3800 [03] 151.411519: 3800:120:R + 3808:120:S
mysqld-3797 [01] 151.411519: 3797:120:S ==> 3805:120:R
sysbench-3810 [05] 151.411520: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.411521: 3791:120:R + 3803:120:S
sysbench-3805 [01] 151.411522: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411523: 3802:120:R + 3810:120:S
sysbench-3805 [01] 151.411532: 3805:120:S ==> 3797:120:R
mysqld-3800 [03] 151.411533: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411535: 3808:120:R + 3800:120:S
mysqld-3797 [01] 151.411536: 3797:120:R + 3805:120:S
mysqld-3796 [00] 151.411546: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411548: 3804:120:R + 3796:120:S
sysbench-3808 [03] 151.411553: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.411553: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.411554: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.411556: 3800:120:R + 3808:120:S
sysbench-3810 [05] 151.411556: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.411556: 3803:120:R + 3791:120:S
sysbench-3804 [00] 151.411559: 3804:120:S ==> 3796:120:R
mysqld-3796 [00] 151.411562: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411566: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.411567: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.411569: 3800:120:S ==> 3808:120:R
mysqld-3791 [07] 151.411569: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.411570: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.411571: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.411578: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411580: 3800:120:R + 3808:120:S
mysqld-3796 [00] 151.411597: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411599: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.411603: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.411603: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.411606: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.411606: 3803:120:R + 3791:120:S
sysbench-3804 [00] 151.411611: 3804:120:S ==> 3796:120:R
mysqld-3796 [00] 151.411615: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411615: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.411618: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.411618: 3800:120:S ==> 3808:120:R
mysqld-3791 [07] 151.411618: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.411620: 3808:120:R + 3800:120:S
mysqld-3802 [05] 151.411621: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.411630: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411632: 3800:120:R + 3808:120:S
mysqld-3796 [00] 151.411647: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411649: 3804:120:R + 3796:120:S
mysqld-3791 [07] 151.411650: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.411651: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.411653: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411653: 3810:120:R + 3802:120:S
sysbench-3804 [00] 151.411661: 3804:120:S ==> 3796:120:R
sysbench-3803 [07] 151.411662: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.411663: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.411663: 3800:120:S ==> 3808:120:R
mysqld-3796 [00] 151.411665: 3796:120:R + 3804:120:S
mysqld-3791 [07] 151.411665: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.411666: 3808:120:R + 3800:120:S
mysqld-3802 [05] 151.411666: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.411674: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411677: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.411699: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.411701: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.411708: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411711: 3808:120:R + 3800:120:S
sysbench-3810 [05] 151.411711: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.411715: 3802:120:R + 3810:120:S
mysqld-3796 [00] 151.411719: 3796:120:S ==> 3804:120:R
sysbench-3808 [03] 151.411720: 3808:120:S ==> 3800:120:R
sysbench-3804 [00] 151.411721: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.411723: 3800:120:R + 3808:120:S
sysbench-3804 [00] 151.411731: 3804:120:S ==> 3796:120:R
mysqld-3796 [00] 151.411734: 3796:120:R + 3804:120:S
sleep-3812 [02] 151.411736: 3812:120:S ==> 3585:120:R
bash-3585 [02] 151.411751: 3585:120:S ==> 3811:120:R
mysqld-3797 [01] 151.411756: 3797:120:S ==> 3805:120:R
mysqld-3800 [03] 151.411757: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411759: 3808:120:R + 3800:120:S
sysbench-3805 [01] 151.411760: 3805:120:R + 3797:120:S
sysbench-3808 [03] 151.411768: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411771: 3800:120:R + 3808:120:S
mysqld-3796 [00] 151.411772: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411775: 3804:120:R + 3796:120:S
sysbench-3805 [01] 151.411777: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.411780: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.411781: 3791:120:S ==> 3803:120:R
sysbench-3804 [00] 151.411783: 3804:120:S ==> 3796:120:R
sysbench-3803 [07] 151.411784: 3803:120:R + 3791:120:S
mysqld-3796 [00] 151.411787: 3796:120:R + 3804:120:S
tracer-3811 [02] 151.411791: 3811:120:S ==> 3798:120:R
mysqld-3798 [02] 151.411796: 3798:120:R + 3799:120:S
sysbench-3803 [07] 151.411801: 3803:120:S ==> 3791:120:R
mysqld-3798 [02] 151.411802: 3798:120:R + 3806:120:S
mysqld-3800 [03] 151.411803: 3800:120:S ==> 3808:120:R
mysqld-3798 [02] 151.411804: 3798:120:R ==> 3806:120:R
mysqld-3791 [07] 151.411805: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.411805: 3808:120:R + 3800:120:S
sysbench-3806 [02] 151.411806: 3806:120:S ==> 3799:120:R
sysbench-3808 [03] 151.411814: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411818: 3800:120:R + 3808:120:S
mysqld-3799 [02] 151.411818: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.411821: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411822: 3796:120:S ==> 3804:120:R
sysbench-3807 [02] 151.411824: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411824: 3804:120:R + 3796:120:S
sysbench-3804 [00] 151.411834: 3804:120:S ==> 3796:120:R
mysqld-3796 [00] 151.411836: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.411849: 3802:120:S ==> 3810:120:R
mysqld-3800 [03] 151.411852: 3800:120:S ==> 3808:120:R
sysbench-3810 [05] 151.411852: 3810:120:R + 3802:120:S
sysbench-3808 [03] 151.411854: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.411862: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411865: 3800:120:R + 3808:120:S
sysbench-3810 [05] 151.411868: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.411871: 3802:120:R + 3810:120:S
mysqld-3799 [02] 151.411875: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.411876: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.411895: 3800:120:S ==> 3808:120:R
sysbench-3807 [02] 151.411896: 3807:120:S ==> 3798:120:R
sysbench-3808 [03] 151.411897: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.411906: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.411909: 3800:120:R + 3808:120:S
mysqld-3791 [07] 151.411915: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.411918: 3803:120:R + 3791:120:S
sysbench-3803 [07] 151.411928: 3803:120:S ==> 3791:120:R
mysqld-3798 [02] 151.411929: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.411931: 3798:120:R ==> 3806:120:R
mysqld-3791 [07] 151.411931: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.411940: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411943: 3808:120:R + 3800:120:S
sysbench-3806 [02] 151.411944: 3806:120:S ==> 3799:120:R
sysbench-3808 [03] 151.411954: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.411955: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.411956: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.411957: 3800:120:R + 3808:120:S
sysbench-3807 [02] 151.411958: 3807:120:S ==> 3799:120:R
mysqld-3802 [05] 151.411986: 3802:120:S ==> 3810:120:R
mysqld-3799 [02] 151.411989: 3799:120:R + 3807:120:S
sysbench-3810 [05] 151.411989: 3810:120:R + 3802:120:S
mysqld-3799 [02] 151.411991: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.411993: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.411995: 3808:120:R + 3800:120:S
mysqld-3796 [00] 151.411997: 3796:120:S ==> 3804:120:R
sysbench-3810 [05] 151.411999: 3810:120:S ==> 3802:120:R
sysbench-3807 [02] 151.412001: 3807:120:S ==> 3798:120:R
mysqld-3802 [05] 151.412002: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.412004: 3808:120:S ==> 3800:120:R
sysbench-3804 [00] 151.412004: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.412007: 3800:120:R + 3808:120:S
mysqld-3798 [02] 151.412010: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412011: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412013: 3806:120:S ==> 3798:120:R
sysbench-3804 [00] 151.412021: 3804:120:S ==> 3796:120:R
mysqld-3797 [01] 151.412023: 3797:120:S ==> 3805:120:R
mysqld-3796 [00] 151.412025: 3796:120:R + 3804:120:S
sysbench-3805 [01] 151.412026: 3805:120:R + 3797:120:S
sysbench-3805 [01] 151.412036: 3805:120:S ==> 3797:120:R
mysqld-3800 [03] 151.412039: 3800:120:S ==> 3808:120:R
mysqld-3797 [01] 151.412039: 3797:120:R + 3805:120:S
sysbench-3808 [03] 151.412041: 3808:120:R + 3800:120:S
mysqld-3798 [02] 151.412041: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412043: 3798:120:R ==> 3806:120:R
sysbench-3808 [03] 151.412050: 3808:120:S ==> 3800:120:R
mysqld-3797 [01] 151.412051: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412053: 3805:120:R + 3797:120:S
mysqld-3800 [03] 151.412054: 3800:120:R + 3808:120:S
sysbench-3806 [02] 151.412055: 3806:120:S ==> 3798:120:R
mysqld-3798 [02] 151.412063: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412064: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412066: 3806:120:S ==> 3799:120:R
sysbench-3805 [01] 151.412071: 3805:120:S ==> 3797:120:R
mysqld-3799 [02] 151.412072: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.412073: 3799:120:R ==> 3807:120:R
mysqld-3797 [01] 151.412074: 3797:120:R + 3805:120:S
sysbench-3807 [02] 151.412075: 3807:120:S ==> 3798:120:R
mysqld-3797 [01] 151.412085: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412087: 3805:120:R + 3797:120:S
sysbench-3805 [01] 151.412094: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412096: 3797:120:R + 3805:120:S
mysqld-3798 [02] 151.412099: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412101: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412111: 3806:120:S ==> 3799:120:R
mysqld-3796 [00] 151.412132: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.412135: 3804:120:R + 3796:120:S
mysqld-3799 [02] 151.412136: 3799:120:R + 3807:120:S
mysqld-3797 [01] 151.412137: 3797:120:S ==> 3805:120:R
mysqld-3799 [02] 151.412139: 3799:120:R ==> 3807:120:R
sysbench-3805 [01] 151.412140: 3805:120:R + 3797:120:S
sysbench-3804 [00] 151.412144: 3804:120:S ==> 3796:120:R
mysqld-3796 [00] 151.412147: 3796:120:R + 3804:120:S
sysbench-3805 [01] 151.412149: 3805:120:S ==> 3797:120:R
sysbench-3807 [02] 151.412150: 3807:120:S ==> 3798:120:R
mysqld-3797 [01] 151.412152: 3797:120:R + 3805:120:S
mysqld-3798 [02] 151.412157: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412159: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412160: 3806:120:S ==> 3799:120:R
mysqld-3799 [02] 151.412167: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.412168: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.412169: 3807:120:S ==> 3798:120:R
mysqld-3800 [03] 151.412171: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.412174: 3808:120:R + 3800:120:S
mysqld-3797 [01] 151.412186: 3797:120:S ==> 3805:120:R
sysbench-3808 [03] 151.412192: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.412196: 3800:120:R + 3808:120:S
mysqld-3798 [02] 151.412201: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412203: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412217: 3806:120:S ==> 3799:120:R
mysqld-3791 [07] 151.412226: 3805:120:S + 3797:120:S
sysbench-3805 [01] 151.412234: 3805:120:S ==> 3797:120:R
mysqld-3791 [07] 151.412252: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.412259: 3802:120:S ==> 3810:120:R
mysqld-3797 [01] 151.412261: 3797:120:R + 3805:120:S
sysbench-3810 [05] 151.412262: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412264: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.412279: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.412281: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.412285: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.412287: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.412292: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412295: 3805:120:R + 3797:120:S
mysqld-3800 [03] 151.412301: 3800:120:S ==> 3808:120:R
sysbench-3805 [01] 151.412304: 3805:120:S ==> 3797:120:R
sysbench-3808 [03] 151.412307: 3808:120:R + 3800:120:S
mysqld-3797 [01] 151.412307: 3797:120:R + 3805:120:S
sysbench-3808 [03] 151.412318: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.412321: 3800:120:R + 3808:120:S
mysqld-3797 [01] 151.412338: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412340: 3805:120:R + 3797:120:S
sysbench-3805 [01] 151.412349: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412352: 3797:120:R + 3805:120:S
mysqld-3799 [02] 151.412356: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.412358: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.412368: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.412373: 3804:120:S ==> 3801:120:R
mysqld-3797 [01] 151.412374: 3797:120:R + 3796:120:S
mysqld-3801 [00] 151.412378: 3801:120:R + 3809:120:S
sysbench-3807 [02] 151.412381: 3807:120:S ==> 3798:120:R
mysqld-3797 [01] 151.412382: 3797:120:R ==> 3805:120:R
sysbench-3805 [01] 151.412384: 3805:120:S ==> 3796:120:R
mysqld-3798 [02] 151.412388: 3798:120:S ==> 3799:120:R
mysqld-3796 [01] 151.412392: 3799:120:R + 3798:120:S
mysqld-3799 [02] 151.412395: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.412398: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.412400: 3807:120:S ==> 3798:120:R
mysqld-3796 [01] 151.412405: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.412405: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412407: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412408: 3806:120:S ==> 3798:120:R
mysqld-3801 [00] 151.412421: 3801:120:S ==> 3809:120:R
mysqld-3796 [01] 151.412427: 3809:120:R + 3787:120:S
mysqld-3796 [01] 151.412430: 3796:120:R + 3801:120:S
mysqld-3796 [01] 151.412433: 3796:120:S ==> 3804:120:R
mysqld-3798 [02] 151.412434: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412436: 3798:120:R ==> 3806:120:R
sysbench-3804 [01] 151.412438: 3804:120:R + 3796:120:S
sysbench-3809 [00] 151.412439: 3809:120:S ==> 3787:120:R
sysbench-3806 [02] 151.412451: 3806:120:S ==> 3798:120:R
mysqld-3798 [02] 151.412458: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412460: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412461: 3806:120:S ==> 3798:120:R
sysbench-3804 [01] 151.412469: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.412473: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.412483: 3798:120:R + 3806:120:S
mysqld-3787 [00] 151.412485: 3787:120:S ==> 0:140:R
mysqld-3798 [02] 151.412485: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412495: 3806:120:S ==> 3798:120:R
mysqld-3798 [02] 151.412503: 3798:120:R + 3806:120:S
mysqld-3791 [07] 151.412504: 3791:120:S ==> 3803:120:R
mysqld-3798 [02] 151.412505: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412506: 3806:120:S ==> 3798:120:R
mysqld-3802 [05] 151.412511: 3803:120:R + 3791:120:S
sysbench-3803 [07] 151.412518: 3803:120:S ==> 3791:120:R
mysqld-3798 [02] 151.412535: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412537: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412547: 3806:120:S ==> 3799:120:R
mysqld-3796 [01] 151.412550: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.412553: 3804:120:S ==> 3797:120:R
mysqld-3802 [05] 151.412556: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412559: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.412561: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.412562: 3797:120:R + 3796:120:S
sysbench-3810 [05] 151.412564: 3810:120:R + 3802:120:S
mysqld-3797 [01] 151.412568: 3797:120:R + 3805:120:S
mysqld-3787 [00] 151.412569: 3787:120:S ==> 0:140:R
sysbench-3810 [05] 151.412574: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.412576: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.412578: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.412579: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412581: 3805:120:R + 3797:120:S
mysqld-3791 [07] 151.412590: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.412592: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.412593: 3803:120:R + 3791:120:S
sysbench-3805 [01] 151.412593: 3805:120:S ==> 3797:120:R
sysbench-3810 [05] 151.412594: 3810:120:R + 3802:120:S
mysqld-3797 [01] 151.412596: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.412602: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.412605: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.412623: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.412626: 3800:120:R + 3808:120:S
mysqld-3797 [01] 151.412630: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412632: 3805:120:R + 3797:120:S
sysbench-3803 [07] 151.412632: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.412634: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.412635: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.412637: 3802:120:R + 3810:120:S
mysqld-3799 [02] 151.412640: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.412642: 3799:120:R ==> 3807:120:R
sysbench-3805 [01] 151.412642: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412646: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.412647: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.412648: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.412649: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.412650: 3810:120:R + 3802:120:S
sysbench-3807 [02] 151.412656: 3807:120:S ==> 3798:120:R
sysbench-3810 [05] 151.412657: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.412657: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.412660: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.412660: 3791:120:R + 3803:120:S
mysqld-3798 [02] 151.412665: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412666: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412668: 3806:120:S ==> 3798:120:R
mysqld-3797 [01] 151.412680: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412682: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.412683: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412686: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.412691: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.412697: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412700: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.412704: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.412705: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.412707: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.412707: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412716: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.412717: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.412720: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.412720: 3791:120:R + 3803:120:S
mysqld-3797 [01] 151.412737: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412739: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.412740: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412743: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.412746: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.412753: 3805:120:S ==> 3797:120:R
mysqld-3791 [07] 151.412754: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.412756: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.412756: 3803:120:R + 3791:120:S
mysqld-3797 [01] 151.412757: 3797:120:R + 3805:120:S
sysbench-3810 [05] 151.412758: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412766: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.412767: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.412769: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.412770: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.412790: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412792: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.412793: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412795: 3805:120:R + 3797:120:S
mysqld-3798 [02] 151.412799: 3798:120:R + 3806:120:S
mysqld-3787 [00] 151.412802: 3787:120:S ==> 0:140:R
mysqld-3798 [02] 151.412802: 3798:120:R ==> 3806:120:R
mysqld-3802 [05] 151.412802: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.412802: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.412806: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412806: 3803:120:R + 3791:120:S
sysbench-3805 [01] 151.412806: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412809: 3797:120:R + 3805:120:S
sysbench-3810 [05] 151.412815: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.412815: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.412818: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.412818: 3791:120:R + 3803:120:S
sysbench-3806 [02] 151.412822: 3806:120:S ==> 3798:120:R
mysqld-3798 [02] 151.412830: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412832: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.412833: 3806:120:S ==> 3798:120:R
mysqld-3797 [01] 151.412844: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412847: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.412847: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412850: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.412851: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.412853: 3810:120:R + 3802:120:S
mysqld-3787 [00] 151.412855: 3787:120:S ==> 0:140:R
mysqld-3791 [07] 151.412858: 3791:120:S ==> 3803:120:R
sysbench-3805 [01] 151.412860: 3805:120:S ==> 3797:120:R
sysbench-3803 [07] 151.412860: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.412863: 3810:120:S ==> 3802:120:R
mysqld-3797 [01] 151.412864: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.412865: 3800:120:S ==> 3808:120:R
mysqld-3802 [05] 151.412866: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.412868: 3808:120:R + 3800:120:S
sysbench-3803 [07] 151.412870: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.412873: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.412879: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.412882: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.412915: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412916: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.412918: 3800:120:S ==> 3808:120:R
mysqld-3802 [05] 151.412919: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.412920: 3791:120:S ==> 3803:120:R
sysbench-3808 [03] 151.412920: 3808:120:R + 3800:120:S
sysbench-3810 [05] 151.412922: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412922: 3803:120:R + 3791:120:S
mysqld-3787 [00] 151.412926: 3787:120:S ==> 0:140:R
mysqld-3798 [02] 151.412929: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.412931: 3798:120:R ==> 3806:120:R
sysbench-3810 [05] 151.412931: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.412932: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.412935: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.412935: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.412935: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.412939: 3800:120:R + 3808:120:S
sysbench-3806 [02] 151.412945: 3806:120:S ==> 3799:120:R
mysqld-3800 [03] 151.412952: 0:140:R + 3787:120:S
<idle>-0 [00] 151.412953: 0:140:R ==> 3787:120:R
mysqld-3799 [02] 151.412954: 3799:120:R + 3807:120:S
mysqld-3800 [03] 151.412954: 3800:120:S ==> 3808:120:R
mysqld-3799 [02] 151.412955: 3799:120:R ==> 3807:120:R
sysbench-3808 [03] 151.412956: 3808:120:R + 3800:120:S
sysbench-3807 [02] 151.412956: 3807:120:S ==> 3799:120:R
mysqld-3787 [00] 151.412959: 3787:120:S ==> 0:140:R
sysbench-3808 [03] 151.412966: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.412967: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.412967: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.412969: 3800:120:R + 3808:120:S
sysbench-3810 [05] 151.412969: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.412969: 3803:120:R + 3791:120:S
mysqld-3797 [01] 151.412973: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.412976: 3805:120:R + 3797:120:S
sysbench-3803 [07] 151.412978: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.412978: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.412981: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.412981: 3791:120:R + 3803:120:S
sysbench-3805 [01] 151.412993: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.412996: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.413010: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413011: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.413012: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.413013: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.413013: 3800:120:S ==> 3808:120:R
sysbench-3810 [05] 151.413015: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.413016: 3803:120:R + 3791:120:S
sysbench-3808 [03] 151.413017: 3808:120:R + 3800:120:S
mysqld-3787 [00] 151.413023: 3787:120:S ==> 0:140:R
sysbench-3810 [05] 151.413025: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.413027: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.413028: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.413029: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.413030: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.413031: 3800:120:R + 3808:120:S
mysqld-3791 [07] 151.413039: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.413042: 3803:120:S + 3791:120:S
sysbench-3803 [07] 151.413044: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.413063: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413065: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.413065: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.413067: 3791:120:R + 3803:120:S
sysbench-3810 [05] 151.413067: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.413071: 3800:120:S ==> 3808:120:R
mysqld-3787 [00] 151.413072: 3787:120:S ==> 0:140:R
sysbench-3808 [03] 151.413074: 3808:120:R + 3800:120:S
mysqld-3791 [07] 151.413076: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.413077: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.413078: 3803:120:R + 3791:120:S
mysqld-3802 [05] 151.413080: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.413083: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413086: 3800:120:R + 3808:120:S
sysbench-3803 [07] 151.413087: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.413090: 3791:120:R + 3803:120:S
mysqld-3797 [01] 151.413106: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413108: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413110: 3797:120:S ==> 3805:120:R
mysqld-3802 [05] 151.413111: 3802:120:S ==> 3810:120:R
sysbench-3805 [01] 151.413112: 3805:120:R + 3797:120:S
sysbench-3810 [05] 151.413114: 3810:120:R + 3802:120:S
mysqld-3787 [00] 151.413115: 3787:120:S ==> 0:140:R
mysqld-3791 [07] 151.413121: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.413122: 3800:120:S ==> 3808:120:R
sysbench-3810 [05] 151.413123: 3810:120:S ==> 3802:120:R
sysbench-3805 [01] 151.413123: 3805:120:S ==> 3797:120:R
sysbench-3803 [07] 151.413124: 3803:120:R + 3791:120:S
sysbench-3808 [03] 151.413124: 3808:120:R + 3800:120:S
mysqld-3802 [05] 151.413126: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.413127: 3797:120:R + 3805:120:S
sysbench-3808 [03] 151.413133: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.413133: 3803:120:S ==> 3791:120:R
mysqld-3800 [03] 151.413136: 3800:120:R + 3808:120:S
mysqld-3791 [07] 151.413136: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.413158: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413160: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.413160: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.413163: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.413167: 3800:120:S ==> 3808:120:R
mysqld-3787 [00] 151.413168: 3787:120:S ==> 0:140:R
mysqld-3791 [07] 151.413169: 3791:120:S ==> 3803:120:R
sysbench-3808 [03] 151.413169: 3808:120:R + 3800:120:S
sysbench-3803 [07] 151.413172: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.413172: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.413175: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.413178: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.413181: 3803:120:S ==> 3791:120:R
mysqld-3800 [03] 151.413182: 3800:120:R + 3808:120:S
mysqld-3791 [07] 151.413184: 3791:120:R + 3803:120:S
mysqld-3799 [02] 151.413213: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.413215: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.413219: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413220: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413222: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413224: 3808:120:R + 3800:120:S
mysqld-3787 [00] 151.413228: 3787:120:S ==> 0:140:R
sysbench-3808 [03] 151.413234: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413237: 3800:120:R + 3808:120:S
sysbench-3807 [02] 151.413237: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.413246: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.413247: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.413248: 3806:120:S ==> 3798:120:R
mysqld-3800 [03] 151.413274: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413275: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413277: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413279: 3808:120:R + 3800:120:S
mysqld-3787 [00] 151.413282: 3787:120:S ==> 0:140:R
sysbench-3808 [03] 151.413289: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413292: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.413300: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413301: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.413303: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.413304: 3791:120:S ==> 3803:120:R
mysqld-3787 [00] 151.413309: 3787:120:S ==> 0:140:R
sysbench-3810 [05] 151.413313: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.413315: 3803:120:R + 3791:120:S
mysqld-3800 [03] 151.413325: 3800:120:S ==> 3808:120:R
sysbench-3810 [05] 151.413329: 3810:120:S ==> 3802:120:R
sysbench-3808 [03] 151.413331: 3808:120:R + 3800:120:S
sysbench-3803 [07] 151.413331: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.413333: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.413334: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.413339: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413342: 3800:120:R + 3808:120:S
mysqld-3797 [01] 151.413380: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413381: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413383: 3797:120:S ==> 3805:120:R
mysqld-3800 [03] 151.413384: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413387: 3808:120:R + 3800:120:S
mysqld-3787 [00] 151.413390: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413390: 3805:120:R + 3797:120:S
sysbench-3808 [03] 151.413396: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413399: 3800:120:R + 3808:120:S
sysbench-3805 [01] 151.413408: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.413411: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.413431: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413432: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413434: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413436: 3808:120:R + 3800:120:S
mysqld-3791 [07] 151.413438: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.413439: 3802:120:S ==> 3810:120:R
mysqld-3787 [00] 151.413439: 3787:120:S ==> 0:140:R
sysbench-3803 [07] 151.413441: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.413442: 3810:120:R + 3802:120:S
sysbench-3808 [03] 151.413446: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.413449: 3800:120:R + 3808:120:S
sysbench-3810 [05] 151.413452: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.413452: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.413455: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.413455: 3802:120:R + 3810:120:S
mysqld-3798 [02] 151.413468: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.413470: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.413482: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413483: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413484: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413486: 3808:120:R + 3800:120:S
sysbench-3806 [02] 151.413491: 3806:120:S ==> 3799:120:R
sysbench-3808 [03] 151.413499: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.413500: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.413502: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.413503: 3800:120:R + 3808:120:S
sysbench-3807 [02] 151.413504: 3807:120:S ==> 3799:120:R
mysqld-3787 [00] 151.413520: 3787:120:S ==> 0:140:R
mysqld-3800 [03] 151.413632: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413633: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413636: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.413639: 3808:120:R + 3800:120:S
mysqld-3797 [01] 151.413640: 3797:120:S ==> 3805:120:R
mysqld-3787 [00] 151.413641: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413643: 3805:120:R + 3797:120:S
sysbench-3805 [01] 151.413654: 3805:120:S ==> 3797:120:R
sysbench-3808 [03] 151.413656: 3808:120:S ==> 3800:120:R
mysqld-3797 [01] 151.413657: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.413659: 3800:120:R + 3808:120:S
mysqld-3797 [01] 151.413671: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413673: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413674: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.413676: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.413680: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413691: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.413695: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.413696: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.413700: 3803:120:R + 3791:120:S
mysqld-3802 [05] 151.413704: 3802:120:S ==> 3810:120:R
mysqld-3797 [01] 151.413706: 3797:120:S ==> 3805:120:R
sysbench-3810 [05] 151.413707: 3810:120:R + 3802:120:S
sysbench-3805 [01] 151.413708: 3805:120:R + 3797:120:S
sysbench-3805 [01] 151.413715: 3805:120:S ==> 3797:120:R
sysbench-3803 [07] 151.413717: 3803:120:S ==> 3791:120:R
mysqld-3797 [01] 151.413718: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.413720: 3791:120:R + 3803:120:S
sysbench-3810 [05] 151.413723: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.413726: 3802:120:R + 3810:120:S
mysqld-3799 [02] 151.413747: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.413749: 3799:120:R ==> 3807:120:R
mysqld-3800 [03] 151.413754: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413755: 0:140:R ==> 3787:120:R
mysqld-3800 [03] 151.413757: 3800:120:S ==> 3808:120:R
mysqld-3797 [01] 151.413757: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.413760: 3805:120:R + 3797:120:S
sysbench-3808 [03] 151.413760: 3808:120:R + 3800:120:S
sysbench-3807 [02] 151.413761: 3807:120:S ==> 3798:120:R
mysqld-3787 [00] 151.413764: 3787:120:S ==> 0:140:R
sysbench-3808 [03] 151.413770: 3808:120:S ==> 3800:120:R
sysbench-3805 [01] 151.413771: 3805:120:S ==> 3797:120:R
mysqld-3798 [02] 151.413771: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.413772: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.413773: 3800:120:R + 3808:120:S
sysbench-3806 [02] 151.413774: 3806:120:S ==> 3798:120:R
mysqld-3797 [01] 151.413774: 3797:120:R + 3805:120:S
mysqld-3797 [01] 151.413808: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413810: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413811: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.413813: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.413816: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413823: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.413826: 3797:120:R + 3805:120:S
mysqld-3797 [01] 151.413857: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413859: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413860: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.413862: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.413865: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413871: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.413874: 3797:120:R + 3805:120:S
mysqld-3797 [01] 151.413905: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413907: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413908: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.413910: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.413913: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413919: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.413922: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.413959: 3791:120:S ==> 3803:120:R
mysqld-3797 [01] 151.413959: 0:140:R + 3787:120:S
<idle>-0 [00] 151.413961: 0:140:R ==> 3787:120:R
mysqld-3797 [01] 151.413963: 3803:120:R + 3791:120:S
mysqld-3802 [05] 151.413964: 3802:120:S ==> 3810:120:R
mysqld-3797 [01] 151.413966: 3797:120:S ==> 3805:120:R
sysbench-3810 [05] 151.413967: 3810:120:R + 3802:120:S
mysqld-3787 [00] 151.413968: 3787:120:S ==> 0:140:R
sysbench-3805 [01] 151.413968: 3805:120:R + 3797:120:S
sysbench-3803 [07] 151.413973: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.413977: 3810:120:S ==> 3802:120:R
sysbench-3805 [01] 151.413979: 3805:120:S ==> 3797:120:R
mysqld-3791 [07] 151.413979: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.413980: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.413983: 3797:120:R + 3805:120:S
mysqld-3798 [02] 151.413997: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.413999: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414014: 3806:120:S ==> 3799:120:R
mysqld-3791 [07] 151.414023: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414025: 0:140:R ==> 3787:120:R
mysqld-3791 [07] 151.414029: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.414029: 3802:120:S ==> 3810:120:R
mysqld-3797 [01] 151.414030: 3797:120:S ==> 3805:120:R
sysbench-3803 [07] 151.414031: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414032: 3810:120:R + 3802:120:S
sysbench-3805 [01] 151.414033: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.414039: 3787:120:S ==> 0:140:R
mysqld-3800 [03] 151.414042: 3800:120:S ==> 3808:120:R
sysbench-3805 [01] 151.414045: 3805:120:S ==> 3797:120:R
sysbench-3808 [03] 151.414045: 3808:120:R + 3800:120:S
mysqld-3799 [02] 151.414047: 3799:120:R + 3807:120:S
mysqld-3797 [01] 151.414048: 3797:120:R + 3805:120:S
mysqld-3799 [02] 151.414049: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414051: 3807:120:S ==> 3799:120:R
sysbench-3810 [05] 151.414053: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.414056: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.414057: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.414059: 3791:120:R + 3803:120:S
mysqld-3799 [02] 151.414061: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414062: 3799:120:R ==> 3807:120:R
sysbench-3808 [03] 151.414064: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.414067: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.414068: 3802:120:S ==> 3810:120:R
mysqld-3800 [03] 151.414069: 3800:120:R ==> 3808:120:R
mysqld-3791 [07] 151.414069: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.414070: 3810:120:R + 3802:120:S
sysbench-3808 [03] 151.414071: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.414071: 3803:120:R + 3791:120:S
mysqld-3797 [01] 151.414077: 3797:120:S ==> 3805:120:R
sysbench-3810 [05] 151.414078: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.414078: 3803:120:S ==> 3791:120:R
sysbench-3805 [01] 151.414080: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.414081: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.414081: 3791:120:R + 3803:120:S
sysbench-3807 [02] 151.414084: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.414090: 3799:120:R + 3807:120:S
sysbench-3805 [01] 151.414091: 3805:120:S ==> 3797:120:R
mysqld-3799 [02] 151.414092: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414093: 3807:120:S ==> 3799:120:R
mysqld-3797 [01] 151.414095: 3797:120:R + 3805:120:S
mysqld-3799 [02] 151.414105: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414106: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414113: 3807:120:S ==> 3798:120:R
mysqld-3791 [07] 151.414122: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.414143: 3803:120:S ==> 0:140:R
mysqld-3802 [05] 151.414167: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414169: 0:140:R ==> 3787:120:R
mysqld-3802 [05] 151.414170: 0:140:R + 3791:120:S
mysqld-3797 [01] 151.414173: 3797:120:S ==> 3805:120:R
<idle>-0 [07] 151.414174: 0:140:R ==> 3791:120:R
mysqld-3798 [02] 151.414175: 3800:120:S + 3800:120:S
sysbench-3805 [01] 151.414175: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.414176: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.414178: 3810:120:R + 3802:120:S
mysqld-3787 [00] 151.414179: 3787:120:S ==> 0:140:R
mysqld-3791 [07] 151.414181: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.414182: 3800:120:R ==> 3796:120:R
mysqld-3798 [02] 151.414184: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414186: 3798:120:R ==> 3806:120:R
sysbench-3805 [01] 151.414187: 3805:120:S ==> 3797:120:R
sysbench-3806 [02] 151.414189: 3806:120:S ==> 3799:120:R
mysqld-3797 [01] 151.414191: 3797:120:R + 3805:120:S
sysbench-3810 [05] 151.414191: 3810:120:S ==> 3802:120:R
mysqld-3799 [02] 151.414194: 3799:120:R + 3807:120:S
mysqld-3802 [05] 151.414194: 3802:120:R + 3810:120:S
mysqld-3799 [02] 151.414195: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414196: 3807:120:S ==> 3799:120:R
mysqld-3791 [07] 151.414217: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414218: 0:140:R ==> 3787:120:R
mysqld-3791 [07] 151.414219: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.414222: 3803:120:R + 3791:120:S
mysqld-3797 [01] 151.414222: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.414225: 3805:120:R + 3797:120:S
mysqld-3787 [00] 151.414226: 3787:120:S ==> 0:140:R
mysqld-3802 [05] 151.414229: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.414231: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.414232: 3803:120:S ==> 3791:120:R
mysqld-3799 [02] 151.414233: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414235: 3799:120:R ==> 3807:120:R
mysqld-3791 [07] 151.414235: 3791:120:R + 3803:120:S
sysbench-3805 [01] 151.414237: 3805:120:S ==> 3797:120:R
mysqld-3797 [01] 151.414240: 3797:120:R + 3805:120:S
sysbench-3810 [05] 151.414241: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.414244: 3802:120:R + 3810:120:S
sysbench-3807 [02] 151.414247: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.414255: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414257: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414258: 3807:120:S ==> 3799:120:R
mysqld-3791 [07] 151.414270: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414272: 0:140:R ==> 3787:120:R
mysqld-3791 [07] 151.414273: 3791:120:S ==> 3803:120:R
mysqld-3797 [01] 151.414274: 3797:120:S ==> 3805:120:R
mysqld-3802 [05] 151.414275: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.414276: 3803:120:R + 3791:120:S
sysbench-3805 [01] 151.414277: 3805:120:R + 3797:120:S
sysbench-3810 [05] 151.414278: 3810:120:R + 3802:120:S
mysqld-3787 [00] 151.414279: 3787:120:S ==> 0:140:R
sysbench-3803 [07] 151.414285: 3803:120:S ==> 3791:120:R
mysqld-3799 [02] 151.414287: 3799:120:R + 3807:120:S
sysbench-3810 [05] 151.414288: 3810:120:S ==> 3802:120:R
mysqld-3799 [02] 151.414288: 3799:120:R ==> 3807:120:R
mysqld-3791 [07] 151.414289: 3791:120:R + 3803:120:S
sysbench-3805 [01] 151.414289: 3805:120:S ==> 3797:120:R
mysqld-3802 [05] 151.414291: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.414292: 3797:120:R + 3805:120:S
sysbench-3807 [02] 151.414300: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.414309: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414310: 3798:120:R ==> 3806:120:R
mysqld-3791 [07] 151.414322: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.414323: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414325: 0:140:R ==> 3787:120:R
sysbench-3806 [02] 151.414327: 3806:120:S ==> 3798:120:R
mysqld-3802 [05] 151.414331: 3803:120:S + 3791:120:S
mysqld-3787 [00] 151.414333: 3787:120:S ==> 0:140:R
sysbench-3803 [07] 151.414333: 3803:120:S ==> 3791:120:R
mysqld-3798 [02] 151.414334: 3798:120:R + 3806:120:S
mysqld-3802 [05] 151.414335: 3802:120:S ==> 3810:120:R
mysqld-3798 [02] 151.414335: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414337: 3806:120:S ==> 3799:120:R
sysbench-3810 [05] 151.414337: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.414343: 3791:120:R + 3803:120:S
mysqld-3799 [02] 151.414343: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414345: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414346: 3807:120:S ==> 3798:120:R
sysbench-3810 [05] 151.414346: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.414349: 3802:120:R + 3810:120:S
mysqld-3796 [03] 151.414351: 3797:120:R + 3804:120:S
mysqld-3798 [02] 151.414354: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414354: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414362: 3806:120:S ==> 3799:120:R
mysqld-3796 [03] 151.414366: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414367: 0:140:R ==> 3787:120:R
mysqld-3796 [03] 151.414369: 3796:120:S ==> 3800:120:R
mysqld-3787 [00] 151.414374: 3787:120:S ==> 0:140:R
mysqld-3791 [07] 151.414379: 3791:120:S ==> 3803:120:R
mysqld-3802 [05] 151.414380: 3802:120:S ==> 3810:120:R
sysbench-3803 [07] 151.414381: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414382: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.414390: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.414391: 3810:120:S ==> 3802:120:R
mysqld-3799 [02] 151.414391: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414393: 3799:120:R ==> 3807:120:R
mysqld-3791 [07] 151.414393: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.414394: 3802:120:R + 3810:120:S
mysqld-3797 [01] 151.414399: 3797:120:S ==> 3804:120:R
sysbench-3807 [02] 151.414403: 3807:120:S ==> 3798:120:R
sysbench-3804 [01] 151.414405: 3800:120:R + 3796:120:S
mysqld-3798 [02] 151.414408: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414410: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414411: 3806:120:S ==> 3798:120:R
sysbench-3804 [01] 151.414418: 3804:120:S ==> 3805:120:R
sysbench-3805 [01] 151.414421: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.414425: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.414425: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.414427: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.414427: 3803:120:R + 3791:120:S
sysbench-3805 [01] 151.414435: 3805:120:S ==> 3797:120:R
sysbench-3810 [05] 151.414436: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.414436: 3803:120:S ==> 3791:120:R
mysqld-3797 [01] 151.414439: 3797:120:R + 3805:120:S
mysqld-3802 [05] 151.414439: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.414439: 3791:120:R + 3803:120:S
mysqld-3798 [02] 151.414443: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414444: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414455: 3806:120:S ==> 3799:120:R
mysqld-3799 [02] 151.414463: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414464: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414465: 3807:120:S ==> 3799:120:R
mysqld-3802 [05] 151.414477: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.414478: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.414480: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.414481: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414488: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.414489: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.414491: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.414492: 3791:120:R + 3803:120:S
mysqld-3799 [02] 151.414494: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414495: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414506: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.414514: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414515: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414516: 3806:120:S ==> 3798:120:R
mysqld-3800 [03] 151.414521: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414523: 3800:120:R ==> 3808:120:R
mysqld-3802 [05] 151.414523: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.414524: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.414525: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.414526: 3803:120:R + 3791:120:S
mysqld-3797 [01] 151.414528: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.414531: 3805:120:R + 3797:120:S
sysbench-3810 [05] 151.414534: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.414535: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.414536: 3808:120:S ==> 3796:120:R
mysqld-3802 [05] 151.414537: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.414538: 3791:120:R + 3803:120:S
mysqld-3796 [03] 151.414540: 3805:120:R + 3804:120:S
mysqld-3798 [02] 151.414542: 3798:120:R + 3806:120:S
sysbench-3805 [01] 151.414542: 3805:120:S ==> 3804:120:R
mysqld-3798 [02] 151.414544: 3798:120:R ==> 3806:120:R
sysbench-3804 [01] 151.414545: 3804:120:S ==> 3797:120:R
mysqld-3797 [01] 151.414548: 3797:120:R + 3805:120:S
mysqld-3796 [03] 151.414553: 3797:120:R + 3804:120:S
sysbench-3806 [02] 151.414556: 3806:120:S ==> 3799:120:R
mysqld-3799 [02] 151.414564: 3799:120:R + 3807:120:S
mysqld-3796 [03] 151.414564: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414565: 0:140:R ==> 3787:120:R
mysqld-3799 [02] 151.414565: 3799:120:R ==> 3807:120:R
mysqld-3796 [03] 151.414566: 3796:120:S ==> 3800:120:R
sysbench-3807 [02] 151.414566: 3807:120:S ==> 3799:120:R
mysqld-3802 [05] 151.414570: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.414572: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.414575: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.414577: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414583: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.414587: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.414587: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.414592: 3791:120:R + 3803:120:S
mysqld-3799 [02] 151.414592: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414594: 3799:120:R ==> 3807:120:R
mysqld-3787 [00] 151.414603: 3787:120:S ==> 0:140:R
sysbench-3807 [02] 151.414605: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.414612: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414614: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414616: 3806:120:S ==> 3798:120:R
mysqld-3802 [05] 151.414619: 3802:120:S ==> 3810:120:R
mysqld-3800 [03] 151.414620: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414622: 3800:120:R ==> 3808:120:R
sysbench-3810 [05] 151.414622: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.414623: 3791:120:S ==> 3803:120:R
sysbench-3808 [03] 151.414623: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.414625: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414630: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.414630: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414632: 3800:120:R ==> 3808:120:R
mysqld-3802 [05] 151.414634: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.414634: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.414637: 3791:120:R + 3803:120:S
mysqld-3798 [02] 151.414642: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414643: 3798:120:R ==> 3806:120:R
sysbench-3808 [03] 151.414652: 3808:120:S ==> 3800:120:R
sysbench-3806 [02] 151.414656: 3806:120:S ==> 3799:120:R
mysqld-3800 [03] 151.414658: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414660: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414661: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.414663: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414664: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414666: 3807:120:S ==> 3799:120:R
mysqld-3800 [03] 151.414667: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414668: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414676: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.414681: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414682: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414683: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.414688: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414689: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414699: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.414706: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414707: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414708: 3806:120:S ==> 3798:120:R
mysqld-3800 [03] 151.414713: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414715: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414726: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.414731: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414732: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.414734: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414735: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414737: 3808:120:S ==> 3800:120:R
sysbench-3806 [02] 151.414741: 3806:120:S ==> 3799:120:R
mysqld-3802 [05] 151.414745: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.414747: 3810:120:R + 3802:120:S
mysqld-3799 [02] 151.414749: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414750: 3799:120:R ==> 3807:120:R
mysqld-3791 [07] 151.414751: 3791:120:S ==> 3803:120:R
sysbench-3807 [02] 151.414752: 3807:120:S ==> 3799:120:R
sysbench-3803 [07] 151.414754: 3803:120:R + 3791:120:S
mysqld-3800 [03] 151.414760: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414762: 3800:120:R ==> 3808:120:R
sysbench-3810 [05] 151.414763: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.414766: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.414770: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.414772: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.414773: 3791:120:R + 3803:120:S
mysqld-3797 [01] 151.414775: 3797:120:S ==> 3804:120:R
mysqld-3799 [02] 151.414777: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414778: 3799:120:R ==> 3807:120:R
sysbench-3804 [01] 151.414779: 3800:120:R + 3796:120:S
mysqld-3800 [03] 151.414781: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414783: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414784: 3808:120:S ==> 3796:120:R
mysqld-3796 [03] 151.414786: 3796:120:S ==> 3800:120:R
sysbench-3807 [02] 151.414789: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.414796: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414798: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414799: 3806:120:S ==> 3798:120:R
sysbench-3804 [01] 151.414801: 3800:120:R + 3796:120:S
sysbench-3804 [01] 151.414803: 3804:120:S ==> 3805:120:R
sysbench-3805 [01] 151.414806: 3805:120:R + 3797:120:S
mysqld-3800 [03] 151.414810: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414811: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414820: 3808:120:S ==> 3796:120:R
sysbench-3805 [01] 151.414821: 3805:120:S ==> 3797:120:R
mysqld-3798 [02] 151.414822: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414823: 3798:120:R ==> 3806:120:R
mysqld-3796 [03] 151.414825: 3797:120:R + 3804:120:S
mysqld-3797 [01] 151.414825: 3797:120:R + 3805:120:S
sysbench-3806 [02] 151.414834: 3806:120:S ==> 3799:120:R
mysqld-3799 [02] 151.414841: 3799:120:R + 3807:120:S
mysqld-3796 [03] 151.414841: 0:140:R + 3787:120:S
<idle>-0 [00] 151.414842: 0:140:R ==> 3787:120:R
mysqld-3799 [02] 151.414842: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414843: 3807:120:S ==> 3799:120:R
mysqld-3796 [03] 151.414844: 3796:120:S ==> 3800:120:R
mysqld-3787 [00] 151.414850: 3787:120:S ==> 0:140:R
mysqld-3800 [03] 151.414850: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414853: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414854: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.414858: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.414861: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.414865: 3791:120:S ==> 3803:120:R
mysqld-3799 [02] 151.414866: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414868: 3799:120:R ==> 3807:120:R
sysbench-3803 [07] 151.414868: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.414872: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.414876: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.414877: 3803:120:S ==> 3791:120:R
sysbench-3807 [02] 151.414877: 3807:120:S ==> 3798:120:R
mysqld-3800 [03] 151.414879: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414881: 3800:120:R ==> 3808:120:R
mysqld-3791 [07] 151.414881: 3791:120:R + 3803:120:S
mysqld-3798 [02] 151.414885: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414886: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414888: 3806:120:S ==> 3798:120:R
sysbench-3808 [03] 151.414891: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.414899: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414900: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414901: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.414914: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414915: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.414924: 3800:120:R + 3808:120:S
sysbench-3806 [02] 151.414925: 3806:120:S ==> 3799:120:R
mysqld-3800 [03] 151.414926: 3800:120:R ==> 3808:120:R
mysqld-3799 [02] 151.414932: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414933: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414934: 3807:120:S ==> 3799:120:R
sysbench-3808 [03] 151.414936: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.414943: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414945: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414946: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.414958: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.414959: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.414969: 3807:120:S ==> 3798:120:R
mysqld-3800 [03] 151.414969: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414971: 3800:120:R ==> 3808:120:R
mysqld-3798 [02] 151.414976: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.414977: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.414979: 3806:120:S ==> 3798:120:R
sysbench-3808 [03] 151.414981: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.414989: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.414990: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.414992: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.415005: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415006: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.415014: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415015: 3800:120:R ==> 3808:120:R
sysbench-3806 [02] 151.415016: 3806:120:S ==> 3799:120:R
mysqld-3799 [02] 151.415023: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415024: 3799:120:R ==> 3807:120:R
sysbench-3808 [03] 151.415025: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.415026: 3807:120:S ==> 3799:120:R
mysqld-3800 [03] 151.415032: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415034: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415036: 3808:120:S ==> 3800:120:R
mysqld-3797 [01] 151.415046: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.415048: 3805:120:R + 3797:120:S
mysqld-3799 [02] 151.415051: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415053: 3799:120:R ==> 3807:120:R
sysbench-3805 [01] 151.415058: 3805:120:S ==> 3804:120:R
mysqld-3800 [03] 151.415059: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415061: 3800:120:R ==> 3808:120:R
sysbench-3807 [02] 151.415063: 3807:120:S ==> 3798:120:R
sysbench-3804 [01] 151.415063: 3804:120:R + 3796:120:S
mysqld-3798 [02] 151.415070: 3798:120:R + 3806:120:S
sysbench-3804 [01] 151.415071: 3804:120:S ==> 3796:120:R
mysqld-3798 [02] 151.415072: 3798:120:R ==> 3806:120:R
sysbench-3808 [03] 151.415072: 3808:120:S ==> 3800:120:R
sysbench-3806 [02] 151.415073: 3806:120:S ==> 3798:120:R
mysqld-3796 [01] 151.415075: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.415080: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415081: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415082: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.415100: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415102: 3798:120:R ==> 3806:120:R
mysqld-3800 [03] 151.415107: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415108: 3800:120:R ==> 3808:120:R
sysbench-3806 [02] 151.415112: 3806:120:S ==> 3799:120:R
mysqld-3802 [05] 151.415114: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415114: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.415117: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415117: 3803:120:R + 3791:120:S
sysbench-3808 [03] 151.415118: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.415121: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415122: 3799:120:R ==> 3807:120:R
mysqld-3796 [01] 151.415123: 3796:120:S ==> 3804:120:R
sysbench-3807 [02] 151.415124: 3807:120:S ==> 3799:120:R
mysqld-3800 [03] 151.415125: 3800:120:R + 3808:120:S
sysbench-3804 [01] 151.415126: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.415126: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415128: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.415133: 3803:120:S ==> 3791:120:R
sysbench-3810 [05] 151.415133: 3810:120:S ==> 3802:120:R
sysbench-3804 [01] 151.415135: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.415137: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415137: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.415138: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.415153: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415155: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415164: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.415171: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415173: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415174: 3808:120:S ==> 3800:120:R
mysqld-3796 [01] 151.415177: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415179: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.415188: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.415191: 3796:120:R + 3804:120:S
mysqld-3796 [01] 151.415221: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415223: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.415231: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.415234: 3796:120:R + 3804:120:S
mysqld-3799 [02] 151.415246: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415248: 3799:120:R ==> 3807:120:R
mysqld-3796 [01] 151.415263: 3796:120:S ==> 3804:120:R
sysbench-3807 [02] 151.415266: 3807:120:S ==> 3798:120:R
sysbench-3804 [01] 151.415266: 3804:120:R + 3796:120:S
mysqld-3798 [02] 151.415274: 3798:120:R + 3806:120:S
sysbench-3804 [01] 151.415275: 3804:120:S ==> 3796:120:R
mysqld-3798 [02] 151.415276: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.415277: 3806:120:S ==> 3798:120:R
mysqld-3796 [01] 151.415278: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.415283: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415285: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415302: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.415303: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415305: 3798:120:R ==> 3806:120:R
mysqld-3796 [01] 151.415309: 3796:120:S ==> 3804:120:R
mysqld-3800 [03] 151.415309: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415311: 3800:120:R ==> 3808:120:R
sysbench-3804 [01] 151.415312: 3804:120:R + 3796:120:S
sysbench-3808 [03] 151.415312: 3808:120:S ==> 3800:120:R
sysbench-3806 [02] 151.415315: 3806:120:S ==> 3798:120:R
sysbench-3804 [01] 151.415320: 3804:120:S ==> 3796:120:R
mysqld-3798 [02] 151.415323: 3798:120:R + 3806:120:S
mysqld-3796 [01] 151.415324: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.415325: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.415326: 3806:120:S ==> 3798:120:R
mysqld-3798 [02] 151.415352: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415353: 3798:120:R ==> 3806:120:R
mysqld-3796 [01] 151.415357: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415360: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.415361: 3802:120:S ==> 3810:120:R
sysbench-3806 [02] 151.415363: 3806:120:S ==> 3798:120:R
mysqld-3791 [07] 151.415365: 3791:120:S ==> 3803:120:R
sysbench-3804 [01] 151.415368: 3804:120:S ==> 3796:120:R
mysqld-3798 [02] 151.415371: 3798:120:R + 3806:120:S
mysqld-3796 [01] 151.415371: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.415372: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.415374: 3806:120:S ==> 3798:120:R
sysbench-3810 [05] 151.415374: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415376: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415383: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.415386: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.415387: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415389: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.415400: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.415402: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415402: 3791:120:S ==> 3803:120:R
mysqld-3796 [01] 151.415403: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.415404: 3803:120:R + 3791:120:S
mysqld-3800 [03] 151.415407: 3800:120:R + 3808:120:S
sysbench-3804 [01] 151.415411: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.415419: 3804:120:S ==> 3796:120:R
mysqld-3800 [03] 151.415421: 3800:120:S ==> 3808:120:R
mysqld-3796 [01] 151.415422: 3796:120:R + 3804:120:S
sysbench-3808 [03] 151.415424: 3808:120:R + 3800:120:S
sysbench-3810 [05] 151.415427: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415430: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415430: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.415433: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.415433: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.415436: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415437: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.415439: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.415441: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415443: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.415443: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415445: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415450: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.415451: 3803:120:S ==> 3791:120:R
mysqld-3796 [01] 151.415452: 3796:120:S ==> 3804:120:R
mysqld-3802 [05] 151.415453: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415454: 3791:120:R + 3803:120:S
sysbench-3804 [01] 151.415454: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.415463: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.415466: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.415475: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415477: 3798:120:R ==> 3806:120:R
mysqld-3802 [05] 151.415492: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415493: 3791:120:S ==> 3803:120:R
sysbench-3806 [02] 151.415494: 3806:120:S ==> 3799:120:R
sysbench-3810 [05] 151.415495: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415495: 3803:120:R + 3791:120:S
mysqld-3796 [01] 151.415499: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415501: 3804:120:R + 3796:120:S
mysqld-3799 [02] 151.415502: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415504: 3799:120:R ==> 3807:120:R
sysbench-3810 [05] 151.415505: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.415505: 3803:120:S ==> 3791:120:R
sysbench-3807 [02] 151.415506: 3807:120:S ==> 3799:120:R
mysqld-3802 [05] 151.415508: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415508: 3791:120:R + 3803:120:S
sysbench-3804 [01] 151.415510: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.415513: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.415541: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415542: 3791:120:S ==> 3803:120:R
mysqld-3796 [01] 151.415543: 3796:120:S ==> 3804:120:R
sysbench-3810 [05] 151.415543: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415544: 3803:120:R + 3791:120:S
sysbench-3804 [01] 151.415545: 3804:120:R + 3796:120:S
sysbench-3810 [05] 151.415552: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.415553: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.415554: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.415556: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415556: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.415557: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.415587: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415588: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.415590: 3810:120:R + 3802:120:S
sysbench-3803 [07] 151.415590: 3803:120:R + 3791:120:S
mysqld-3799 [02] 151.415591: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.415592: 3799:120:R ==> 3807:120:R
sysbench-3810 [05] 151.415598: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.415599: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.415601: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415602: 3791:120:R + 3803:120:S
sysbench-3807 [02] 151.415603: 3807:120:S ==> 3798:120:R
mysqld-3798 [02] 151.415611: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415612: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.415613: 3806:120:S ==> 3798:120:R
mysqld-3802 [05] 151.415631: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.415633: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415634: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.415636: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415641: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415644: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415645: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.415648: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.415660: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415662: 3800:120:R ==> 3808:120:R
mysqld-3796 [01] 151.415667: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415670: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.415675: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.415677: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415678: 3791:120:S ==> 3803:120:R
sysbench-3808 [03] 151.415680: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.415680: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415686: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.415688: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415689: 3800:120:R ==> 3808:120:R
mysqld-3802 [05] 151.415689: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415689: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.415690: 3804:120:S ==> 3796:120:R
sysbench-3808 [03] 151.415690: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.415692: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.415693: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.415702: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.415704: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.415714: 3806:120:S ==> 3799:120:R
mysqld-3802 [05] 151.415721: 3802:120:S ==> 3810:120:R
mysqld-3799 [02] 151.415722: 3799:120:R + 3807:120:S
sysbench-3810 [05] 151.415723: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415724: 3791:120:S ==> 3803:120:R
mysqld-3799 [02] 151.415724: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.415725: 3807:120:S ==> 3799:120:R
sysbench-3803 [07] 151.415727: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415732: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415735: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415735: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.415738: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.415764: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.415767: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415768: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.415770: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415775: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415778: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415778: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.415781: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.415784: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.415786: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.415796: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.415799: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.415808: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.415810: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.415811: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.415814: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.415818: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415821: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415822: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.415825: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.415845: 3796:120:R ==> 3804:120:R
sysbench-3804 [01] 151.415847: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.415851: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.415852: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.415854: 3810:120:R + 3802:120:S
sysbench-3810 [05] 151.415863: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415867: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.415888: 3803:120:S ==> 0:140:R
mysqld-3796 [01] 151.415893: 0:140:R + 3791:120:S
mysqld-3802 [05] 151.415895: 3802:120:S ==> 3810:120:R
<idle>-0 [07] 151.415896: 0:140:R ==> 3791:120:R
sysbench-3810 [05] 151.415898: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.415902: 3800:120:R + 3808:120:S
sysbench-3810 [05] 151.415906: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.415910: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.415911: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.415919: 3791:120:S ==> 3803:120:R
mysqld-3800 [03] 151.415921: 3800:120:S ==> 3808:120:R
sysbench-3803 [07] 151.415922: 3803:120:R + 3791:120:S
sysbench-3808 [03] 151.415924: 3808:120:R + 3800:120:S
sysbench-3803 [07] 151.415931: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.415934: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.415934: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.415937: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.415949: 3800:120:S ==> 3808:120:R
mysqld-3799 [02] 151.415950: 3799:120:R + 3807:120:S
sysbench-3808 [03] 151.415952: 3808:120:R + 3800:120:S
mysqld-3799 [02] 151.415960: 3799:120:S ==> 3807:120:R
sysbench-3807 [02] 151.415963: 3807:120:R + 3799:120:S
mysqld-3791 [07] 151.415965: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.415967: 3803:120:R + 3791:120:S
sysbench-3808 [03] 151.415969: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.415972: 3800:120:R + 3808:120:S
sysbench-3803 [07] 151.415976: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.415979: 3791:120:R + 3803:120:S
sysbench-3807 [02] 151.415982: 3807:120:S ==> 3799:120:R
mysqld-3800 [03] 151.415983: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.415985: 3808:120:R + 3800:120:S
mysqld-3799 [02] 151.415985: 3799:120:R + 3807:120:S
sysbench-3808 [03] 151.415992: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.415994: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.416020: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416023: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.416034: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.416037: 3808:120:R + 3800:120:S
sysbench-3810 [05] 151.416038: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416041: 3802:120:R + 3810:120:S
sysbench-3808 [03] 151.416046: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.416049: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416079: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.416081: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.416090: 3808:120:S ==> 3800:120:R
mysqld-3796 [01] 151.416092: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.416093: 3800:120:R + 3808:120:S
mysqld-3791 [07] 151.416099: 3791:120:S ==> 3803:120:R
mysqld-3796 [01] 151.416101: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.416102: 3803:120:R + 3791:120:S
sysbench-3804 [01] 151.416104: 3804:120:R + 3796:120:S
sysbench-3803 [07] 151.416117: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.416120: 3804:120:S ==> 3796:120:R
mysqld-3791 [07] 151.416121: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.416122: 3800:120:S ==> 3808:120:R
mysqld-3796 [01] 151.416124: 3796:120:R + 3804:120:S
sysbench-3808 [03] 151.416125: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.416133: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.416136: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.416143: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416146: 3810:120:R + 3802:120:S
sysbench-3810 [05] 151.416155: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416158: 3802:120:R + 3810:120:S
mysqld-3800 [03] 151.416165: 3800:120:S ==> 3808:120:R
sysbench-3808 [03] 151.416168: 3808:120:R + 3800:120:S
sysbench-3808 [03] 151.416176: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.416179: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416205: 3800:120:S ==> 3808:120:R
mysqld-3791 [07] 151.416218: 3791:120:S ==> 3803:120:R
mysqld-3799 [02] 151.416225: 3799:120:S ==> 3807:120:R
sysbench-3803 [07] 151.416235: 3803:120:R + 3791:120:S
sysbench-3803 [07] 151.416244: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.416248: 3791:120:R + 3803:120:S
sysbench-3807 [02] 151.416249: 3807:120:R + 3799:120:S
sysbench-3808 [03] 151.416252: 3808:120:S ==> 3800:120:R
mysqld-3802 [05] 151.416253: 3808:120:S + 3800:120:S
mysqld-3800 [03] 151.416257: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416258: 3800:120:R ==> 3808:120:R
sysbench-3807 [02] 151.416260: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416263: 3799:120:R + 3807:120:S
sysbench-3808 [03] 151.416269: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.416276: 3800:120:R + 3808:120:S
mysqld-3799 [02] 151.416277: 3799:120:S ==> 3807:120:R
mysqld-3800 [03] 151.416278: 3800:120:R ==> 3808:120:R
sysbench-3807 [02] 151.416279: 3807:120:R + 3799:120:S
sysbench-3808 [03] 151.416279: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.416294: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416297: 3799:120:R + 3807:120:S
mysqld-3800 [03] 151.416304: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416305: 3800:120:R ==> 3808:120:R
mysqld-3799 [02] 151.416307: 3799:120:S ==> 3807:120:R
sysbench-3807 [02] 151.416308: 3807:120:R + 3799:120:S
sysbench-3808 [03] 151.416315: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.416315: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416318: 3799:120:R + 3807:120:S
mysqld-3800 [03] 151.416322: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416324: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416325: 3808:120:S ==> 3800:120:R
mysqld-3800 [03] 151.416348: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416350: 3800:120:R ==> 3808:120:R
mysqld-3799 [02] 151.416359: 3799:120:S ==> 3807:120:R
sysbench-3808 [03] 151.416360: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.416362: 3807:120:R + 3799:120:S
mysqld-3800 [03] 151.416367: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416368: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416369: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.416371: 3807:120:S ==> 3799:120:R
mysqld-3796 [01] 151.416373: 3796:120:S ==> 3804:120:R
mysqld-3799 [02] 151.416374: 3799:120:R + 3807:120:S
sysbench-3804 [01] 151.416376: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.416389: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416392: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.416393: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416394: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416404: 3808:120:S ==> 3800:120:R
mysqld-3796 [01] 151.416405: 3796:120:S ==> 3804:120:R
mysqld-3799 [02] 151.416410: 3799:120:S ==> 3807:120:R
sysbench-3804 [01] 151.416410: 3804:120:R + 3796:120:S
sysbench-3807 [02] 151.416412: 3807:120:R + 3799:120:S
mysqld-3800 [03] 151.416415: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416417: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416418: 3808:120:S ==> 3800:120:R
sysbench-3807 [02] 151.416421: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416424: 3799:120:R + 3807:120:S
sysbench-3804 [01] 151.416426: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416429: 3796:120:R + 3804:120:S
mysqld-3796 [01] 151.416440: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416442: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.416443: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416445: 3800:120:R ==> 3808:120:R
sysbench-3804 [01] 151.416449: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.416451: 3802:120:S ==> 3810:120:R
mysqld-3796 [01] 151.416452: 3796:120:R + 3804:120:S
sysbench-3808 [03] 151.416455: 3808:120:S ==> 3800:120:R
mysqld-3799 [02] 151.416457: 3799:120:S ==> 3807:120:R
sysbench-3810 [05] 151.416459: 3810:120:R + 3802:120:S
sysbench-3807 [02] 151.416459: 3807:120:R + 3799:120:S
mysqld-3800 [03] 151.416463: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416464: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416465: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.416467: 3791:120:S ==> 3803:120:R
sysbench-3807 [02] 151.416468: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416471: 3799:120:R + 3807:120:S
sysbench-3803 [07] 151.416474: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.416475: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416478: 3802:120:R + 3810:120:S
mysqld-3796 [01] 151.416489: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.416490: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.416491: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.416492: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416493: 3800:120:R ==> 3808:120:R
mysqld-3791 [07] 151.416493: 3791:120:R + 3803:120:S
sysbench-3804 [01] 151.416501: 3804:120:S ==> 3796:120:R
sysbench-3808 [03] 151.416503: 3808:120:S ==> 3800:120:R
mysqld-3796 [01] 151.416504: 3796:120:R + 3804:120:S
mysqld-3799 [02] 151.416504: 3799:120:S ==> 3807:120:R
sysbench-3807 [02] 151.416507: 3807:120:R + 3799:120:S
sysbench-3807 [02] 151.416515: 3807:120:S ==> 3799:120:R
mysqld-3799 [02] 151.416518: 3799:120:R + 3807:120:S
mysqld-3799 [02] 151.416519: 3799:120:R ==> 3807:120:R
sysbench-3807 [02] 151.416521: 3807:120:S ==> 3798:120:R
mysqld-3800 [03] 151.416523: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416524: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416526: 3808:120:S ==> 3800:120:R
mysqld-3798 [02] 151.416528: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.416529: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.416531: 3806:120:S ==> 3798:120:R
mysqld-3796 [01] 151.416535: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416538: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.416546: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416549: 3796:120:R + 3804:120:S
mysqld-3796 [01] 151.416580: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416582: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.416590: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416594: 3796:120:R + 3804:120:S
mysqld-3796 [01] 151.416621: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416623: 3804:120:R + 3796:120:S
sysbench-3804 [01] 151.416632: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416635: 3796:120:R + 3804:120:S
mysqld-3800 [03] 151.416645: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416646: 3800:120:R ==> 3808:120:R
mysqld-3796 [01] 151.416662: 3796:120:S ==> 3804:120:R
sysbench-3808 [03] 151.416663: 3808:120:S ==> 3800:120:R
sysbench-3804 [01] 151.416664: 3804:120:R + 3796:120:S
mysqld-3800 [03] 151.416670: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416672: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.416673: 3808:120:S ==> 3800:120:R
sysbench-3804 [01] 151.416673: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416676: 3796:120:R + 3804:120:S
mysqld-3796 [01] 151.416706: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416709: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.416709: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416712: 3810:120:R + 3802:120:S
sysbench-3804 [01] 151.416717: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416720: 3796:120:R + 3804:120:S
mysqld-3791 [07] 151.416721: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.416722: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.416724: 3803:120:R + 3791:120:S
mysqld-3802 [05] 151.416725: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.416733: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.416736: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.416737: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416739: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.416750: 3791:120:S ==> 3803:120:R
mysqld-3796 [01] 151.416750: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.416752: 3803:120:R + 3791:120:S
sysbench-3804 [01] 151.416752: 3804:120:R + 3796:120:S
sysbench-3810 [05] 151.416754: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.416757: 3800:120:R + 3808:120:S
mysqld-3802 [05] 151.416757: 3802:120:R + 3810:120:S
mysqld-3800 [03] 151.416759: 3800:120:R ==> 3808:120:R
sysbench-3804 [01] 151.416761: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416764: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.416766: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.416768: 3802:120:S ==> 3810:120:R
mysqld-3791 [07] 151.416769: 3791:120:R + 3803:120:S
sysbench-3808 [03] 151.416769: 3808:120:S ==> 3800:120:R
sysbench-3810 [05] 151.416770: 3810:120:R + 3802:120:S
sysbench-3810 [05] 151.416777: 3810:120:S ==> 3802:120:R
mysqld-3800 [03] 151.416777: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416779: 3800:120:R ==> 3808:120:R
mysqld-3802 [05] 151.416780: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.416780: 3791:120:S ==> 3803:120:R
sysbench-3808 [03] 151.416781: 3808:120:S ==> 3800:120:R
sysbench-3803 [07] 151.416782: 3803:120:R + 3791:120:S
mysqld-3798 [02] 151.416783: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.416785: 3798:120:R ==> 3806:120:R
sysbench-3803 [07] 151.416789: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.416792: 3791:120:R + 3803:120:S
mysqld-3796 [01] 151.416794: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.416796: 3804:120:R + 3796:120:S
sysbench-3806 [02] 151.416804: 3806:120:S ==> 3798:120:R
sysbench-3804 [01] 151.416805: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416808: 3796:120:R + 3804:120:S
mysqld-3798 [02] 151.416812: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.416813: 3798:120:R ==> 3806:120:R
sysbench-3806 [02] 151.416815: 3806:120:S ==> 3798:120:R
mysqld-3802 [05] 151.416817: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416820: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.416828: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.416829: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.416830: 3803:120:R + 3791:120:S
mysqld-3802 [05] 151.416832: 3802:120:R + 3810:120:S
mysqld-3796 [01] 151.416838: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.416839: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.416840: 3804:120:R + 3796:120:S
mysqld-3791 [07] 151.416842: 3791:120:R + 3803:120:S
sysbench-3804 [01] 151.416849: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416852: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.416865: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416867: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.416873: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.416875: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.416876: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416879: 3802:120:R + 3810:120:S
mysqld-3796 [01] 151.416882: 3796:120:S ==> 3804:120:R
sysbench-3803 [07] 151.416884: 3803:120:S ==> 3791:120:R
sysbench-3804 [01] 151.416885: 3804:120:R + 3796:120:S
mysqld-3791 [07] 151.416887: 3791:120:R + 3803:120:S
sysbench-3804 [01] 151.416893: 3804:120:S ==> 3796:120:R
mysqld-3796 [01] 151.416896: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.416909: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416911: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.416916: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.416918: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.416920: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416923: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.416927: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.416930: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.416951: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416953: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.416958: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.416960: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.416961: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.416964: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.416969: 3803:120:S ==> 3791:120:R
mysqld-3791 [07] 151.416972: 3791:120:R + 3803:120:S
mysqld-3802 [05] 151.416993: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.416996: 3810:120:R + 3802:120:S
mysqld-3800 [03] 151.416996: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.416998: 3800:120:R ==> 3808:120:R
mysqld-3791 [07] 151.417002: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.417004: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.417004: 3810:120:S ==> 3802:120:R
mysqld-3802 [05] 151.417007: 3802:120:R + 3810:120:S
mysqld-3796 [01] 151.417008: 3796:120:S ==> 3804:120:R
sysbench-3804 [01] 151.417010: 3804:120:R + 3796:120:S
sysbench-3803 [07] 151.417013: 3803:120:S ==> 3791:120:R
sysbench-3808 [03] 151.417015: 3808:120:S ==> 3800:120:R
mysqld-3791 [07] 151.417016: 3791:120:R + 3803:120:S
mysqld-3800 [03] 151.417023: 3800:120:R + 3808:120:S
mysqld-3800 [03] 151.417024: 3800:120:R ==> 3808:120:R
sysbench-3808 [03] 151.417025: 3808:120:S ==> 3800:120:R
sysbench-3804 [01] 151.417026: 3804:120:S ==> 3796:120:R
mysqld-3798 [02] 151.417027: 3798:120:R + 3806:120:S
mysqld-3798 [02] 151.417029: 3798:120:R ==> 3806:120:R
mysqld-3796 [01] 151.417030: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.417036: 3802:120:S ==> 3810:120:R
sysbench-3810 [05] 151.417039: 3810:120:R + 3802:120:S
sysbench-3806 [02] 151.417039: 3806:120:S ==> 3798:120:R
mysqld-3791 [07] 151.417045: 3791:120:S ==> 3803:120:R
sysbench-3810 [05] 151.417047: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.417048: 3803:120:R + 3791:120:S
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 11:30 ` Lin Ming
@ 2008-09-04 11:35 ` Ingo Molnar
2008-09-04 12:19 ` Lin Ming
0 siblings, 1 reply; 19+ messages in thread
From: Ingo Molnar @ 2008-09-04 11:35 UTC (permalink / raw)
To: Lin Ming; +Cc: Peter Zijlstra, linux-kernel, Zhang, Yanmin
* Lin Ming <ming.m.lin@intel.com> wrote:
> A huge trace result file, see the attachment for more.
> # tracer: sched_switch
> #
> # TASK-PID CPU# TIMESTAMP FUNCTION
> # | | | | |
> mysqld-3799 [07] 151.411009: 3799:120:R ==> 3807:120:R
...
> sysbench-3803 [07] 151.417048: 3803:120:R + 3791:120:S
ok, this only spans 6 milliseconds, which is probably a bit too light.
Could you please increase the trace size 32-fold (via
/debug/tracing/trace_entries), and send us the (large!) trace file
compressed via bzip2 -9, off-list (or via a web link)? [anyone on-list
can have a look at the smaller trace file already]
Thanks,
Ingo
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 11:06 ` Peter Zijlstra
@ 2008-09-04 12:12 ` Lin Ming
2008-09-04 12:26 ` Peter Zijlstra
2008-09-04 13:50 ` Gregory Haskins
1 sibling, 1 reply; 19+ messages in thread
From: Lin Ming @ 2008-09-04 12:12 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo, Gregory Haskins
> > > Thats bizarre... that just indicates the better clock, which should give
> > > better (read fairer) scheduling hurts your workload.
> > >
> > > Is there anything I can run to see if we can fix the scheduler perhaps?
> >
> > I observed schedstats of sysbench, there's more
> > "nr_failed_migrations_hot"
> >
> > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> >
> > task migration failed because of task_hot, the system is un-balanced?
>
> Ah, that makes sense, a more accurate clock could indeed make more tasks
> hot.
>
> Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
sched_migration_cost regression
---------------------- -------------
50000 ~6%
0 ~8%
500000 (default) ~10%
5000000 ~14%
-1 ~19%
>
> Also, we used to have some auto-tuning in there, which dissapeared some
> time ago, gregory brought it back to live recently, perhaps he likes to
> share? :-)
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 11:35 ` Ingo Molnar
@ 2008-09-04 12:19 ` Lin Ming
0 siblings, 0 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-04 12:19 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Peter Zijlstra, linux-kernel, Zhang, Yanmin
On Thu, 2008-09-04 at 13:35 +0200, Ingo Molnar wrote:
> * Lin Ming <ming.m.lin@intel.com> wrote:
>
> > A huge trace result file, see the attachment for more.
>
> > # tracer: sched_switch
> > #
> > # TASK-PID CPU# TIMESTAMP FUNCTION
> > # | | | | |
> > mysqld-3799 [07] 151.411009: 3799:120:R ==> 3807:120:R
> ...
> > sysbench-3803 [07] 151.417048: 3803:120:R + 3791:120:S
>
> ok, this only spans 6 milliseconds, which is probably a bit too light.
>
> Could you please increase the trace size 32-fold (via
> /debug/tracing/trace_entries), and send us the (large!) trace file
> compressed via bzip2 -9, off-list (or via a web link)? [anyone on-list
> can have a look at the smaller trace file already]
The large trace file also available at below link,
http://myfreefilehosting.com/f/94991018dc_1.31MB
>
> Thanks,
>
> Ingo
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 12:12 ` Lin Ming
@ 2008-09-04 12:26 ` Peter Zijlstra
2008-09-04 12:42 ` Lin Ming
0 siblings, 1 reply; 19+ messages in thread
From: Peter Zijlstra @ 2008-09-04 12:26 UTC (permalink / raw)
To: Lin Ming; +Cc: linux-kernel, Zhang, Yanmin, mingo, Gregory Haskins
On Thu, 2008-09-04 at 20:12 +0800, Lin Ming wrote:
> > > > Thats bizarre... that just indicates the better clock, which should give
> > > > better (read fairer) scheduling hurts your workload.
> > > >
> > > > Is there anything I can run to see if we can fix the scheduler perhaps?
> > >
> > > I observed schedstats of sysbench, there's more
> > > "nr_failed_migrations_hot"
> > >
> > > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> > >
> > > task migration failed because of task_hot, the system is un-balanced?
> >
> > Ah, that makes sense, a more accurate clock could indeed make more tasks
> > hot.
> >
> > Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
>
> sched_migration_cost regression
> ---------------------- -------------
> 50000 ~6%
> 0 ~8%
> 500000 (default) ~10%
> 5000000 ~14%
> -1 ~19%
at 50000 (~6%), is the predominant difference in schedstats still the
nr_failed_migrations_hot?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 12:26 ` Peter Zijlstra
@ 2008-09-04 12:42 ` Lin Ming
0 siblings, 0 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-04 12:42 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo, Gregory Haskins
On Thu, 2008-09-04 at 14:26 +0200, Peter Zijlstra wrote:
> On Thu, 2008-09-04 at 20:12 +0800, Lin Ming wrote:
> > > > > Thats bizarre... that just indicates the better clock, which should give
> > > > > better (read fairer) scheduling hurts your workload.
> > > > >
> > > > > Is there anything I can run to see if we can fix the scheduler perhaps?
> > > >
> > > > I observed schedstats of sysbench, there's more
> > > > "nr_failed_migrations_hot"
> > > >
> > > > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > > > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> > > >
> > > > task migration failed because of task_hot, the system is un-balanced?
> > >
> > > Ah, that makes sense, a more accurate clock could indeed make more tasks
> > > hot.
> > >
> > > Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
> >
> > sched_migration_cost regression
> > ---------------------- -------------
> > 50000 ~6%
> > 0 ~8%
> > 500000 (default) ~10%
> > 5000000 ~14%
> > -1 ~19%
>
> at 50000 (~6%), is the predominant difference in schedstats still the
> nr_failed_migrations_hot?
Yes, it's strange that nr_failed_migrations_hot (cost=50000) is larger
than (cost=500000)
sched_migration_cost = 50000
----------------------------------------------------------
se.exec_start : 3475158.689868
se.vruntime : 935215.526989
se.sum_exec_runtime : 14358.431942
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 3475158.689868
se.block_start : 0.000000
se.sleep_max : 37.910779
se.block_max : 0.033796
se.exec_max : 0.121028
se.slice_max : 0.000000
se.wait_max : 6.007209
se.wait_sum : 25029.649588
se.wait_count : 1954948
sched_info.bkl_count : 0
se.nr_migrations : 2521
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 147
se.nr_failed_migrations_hot : 123
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1236
se.nr_wakeups : 1952827
se.nr_wakeups_sync : 1944785
se.nr_wakeups_migrate : 2479
se.nr_wakeups_local : 1930122
se.nr_wakeups_remote : 22705
se.nr_wakeups_affine : 13252
se.nr_wakeups_affine_attempts : 825950
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007355
avg_per_cpu : 5.695530
nr_switches : 1951994
nr_voluntary_switches : 1949963
nr_involuntary_switches : 2031
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 81
sched_migration_cost = 500000
----------------------------------------------------------
se.exec_start : 3719706.806593
se.vruntime : 1049301.798712
se.sum_exec_runtime : 14636.764384
se.avg_overlap : 0.000121
se.wait_start : 0.000000
se.sleep_start : 3719706.806593
se.block_start : 0.000000
se.sleep_max : 214.989106
se.block_max : 0.036240
se.exec_max : 0.147829
se.slice_max : 0.000000
se.wait_max : 6.475864
se.wait_sum : 24932.530233
se.wait_count : 1922627
sched_info.bkl_count : 0
se.nr_migrations : 2314
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 71
se.nr_failed_migrations_hot : 76
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1735
se.nr_wakeups : 1920337
se.nr_wakeups_sync : 1908767
se.nr_wakeups_migrate : 2272
se.nr_wakeups_local : 1898194
se.nr_wakeups_remote : 22143
se.nr_wakeups_affine : 1218
se.nr_wakeups_affine_attempts : 820118
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007624
avg_per_cpu : 6.325308
nr_switches : 1919596
nr_voluntary_switches : 1917396
nr_involuntary_switches : 2200
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 69
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 12:12 ` Lin Ming
@ 2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
` (3 more replies)
1 sibling, 4 replies; 19+ messages in thread
From: Gregory Haskins @ 2008-09-04 13:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: peterz, Lin Ming, linux-kernel, yanmin_zhang, mingo
Hi Peter,
Peter Zijlstra wrote:
>
>
> Also, we used to have some auto-tuning in there, which dissapeared some
> time ago, gregory brought it back to live recently, perhaps he likes to
> share? :-)
>
Per your request, I brought those "revert" patches I had in the PQ branch
forward to 2.6.27-rc5. They are completely unbuilt and untested, but here
for your experimentation.
-Greg
---
Gregory Haskins (4):
sched: make task_hot() once again use sd->cache_hot_time
Revert "sched: zap the migration init / cache-hot balancing code"
Revert "[PATCH] sched: remove cache_hot_time"
revert "sched: sched_cacheflush is now unused"
Documentation/kernel-parameters.txt | 43 +++
arch/ia64/kernel/setup.c | 15 +
arch/mips/kernel/smp.c | 11 +
arch/sparc/kernel/smp.c | 10 +
arch/sparc64/kernel/smp.c | 27 ++
arch/x86/kernel/smpboot.c | 12 +
include/asm-m32r/system.h | 10 +
include/asm-mips/system.h | 10 +
include/asm-parisc/system.h | 11 +
include/asm-x86/system.h | 9 +
include/linux/sched.h | 7
include/linux/topology.h | 1
kernel/sched.c | 499 ++++++++++++++++++++++++++++++++++-
13 files changed, 656 insertions(+), 9 deletions(-)
--
Signature
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 1/4] revert "sched: sched_cacheflush is now unused"
2008-09-04 13:50 ` Gregory Haskins
@ 2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time" Gregory Haskins
` (2 subsequent siblings)
3 siblings, 0 replies; 19+ messages in thread
From: Gregory Haskins @ 2008-09-04 13:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: peterz, Lin Ming, linux-kernel, yanmin_zhang, mingo
revert c41917df8a1adde34864116ce2231a7fe308d2ff
Author: Ralf Baechle <ralf@linux-mips.org>
Date: Thu Jul 19 21:28:35 2007 +0200
[PATCH] sched: sched_cacheflush is now unused
Since Ingo's recent scheduler rewrite which was merged as commit
0437e109e1841607f2988891eaa36c531c6aa6ac sched_cacheflush is unused.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
We want to restore sd->cache_hot_time for better migration decisions.
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
---
arch/ia64/kernel/setup.c | 9 +++++++++
include/asm-m32r/system.h | 10 ++++++++++
include/asm-mips/system.h | 10 ++++++++++
include/asm-parisc/system.h | 11 +++++++++++
include/asm-x86/system.h | 9 +++++++++
5 files changed, 49 insertions(+), 0 deletions(-)
diff --git a/arch/ia64/kernel/setup.c b/arch/ia64/kernel/setup.c
index c27d5b2..c0050ab 100644
--- a/arch/ia64/kernel/setup.c
+++ b/arch/ia64/kernel/setup.c
@@ -1051,6 +1051,15 @@ cpu_init (void)
pm_idle = default_idle;
}
+/*
+ * On SMP systems, when the scheduler does migration-cost autodetection,
+ * it needs a way to flush as much of the CPU's caches as possible.
+ */
+void sched_cacheflush(void)
+{
+ ia64_sal_cache_flush(3);
+}
+
void __init
check_bugs (void)
{
diff --git a/include/asm-m32r/system.h b/include/asm-m32r/system.h
index 70a57c8..d2f9559 100644
--- a/include/asm-m32r/system.h
+++ b/include/asm-m32r/system.h
@@ -54,6 +54,16 @@
); \
} while(0)
+/*
+ * On SMP systems, when the scheduler does migration-cost autodetection,
+ * it needs a way to flush as much of the CPU's caches as possible.
+ *
+ * TODO: fill this in!
+ */
+static inline void sched_cacheflush(void)
+{
+}
+
/* Interrupt Control */
#if !defined(CONFIG_CHIP_M32102) && !defined(CONFIG_CHIP_M32104)
#define local_irq_enable() \
diff --git a/include/asm-mips/system.h b/include/asm-mips/system.h
index a944eda..aaba821 100644
--- a/include/asm-mips/system.h
+++ b/include/asm-mips/system.h
@@ -78,6 +78,16 @@ do { \
write_c0_userlocal(current_thread_info()->tp_value); \
} while (0)
+/*
+ * On SMP systems, when the scheduler does migration-cost autodetection,
+ * it needs a way to flush as much of the CPU's caches as possible.
+ *
+ * TODO: fill this in!
+ */
+static inline void sched_cacheflush(void)
+{
+}
+
static inline unsigned long __xchg_u32(volatile int * m, unsigned int val)
{
__u32 retval;
diff --git a/include/asm-parisc/system.h b/include/asm-parisc/system.h
index ee80c92..21fbfc5 100644
--- a/include/asm-parisc/system.h
+++ b/include/asm-parisc/system.h
@@ -48,6 +48,17 @@ extern struct task_struct *_switch_to(struct task_struct *, struct task_struct *
(last) = _switch_to(prev, next); \
} while(0)
+/*
+ * On SMP systems, when the scheduler does migration-cost autodetection,
+ * it needs a way to flush as much of the CPU's caches as possible.
+ *
+ * TODO: fill this in!
+ */
+static inline void sched_cacheflush(void)
+{
+}
+
+
/* interrupt control */
#define local_save_flags(x) __asm__ __volatile__("ssm 0, %0" : "=r" (x) : : "memory")
#define local_irq_disable() __asm__ __volatile__("rsm %0,%%r0\n" : : "i" (PSW_I) : "memory" )
diff --git a/include/asm-x86/system.h b/include/asm-x86/system.h
index 983ce37..7847702 100644
--- a/include/asm-x86/system.h
+++ b/include/asm-x86/system.h
@@ -306,6 +306,15 @@ void enable_hlt(void);
void cpu_idle_wait(void);
+/*
+ * On SMP systems, when the scheduler does migration-cost autodetection,
+ * it needs a way to flush as much of the CPU's caches as possible:
+ */
+static inline void sched_cacheflush(void)
+{
+ wbinvd();
+}
+
extern unsigned long arch_align_stack(unsigned long sp);
extern void free_init_pages(char *what, unsigned long begin, unsigned long end);
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time"
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
@ 2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code" Gregory Haskins
2008-09-04 13:50 ` [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time Gregory Haskins
3 siblings, 0 replies; 19+ messages in thread
From: Gregory Haskins @ 2008-09-04 13:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: peterz, Lin Ming, linux-kernel, yanmin_zhang, mingo
This reverts commit 362a7016637648c6aefc98b706298baedfaa1543.
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
---
include/linux/sched.h | 1 +
include/linux/topology.h | 1 +
kernel/sched.c | 8 +++++---
3 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index cfb0d87..5619f3c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -777,6 +777,7 @@ struct sched_domain {
unsigned long max_interval; /* Maximum balance interval ms */
unsigned int busy_factor; /* less balancing by factor if busy */
unsigned int imbalance_pct; /* No balance until over watermark */
+ unsigned long long cache_hot_time; /* Task considered cache hot (ns) */
unsigned int cache_nice_tries; /* Leave cache hot tasks for # tries */
unsigned int busy_idx;
unsigned int idle_idx;
diff --git a/include/linux/topology.h b/include/linux/topology.h
index 2158fc0..f6a7928 100644
--- a/include/linux/topology.h
+++ b/include/linux/topology.h
@@ -162,6 +162,7 @@ void arch_update_cpu_topology(void);
.max_interval = 64*num_online_cpus(), \
.busy_factor = 128, \
.imbalance_pct = 133, \
+ .cache_hot_time = (10*1000000), \
.cache_nice_tries = 1, \
.busy_idx = 3, \
.idle_idx = 3, \
diff --git a/kernel/sched.c b/kernel/sched.c
index 9a1ddb8..0ca5218 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -6223,7 +6223,7 @@ set_table_entry(struct ctl_table *entry,
static struct ctl_table *
sd_alloc_ctl_domain_table(struct sched_domain *sd)
{
- struct ctl_table *table = sd_alloc_ctl_entry(12);
+ struct ctl_table *table = sd_alloc_ctl_entry(13);
if (table == NULL)
return NULL;
@@ -6249,9 +6249,11 @@ sd_alloc_ctl_domain_table(struct sched_domain *sd)
set_table_entry(&table[9], "cache_nice_tries",
&sd->cache_nice_tries,
sizeof(int), 0644, proc_dointvec_minmax);
- set_table_entry(&table[10], "flags", &sd->flags,
+ set_table_entry(&table[10], "cache_hot_time", &sd->cache_hot_time,
+ sizeof(long long), 0644, proc_doulongvec_minmax);
+ set_table_entry(&table[11], "flags", &sd->flags,
sizeof(int), 0644, proc_dointvec_minmax);
- /* &table[11] is terminator */
+ /* &table[12] is terminator */
return table;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code"
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
2008-09-04 13:50 ` [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time" Gregory Haskins
@ 2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time Gregory Haskins
3 siblings, 0 replies; 19+ messages in thread
From: Gregory Haskins @ 2008-09-04 13:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: peterz, Lin Ming, linux-kernel, yanmin_zhang, mingo
>From commit: 0437e109e1841607f2988891eaa36c531c6aa6ac
We want to restore the concept of using empirical cache_hot data for
managing migrations.
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
---
Documentation/kernel-parameters.txt | 43 +++
arch/ia64/kernel/setup.c | 6
arch/mips/kernel/smp.c | 11 +
arch/sparc/kernel/smp.c | 10 +
arch/sparc64/kernel/smp.c | 27 ++
arch/x86/kernel/smpboot.c | 12 +
include/linux/sched.h | 6
kernel/sched.c | 483 +++++++++++++++++++++++++++++++++++
8 files changed, 598 insertions(+), 0 deletions(-)
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 1150444..6e7b78f 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1247,6 +1247,49 @@ and is between 256 and 4096 characters. It is defined in the file
mga= [HW,DRM]
+ migration_cost=
+ [KNL,SMP] debug: override scheduler migration costs
+ Format: <level-1-usecs>,<level-2-usecs>,...
+ This debugging option can be used to override the
+ default scheduler migration cost matrix. The numbers
+ are indexed by 'CPU domain distance'.
+ E.g. migration_cost=1000,2000,3000 on an SMT NUMA
+ box will set up an intra-core migration cost of
+ 1 msec, an inter-core migration cost of 2 msecs,
+ and an inter-node migration cost of 3 msecs.
+
+ WARNING: using the wrong values here can break
+ scheduler performance, so it's only for scheduler
+ development purposes, not production environments.
+
+ migration_debug=
+ [KNL,SMP] migration cost auto-detect verbosity
+ Format=<0|1|2>
+ If a system's migration matrix reported at bootup
+ seems erroneous then this option can be used to
+ increase verbosity of the detection process.
+ We default to 0 (no extra messages), 1 will print
+ some more information, and 2 will be really
+ verbose (probably only useful if you also have a
+ serial console attached to the system).
+
+ migration_factor=
+ [KNL,SMP] multiply/divide migration costs by a factor
+ Format=<percent>
+ This debug option can be used to proportionally
+ increase or decrease the auto-detected migration
+ costs for all entries of the migration matrix.
+ E.g. migration_factor=150 will increase migration
+ costs by 50%. (and thus the scheduler will be less
+ eager migrating cache-hot tasks)
+ migration_factor=80 will decrease migration costs
+ by 20%. (thus the scheduler will be more eager to
+ migrate tasks)
+
+ WARNING: using the wrong values here can break
+ scheduler performance, so it's only for scheduler
+ development purposes, not production environments.
+
mminit_loglevel=
[KNL] When CONFIG_DEBUG_MEMORY_INIT is set, this
parameter allows control of the logging verbosity for
diff --git a/arch/ia64/kernel/setup.c b/arch/ia64/kernel/setup.c
index c0050ab..d2e1724 100644
--- a/arch/ia64/kernel/setup.c
+++ b/arch/ia64/kernel/setup.c
@@ -854,6 +854,7 @@ static void __cpuinit
get_max_cacheline_size (void)
{
unsigned long line_size, max = 1;
+ unsigned int cache_size = 0;
u64 l, levels, unique_caches;
pal_cache_config_info_t cci;
s64 status;
@@ -883,6 +884,8 @@ get_max_cacheline_size (void)
line_size = 1 << cci.pcci_line_size;
if (line_size > max)
max = line_size;
+ if (cache_size < cci.pcci_cache_size)
+ cache_size = cci.pcci_cache_size;
if (!cci.pcci_unified) {
status = ia64_pal_cache_config_info(l,
/* cache_type (instruction)= */ 1,
@@ -899,6 +902,9 @@ get_max_cacheline_size (void)
ia64_i_cache_stride_shift = cci.pcci_stride;
}
out:
+#ifdef CONFIG_SMP
+ max_cache_size = max(max_cache_size, cache_size);
+#endif
if (max > ia64_max_cacheline_size)
ia64_max_cacheline_size = max;
}
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 4410f17..cb63c56 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -53,6 +53,16 @@ int __cpu_logical_map[NR_CPUS]; /* Map logical to physical */
EXPORT_SYMBOL(phys_cpu_present_map);
EXPORT_SYMBOL(cpu_online_map);
+/* This happens early in bootup, can't really do it better */
+static void smp_tune_scheduling (void)
+{
+ struct cache_desc *cd = ¤t_cpu_data.scache;
+ unsigned long cachesize = cd->linesz * cd->sets * cd->ways;
+
+ if (cachesize > max_cache_size)
+ max_cache_size = cachesize;
+}
+
extern void cpu_idle(void);
/* Number of TCs (or siblings in Intel speak) per CPU core */
@@ -181,6 +191,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
{
init_new_context(current, &init_mm);
current_thread_info()->cpu = 0;
+ smp_tune_scheduling();
mp_ops->prepare_cpus(max_cpus);
set_cpu_sibling_map(0);
#ifndef CONFIG_HOTPLUG_CPU
diff --git a/arch/sparc/kernel/smp.c b/arch/sparc/kernel/smp.c
index 1619ec1..44a0448 100644
--- a/arch/sparc/kernel/smp.c
+++ b/arch/sparc/kernel/smp.c
@@ -63,6 +63,16 @@ void __cpuinit smp_store_cpu_info(int id)
cpu_data(id).prom_node = cpu_node;
cpu_data(id).mid = cpu_get_hwmid(cpu_node);
+ /* this is required to tune the scheduler correctly */
+ /* is it possible to have CPUs with different cache sizes? */
+ if (id == boot_cpu_id) {
+ int cache_line,cache_nlines;
+ cache_line = 0x20;
+ cache_line = prom_getintdefault(cpu_node, "ecache-line-size", cache_line);
+ cache_nlines = 0x8000;
+ cache_nlines = prom_getintdefault(cpu_node, "ecache-nlines", cache_nlines);
+ max_cache_size = cache_line * cache_nlines;
+ }
if (cpu_data(id).mid < 0)
panic("No MID found for CPU%d at node 0x%08d", id, cpu_node);
}
diff --git a/arch/sparc64/kernel/smp.c b/arch/sparc64/kernel/smp.c
index 743ccad..926072b 100644
--- a/arch/sparc64/kernel/smp.c
+++ b/arch/sparc64/kernel/smp.c
@@ -1174,8 +1174,35 @@ int setup_profiling_timer(unsigned int multiplier)
return -EINVAL;
}
+static void __init smp_tune_scheduling(void)
+{
+ unsigned int smallest = ~0U;
+ int i;
+
+ for (i = 0; i < NR_CPUS; i++) {
+ unsigned int val = cpu_data(i).ecache_size;
+
+ if (val && val < smallest)
+ smallest = val;
+ }
+
+ /* Any value less than 256K is nonsense. */
+ if (smallest < (256U * 1024U))
+ smallest = 256 * 1024;
+
+ max_cache_size = smallest;
+
+ if (smallest < 1U * 1024U * 1024U)
+ printk(KERN_INFO "Using max_cache_size of %uKB\n",
+ smallest / 1024U);
+ else
+ printk(KERN_INFO "Using max_cache_size of %uMB\n",
+ smallest / 1024U / 1024U);
+}
+
void __init smp_prepare_cpus(unsigned int max_cpus)
{
+ smp_tune_scheduling();
}
void __devinit smp_prepare_boot_cpu(void)
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 7985c5b..c98fdc5 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -1029,6 +1029,17 @@ int __cpuinit native_cpu_up(unsigned int cpu)
return 0;
}
+static void smp_tune_scheduling(void)
+{
+ if (cpu_khz) {
+ /* cache size in kB */
+ long cachesize = boot_cpu_data.x86_cache_size;
+
+ if (cachesize > 0)
+ max_cache_size = cachesize * 1024;
+ }
+}
+
/*
* Fall back to non SMP mode after errors.
*
@@ -1177,6 +1188,7 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus)
smp_store_cpu_info(0); /* Final full version of the data */
boot_cpu_logical_apicid = logical_smp_processor_id();
current_thread_info()->cpu = 0; /* needed? */
+ smp_tune_scheduling();
set_cpu_sibling_map(0);
if (smp_sanity_check(max_cpus) < 0) {
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 5619f3c..5046e3a 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -831,6 +831,12 @@ extern void partition_sched_domains(int ndoms_new, cpumask_t *doms_new,
struct sched_domain_attr *dattr_new);
extern int arch_reinit_sched_domains(void);
+/*
+ * Maximum cache size the migration-costs auto-tuning code will
+ * search from:
+ */
+extern unsigned int max_cache_size;
+
#else /* CONFIG_SMP */
struct sched_domain_attr;
diff --git a/kernel/sched.c b/kernel/sched.c
index 0ca5218..fd28b64 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -6838,6 +6838,483 @@ init_sched_build_groups(const cpumask_t *span, const cpumask_t *cpu_map,
#define SD_NODES_PER_DOMAIN 16
+/*
+ * Self-tuning task migration cost measurement between source and target CPUs.
+ *
+ * This is done by measuring the cost of manipulating buffers of varying
+ * sizes. For a given buffer-size here are the steps that are taken:
+ *
+ * 1) the source CPU reads+dirties a shared buffer
+ * 2) the target CPU reads+dirties the same shared buffer
+ *
+ * We measure how long they take, in the following 4 scenarios:
+ *
+ * - source: CPU1, target: CPU2 | cost1
+ * - source: CPU2, target: CPU1 | cost2
+ * - source: CPU1, target: CPU1 | cost3
+ * - source: CPU2, target: CPU2 | cost4
+ *
+ * We then calculate the cost3+cost4-cost1-cost2 difference - this is
+ * the cost of migration.
+ *
+ * We then start off from a small buffer-size and iterate up to larger
+ * buffer sizes, in 5% steps - measuring each buffer-size separately, and
+ * doing a maximum search for the cost. (The maximum cost for a migration
+ * normally occurs when the working set size is around the effective cache
+ * size.)
+ */
+#define SEARCH_SCOPE 2
+#define MIN_CACHE_SIZE (64*1024U)
+#define DEFAULT_CACHE_SIZE (5*1024*1024U)
+#define ITERATIONS 1
+#define SIZE_THRESH 130
+#define COST_THRESH 130
+
+/*
+ * The migration cost is a function of 'domain distance'. Domain
+ * distance is the number of steps a CPU has to iterate down its
+ * domain tree to share a domain with the other CPU. The farther
+ * two CPUs are from each other, the larger the distance gets.
+ *
+ * Note that we use the distance only to cache measurement results,
+ * the distance value is not used numerically otherwise. When two
+ * CPUs have the same distance it is assumed that the migration
+ * cost is the same. (this is a simplification but quite practical)
+ */
+#define MAX_DOMAIN_DISTANCE 32
+
+static unsigned long long migration_cost[MAX_DOMAIN_DISTANCE] =
+ { [ 0 ... MAX_DOMAIN_DISTANCE-1 ] =
+/*
+ * Architectures may override the migration cost and thus avoid
+ * boot-time calibration. Unit is nanoseconds. Mostly useful for
+ * virtualized hardware:
+ */
+#ifdef CONFIG_DEFAULT_MIGRATION_COST
+ CONFIG_DEFAULT_MIGRATION_COST
+#else
+ -1LL
+#endif
+};
+
+/*
+ * Allow override of migration cost - in units of microseconds.
+ * E.g. migration_cost=1000,2000,3000 will set up a level-1 cost
+ * of 1 msec, level-2 cost of 2 msecs and level3 cost of 3 msecs:
+ */
+static int __init migration_cost_setup(char *str)
+{
+ int ints[MAX_DOMAIN_DISTANCE+1], i;
+
+ str = get_options(str, ARRAY_SIZE(ints), ints);
+
+ printk("#ints: %d\n", ints[0]);
+ for (i = 1; i <= ints[0]; i++) {
+ migration_cost[i-1] = (unsigned long long)ints[i]*1000;
+ printk("migration_cost[%d]: %Ld\n", i-1, migration_cost[i-1]);
+ }
+ return 1;
+}
+
+__setup ("migration_cost=", migration_cost_setup);
+
+/*
+ * Global multiplier (divisor) for migration-cutoff values,
+ * in percentiles. E.g. use a value of 150 to get 1.5 times
+ * longer cache-hot cutoff times.
+ *
+ * (We scale it from 100 to 128 to long long handling easier.)
+ */
+
+#define MIGRATION_FACTOR_SCALE 128
+
+static unsigned int migration_factor = MIGRATION_FACTOR_SCALE;
+
+static int __init setup_migration_factor(char *str)
+{
+ get_option(&str, &migration_factor);
+ migration_factor = migration_factor * MIGRATION_FACTOR_SCALE / 100;
+ return 1;
+}
+
+__setup("migration_factor=", setup_migration_factor);
+
+/*
+ * Estimated distance of two CPUs, measured via the number of domains
+ * we have to pass for the two CPUs to be in the same span:
+ */
+static unsigned long domain_distance(int cpu1, int cpu2)
+{
+ unsigned long distance = 0;
+ struct sched_domain *sd;
+
+ for_each_domain(cpu1, sd) {
+ WARN_ON(!cpu_isset(cpu1, sd->span));
+ if (cpu_isset(cpu2, sd->span))
+ return distance;
+ distance++;
+ }
+ if (distance >= MAX_DOMAIN_DISTANCE) {
+ WARN_ON(1);
+ distance = MAX_DOMAIN_DISTANCE-1;
+ }
+
+ return distance;
+}
+
+static unsigned int migration_debug;
+
+static int __init setup_migration_debug(char *str)
+{
+ get_option(&str, &migration_debug);
+ return 1;
+}
+
+__setup("migration_debug=", setup_migration_debug);
+
+/*
+ * Maximum cache-size that the scheduler should try to measure.
+ * Architectures with larger caches should tune this up during
+ * bootup. Gets used in the domain-setup code (i.e. during SMP
+ * bootup).
+ */
+unsigned int max_cache_size;
+
+static int __init setup_max_cache_size(char *str)
+{
+ get_option(&str, &max_cache_size);
+ return 1;
+}
+
+__setup("max_cache_size=", setup_max_cache_size);
+
+/*
+ * Dirty a big buffer in a hard-to-predict (for the L2 cache) way. This
+ * is the operation that is timed, so we try to generate unpredictable
+ * cachemisses that still end up filling the L2 cache:
+ */
+static void touch_cache(void *__cache, unsigned long __size)
+{
+ unsigned long size = __size / sizeof(long);
+ unsigned long chunk1 = size / 3;
+ unsigned long chunk2 = 2 * size / 3;
+ unsigned long *cache = __cache;
+ int i;
+
+ for (i = 0; i < size/6; i += 8) {
+ switch (i % 6) {
+ case 0: cache[i]++;
+ case 1: cache[size-1-i]++;
+ case 2: cache[chunk1-i]++;
+ case 3: cache[chunk1+i]++;
+ case 4: cache[chunk2-i]++;
+ case 5: cache[chunk2+i]++;
+ }
+ }
+}
+
+/*
+ * Measure the cache-cost of one task migration. Returns in units of nsec.
+ */
+static unsigned long long
+measure_one(void *cache, unsigned long size, int source, int target)
+{
+ cpumask_t mask, saved_mask;
+ unsigned long long t0, t1, t2, t3, cost;
+
+ saved_mask = current->cpus_allowed;
+
+ /*
+ * Flush source caches to RAM and invalidate them:
+ */
+ sched_cacheflush();
+
+ /*
+ * Migrate to the source CPU:
+ */
+ mask = cpumask_of_cpu(source);
+ set_cpus_allowed(current, mask);
+ WARN_ON(smp_processor_id() != source);
+
+ /*
+ * Dirty the working set:
+ */
+ t0 = sched_clock();
+ touch_cache(cache, size);
+ t1 = sched_clock();
+
+ /*
+ * Migrate to the target CPU, dirty the L2 cache and access
+ * the shared buffer. (which represents the working set
+ * of a migrated task.)
+ */
+ mask = cpumask_of_cpu(target);
+ set_cpus_allowed(current, mask);
+ WARN_ON(smp_processor_id() != target);
+
+ t2 = sched_clock();
+ touch_cache(cache, size);
+ t3 = sched_clock();
+
+ cost = t1-t0 + t3-t2;
+
+ if (migration_debug >= 2)
+ printk("[%d->%d]: %8Ld %8Ld %8Ld => %10Ld.\n",
+ source, target, t1-t0, t1-t0, t3-t2, cost);
+ /*
+ * Flush target caches to RAM and invalidate them:
+ */
+ sched_cacheflush();
+
+ set_cpus_allowed(current, saved_mask);
+
+ return cost;
+}
+
+/*
+ * Measure a series of task migrations and return the average
+ * result. Since this code runs early during bootup the system
+ * is 'undisturbed' and the average latency makes sense.
+ *
+ * The algorithm in essence auto-detects the relevant cache-size,
+ * so it will properly detect different cachesizes for different
+ * cache-hierarchies, depending on how the CPUs are connected.
+ *
+ * Architectures can prime the upper limit of the search range via
+ * max_cache_size, otherwise the search range defaults to 20MB...64K.
+ */
+static unsigned long long
+measure_cost(int cpu1, int cpu2, void *cache, unsigned int size)
+{
+ unsigned long long cost1, cost2;
+ int i;
+
+ /*
+ * Measure the migration cost of 'size' bytes, over an
+ * average of 10 runs:
+ *
+ * (We perturb the cache size by a small (0..4k)
+ * value to compensate size/alignment related artifacts.
+ * We also subtract the cost of the operation done on
+ * the same CPU.)
+ */
+ cost1 = 0;
+
+ /*
+ * dry run, to make sure we start off cache-cold on cpu1,
+ * and to get any vmalloc pagefaults in advance:
+ */
+ measure_one(cache, size, cpu1, cpu2);
+ for (i = 0; i < ITERATIONS; i++)
+ cost1 += measure_one(cache, size - i * 1024, cpu1, cpu2);
+
+ measure_one(cache, size, cpu2, cpu1);
+ for (i = 0; i < ITERATIONS; i++)
+ cost1 += measure_one(cache, size - i * 1024, cpu2, cpu1);
+
+ /*
+ * (We measure the non-migrating [cached] cost on both
+ * cpu1 and cpu2, to handle CPUs with different speeds)
+ */
+ cost2 = 0;
+
+ measure_one(cache, size, cpu1, cpu1);
+ for (i = 0; i < ITERATIONS; i++)
+ cost2 += measure_one(cache, size - i * 1024, cpu1, cpu1);
+
+ measure_one(cache, size, cpu2, cpu2);
+ for (i = 0; i < ITERATIONS; i++)
+ cost2 += measure_one(cache, size - i * 1024, cpu2, cpu2);
+
+ /*
+ * Get the per-iteration migration cost:
+ */
+ do_div(cost1, 2 * ITERATIONS);
+ do_div(cost2, 2 * ITERATIONS);
+
+ return cost1 - cost2;
+}
+
+static unsigned long long measure_migration_cost(int cpu1, int cpu2)
+{
+ unsigned long long max_cost = 0, fluct = 0, avg_fluct = 0;
+ unsigned int max_size, size, size_found = 0;
+ long long cost = 0, prev_cost;
+ void *cache;
+
+ /*
+ * Search from max_cache_size*5 down to 64K - the real relevant
+ * cachesize has to lie somewhere inbetween.
+ */
+ if (max_cache_size) {
+ max_size = max(max_cache_size * SEARCH_SCOPE, MIN_CACHE_SIZE);
+ size = max(max_cache_size / SEARCH_SCOPE, MIN_CACHE_SIZE);
+ } else {
+ /*
+ * Since we have no estimation about the relevant
+ * search range
+ */
+ max_size = DEFAULT_CACHE_SIZE * SEARCH_SCOPE;
+ size = MIN_CACHE_SIZE;
+ }
+
+ if (!cpu_online(cpu1) || !cpu_online(cpu2)) {
+ printk("cpu %d and %d not both online!\n", cpu1, cpu2);
+ return 0;
+ }
+
+ /*
+ * Allocate the working set:
+ */
+ cache = vmalloc(max_size);
+ if (!cache) {
+ printk("could not vmalloc %d bytes for cache!\n", 2 * max_size);
+ return 1000000; /* return 1 msec on very small boxen */
+ }
+
+ while (size <= max_size) {
+ prev_cost = cost;
+ cost = measure_cost(cpu1, cpu2, cache, size);
+
+ /*
+ * Update the max:
+ */
+ if (cost > 0) {
+ if (max_cost < cost) {
+ max_cost = cost;
+ size_found = size;
+ }
+ }
+ /*
+ * Calculate average fluctuation, we use this to prevent
+ * noise from triggering an early break out of the loop:
+ */
+ fluct = abs(cost - prev_cost);
+ avg_fluct = (avg_fluct + fluct)/2;
+
+ if (migration_debug)
+ printk("-> [%d][%d][%7d] %3ld.%ld [%3ld.%ld] (%ld): "
+ "(%8Ld %8Ld)\n",
+ cpu1, cpu2, size,
+ (long)cost / 1000000,
+ ((long)cost / 100000) % 10,
+ (long)max_cost / 1000000,
+ ((long)max_cost / 100000) % 10,
+ domain_distance(cpu1, cpu2),
+ cost, avg_fluct);
+
+ /*
+ * If we iterated at least 20% past the previous maximum,
+ * and the cost has dropped by more than 20% already,
+ * (taking fluctuations into account) then we assume to
+ * have found the maximum and break out of the loop early:
+ */
+ if (size_found && (size*100 > size_found*SIZE_THRESH))
+ if (cost+avg_fluct <= 0 ||
+ max_cost*100 > (cost+avg_fluct)*COST_THRESH) {
+
+ if (migration_debug)
+ printk("-> found max.\n");
+ break;
+ }
+ /*
+ * Increase the cachesize in 10% steps:
+ */
+ size = size * 10 / 9;
+ }
+
+ if (migration_debug)
+ printk("[%d][%d] working set size found: %d, cost: %Ld\n",
+ cpu1, cpu2, size_found, max_cost);
+
+ vfree(cache);
+
+ /*
+ * A task is considered 'cache cold' if at least 2 times
+ * the worst-case cost of migration has passed.
+ *
+ * (this limit is only listened to if the load-balancing
+ * situation is 'nice' - if there is a large imbalance we
+ * ignore it for the sake of CPU utilization and
+ * processing fairness.)
+ */
+ return 2 * max_cost * migration_factor / MIGRATION_FACTOR_SCALE;
+}
+
+static void calibrate_migration_costs(const cpumask_t *cpu_map)
+{
+ int cpu1 = -1, cpu2 = -1, cpu, orig_cpu = raw_smp_processor_id();
+ unsigned long j0, j1, distance, max_distance = 0;
+ struct sched_domain *sd;
+
+ j0 = jiffies;
+
+ /*
+ * First pass - calculate the cacheflush times:
+ */
+ for_each_cpu_mask(cpu1, *cpu_map) {
+ for_each_cpu_mask(cpu2, *cpu_map) {
+ if (cpu1 == cpu2)
+ continue;
+ distance = domain_distance(cpu1, cpu2);
+ max_distance = max(max_distance, distance);
+ /*
+ * No result cached yet?
+ */
+ if (migration_cost[distance] == -1LL)
+ migration_cost[distance] =
+ measure_migration_cost(cpu1, cpu2);
+ }
+ }
+ /*
+ * Second pass - update the sched domain hierarchy with
+ * the new cache-hot-time estimations:
+ */
+ for_each_cpu_mask(cpu, *cpu_map) {
+ distance = 0;
+ for_each_domain(cpu, sd) {
+ sd->cache_hot_time = migration_cost[distance];
+ distance++;
+ }
+ }
+ /*
+ * Print the matrix:
+ */
+ if (migration_debug)
+ printk("migration: max_cache_size: %d, cpu: %d MHz:\n",
+ max_cache_size,
+#ifdef CONFIG_X86
+ cpu_khz/1000
+#else
+ -1
+#endif
+ );
+ if (system_state == SYSTEM_BOOTING && num_online_cpus() > 1) {
+ printk("migration_cost=");
+ for (distance = 0; distance <= max_distance; distance++) {
+ if (distance)
+ printk(",");
+ printk("%ld", (long)migration_cost[distance] / 1000);
+ }
+ printk("\n");
+ }
+ j1 = jiffies;
+ if (migration_debug)
+ printk("migration: %ld seconds\n", (j1-j0) / HZ);
+
+ /*
+ * Move back to the original CPU. NUMA-Q gets confused
+ * if we migrate to another quad during bootup.
+ */
+ if (raw_smp_processor_id() != orig_cpu) {
+ cpumask_t mask = cpumask_of_cpu(orig_cpu),
+ saved_mask = current->cpus_allowed;
+
+ set_cpus_allowed(current, mask);
+ set_cpus_allowed(current, saved_mask);
+ }
+}
+
#ifdef CONFIG_NUMA
/**
@@ -7528,6 +8005,12 @@ static int __build_sched_domains(const cpumask_t *cpu_map,
}
SCHED_CPUMASK_FREE((void *)allmasks);
+
+ /*
+ * Tune cache-hot values:
+ */
+ calibrate_migration_costs(cpu_map);
+
return 0;
#ifdef CONFIG_NUMA
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time
2008-09-04 13:50 ` Gregory Haskins
` (2 preceding siblings ...)
2008-09-04 13:50 ` [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code" Gregory Haskins
@ 2008-09-04 13:50 ` Gregory Haskins
3 siblings, 0 replies; 19+ messages in thread
From: Gregory Haskins @ 2008-09-04 13:50 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: peterz, Lin Ming, linux-kernel, yanmin_zhang, mingo
Signed-off-by: Gregory Haskins <ghaskins@novell.com>
---
kernel/sched.c | 8 ++------
1 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/kernel/sched.c b/kernel/sched.c
index fd28b64..b5f8640 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1778,6 +1778,7 @@ static int
task_hot(struct task_struct *p, u64 now, struct sched_domain *sd)
{
s64 delta;
+ s64 migration_cost = (s64)(sd ? sd->cache_hot_time : sysctl_sched_migration_cost);
/*
* Buddy candidates are cache hot:
@@ -1788,14 +1789,9 @@ task_hot(struct task_struct *p, u64 now, struct sched_domain *sd)
if (p->sched_class != &fair_sched_class)
return 0;
- if (sysctl_sched_migration_cost == -1)
- return 1;
- if (sysctl_sched_migration_cost == 0)
- return 0;
-
delta = now - p->se.exec_start;
- return delta < (s64)sysctl_sched_migration_cost;
+ return delta < migration_cost;
}
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 9:03 ` Peter Zijlstra
2008-09-04 10:52 ` Lin Ming
@ 2008-09-05 1:26 ` Lin Ming
1 sibling, 0 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-05 1:26 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo, Gregory Haskins
It does not help after these 4 patches applied.
Gregory Haskins (4):
sched: make task_hot() once again use sd->cache_hot_time
Revert "sched: zap the migration init / cache-hot balancing code"
Revert "[PATCH] sched: remove cache_hot_time"
revert "sched: sched_cacheflush is now unused"
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-04 8:51 oltp ~10% regression with 2.6.27-rc5 on stoakley machine Lin Ming
2008-09-04 9:03 ` Peter Zijlstra
@ 2008-09-20 21:38 ` Peter Zijlstra
2008-09-26 2:00 ` Lin Ming
1 sibling, 1 reply; 19+ messages in thread
From: Peter Zijlstra @ 2008-09-20 21:38 UTC (permalink / raw)
To: Lin Ming; +Cc: linux-kernel, Zhang, Yanmin, mingo, Mike Galbraith
>
> ------- Comment #4 from ming.m.lin@intel.com 2008-09-17 17:55 -------
> sched_switch trace of oltp in 2.6.27-rc4
> http://myfreefilehosting.com/f/fc6c8eaacf_1.31MB
>
> sched_switch trace of oltp in 2.6.27-rc5
> http://myfreefilehosting.com/f/a2f9aea1b0_0.42MB
>
> Compared the 2 trace files, you can find that (at most time),
> with 2.6.27-rc5, mysql switches to sysbench when it's going to sleep
> mysqld-3791 [07] 151.421836: 3791:120:R + 3803:120:S
> mysqld-3791 [07] 151.421876: 3791:120:S ==> 3803:120:R
> sysbench-3803 [07] 151.421878: 3803:120:R + 3791:120:S
> sysbench-3803 [07] 151.421887: 3803:120:S ==> 3791:120:R
>
> with 2.6.27-rc4, mysql switches to sysbench when it's still running
> mysqld-3674 [07] 95.960220: 3674:120:R + 3687:120:S
> mysqld-3674 [07] 95.960220: 3674:120:R ==> 3687:120:R
> sysbench-3687 [07] 95.960220: 3687:120:S ==> 3674:120:R
>
> So with 2.6.27-rc5, sysbench wakes up mysql and then switches to it (mysql is
> in sleep state)
>
> With 2.6.27-rc4, sysbench switches to mysql, no need to wake up it (mysql is
> still in running state)
Ming, how does this work for you?
---
Subject: sched: wakeup preempt when small overlap
Aggresively preempt a task if its avg overlap is very small, this should
avoid the task going to sleep and find it still running when we schedule
back to it - saving a wakeup.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 7b4592c..cb44774 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -897,7 +897,7 @@ struct sched_class {
void (*yield_task) (struct rq *rq);
int (*select_task_rq)(struct task_struct *p, int sync);
- void (*check_preempt_curr) (struct rq *rq, struct task_struct *p);
+ void (*check_preempt_curr) (struct rq *rq, struct task_struct *p, int sync);
struct task_struct * (*pick_next_task) (struct rq *rq);
void (*put_prev_task) (struct rq *rq, struct task_struct *p);
diff --git a/kernel/sched.c b/kernel/sched.c
index 32d56d6..e17d506 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -609,9 +609,9 @@ struct rq {
static DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
-static inline void check_preempt_curr(struct rq *rq, struct task_struct *p)
+static inline void check_preempt_curr(struct rq *rq, struct task_struct *p, int sync)
{
- rq->curr->sched_class->check_preempt_curr(rq, p);
+ rq->curr->sched_class->check_preempt_curr(rq, p, sync);
}
static inline int cpu_of(struct rq *rq)
@@ -2299,7 +2299,7 @@ out_activate:
out_running:
trace_sched_wakeup(rq, p);
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, sync);
p->state = TASK_RUNNING;
#ifdef CONFIG_SMP
@@ -2432,7 +2432,7 @@ void wake_up_new_task(struct task_struct *p, unsigned long clone_flags)
inc_nr_running(rq);
}
trace_sched_wakeup_new(rq, p);
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, 0);
#ifdef CONFIG_SMP
if (p->sched_class->task_wake_up)
p->sched_class->task_wake_up(rq, p);
@@ -2889,7 +2889,7 @@ static void pull_task(struct rq *src_rq, struct task_struct *p,
* Note that idle threads have a prio of MAX_PRIO, for this test
* to be always true for them.
*/
- check_preempt_curr(this_rq, p);
+ check_preempt_curr(this_rq, p, 0);
}
/*
@@ -6015,7 +6015,7 @@ static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)
set_task_cpu(p, dest_cpu);
if (on_rq) {
activate_task(rq_dest, p, 0);
- check_preempt_curr(rq_dest, p);
+ check_preempt_curr(rq_dest, p, 0);
}
done:
ret = 1;
diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index a10ac0b..82a907c 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -1331,7 +1331,7 @@ static inline int depth_se(struct sched_entity *se)
/*
* Preempt the current task with a newly woken task if needed:
*/
-static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
+static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
{
struct task_struct *curr = rq->curr;
struct cfs_rq *cfs_rq = task_cfs_rq(curr);
@@ -1348,6 +1348,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
if (unlikely(se == pse))
return;
+ cfs_rq_of(pse)->next = pse;
+
/*
* We can come here with TIF_NEED_RESCHED already set from new task
* wake up path.
@@ -1355,8 +1357,6 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
if (test_tsk_need_resched(curr))
return;
- cfs_rq_of(pse)->next = pse;
-
/*
* Batch tasks do not preempt (their preemption is driven by
* the tick):
@@ -1367,6 +1367,13 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
if (!sched_feat(WAKEUP_PREEMPT))
return;
+ if (sched_feat(WAKEUP_OVERLAP) && sync &&
+ se->avg_overlap < sysctl_sched_migration_cost &&
+ pse->avg_overlap < sysctl_sched_migration_cost) {
+ resched_task(curr);
+ return;
+ }
+
/*
* preemption test can be made between sibling entities who are in the
* same cfs_rq i.e who have a common parent. Walk up the hierarchy of
@@ -1649,7 +1656,7 @@ static void prio_changed_fair(struct rq *rq, struct task_struct *p,
if (p->prio > oldprio)
resched_task(rq->curr);
} else
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, 0);
}
/*
@@ -1666,7 +1673,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p,
if (running)
resched_task(rq->curr);
else
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, 0);
}
/* Account for a task changing its policy or group.
diff --git a/kernel/sched_features.h b/kernel/sched_features.h
index 9353ca7..bf027a7 100644
--- a/kernel/sched_features.h
+++ b/kernel/sched_features.h
@@ -11,3 +11,4 @@ SCHED_FEAT(ASYM_GRAN, 1)
SCHED_FEAT(LB_BIAS, 1)
SCHED_FEAT(LB_WAKEUP_UPDATE, 1)
SCHED_FEAT(ASYM_EFF_LOAD, 1)
+SCHED_FEAT(WAKEUP_OVERLAP, 1)
diff --git a/kernel/sched_idletask.c b/kernel/sched_idletask.c
index 3a4f92d..dec4cca 100644
--- a/kernel/sched_idletask.c
+++ b/kernel/sched_idletask.c
@@ -14,7 +14,7 @@ static int select_task_rq_idle(struct task_struct *p, int sync)
/*
* Idle tasks are unconditionally rescheduled:
*/
-static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p)
+static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int sync)
{
resched_task(rq->idle);
}
@@ -76,7 +76,7 @@ static void switched_to_idle(struct rq *rq, struct task_struct *p,
if (running)
resched_task(rq->curr);
else
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, 0);
}
static void prio_changed_idle(struct rq *rq, struct task_struct *p,
@@ -93,7 +93,7 @@ static void prio_changed_idle(struct rq *rq, struct task_struct *p,
if (p->prio > oldprio)
resched_task(rq->curr);
} else
- check_preempt_curr(rq, p);
+ check_preempt_curr(rq, p, 0);
}
/*
diff --git a/kernel/sched_rt.c b/kernel/sched_rt.c
index 04875ef..2e228bd 100644
--- a/kernel/sched_rt.c
+++ b/kernel/sched_rt.c
@@ -787,7 +787,7 @@ static void check_preempt_equal_prio(struct rq *rq, struct task_struct *p)
/*
* Preempt the current task with a newly woken task if needed:
*/
-static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p)
+static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p, int sync)
{
if (p->prio < rq->curr->prio) {
resched_task(rq->curr);
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
2008-09-20 21:38 ` Peter Zijlstra
@ 2008-09-26 2:00 ` Lin Ming
0 siblings, 0 replies; 19+ messages in thread
From: Lin Ming @ 2008-09-26 2:00 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-kernel, Zhang, Yanmin, mingo, Mike Galbraith
This patch restores ~5% performance.
kernel peformance
-----------------------------------
27-rc4 100%
27-rc5 89.6%
27-rc5+peter's patch 94.5%
Thanks,
Lin Ming
On Sat, 2008-09-20 at 15:38 -0600, Peter Zijlstra wrote:
> >
> > ------- Comment #4 from ming.m.lin@intel.com 2008-09-17 17:55 -------
> > sched_switch trace of oltp in 2.6.27-rc4
> > http://myfreefilehosting.com/f/fc6c8eaacf_1.31MB
> >
> > sched_switch trace of oltp in 2.6.27-rc5
> > http://myfreefilehosting.com/f/a2f9aea1b0_0.42MB
> >
> > Compared the 2 trace files, you can find that (at most time),
> > with 2.6.27-rc5, mysql switches to sysbench when it's going to sleep
> > mysqld-3791 [07] 151.421836: 3791:120:R + 3803:120:S
> > mysqld-3791 [07] 151.421876: 3791:120:S ==> 3803:120:R
> > sysbench-3803 [07] 151.421878: 3803:120:R + 3791:120:S
> > sysbench-3803 [07] 151.421887: 3803:120:S ==> 3791:120:R
> >
> > with 2.6.27-rc4, mysql switches to sysbench when it's still running
> > mysqld-3674 [07] 95.960220: 3674:120:R + 3687:120:S
> > mysqld-3674 [07] 95.960220: 3674:120:R ==> 3687:120:R
> > sysbench-3687 [07] 95.960220: 3687:120:S ==> 3674:120:R
> >
> > So with 2.6.27-rc5, sysbench wakes up mysql and then switches to it (mysql is
> > in sleep state)
> >
> > With 2.6.27-rc4, sysbench switches to mysql, no need to wake up it (mysql is
> > still in running state)
>
> Ming, how does this work for you?
>
> ---
> Subject: sched: wakeup preempt when small overlap
>
> Aggresively preempt a task if its avg overlap is very small, this should
> avoid the task going to sleep and find it still running when we schedule
> back to it - saving a wakeup.
>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 7b4592c..cb44774 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -897,7 +897,7 @@ struct sched_class {
> void (*yield_task) (struct rq *rq);
> int (*select_task_rq)(struct task_struct *p, int sync);
>
> - void (*check_preempt_curr) (struct rq *rq, struct task_struct *p);
> + void (*check_preempt_curr) (struct rq *rq, struct task_struct *p, int sync);
>
> struct task_struct * (*pick_next_task) (struct rq *rq);
> void (*put_prev_task) (struct rq *rq, struct task_struct *p);
> diff --git a/kernel/sched.c b/kernel/sched.c
> index 32d56d6..e17d506 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -609,9 +609,9 @@ struct rq {
>
> static DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
>
> -static inline void check_preempt_curr(struct rq *rq, struct task_struct *p)
> +static inline void check_preempt_curr(struct rq *rq, struct task_struct *p, int sync)
> {
> - rq->curr->sched_class->check_preempt_curr(rq, p);
> + rq->curr->sched_class->check_preempt_curr(rq, p, sync);
> }
>
> static inline int cpu_of(struct rq *rq)
> @@ -2299,7 +2299,7 @@ out_activate:
>
> out_running:
> trace_sched_wakeup(rq, p);
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, sync);
>
> p->state = TASK_RUNNING;
> #ifdef CONFIG_SMP
> @@ -2432,7 +2432,7 @@ void wake_up_new_task(struct task_struct *p, unsigned long clone_flags)
> inc_nr_running(rq);
> }
> trace_sched_wakeup_new(rq, p);
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, 0);
> #ifdef CONFIG_SMP
> if (p->sched_class->task_wake_up)
> p->sched_class->task_wake_up(rq, p);
> @@ -2889,7 +2889,7 @@ static void pull_task(struct rq *src_rq, struct task_struct *p,
> * Note that idle threads have a prio of MAX_PRIO, for this test
> * to be always true for them.
> */
> - check_preempt_curr(this_rq, p);
> + check_preempt_curr(this_rq, p, 0);
> }
>
> /*
> @@ -6015,7 +6015,7 @@ static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)
> set_task_cpu(p, dest_cpu);
> if (on_rq) {
> activate_task(rq_dest, p, 0);
> - check_preempt_curr(rq_dest, p);
> + check_preempt_curr(rq_dest, p, 0);
> }
> done:
> ret = 1;
> diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
> index a10ac0b..82a907c 100644
> --- a/kernel/sched_fair.c
> +++ b/kernel/sched_fair.c
> @@ -1331,7 +1331,7 @@ static inline int depth_se(struct sched_entity *se)
> /*
> * Preempt the current task with a newly woken task if needed:
> */
> -static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
> +static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
> {
> struct task_struct *curr = rq->curr;
> struct cfs_rq *cfs_rq = task_cfs_rq(curr);
> @@ -1348,6 +1348,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
> if (unlikely(se == pse))
> return;
>
> + cfs_rq_of(pse)->next = pse;
> +
> /*
> * We can come here with TIF_NEED_RESCHED already set from new task
> * wake up path.
> @@ -1355,8 +1357,6 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
> if (test_tsk_need_resched(curr))
> return;
>
> - cfs_rq_of(pse)->next = pse;
> -
> /*
> * Batch tasks do not preempt (their preemption is driven by
> * the tick):
> @@ -1367,6 +1367,13 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p)
> if (!sched_feat(WAKEUP_PREEMPT))
> return;
>
> + if (sched_feat(WAKEUP_OVERLAP) && sync &&
> + se->avg_overlap < sysctl_sched_migration_cost &&
> + pse->avg_overlap < sysctl_sched_migration_cost) {
> + resched_task(curr);
> + return;
> + }
> +
> /*
> * preemption test can be made between sibling entities who are in the
> * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
> @@ -1649,7 +1656,7 @@ static void prio_changed_fair(struct rq *rq, struct task_struct *p,
> if (p->prio > oldprio)
> resched_task(rq->curr);
> } else
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, 0);
> }
>
> /*
> @@ -1666,7 +1673,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p,
> if (running)
> resched_task(rq->curr);
> else
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, 0);
> }
>
> /* Account for a task changing its policy or group.
> diff --git a/kernel/sched_features.h b/kernel/sched_features.h
> index 9353ca7..bf027a7 100644
> --- a/kernel/sched_features.h
> +++ b/kernel/sched_features.h
> @@ -11,3 +11,4 @@ SCHED_FEAT(ASYM_GRAN, 1)
> SCHED_FEAT(LB_BIAS, 1)
> SCHED_FEAT(LB_WAKEUP_UPDATE, 1)
> SCHED_FEAT(ASYM_EFF_LOAD, 1)
> +SCHED_FEAT(WAKEUP_OVERLAP, 1)
> diff --git a/kernel/sched_idletask.c b/kernel/sched_idletask.c
> index 3a4f92d..dec4cca 100644
> --- a/kernel/sched_idletask.c
> +++ b/kernel/sched_idletask.c
> @@ -14,7 +14,7 @@ static int select_task_rq_idle(struct task_struct *p, int sync)
> /*
> * Idle tasks are unconditionally rescheduled:
> */
> -static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p)
> +static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int sync)
> {
> resched_task(rq->idle);
> }
> @@ -76,7 +76,7 @@ static void switched_to_idle(struct rq *rq, struct task_struct *p,
> if (running)
> resched_task(rq->curr);
> else
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, 0);
> }
>
> static void prio_changed_idle(struct rq *rq, struct task_struct *p,
> @@ -93,7 +93,7 @@ static void prio_changed_idle(struct rq *rq, struct task_struct *p,
> if (p->prio > oldprio)
> resched_task(rq->curr);
> } else
> - check_preempt_curr(rq, p);
> + check_preempt_curr(rq, p, 0);
> }
>
> /*
> diff --git a/kernel/sched_rt.c b/kernel/sched_rt.c
> index 04875ef..2e228bd 100644
> --- a/kernel/sched_rt.c
> +++ b/kernel/sched_rt.c
> @@ -787,7 +787,7 @@ static void check_preempt_equal_prio(struct rq *rq, struct task_struct *p)
> /*
> * Preempt the current task with a newly woken task if needed:
> */
> -static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p)
> +static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p, int sync)
> {
> if (p->prio < rq->curr->prio) {
> resched_task(rq->curr);
>
>
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2008-09-26 2:01 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-09-04 8:51 oltp ~10% regression with 2.6.27-rc5 on stoakley machine Lin Ming
2008-09-04 9:03 ` Peter Zijlstra
2008-09-04 10:52 ` Lin Ming
2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 12:12 ` Lin Ming
2008-09-04 12:26 ` Peter Zijlstra
2008-09-04 12:42 ` Lin Ming
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
2008-09-04 13:50 ` [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time" Gregory Haskins
2008-09-04 13:50 ` [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code" Gregory Haskins
2008-09-04 13:50 ` [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time Gregory Haskins
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
2008-09-04 11:30 ` Lin Ming
2008-09-04 11:35 ` Ingo Molnar
2008-09-04 12:19 ` Lin Ming
2008-09-05 1:26 ` Lin Ming
2008-09-20 21:38 ` Peter Zijlstra
2008-09-26 2:00 ` Lin Ming
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox