* accuracy of btt result on btreplay?
@ 2010-03-03 7:41 Xuekun Hu
2010-03-03 14:05 ` Alan D. Brunelle
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-03 7:41 UTC (permalink / raw)
To: linux-btrace
Hi, Guys
I used blktrace to collect IOs traces on a workload. Then replayed it
on the same disk with btreplay, meanwhile I used blktrace to gather
the traces again. I would like to compare the two traces to evaluate
btreplay accuracy. For io direction, io offset, io size, the two
traces were matched. However for the btt output on average component
times across all IOs, the number of request are not matched.
Here is the btt output on the workload:
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000003 0.000227312 0.302266672 438801
Q2G 0.000000368 0.000003045 0.004852484 175920
G2I 0.000000157 0.000001230 0.000056954 175920
Q2M 0.000000123 0.000000404 0.000005004 262882
I2D 0.000000286 0.000010544 0.008151437 175920
M2D 0.000001543 0.000010717 0.003731306 262882
D2C 0.000035010 0.000188685 0.117930905 438801
Q2C 0.000038161 0.000201288 0.117934610 438801
Here is the btt output on the btreplay:
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000000001 0.000238955 0.302027673 416343
Q2G 0.000000339 0.000017235 0.144473266 429721
S2G 0.006854780 0.079412795 0.144471318 15
G2I 0.000000023 0.000000579 0.000022893 429666
Q2M 0.000000163 0.000382036 0.028901367 8970
I2D 0.000000336 0.000672347 0.442394220 430233
M2D 0.000000707 0.067473630 0.439112467 7846
D2C 0.000000601 0.001189618 0.179906289 416325
Q2C 0.000000127 0.003043274 0.510393670 416344
Observation:
1. For both traces, blkparse reported 438802 request queued.
2. In the first btt output, seems like the number of Q2C missed one
that 438801 vs. 438802. Other number of events were correct.
3. In the second btt output, all number of events were below 438802.
And some data were misunderstanding. like the number of Q2G was
greater than the number of Q2Q, I2D > Q2C etc. However its blkparse
reported the correct 438802 request queued.
So how to explain the btt result on btreplay? Your comments are appreciated.
Thx, Xuekun
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accuracy of btt result on btreplay?
2010-03-03 7:41 accuracy of btt result on btreplay? Xuekun Hu
@ 2010-03-03 14:05 ` Alan D. Brunelle
2010-03-04 1:37 ` Xuekun Hu
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Alan D. Brunelle @ 2010-03-03 14:05 UTC (permalink / raw)
To: linux-btrace
Hi Xuekun -
Not sure why they Q2Q N value is down 5%, but the AVG Q2Q is pretty
close - only 11msec (again, about 5%). After that, things are affected
by merge rates most likely - note that there were 262882 merges on the
first run whilst there were only 8970 on the second run. Perhaps you
could look at the blkparse output from both and get a feel for the vast
reduction in merges. [Any changes on the system between the runs?]
Regards,
Alan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accuracy of btt result on btreplay?
2010-03-03 7:41 accuracy of btt result on btreplay? Xuekun Hu
2010-03-03 14:05 ` Alan D. Brunelle
@ 2010-03-04 1:37 ` Xuekun Hu
2010-03-05 7:30 ` Xuekun Hu
2010-03-08 8:38 ` Xuekun Hu
3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-04 1:37 UTC (permalink / raw)
To: linux-btrace
Hi, Alan
It should be no changes on the system.
I compared the per io dump info, and found the merges did decrease a
lot. Another interesting thing I found from the blkparse output is the
write requests has much more "WS" (synchronous) flag in the replay
run. Does the merges decrease due to the "S" flag?
Perhaps I need to tune the "-max-pkts"? In my case, to increase its
value to let more requests have more opportunities to merge?
BTW. I also tried replaying "dd" traces, and found the number of
merges also decreases.
Thx, Xuekun
On Wed, Mar 3, 2010 at 10:05 PM, Alan D. Brunelle <Alan.Brunelle@hp.com> wrote:
>
> Hi Xuekun -
>
> Not sure why they Q2Q N value is down 5%, but the AVG Q2Q is pretty
> close - only 11msec (again, about 5%). After that, things are affected
> by merge rates most likely - note that there were 262882 merges on the
> first run whilst there were only 8970 on the second run. Perhaps you
> could look at the blkparse output from both and get a feel for the vast
> reduction in merges. [Any changes on the system between the runs?]
>
> Regards,
> Alan
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accuracy of btt result on btreplay?
2010-03-03 7:41 accuracy of btt result on btreplay? Xuekun Hu
2010-03-03 14:05 ` Alan D. Brunelle
2010-03-04 1:37 ` Xuekun Hu
@ 2010-03-05 7:30 ` Xuekun Hu
2010-03-08 8:38 ` Xuekun Hu
3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-05 7:30 UTC (permalink / raw)
To: linux-btrace
Updates. I decreased/increased max-pkts to 1/16, however nothing
changed, still less merges in the replay run.
I think the "S" attribute for "write" should be the reason. First I
suspected the "S" attribute is brought by io_submit system call,
however I tested aio-stress tool which uses io_submit to send AIO
also. In the gathering traces, the write operations doesn't have "S"
attribute, however during replay, the "S" attribute happened again.
So I wonder where the "S" comes from? Does anyone have suggestions?
Thanks in advance.
Thx, Xuekun
On Thu, Mar 4, 2010 at 9:37 AM, Xuekun Hu <xuekun.hu@gmail.com> wrote:
> Hi, Alan
>
> It should be no changes on the system.
>
> I compared the per io dump info, and found the merges did decrease a
> lot. Another interesting thing I found from the blkparse output is the
> write requests has much more "WS" (synchronous) flag in the replay
> run. Does the merges decrease due to the "S" flag?
>
> Perhaps I need to tune the "-max-pkts"? In my case, to increase its
> value to let more requests have more opportunities to merge?
>
> BTW. I also tried replaying "dd" traces, and found the number of
> merges also decreases.
>
> Thx, Xuekun
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: accuracy of btt result on btreplay?
2010-03-03 7:41 accuracy of btt result on btreplay? Xuekun Hu
` (2 preceding siblings ...)
2010-03-05 7:30 ` Xuekun Hu
@ 2010-03-08 8:38 ` Xuekun Hu
3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-08 8:38 UTC (permalink / raw)
To: linux-btrace
I think I found where the "S" flag was put for write IO request. In
the kernel function __blockdev_direct_IO(), every write request will
be tagged as "sync". Since btreplay uses O_DIRECT + AIO, which should
go into __blockdev_direct_IO.
So now my views on btreplay:
1. for write request, btreplay will decrease merge rates, thus impact
the latency etc.
2. for read request, btrecord may adjust "-max-pkts"/"-max-bunch-time"
to control the size of bunch to close to the real read IO merge rates.
Thx, Xuekun
On Fri, Mar 5, 2010 at 3:30 PM, Xuekun Hu <xuekun.hu@gmail.com> wrote:
> Updates. I decreased/increased max-pkts to 1/16, however nothing
> changed, still less merges in the replay run.
>
> I think the "S" attribute for "write" should be the reason. First I
> suspected the "S" attribute is brought by io_submit system call,
> however I tested aio-stress tool which uses io_submit to send AIO
> also. In the gathering traces, the write operations doesn't have "S"
> attribute, however during replay, the "S" attribute happened again.
>
> So I wonder where the "S" comes from? Does anyone have suggestions?
> Thanks in advance.
>
> Thx, Xuekun
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-03-08 8:38 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-03-03 7:41 accuracy of btt result on btreplay? Xuekun Hu
2010-03-03 14:05 ` Alan D. Brunelle
2010-03-04 1:37 ` Xuekun Hu
2010-03-05 7:30 ` Xuekun Hu
2010-03-08 8:38 ` Xuekun Hu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox