Linux btrace development
 help / color / mirror / Atom feed
* accuracy of btt result on btreplay?
@ 2010-03-03  7:41 Xuekun Hu
  2010-03-03 14:05 ` Alan D. Brunelle
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-03  7:41 UTC (permalink / raw)
  To: linux-btrace

Hi, Guys

I used blktrace to collect IOs traces on a workload. Then replayed it
on the same disk with btreplay, meanwhile I used blktrace to gather
the traces again. I would like to compare the two traces to evaluate
btreplay accuracy. For io direction, io offset, io size, the two
traces were matched. However for the btt output on average component
times across all IOs, the number of request are not matched.

Here is the btt output on the workload:

           ALL           MIN           AVG           MAX           N
--------------- ------------- ------------- ------------- -----------

Q2Q               0.000000003   0.000227312   0.302266672      438801
Q2G               0.000000368   0.000003045   0.004852484      175920
G2I               0.000000157   0.000001230   0.000056954      175920
Q2M               0.000000123   0.000000404   0.000005004      262882
I2D               0.000000286   0.000010544   0.008151437      175920
M2D               0.000001543   0.000010717   0.003731306      262882
D2C               0.000035010   0.000188685   0.117930905      438801
Q2C               0.000038161   0.000201288   0.117934610      438801

Here is the btt output on the btreplay:

            ALL           MIN           AVG           MAX           N
--------------- ------------- ------------- ------------- -----------

Q2Q               0.000000001   0.000238955   0.302027673      416343
Q2G               0.000000339   0.000017235   0.144473266      429721
S2G               0.006854780   0.079412795   0.144471318          15
G2I               0.000000023   0.000000579   0.000022893      429666
Q2M               0.000000163   0.000382036   0.028901367        8970
I2D               0.000000336   0.000672347   0.442394220      430233
M2D               0.000000707   0.067473630   0.439112467        7846
D2C               0.000000601   0.001189618   0.179906289      416325
Q2C               0.000000127   0.003043274   0.510393670      416344

Observation:
1. For both traces, blkparse reported 438802 request queued.
2. In the first btt output, seems like the number of Q2C missed one
that 438801 vs. 438802. Other number of events were correct.
3. In the second btt output, all number of events were below 438802.
And some data were misunderstanding. like the number of Q2G was
greater than the number of Q2Q, I2D > Q2C etc. However its blkparse
reported the correct 438802 request queued.

So how to explain the btt result on btreplay? Your comments are appreciated.

Thx, Xuekun

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: accuracy of btt result on btreplay?
  2010-03-03  7:41 accuracy of btt result on btreplay? Xuekun Hu
@ 2010-03-03 14:05 ` Alan D. Brunelle
  2010-03-04  1:37 ` Xuekun Hu
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Alan D. Brunelle @ 2010-03-03 14:05 UTC (permalink / raw)
  To: linux-btrace


Hi Xuekun - 

Not sure why they Q2Q N value is down 5%, but the AVG Q2Q is pretty
close - only 11msec (again, about 5%). After that, things are affected
by merge rates most likely - note that there were 262882 merges on the
first run whilst there were only 8970 on the second run. Perhaps you
could look at the blkparse output from both and get a feel for the vast
reduction in merges. [Any changes on the system between the runs?]

Regards,
Alan


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: accuracy of btt result on btreplay?
  2010-03-03  7:41 accuracy of btt result on btreplay? Xuekun Hu
  2010-03-03 14:05 ` Alan D. Brunelle
@ 2010-03-04  1:37 ` Xuekun Hu
  2010-03-05  7:30 ` Xuekun Hu
  2010-03-08  8:38 ` Xuekun Hu
  3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-04  1:37 UTC (permalink / raw)
  To: linux-btrace

Hi, Alan

It should be no changes on the system.

I compared the per io dump info, and found the merges did decrease a
lot. Another interesting thing I found from the blkparse output is the
write requests has much more "WS" (synchronous) flag in the replay
run.  Does the merges decrease due to the "S" flag?

Perhaps  I need to tune the "-max-pkts"? In my case, to increase its
value to let more requests have more opportunities to merge?

BTW. I also tried replaying "dd" traces, and found the number of
merges also decreases.

Thx, Xuekun

On Wed, Mar 3, 2010 at 10:05 PM, Alan D. Brunelle <Alan.Brunelle@hp.com> wrote:
>
> Hi Xuekun -
>
> Not sure why they Q2Q N value is down 5%, but the AVG Q2Q is pretty
> close - only 11msec (again, about 5%). After that, things are affected
> by merge rates most likely - note that there were 262882 merges on the
> first run whilst there were only 8970 on the second run. Perhaps you
> could look at the blkparse output from both and get a feel for the vast
> reduction in merges. [Any changes on the system between the runs?]
>
> Regards,
> Alan
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: accuracy of btt result on btreplay?
  2010-03-03  7:41 accuracy of btt result on btreplay? Xuekun Hu
  2010-03-03 14:05 ` Alan D. Brunelle
  2010-03-04  1:37 ` Xuekun Hu
@ 2010-03-05  7:30 ` Xuekun Hu
  2010-03-08  8:38 ` Xuekun Hu
  3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-05  7:30 UTC (permalink / raw)
  To: linux-btrace

Updates. I decreased/increased max-pkts to 1/16, however nothing
changed, still less merges in the replay run.

I think the "S" attribute for "write" should be the reason. First I
suspected the "S" attribute is brought by io_submit system call,
however I tested aio-stress tool which uses io_submit to send AIO
also. In the gathering traces, the write operations doesn't have "S"
attribute, however during replay, the "S" attribute happened again.

So I wonder where the "S" comes from? Does anyone have suggestions?
Thanks in advance.

Thx, Xuekun

On Thu, Mar 4, 2010 at 9:37 AM, Xuekun Hu <xuekun.hu@gmail.com> wrote:
> Hi, Alan
>
> It should be no changes on the system.
>
> I compared the per io dump info, and found the merges did decrease a
> lot. Another interesting thing I found from the blkparse output is the
> write requests has much more "WS" (synchronous) flag in the replay
> run.  Does the merges decrease due to the "S" flag?
>
> Perhaps  I need to tune the "-max-pkts"? In my case, to increase its
> value to let more requests have more opportunities to merge?
>
> BTW. I also tried replaying "dd" traces, and found the number of
> merges also decreases.
>
> Thx, Xuekun
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: accuracy of btt result on btreplay?
  2010-03-03  7:41 accuracy of btt result on btreplay? Xuekun Hu
                   ` (2 preceding siblings ...)
  2010-03-05  7:30 ` Xuekun Hu
@ 2010-03-08  8:38 ` Xuekun Hu
  3 siblings, 0 replies; 5+ messages in thread
From: Xuekun Hu @ 2010-03-08  8:38 UTC (permalink / raw)
  To: linux-btrace

I think I found where the "S" flag was put for write IO request. In
the kernel function __blockdev_direct_IO(), every write request will
be tagged as "sync". Since btreplay uses O_DIRECT + AIO, which should
go into __blockdev_direct_IO.

So now my views on btreplay:
1. for write request, btreplay will decrease merge rates, thus impact
the latency etc.
2. for read request, btrecord may adjust "-max-pkts"/"-max-bunch-time"
to control the size of bunch to close to the real read IO merge rates.

Thx, Xuekun

On Fri, Mar 5, 2010 at 3:30 PM, Xuekun Hu <xuekun.hu@gmail.com> wrote:
> Updates. I decreased/increased max-pkts to 1/16, however nothing
> changed, still less merges in the replay run.
>
> I think the "S" attribute for "write" should be the reason. First I
> suspected the "S" attribute is brought by io_submit system call,
> however I tested aio-stress tool which uses io_submit to send AIO
> also. In the gathering traces, the write operations doesn't have "S"
> attribute, however during replay, the "S" attribute happened again.
>
> So I wonder where the "S" comes from? Does anyone have suggestions?
> Thanks in advance.
>
> Thx, Xuekun
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2010-03-08  8:38 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-03-03  7:41 accuracy of btt result on btreplay? Xuekun Hu
2010-03-03 14:05 ` Alan D. Brunelle
2010-03-04  1:37 ` Xuekun Hu
2010-03-05  7:30 ` Xuekun Hu
2010-03-08  8:38 ` Xuekun Hu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox