using blktrace, blkparse, and fio

using blktrace, blkparse, and fio

[Hsiao Su]

Hi,

I'm trying to use these 3 tools to capture the IO workload of an application.� This application uses several different IO libraries.� However, I'm not sure if I'm doing this right, so I've been trying to do this on a much smaller scale.� For this small scale test, my workload is simply to use the cp command to copy a large file onto the same filesystem, then use blktrace/blkparse/fio to replay that.

Some questions:

1) Is using blkparse necessary?� I've actually found that you can skip blkparse, and simply use the output of blktrace and input it into fio.� The result is that the replay is much much longer.� My cp command took only 1.5 minutes, but if I skip blkparse, the replay via fio would take almost an hour.

2) Because of #1, I suspect that blkparse is a must.� So I've included the blkparse step.� But when I replay it via fio on the same device, it took about 4-5 minutes.� Does this sound about right?� Should I expect the fio replay to be 3-4 times longer than the original workload?� I was expecting that their time should be the same.

3) I'm seeing so possible file system corruption now.� When I remove a file, I get an Input/Output Error.� Is this possibly due to what I had done in #1 and #2?� In general, when using fio, what steps should I take to prevent a file system corruption?


---
Hsiao "Shao" Su
MarkLogic
Senior Performance Engineer
650 287 2545 (W)

Re: using blktrace, blkparse, and fio

[Jens Axboe]

On 2011-02-24 20:13, Hsiao Su wrote:
> 
> Hi,
> 
> I'm trying to use these 3 tools to capture the IO workload of an
> application.  This application uses several different IO libraries.
> However, I'm not sure if I'm doing this right, so I've been trying to
> do this on a much smaller scale.  For this small scale test, my
> workload is simply to use the cp command to copy a large file onto the
> same filesystem, then use blktrace/blkparse/fio to replay that.
> 
> Some questions:
> 
> 1) Is using blkparse necessary?  I've actually found that you can skip
> blkparse, and simply use the output of blktrace and input it into fio.
> The result is that the replay is much much longer.  My cp command took
> only 1.5 minutes, but if I skip blkparse, the replay via fio would
> take almost an hour.
> 
> 2) Because of #1, I suspect that blkparse is a must.  So I've included
> the blkparse step.  But when I replay it via fio on the same device,
> it took about 4-5 minutes.  Does this sound about right?  Should I
> expect the fio replay to be 3-4 times longer than the original
> workload?  I was expecting that their time should be the same.

These are really the same question, so I'll answer here. You need to run
blkparse to generate a binary trace file for fio. It'll have the same
format as the origina file(s), the difference is that blkparse will
collect them into a single unified trace. It's what blkparse does for
the ascii display of events as well.

As to performance, what fio version are you using? You could try and use
replay_no_stall=1 option to see if that improves things.

> 3) I'm seeing so possible file system corruption now.  When I remove a
> file, I get an Input/Output Error.  Is this possibly due to what I had
> done in #1 and #2?  In general, when using fio, what steps should I
> take to prevent a file system corruption?

Since blktrace replay works on the device in question and not through
the file system, it will screw up the data on the device. So never use
it on anything you can't just regenerate, like test data. In general be
careful with raw device access, ideally use a dedicated drive for that
so you don't risk corrupting data you care about.

-- 
Jens Axboe

RE: using blktrace, blkparse, and fio

[Hsiao Su]

Thanks,

A follow up question.  Suppose that I use fio directly, without blktrace and blkparse.  In this case, I would have to guess the real IO workload that my app is doing, and specify that in the job file.  Would I still have to worry about file system corruption?

My guess is no for this one, that fio would create its own file for replaying the workload.

---
Hsiao "Shao" Su
MarkLogic
Senior Performance Engineer
650 287 2545 (W)


-----Original Message-----
From: Jens Axboe [mailto:jaxboe@fusionio.com] 
Sent: Thursday, February 24, 2011 11:58 PM
To: Hsiao Su
Cc: fio@vger.kernel.org
Subject: Re: using blktrace, blkparse, and fio

On 2011-02-24 20:13, Hsiao Su wrote:
> 
> Hi,
> 
> I'm trying to use these 3 tools to capture the IO workload of an
> application.  This application uses several different IO libraries.
> However, I'm not sure if I'm doing this right, so I've been trying to
> do this on a much smaller scale.  For this small scale test, my
> workload is simply to use the cp command to copy a large file onto the
> same filesystem, then use blktrace/blkparse/fio to replay that.
> 
> Some questions:
> 
> 1) Is using blkparse necessary?  I've actually found that you can skip
> blkparse, and simply use the output of blktrace and input it into fio.
> The result is that the replay is much much longer.  My cp command took
> only 1.5 minutes, but if I skip blkparse, the replay via fio would
> take almost an hour.
> 
> 2) Because of #1, I suspect that blkparse is a must.  So I've included
> the blkparse step.  But when I replay it via fio on the same device,
> it took about 4-5 minutes.  Does this sound about right?  Should I
> expect the fio replay to be 3-4 times longer than the original
> workload?  I was expecting that their time should be the same.

These are really the same question, so I'll answer here. You need to run
blkparse to generate a binary trace file for fio. It'll have the same
format as the origina file(s), the difference is that blkparse will
collect them into a single unified trace. It's what blkparse does for
the ascii display of events as well.

As to performance, what fio version are you using? You could try and use
replay_no_stall=1 option to see if that improves things.

> 3) I'm seeing so possible file system corruption now.  When I remove a
> file, I get an Input/Output Error.  Is this possibly due to what I had
> done in #1 and #2?  In general, when using fio, what steps should I
> take to prevent a file system corruption?

Since blktrace replay works on the device in question and not through
the file system, it will screw up the data on the device. So never use
it on anything you can't just regenerate, like test data. In general be
careful with raw device access, ideally use a dedicated drive for that
so you don't risk corrupting data you care about.

-- 
Jens Axboe

Re: using blktrace, blkparse, and fio

[Jens Axboe]

On 2011-02-25 13:24, Hsiao Su wrote:
> 
> Thanks,
> 
> A follow up question.  Suppose that I use fio directly, without
> blktrace and blkparse.  In this case, I would have to guess the real
> IO workload that my app is doing, and specify that in the job file.
> Would I still have to worry about file system corruption?
> 
> My guess is no for this one, that fio would create its own file for
> replaying the workload.

Depends on what the job looks like. If you stick to writing to files in
a mounted fs, it'll work fine. If you tell fio to use
--filename=/dev/sda or whatever the device is, you'll corrupt the file
system(s) if that job file is doing writes.

-- 
Jens Axboe

RE: using blktrace, blkparse, and fio

[Hsiao Su]

Thanks.

Still trying to think of possible workarounds for this.  This is a different idea:

Suppose that I re-partition the /dev/sda device into 2.  (Not sure exactly how to do this yet, either via LVM or use some other utility.)  And I use the first partition for real work, and the second partition for fio replay (using captures from blktrace/blkparse).

Now, this would corrupt any filesystem that I have on the second partition.  But would it corrupt the first partition?

I would think the answer is no.  Could you confirm?

---
Hsiao "Shao" Su
MarkLogic
Senior Performance Engineer
650 287 2545 (W)


-----Original Message-----
From: Jens Axboe [mailto:jaxboe@fusionio.com] 
Sent: Monday, February 28, 2011 5:07 AM
To: Hsiao Su
Cc: fio@vger.kernel.org
Subject: Re: using blktrace, blkparse, and fio

On 2011-02-25 13:24, Hsiao Su wrote:
> 
> Thanks,
> 
> A follow up question.  Suppose that I use fio directly, without
> blktrace and blkparse.  In this case, I would have to guess the real
> IO workload that my app is doing, and specify that in the job file.
> Would I still have to worry about file system corruption?
> 
> My guess is no for this one, that fio would create its own file for
> replaying the workload.

Depends on what the job looks like. If you stick to writing to files in
a mounted fs, it'll work fine. If you tell fio to use
--filename=/dev/sda or whatever the device is, you'll corrupt the file
system(s) if that job file is doing writes.

-- 
Jens Axboe