test definition help needed

test definition help needed

[Steven Pratt]

I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:

[global]
bs=4k
time_based=1
runtime=15m
iodepth=4
rw=read
ioengine=libaio
time_based=1
ramp_time=600s
norandommap

[job1]
opendir=/${FIO_MOUNT}/session1/small_file1
file_service_type=sequential
numjobs=8



I used file_service_type=sequential because tought without it it would only do a single read (block) from the file before switching to a different file, which is not what I want. The issue with this test as written is it seems like all the fio processes choose files in the same order so I get way more cache hits than I want. I want this to be more of a random file selection, but with reading whole file.  Any advice?

Steve

Re: test definition help needed

[Jens Axboe]

On 2011-02-04 20:21, Steven Pratt wrote:
> I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:
> 
> [global]
> bs=4k
> time_based=1
> runtime=15m
> iodepth=4
> rw=read
> ioengine=libaio
> time_based=1
> ramp_time=600s
> norandommap
> 
> [job1]
> opendir=/${FIO_MOUNT}/session1/small_file1
> file_service_type=sequential
> numjobs=8
> 
> 
> 
> I used file_service_type=sequential because tought without it it would
> only do a single read (block) from the file before switching to a
> different file, which is not what I want. The issue with this test as
> written is it seems like all the fio processes choose files in the
> same order so I get way more cache hits than I want. I want this to be
> more of a random file selection, but with reading whole file.  Any
> advice?

file_service_type=random:<largenum>

should do what you need, I think. If you ensure that <largenum> is
sufficiently large that the file will always be finished before you run
out, then that should work.

-- 
Jens Axboe

Re: test definition help needed

[Jeff Moyer]

Jens Axboe <jaxboe@fusionio.com> writes:

> On 2011-02-04 20:21, Steven Pratt wrote:
>> I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:
>> 
>> [global]
>> bs=4k
>> time_based=1
>> runtime=15m
>> iodepth=4
>> rw=read
>> ioengine=libaio
>> time_based=1
>> ramp_time=600s
>> norandommap
>> 
>> [job1]
>> opendir=/${FIO_MOUNT}/session1/small_file1
>> file_service_type=sequential
>> numjobs=8
>> 
>> 
>> 
>> I used file_service_type=sequential because tought without it it would
>> only do a single read (block) from the file before switching to a
>> different file, which is not what I want. The issue with this test as
>> written is it seems like all the fio processes choose files in the
>> same order so I get way more cache hits than I want. I want this to be
>> more of a random file selection, but with reading whole file.  Any
>> advice?
>
> file_service_type=random:<largenum>
>
> should do what you need, I think. If you ensure that <largenum> is
> sufficiently large that the file will always be finished before you run
> out, then that should work.

Also note that with ioengine=libaio, you'll also want to specify direct
io (otherwise io_submit will block until the I/O is complete).  If you
really want buffered, then you need to choose a different io engine.

Cheers,
Jeff

Re: test definition help needed

[Jens Axboe]

On 2011-02-07 15:35, Jeff Moyer wrote:
> Jens Axboe <jaxboe@fusionio.com> writes:
> 
>> On 2011-02-04 20:21, Steven Pratt wrote:
>>> I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:
>>>
>>> [global]
>>> bs=4k
>>> time_based=1
>>> runtime=15m
>>> iodepth=4
>>> rw=read
>>> ioengine=libaio
>>> time_based=1
>>> ramp_time=600s
>>> norandommap
>>>
>>> [job1]
>>> opendir=/${FIO_MOUNT}/session1/small_file1
>>> file_service_type=sequential
>>> numjobs=8
>>>
>>>
>>>
>>> I used file_service_type=sequential because tought without it it would
>>> only do a single read (block) from the file before switching to a
>>> different file, which is not what I want. The issue with this test as
>>> written is it seems like all the fio processes choose files in the
>>> same order so I get way more cache hits than I want. I want this to be
>>> more of a random file selection, but with reading whole file.  Any
>>> advice?
>>
>> file_service_type=random:<largenum>
>>
>> should do what you need, I think. If you ensure that <largenum> is
>> sufficiently large that the file will always be finished before you run
>> out, then that should work.
> 
> Also note that with ioengine=libaio, you'll also want to specify direct
> io (otherwise io_submit will block until the I/O is complete).  If you
> really want buffered, then you need to choose a different io engine.

Good point. I used to have a warning for that, but dropped it since it
got annoying when I was testing the buffered aio patches. Perhaps I
should reinstate it.

In any case, if the result is inspected, it'll be apparent that the
queue depth was 1 throughout the run.

-- 
Jens Axboe

Re: test definition help needed

[Jeff Moyer]

Jens Axboe <jaxboe@fusionio.com> writes:

> Good point. I used to have a warning for that, but dropped it since it
> got annoying when I was testing the buffered aio patches. Perhaps I
> should reinstate it.
>
> In any case, if the result is inspected, it'll be apparent that the
> queue depth was 1 throughout the run.

It might be apparent, but it won't be obvious *why* it's happening for
the uninitiated.  ;-)  It may well be worth putting the warning back
in.

Cheers,
Jeff

Re: test definition help needed

[Steven Pratt]

On 02/06/2011 03:00 PM, Jens Axboe wrote:
> On 2011-02-04 20:21, Steven Pratt wrote:
>> I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:
>>
>> [global]
>> bs=4k
>> time_based=1
>> runtime=15m
>> iodepth=4
>> rw=read
>> ioengine=libaio
>> time_based=1
>> ramp_time=600s
>> norandommap
>>
>> [job1]
>> opendir=/${FIO_MOUNT}/session1/small_file1
>> file_service_type=sequential
>> numjobs=8
>>
>>
>>
>> I used file_service_type=sequential because tought without it it would
>> only do a single read (block) from the file before switching to a
>> different file, which is not what I want. The issue with this test as
>> written is it seems like all the fio processes choose files in the
>> same order so I get way more cache hits than I want. I want this to be
>> more of a random file selection, but with reading whole file.  Any
>> advice?
> file_service_type=random:<largenum>
>
> should do what you need, I think. If you ensure that <largenum> is
> sufficiently large that the file will always be finished before you run
> out, then that should work.
Had thought of that, but seemed kind of hackish.  Any way gave it a shot and it kind of worked.  It seemed to randomly select a file to operate on, however fio eventually went into a tight spin loop and never exited. Traced that back to get_next_file_rand where apparently if you ever access every file you will never exit(get stuck in  "if (fio_file_done(f))  continue;" forever).  So now I have 2 problems.  One is this infinite loop which is obviously not good. The second (which may help solve the first) is that I want to randomly reuse files. I don't want a perfect no reuse case, and I can't find any way to get that to happen. If reuse were possible, this loop would not be a problem.  I guess for now I can just remove the continue and it will do what I want and not loop.  Thoughts on a better solution?

Steve

Re: test definition help needed

[Jens Axboe]

On 2011-02-07 21:51, Steven Pratt wrote:
> On 02/06/2011 03:00 PM, Jens Axboe wrote:
>> On 2011-02-04 20:21, Steven Pratt wrote:
>>> I am trying to create a job file that randomly select a file form an imported list and reads the entire file sequentially. Them moves to the next file. I also want multiple jobs(processe) running the same workload. I have this:
>>>
>>> [global]
>>> bs=4k
>>> time_based=1
>>> runtime=15m
>>> iodepth=4
>>> rw=read
>>> ioengine=libaio
>>> time_based=1
>>> ramp_time=600s
>>> norandommap
>>>
>>> [job1]
>>> opendir=/${FIO_MOUNT}/session1/small_file1
>>> file_service_type=sequential
>>> numjobs=8
>>>
>>>
>>>
>>> I used file_service_type=sequential because tought without it it would
>>> only do a single read (block) from the file before switching to a
>>> different file, which is not what I want. The issue with this test as
>>> written is it seems like all the fio processes choose files in the
>>> same order so I get way more cache hits than I want. I want this to be
>>> more of a random file selection, but with reading whole file.  Any
>>> advice?
>> file_service_type=random:<largenum>
>>
>> should do what you need, I think. If you ensure that <largenum> is
>> sufficiently large that the file will always be finished before you run
>> out, then that should work.
> Had thought of that, but seemed kind of hackish.  Any way gave it a

It is sort-of hackish. I guess it would be cleaner to add a separate
file_service_type define for that.

> shot and it kind of worked.  It seemed to randomly select a file to
> operate on, however fio eventually went into a tight spin loop and
> never exited. Traced that back to get_next_file_rand where apparently
> if you ever access every file you will never exit(get stuck in  "if
> (fio_file_done(f))  continue;" forever).  So now I have 2 problems.
> One is this infinite loop which is obviously not good. The second
> (which may help solve the first) is that I want to randomly reuse
> files. I don't want a perfect no reuse case, and I can't find any way
> to get that to happen. If reuse were possible, this loop would not be
> a problem.  I guess for now I can just remove the continue and it will
> do what I want and not loop.  Thoughts on a better solution?

I'll look into that tomorrow, sounds like a bug that has been there
since that service type was introduced.

-- 
Jens Axboe