Generating almost-sequential traffic

Generating almost-sequential traffic

[Carl Zwanzig]

Hi,

I need to test some devices with "almost-sequential" data traffic. This would have anywhere from 10 to 50 I/Os in order then either drop one from the sequence "1 2 3 (no 4) 5 6" or transpose two adjacent ones "1 2 (4 3) 5 6". This could be done with an iolog, but that seems both wasteful (huge files, 131k lines/MB) and slow (for reading and processing the files). To properly test, I need to move something like 3-4 Tb in 8k blocks.

It's also inelegant :). 

It looks like a drop sequence can be done with something like:
	blocksize=64k
	rw=write:8k
but I'm at a loss for how to do transposition.  Could be a hack into the get-next-block code in io_u.c, but I don't see how to actually implement that.

Any ideas (or is an iolog the best way)?

Thanks,

z!
Carl Zwanzig

Re: Generating almost-sequential traffic

[Jens Axboe]

On Wed, Jun 26 2013, Carl Zwanzig wrote:
> Hi,
> 
> I need to test some devices with "almost-sequential" data traffic. This would have anywhere from 10 to 50 I/Os in order then either drop one from the sequence "1 2 3 (no 4) 5 6" or transpose two adjacent ones "1 2 (4 3) 5 6". This could be done with an iolog, but that seems both wasteful (huge files, 131k lines/MB) and slow (for reading and processing the files). To properly test, I need to move something like 3-4 Tb in 8k blocks.
> 
> It's also inelegant :). 
> 
> It looks like a drop sequence can be done with something like:
> 	blocksize=64k
> 	rw=write:8k
> but I'm at a loss for how to do transposition.  Could be a hack into the get-next-block code in io_u.c, but I don't see how to actually implement that.
> 
> Any ideas (or is an iolog the best way)?

See percentage_random in the man page (or --cmdhelp=percentage_random)

-- 
Jens Axboe

RE: Generating almost-sequential traffic

[Carl Zwanzig]

> From: Jens Axboe [mailto:axboe@kernel.dk]
> Sent: Thursday, June 27, 2013 2:01 AM

> See percentage_random in the man page (or --cmdhelp=percentage_random)

In the tests I'm running, the sequence has to continue essentially uninterrupted with the occasional drop/transposition, but it can't contain LBAs that are wildly off. So '1 2 3 (4) 5 6' or '1 2 3 (5 4) 6' are good, '1 2 3 9999 4 5 6' is not. They trigger different specific behaviors in some hard drives.

Is there a good place in the code to insert this sort or thing?

z!

Re: Generating almost-sequential traffic

[Jens Axboe]

On Thu, Jun 27 2013, Carl Zwanzig wrote:
> > From: Jens Axboe [mailto:axboe@kernel.dk]
> > Sent: Thursday, June 27, 2013 2:01 AM
> 
> > See percentage_random in the man page (or --cmdhelp=percentage_random)
> 
> In the tests I'm running, the sequence has to continue essentially
> uninterrupted with the occasional drop/transposition, but it can't
> contain LBAs that are wildly off. So '1 2 3 (4) 5 6' or '1 2 3 (5 4)
> 6' are good, '1 2 3 9999 4 5 6' is not. They trigger different
> specific behaviors in some hard drives.
> 
> Is there a good place in the code to insert this sort or thing?

I would look at the profile support in fio. That allows you to hook into
io_u.c:get_next_offset(), which is where fio decides what the next
offset should be. If a profile defines ops->fill_io_u(), then that is
called instead. That gives you absolute control of the IO pattern,
without having to modify core fio code.

-- 
Jens Axboe