fio file test patterns

fio file test patterns

[Brian Fallik]

Hi,

Apologies if this is documented somewhere else but I couldn't find it
in the fio man page, example job files, or list archives.

I'm exploring fio as a testing tool and it seems very well suited for
my needs.  I'm currently running experiments with N sequential writers
all writing at 200k.  The jobs file is very simple:
    [global]
    size=10m
    directory=.

    [foo1]
    rw=write
    rate=200k

    [foo2]
    ...
fio creates various foo* files as part of its test but they all seem
to contain the same content.  I would have expected fio to generate
random data in each file to avoid potential optimizations like
deduplication.   Am I missing the flag to generate random test
patterns or is this behavior intentional?

I'm currently using fio 1.57.

Thanks,
brian

Re: fio file test patterns

[Jeff Moyer]

Brian Fallik <bfallik@bamboom.com> writes:

> Hi,
>
> Apologies if this is documented somewhere else but I couldn't find it
> in the fio man page, example job files, or list archives.
>
> I'm exploring fio as a testing tool and it seems very well suited for
> my needs.  I'm currently running experiments with N sequential writers
> all writing at 200k.  The jobs file is very simple:
>     [global]
>     size=10m
>     directory=.
>
>     [foo1]
>     rw=write
>     rate=200k
>
>     [foo2]
>     ...
> fio creates various foo* files as part of its test but they all seem
> to contain the same content.  I would have expected fio to generate
> random data in each file to avoid potential optimizations like
> deduplication.   Am I missing the flag to generate random test
> patterns or is this behavior intentional?

       refill_buffers
              If this option is given, fio will refill the IO buffers on every
              submit. The default is to only fill it at init  time and reuse
              that  data.  Only  makes  sense if zero_buffers isn't specified,
              naturally. If data verification is  enabled, refill_buffers is
              also automatically enabled.

Looks like what you want.

Cheers,
Jeff

Re: fio file test patterns

[Jens Axboe]

On 2011-08-31 12:30, Jeff Moyer wrote:
> Brian Fallik <bfallik@bamboom.com> writes:
> 
>> Hi,
>>
>> Apologies if this is documented somewhere else but I couldn't find it
>> in the fio man page, example job files, or list archives.
>>
>> I'm exploring fio as a testing tool and it seems very well suited for
>> my needs.  I'm currently running experiments with N sequential writers
>> all writing at 200k.  The jobs file is very simple:
>>     [global]
>>     size=10m
>>     directory=.
>>
>>     [foo1]
>>     rw=write
>>     rate=200k
>>
>>     [foo2]
>>     ...
>> fio creates various foo* files as part of its test but they all seem
>> to contain the same content.  I would have expected fio to generate
>> random data in each file to avoid potential optimizations like
>> deduplication.   Am I missing the flag to generate random test
>> patterns or is this behavior intentional?
> 
>        refill_buffers
>               If this option is given, fio will refill the IO buffers on every
>               submit. The default is to only fill it at init  time and reuse
>               that  data.  Only  makes  sense if zero_buffers isn't specified,
>               naturally. If data verification is  enabled, refill_buffers is
>               also automatically enabled.

Yes. Fio does use random data by default, but for to avoid slowing down
too much, it also defaults to reusing the same random data all the time.
If you set the above option, you get fully fresh random data for every
write, thus fully defeating any de-dupe/compression attempts on the
target.

-- 
Jens Axboe

Re: fio file test patterns

[Brian Fallik]

Hi,

Hmm.  I'm still unable to generate random test data using fio.   My
new job control file is:
  [foo0]
  rate =,200k
  rw =write
  refill_buffers =1
  size =4m

  [foo1]
  rate =,200k
  rw =write
  refill_buffers =1
  size =4m
But:
  $ diff foo0.1.0 foo1.2.0
reports they're identical.  I also tried enabling data verification by
adding "verify=crc32c-intel" but that had no effect.  Am I missing
something obvious or is this potentially broken in fio 1.57?

I also have another (maybe related?) question.  Apologies if this
belongs in a separate thread, but are there any notes explaining why
fio lays out the files before starting sequential writes?  The
workload I was hoping to simulate is sustained, sequential writes to
disk.  I'm trying to answer the question "How many simultaneous
200kBps writers can we support?"  Using my current jobs file, fio
starts by creating the files (e.g "foo0: Laying out IO file(s) (1
file(s) / 4MB)") before it starts processing.  However, creating the
files in advance accounts for a chunk of performance that doesn't seem
to be measured by fio.  Am I misunderstanding how to configure fio or
its intended usage?

Thanks,
brian


On Wed, Aug 31, 2011 at 3:56 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-08-31 12:30, Jeff Moyer wrote:
>> Brian Fallik <bfallik@bamboom.com> writes:
>>
>>> Hi,
>>>
>>> Apologies if this is documented somewhere else but I couldn't find it
>>> in the fio man page, example job files, or list archives.
>>>
>>> I'm exploring fio as a testing tool and it seems very well suited for
>>> my needs.  I'm currently running experiments with N sequential writers
>>> all writing at 200k.  The jobs file is very simple:
>>>     [global]
>>>     size=10m
>>>     directory=.
>>>
>>>     [foo1]
>>>     rw=write
>>>     rate=200k
>>>
>>>     [foo2]
>>>     ...
>>> fio creates various foo* files as part of its test but they all seem
>>> to contain the same content.  I would have expected fio to generate
>>> random data in each file to avoid potential optimizations like
>>> deduplication.   Am I missing the flag to generate random test
>>> patterns or is this behavior intentional?
>>
>>        refill_buffers
>>               If this option is given, fio will refill the IO buffers on every
>>               submit. The default is to only fill it at init  time and reuse
>>               that  data.  Only  makes  sense if zero_buffers isn't specified,
>>               naturally. If data verification is  enabled, refill_buffers is
>>               also automatically enabled.
>
> Yes. Fio does use random data by default, but for to avoid slowing down
> too much, it also defaults to reusing the same random data all the time.
> If you set the above option, you get fully fresh random data for every
> write, thus fully defeating any de-dupe/compression attempts on the
> target.
>
> --
> Jens Axboe
>
>

Re: fio file test patterns

[Jens Axboe]

On 2011-08-31 14:34, Brian Fallik wrote:
> Hi,
> 
> Hmm.  I'm still unable to generate random test data using fio.   My
> new job control file is:
>   [foo0]
>   rate =,200k
>   rw =write
>   refill_buffers =1
>   size =4m
> 
>   [foo1]
>   rate =,200k
>   rw =write
>   refill_buffers =1
>   size =4m
> But:
>   $ diff foo0.1.0 foo1.2.0
> reports they're identical.  I also tried enabling data verification by
> adding "verify=crc32c-intel" but that had no effect.  Am I missing
> something obvious or is this potentially broken in fio 1.57?

It's random, just not across jobs apparently... Ooops. The below patch
should rectify that. It's also committed, jfyi.

> I also have another (maybe related?) question.  Apologies if this
> belongs in a separate thread, but are there any notes explaining why
> fio lays out the files before starting sequential writes?  The
> workload I was hoping to simulate is sustained, sequential writes to
> disk.  I'm trying to answer the question "How many simultaneous
> 200kBps writers can we support?"  Using my current jobs file, fio
> starts by creating the files (e.g "foo0: Laying out IO file(s) (1
> file(s) / 4MB)") before it starts processing.  However, creating the
> files in advance accounts for a chunk of performance that doesn't seem
> to be measured by fio.  Am I misunderstanding how to configure fio or
> its intended usage?

You should be able to set overwrite=0 to avoid that. Are they random
writes?


commit 3545a109a2cfe5ab22969ef453dc049db47f0b68
Author: Jens Axboe <jaxboe@fusionio.com>
Date:   Wed Aug 31 15:20:15 2011 -0600

    Ensure that buffer contents are random across jobs as well
    
    Signed-off-by: Jens Axboe <jaxboe@fusionio.com>

diff --git a/fio.c b/fio.c
index 9c1bed3..4514840 100644
--- a/fio.c
+++ b/fio.c
@@ -874,9 +874,9 @@ static int init_io_u(struct thread_data *td)
 			io_u->buf = p + max_bs * i;
 			dprint(FD_MEM, "io_u %p, mem %p\n", io_u, io_u->buf);
 
-			if (td_write(td) && !td->o.refill_buffers)
+			if (td_write(td))
 				io_u_fill_buffer(td, io_u, max_bs);
-			else if (td_write(td) && td->o.verify_pattern_bytes) {
+			if (td_write(td) && td->o.verify_pattern_bytes) {
 				/*
 				 * Fill the buffer with the pattern if we are
 				 * going to be doing writes.
@@ -1699,7 +1699,6 @@ int main(int argc, char *argv[], char *envp[])
 	arch_init(envp);
 
 	sinit();
-	init_rand(&__fio_rand_state);
 
 	/*
 	 * We need locale for number printing, if it isn't set then just
diff --git a/fio.h b/fio.h
index 8401eda..6eb270d 100644
--- a/fio.h
+++ b/fio.h
@@ -460,7 +460,7 @@ struct thread_data {
 
 	char *sysfs_root;
 
-	unsigned long rand_seeds[7];
+	unsigned long rand_seeds[8];
 
 	union {
 		os_random_state_t bsrange_state;
@@ -475,6 +475,8 @@ struct thread_data {
 		struct frand_state __trim_state;
 	};
 
+	struct frand_state buf_state;
+
 	unsigned int verify_batch;
 	unsigned int trim_batch;
 
diff --git a/init.c b/init.c
index a920c6e..ed34269 100644
--- a/init.c
+++ b/init.c
@@ -515,6 +515,8 @@ void td_fill_rand_seeds(struct thread_data *td)
 		td_fill_rand_seeds_os(td);
 	else
 		td_fill_rand_seeds_internal(td);
+
+	init_rand_seed(&td->buf_state, td->rand_seeds[7]);
 }
 
 /*
diff --git a/io_u.c b/io_u.c
index 16c98b1..a87c58e 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1450,7 +1450,7 @@ void io_u_fill_buffer(struct thread_data *td, struct io_u *io_u,
 	io_u->buf_filled_len = 0;
 
 	if (!td->o.zero_buffers)
-		fill_random_buf(io_u->buf, max_bs);
+		fill_random_buf(&td->buf_state, io_u->buf, max_bs);
 	else
 		memset(io_u->buf, 0, max_bs);
 }
diff --git a/lib/rand.c b/lib/rand.c
index 3b2d67a..7c6fed1 100644
--- a/lib/rand.c
+++ b/lib/rand.c
@@ -36,8 +36,6 @@
 #include "rand.h"
 #include "../hash.h"
 
-struct frand_state __fio_rand_state;
-
 static inline int __seed(unsigned int x, unsigned int m)
 {
 	return (x < m) ? x + m : x;
@@ -79,12 +77,13 @@ void __fill_random_buf(void *buf, unsigned int len, unsigned long seed)
 	}
 }
 
-unsigned long fill_random_buf(void *buf, unsigned int len)
+unsigned long fill_random_buf(struct frand_state *fs, void *buf,
+			      unsigned int len)
 {
-	unsigned long r = __rand(&__fio_rand_state);
+	unsigned long r = __rand(fs);
 
 	if (sizeof(int) != sizeof(long *))
-		r *= (unsigned long) __rand(&__fio_rand_state);
+		r *= (unsigned long) __rand(fs);
 
 	__fill_random_buf(buf, len, r);
 	return r;
diff --git a/lib/rand.h b/lib/rand.h
index f80c111..6b9e13c 100644
--- a/lib/rand.h
+++ b/lib/rand.h
@@ -7,8 +7,6 @@ struct frand_state {
 	unsigned int s1, s2, s3;
 };
 
-extern struct frand_state __fio_rand_state;
-
 static inline unsigned int __rand(struct frand_state *state)
 {
 #define TAUSWORTHE(s,a,b,c,d) ((s&c)<<d) ^ (((s <<a) ^ s)>>b)
@@ -23,6 +21,6 @@ static inline unsigned int __rand(struct frand_state *state)
 extern void init_rand(struct frand_state *);
 extern void init_rand_seed(struct frand_state *, unsigned int seed);
 extern void __fill_random_buf(void *buf, unsigned int len, unsigned long seed);
-extern unsigned long fill_random_buf(void *buf, unsigned int len);
+extern unsigned long fill_random_buf(struct frand_state *, void *buf, unsigned int len);
 
 #endif
diff --git a/verify.c b/verify.c
index fc207cf..c450e88 100644
--- a/verify.c
+++ b/verify.c
@@ -36,7 +36,7 @@ void fill_pattern(struct thread_data *td, void *p, unsigned int len, struct io_u
 		if (use_seed)
 			__fill_random_buf(p, len, seed);
 		else
-			io_u->rand_seed = fill_random_buf(p, len);
+			io_u->rand_seed = fill_random_buf(&td->buf_state, p, len);
 		break;
 	case 1:
 		/*

-- 
Jens Axboe

Re: fio file test patterns

[Jens Axboe]

On 2011-08-31 15:21, Jens Axboe wrote:
>> I also have another (maybe related?) question.  Apologies if this
>> belongs in a separate thread, but are there any notes explaining why
>> fio lays out the files before starting sequential writes?  The
>> workload I was hoping to simulate is sustained, sequential writes to
>> disk.  I'm trying to answer the question "How many simultaneous
>> 200kBps writers can we support?"  Using my current jobs file, fio
>> starts by creating the files (e.g "foo0: Laying out IO file(s) (1
>> file(s) / 4MB)") before it starts processing.  However, creating the
>> files in advance accounts for a chunk of performance that doesn't seem
>> to be measured by fio.  Am I misunderstanding how to configure fio or
>> its intended usage?
> 
> You should be able to set overwrite=0 to avoid that. Are they random
> writes?

overwrite=0 is even default. I'm thinking the "Laying out IO file"
message is confusing, it wont actually write contents first unless you
ask it to (with eg overwrite=1).


-- 
Jens Axboe

Re: fio file test patterns

[Brian Fallik]

Perhaps I should describe what I'm seeing.  I have two terminals open.
 In one, I launch:
  $ watch "ls -l foo* | head -n 20"
In the other terminal I kick off fio.  fio first spits out the "Laying
out" messages, one per job.  Immediately the full 4MB files appear in
the directory listing.  *Then* fio starts writing to the files, as
indicated by the fio output.  The size of each file never changes
during the test.

I wasn't expecting the full 4Mb files to be created before the
sequential writes.  I expected to not see any files initially, and
then a file for each job growing at a rate of 200Kbs throughout the
test until reaching the 4MB limit.

Perhaps I'm just configuring fio incorrectly?

brian


On Wed, Aug 31, 2011 at 5:38 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2011-08-31 15:21, Jens Axboe wrote:
>>> I also have another (maybe related?) question.  Apologies if this
>>> belongs in a separate thread, but are there any notes explaining why
>>> fio lays out the files before starting sequential writes?  The
>>> workload I was hoping to simulate is sustained, sequential writes to
>>> disk.  I'm trying to answer the question "How many simultaneous
>>> 200kBps writers can we support?"  Using my current jobs file, fio
>>> starts by creating the files (e.g "foo0: Laying out IO file(s) (1
>>> file(s) / 4MB)") before it starts processing.  However, creating the
>>> files in advance accounts for a chunk of performance that doesn't seem
>>> to be measured by fio.  Am I misunderstanding how to configure fio or
>>> its intended usage?
>>
>> You should be able to set overwrite=0 to avoid that. Are they random
>> writes?
>
> overwrite=0 is even default. I'm thinking the "Laying out IO file"
> message is confusing, it wont actually write contents first unless you
> ask it to (with eg overwrite=1).
>
>
> --
> Jens Axboe
>
>

Re: fio file test patterns

[Jens Axboe]

On 2011-09-01 08:18, Brian Fallik wrote:
> Perhaps I should describe what I'm seeing.  I have two terminals open.
>  In one, I launch:
>   $ watch "ls -l foo* | head -n 20"
> In the other terminal I kick off fio.  fio first spits out the "Laying
> out" messages, one per job.  Immediately the full 4MB files appear in
> the directory listing.  *Then* fio starts writing to the files, as
> indicated by the fio output.  The size of each file never changes
> during the test.
> 
> I wasn't expecting the full 4Mb files to be created before the
> sequential writes.  I expected to not see any files initially, and
> then a file for each job growing at a rate of 200Kbs throughout the
> test until reaching the 4MB limit.
> 
> Perhaps I'm just configuring fio incorrectly?

That you see them there does not mean they are actually written. Fio
will use truncate to setup the files, but only actually fill them with
data if you are going to be reading them (or requested overwrite).

BTW, please don't top-post. Reply beneath the original text, like a
normal flow of conversation.

-- 
Jens Axboe

Re: fio file test patterns

[Brian Fallik]

On Thu, Sep 1, 2011 at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2011-09-01 08:18, Brian Fallik wrote:
>> Perhaps I should describe what I'm seeing.  I have two terminals open.
>>  In one, I launch:
>>   $ watch "ls -l foo* | head -n 20"
>> In the other terminal I kick off fio.  fio first spits out the "Laying
>> out" messages, one per job.  Immediately the full 4MB files appear in
>> the directory listing.  *Then* fio starts writing to the files, as
>> indicated by the fio output.  The size of each file never changes
>> during the test.
>>
>> I wasn't expecting the full 4Mb files to be created before the
>> sequential writes.  I expected to not see any files initially, and
>> then a file for each job growing at a rate of 200Kbs throughout the
>> test until reaching the 4MB limit.
>>
>> Perhaps I'm just configuring fio incorrectly?
>
> That you see them there does not mean they are actually written. Fio
> will use truncate to setup the files, but only actually fill them with
> data if you are going to be reading them (or requested overwrite).

Digging deeper, I ran fio under strace:
  25445 stat("foo0", 0x7ffff0a586b0)      = -1 ENOENT (No such file or
directory)
  25445 unlink("foo0")                    = -1 ENOENT (No such file or
directory)
  25445 open("foo0", O_WRONLY|O_CREAT, 0644) = 3
  25445 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
  25445 fstatfs(3, {f_type=0x58465342, f_bsize=4096,
f_blocks=4880335840, f_bfree=4880324644, f_bavail=4880324644,
f_files=19521474432, f_ffree=19521474273, f_fsid={2065, 0},
f_namelen=255, f_frsize=4096}) = 0
  25445 pwrite(3, "\0", 1, 4095)          = 1
  25445 pwrite(3, "\0", 1, 8191)          = 1
  ...
  25445 pwrite(3, "\0", 1, 4194303)       = 1
  25445 fadvise64(3, 0, 4194304, POSIX_FADV_DONTNEED) = 0
  25445 close(3)
and then later in the log:
  25447 open("foo0", O_RDWR|O_CREAT, 0600) = 3
  25447 fadvise64(3, 0, 4194304, POSIX_FADV_DONTNEED) = 0
  25447 fadvise64(3, 0, 4194304, POSIX_FADV_SEQUENTIAL) = 0
  25447 lseek(3, 0, SEEK_SET)             = 0
  25447 write(3,
"\324\310\21i7\345\213g\32\271\323\214?\216)\33#7\255K\177ow\34\344F\306\2677\2630\24"...,
4096) = 4096
  25447 nanosleep({0, 19559000},  <unfinished ...>
  25447 <... nanosleep resumed> NULL)     = 0
  25447 write(3,
"\354\316\3126;\355\311A\335\331oj\320{\252\30;[\335\323IL\204\ngK\361\332\344m\344\37"...,
4096) = 4096
  ...
until the end of the test.

If I'm reading this correctly, during startup fio writes EOF to
successively larger file positions until it achieves the desired file
size.  Then, during the test itself, fio writes sequentially from the
start of the file.  I captured this on RHEL5 with xfs.  When I perform
the same test on Fedora 15 / ext4, fio does not use pwrite() but
instead calls fallocate().

I'm not sure why fallocate() isn't being used in both cases, but
regardless it seems like fio is instructing the filesystem to allocate
the blocks for the file before the sequential write test starts.  I
may be misunderstanding your last response, but I was expecting to see
an strace log containing:
  open()
  truncate()
to setup the test file and then:
  open()
  write()
  ...
during the actual test.

>
> BTW, please don't top-post. Reply beneath the original text, like a
> normal flow of conversation.

Apologies.


Thanks,
brian

Re: fio file test patterns

[Jens Axboe]

On 2011-09-01 12:17, Brian Fallik wrote:
> On Thu, Sep 1, 2011 at 12:06 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 2011-09-01 08:18, Brian Fallik wrote:
>>> Perhaps I should describe what I'm seeing.  I have two terminals open.
>>>  In one, I launch:
>>>   $ watch "ls -l foo* | head -n 20"
>>> In the other terminal I kick off fio.  fio first spits out the "Laying
>>> out" messages, one per job.  Immediately the full 4MB files appear in
>>> the directory listing.  *Then* fio starts writing to the files, as
>>> indicated by the fio output.  The size of each file never changes
>>> during the test.
>>>
>>> I wasn't expecting the full 4Mb files to be created before the
>>> sequential writes.  I expected to not see any files initially, and
>>> then a file for each job growing at a rate of 200Kbs throughout the
>>> test until reaching the 4MB limit.
>>>
>>> Perhaps I'm just configuring fio incorrectly?
>>
>> That you see them there does not mean they are actually written. Fio
>> will use truncate to setup the files, but only actually fill them with
>> data if you are going to be reading them (or requested overwrite).
> 
> Digging deeper, I ran fio under strace:
>   25445 stat("foo0", 0x7ffff0a586b0)      = -1 ENOENT (No such file or
> directory)
>   25445 unlink("foo0")                    = -1 ENOENT (No such file or
> directory)
>   25445 open("foo0", O_WRONLY|O_CREAT, 0644) = 3
>   25445 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>   25445 fstatfs(3, {f_type=0x58465342, f_bsize=4096,
> f_blocks=4880335840, f_bfree=4880324644, f_bavail=4880324644,
> f_files=19521474432, f_ffree=19521474273, f_fsid={2065, 0},
> f_namelen=255, f_frsize=4096}) = 0
>   25445 pwrite(3, "\0", 1, 4095)          = 1
>   25445 pwrite(3, "\0", 1, 8191)          = 1
>   ...
>   25445 pwrite(3, "\0", 1, 4194303)       = 1
>   25445 fadvise64(3, 0, 4194304, POSIX_FADV_DONTNEED) = 0
>   25445 close(3)

A quick guess would be that your glibc is too old, hence it "emulates"
truncate by helpfully zero filling the file.

-- 
Jens Axboe