Tell fio to only lay out files

Tell fio to only lay out files

[Georg Schönberger]

Hello Everyone,

is it possible to tell fio to only lay out the data files and not carry
out any tests?
The thing is that I need to sync my Flashcache device after laying out
the files and before starting the read tests. Therefore it would be
helpful if I could do the following:
-Start fio and tell him to only lay out data files
-Sync my cache
-Start fio again with the previous layed out files and do the tests

Another option would be the parameter "filename", which tells fio to use
that file. But what if I am using multiple processes, can I specify a
file on command line for each process?

Thanks, Georg

Re: Tell fio to only lay out files

[Jens Axboe]

On 05/02/2012 11:15 AM, Georg Schï¿œnberger wrote:
> Hello Everyone,
> 
> is it possible to tell fio to only lay out the data files and not carry
> out any tests?
> The thing is that I need to sync my Flashcache device after laying out
> the files and before starting the read tests. Therefore it would be
> helpful if I could do the following:
> -Start fio and tell him to only lay out data files
> -Sync my cache
> -Start fio again with the previous layed out files and do the tests

That's not a bad idea. The below patch will add a create_only option. If
set, fio will exit after file creation is completed.

> Another option would be the parameter "filename", which tells fio to use
> that file. But what if I am using multiple processes, can I specify a
> file on command line for each process?

You could do that as well, but it requires you to know how fio names
its files for the jobs. Which isn't tricky, it's basically:

        name.threadno.fileno

So if you have

[foo]
numjobs=4

as the first entry, you'll have foo.0.[1-4] as the file names. But the
setup only option is cleaner and then you don't have to document and
stick to the file naming :-)

diff --git a/filesetup.c b/filesetup.c
index f3d3829..a1ad026 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -665,7 +665,7 @@ int setup_files(struct thread_data *td)
 	dprint(FD_FILE, "setup files\n");
 
 	if (td->o.read_iolog_file)
-		return 0;
+		goto done;
 
 	/*
 	 * if ioengine defines a setup() method, it's responsible for
@@ -816,6 +816,11 @@ int setup_files(struct thread_data *td)
 	 */
 	if (!td->o.read_iolog_file)
 		td->total_io_size = td->o.size * td->o.loops;
+
+done:
+	if (td->o.create_only)
+		td->done = 1;
+
 	return 0;
 err_offset:
 	log_err("%s: you need to specify valid offset=\n", td->o.name);
diff --git a/fio.h b/fio.h
index 6da22f0..f2a5a1f 100644
--- a/fio.h
+++ b/fio.h
@@ -125,6 +125,7 @@ struct thread_options {
 	unsigned int create_serialize;
 	unsigned int create_fsync;
 	unsigned int create_on_open;
+	unsigned int create_only;
 	unsigned int end_fsync;
 	unsigned int pre_read;
 	unsigned int sync_io;
diff --git a/options.c b/options.c
index f8927ee..28a228c 100644
--- a/options.c
+++ b/options.c
@@ -1885,6 +1885,13 @@ static struct fio_option options[FIO_MAX_OPTS] = {
 		.def	= "0",
 	},
 	{
+		.name	= "create_only",
+		.type	= FIO_OPT_BOOL,
+		.off1	= td_var_offset(create_only),
+		.help	= "Only perform file creation phase",
+		.def	= "0",
+	},
+	{
 		.name	= "pre_read",
 		.type	= FIO_OPT_BOOL,
 		.off1	= td_var_offset(pre_read),

-- 
Jens Axboe

Re: Tell fio to only lay out files

[Georg Schönberger]

Thanks for your fast patch, I really appreciate that :-)
The create_only option now helps me to start with a consistent cache
state for every test run.

Georg

Am 2012-05-02 12:35, schrieb Jens Axboe:
> On 05/02/2012 11:15 AM, Georg Schönberger wrote:
>> Hello Everyone,
>>
>> is it possible to tell fio to only lay out the data files and not carry
>> out any tests?
>> The thing is that I need to sync my Flashcache device after laying out
>> the files and before starting the read tests. Therefore it would be
>> helpful if I could do the following:
>> -Start fio and tell him to only lay out data files
>> -Sync my cache
>> -Start fio again with the previous layed out files and do the tests
> That's not a bad idea. The below patch will add a create_only option. If
> set, fio will exit after file creation is completed.
>
>> Another option would be the parameter "filename", which tells fio to use
>> that file. But what if I am using multiple processes, can I specify a
>> file on command line for each process?
> You could do that as well, but it requires you to know how fio names
> its files for the jobs. Which isn't tricky, it's basically:
>
>         name.threadno.fileno
>
> So if you have
>
> [foo]
> numjobs=4
>
> as the first entry, you'll have foo.0.[1-4] as the file names. But the
> setup only option is cleaner and then you don't have to document and
> stick to the file naming :-)
>
> diff --git a/filesetup.c b/filesetup.c
> index f3d3829..a1ad026 100644
> --- a/filesetup.c
> +++ b/filesetup.c
> @@ -665,7 +665,7 @@ int setup_files(struct thread_data *td)
>  	dprint(FD_FILE, "setup files\n");
>  
>  	if (td->o.read_iolog_file)
> -		return 0;
> +		goto done;
>  
>  	/*
>  	 * if ioengine defines a setup() method, it's responsible for
> @@ -816,6 +816,11 @@ int setup_files(struct thread_data *td)
>  	 */
>  	if (!td->o.read_iolog_file)
>  		td->total_io_size = td->o.size * td->o.loops;
> +
> +done:
> +	if (td->o.create_only)
> +		td->done = 1;
> +
>  	return 0;
>  err_offset:
>  	log_err("%s: you need to specify valid offset=\n", td->o.name);
> diff --git a/fio.h b/fio.h
> index 6da22f0..f2a5a1f 100644
> --- a/fio.h
> +++ b/fio.h
> @@ -125,6 +125,7 @@ struct thread_options {
>  	unsigned int create_serialize;
>  	unsigned int create_fsync;
>  	unsigned int create_on_open;
> +	unsigned int create_only;
>  	unsigned int end_fsync;
>  	unsigned int pre_read;
>  	unsigned int sync_io;
> diff --git a/options.c b/options.c
> index f8927ee..28a228c 100644
> --- a/options.c
> +++ b/options.c
> @@ -1885,6 +1885,13 @@ static struct fio_option options[FIO_MAX_OPTS] = {
>  		.def	= "0",
>  	},
>  	{
> +		.name	= "create_only",
> +		.type	= FIO_OPT_BOOL,
> +		.off1	= td_var_offset(create_only),
> +		.help	= "Only perform file creation phase",
> +		.def	= "0",
> +	},
> +	{
>  		.name	= "pre_read",
>  		.type	= FIO_OPT_BOOL,
>  		.off1	= td_var_offset(pre_read),
>

Re: Tell fio to only lay out files

[Jens Axboe]

On 05/02/2012 01:51 PM, Georg Schï¿œnberger wrote:
> Thanks for your fast patch, I really appreciate that :-)
> The create_only option now helps me to start with a consistent cache
> state for every test run.

Did you test it? I'll add the option to the man page and HOWTO and
commit, then.

-- 
Jens Axboe

Re: Tell fio to only lay out files

[Georg Schönberger]

I get the following error after applying the patch:
$ patch -b -p1 < fio.patch
...make and install...
$ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
--rw=read --size=10M --bs=4k --numjobs=4 --group_reporting
max value out of range: 4 (1 max)
max value out of range: 4 (1 max)

Georg

Am 2012-05-02 13:55, schrieb Jens Axboe:
> On 05/02/2012 01:51 PM, Georg Schï¿œnberger wrote:
>> Thanks for your fast patch, I really appreciate that :-)
>> The create_only option now helps me to start with a consistent cache
>> state for every test run.
> Did you test it? I'll add the option to the man page and HOWTO and
> commit, then.
>

Re: Tell fio to only lay out files

[Jens Axboe]

On 05/02/2012 02:06 PM, Georg Schï¿œnberger wrote:
> I get the following error after applying the patch:
> $ patch -b -p1 < fio.patch
> ...make and install...
> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
> --rw=read --size=10M --bs=4k --numjobs=4 --group_reporting
> max value out of range: 4 (1 max)
> max value out of range: 4 (1 max)

You probably want to make clean first.

-- 
Jens Axboe

Re: Tell fio to only lay out files

[Georg Schönberger]

Same error after a clean compilation.
$ fio --version
fio-2.0.7-13-g3b7f
$ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
--rw=read --size=10M --bs=4k --numjobs=4
max value out of range: 4 (1 max)
max value out of range: 4 (1 max)

Am 2012-05-02 14:08, schrieb Jens Axboe:
> On 05/02/2012 02:06 PM, Georg Schï¿œnberger wrote:
>> I get the following error after applying the patch:
>> $ patch -b -p1 < fio.patch
>> ...make and install...
>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>> --rw=read --size=10M --bs=4k --numjobs=4 --group_reporting
>> max value out of range: 4 (1 max)
>> max value out of range: 4 (1 max)
> You probably want to make clean first.
>

Re: Tell fio to only lay out files

[Jens Axboe]

On 05/02/2012 02:12 PM, Georg Schï¿œnberger wrote:
> Same error after a clean compilation.
> $ fio --version
> fio-2.0.7-13-g3b7f
> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
> --rw=read --size=10M --bs=4k --numjobs=4
> max value out of range: 4 (1 max)
> max value out of range: 4 (1 max)

Please don't top post, reply beneath what you are replying to.

Parser looks confused, because you forgot to add the value for the
create_only option. If you use --create_only=1 instead, it should work.

I wonder why that isn't caught in the command line variant, if you put
it in a job file it does catch it:

axboe@nelson:/home/axboe/git/fio $ ./fio seq-null 
Option create_only requires an argument
fio: failed parsing create_only
fio: job global dropped

I'll take a look at that separately.

-- 
Jens Axboe

Re: Tell fio to only lay out files

[Georg Schönberger]

Am 2012-05-02 14:15, schrieb Jens Axboe:
> On 05/02/2012 02:12 PM, Georg Schï¿œnberger wrote:
>> Same error after a clean compilation.
>> $ fio --version
>> fio-2.0.7-13-g3b7f
>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>> --rw=read --size=10M --bs=4k --numjobs=4
>> max value out of range: 4 (1 max)
>> max value out of range: 4 (1 max)
> Please don't top post, reply beneath what you are replying to.
>
> Parser looks confused, because you forgot to add the value for the
> create_only option. If you use --create_only=1 instead, it should work.
>
> I wonder why that isn't caught in the command line variant, if you put
> it in a job file it does catch it:
>
> axboe@nelson:/home/axboe/git/fio $ ./fio seq-null 
> Option create_only requires an argument
> fio: failed parsing create_only
> fio: job global dropped
>
> I'll take a look at that separately.
>
That works now:
$ fio --create_only=1 --name=test --directory=./tests --rw=read
--size=10M --bs=4k --numjobs=4
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.7-13-g3b7f
Starting 4 processes
test: Laying out IO file(s) (1 file(s) / 10MB)
test: Laying out IO file(s) (1 file(s) / 10MB)
test: Laying out IO file(s) (1 file(s) / 10MB)
test: Laying out IO file(s) (1 file(s) / 10MB)


Run status group 0 (all jobs):
$ ls tests/
test.1.0  test.2.0  test.3.0  test.4.0

Nice!
Thanks, Georg

P.S.: Sorry for top posting

Re: Tell fio to only lay out files

[Vikram Seth]

On Wed, May 2, 2012 at 5:25 AM, Georg Schönberger
<gschoenberger@thomas-krenn.com> wrote:
> Am 2012-05-02 14:15, schrieb Jens Axboe:
>> On 05/02/2012 02:12 PM, Georg Schönberger wrote:
>>> Same error after a clean compilation.
>>> $ fio --version
>>> fio-2.0.7-13-g3b7f
>>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>>> --rw=read --size=10M --bs=4k --numjobs=4
>>> max value out of range: 4 (1 max)
>>> max value out of range: 4 (1 max)
>> Please don't top post, reply beneath what you are replying to.
>>
>> Parser looks confused, because you forgot to add the value for the
>> create_only option. If you use --create_only=1 instead, it should work.
>>
>> I wonder why that isn't caught in the command line variant, if you put
>> it in a job file it does catch it:
>>
>> axboe@nelson:/home/axboe/git/fio $ ./fio seq-null
>> Option create_only requires an argument
>> fio: failed parsing create_only
>> fio: job global dropped
>>
>> I'll take a look at that separately.
>>
> That works now:
> $ fio --create_only=1 --name=test --directory=./tests --rw=read
> --size=10M --bs=4k --numjobs=4
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> ...
> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
> fio-2.0.7-13-g3b7f
> Starting 4 processes
> test: Laying out IO file(s) (1 file(s) / 10MB)
> test: Laying out IO file(s) (1 file(s) / 10MB)
> test: Laying out IO file(s) (1 file(s) / 10MB)
> test: Laying out IO file(s) (1 file(s) / 10MB)
>
>
> Run status group 0 (all jobs):
> $ ls tests/
> test.1.0  test.2.0  test.3.0  test.4.0
>
> Nice!
> Thanks, Georg
>
> P.S.: Sorry for top posting
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Jens,

Does this mean that if we create the files with appropriate names and
size before starting fio, then fio will not lay them out explicitly?
Just confirming .. this will be very useful (reduce runtime) when
performing tests on large filesystems.

Thanks,

Vikram.

Re: Tell fio to only lay out files

[Jens Axboe]

On 2012-05-09 22:52, Vikram Seth wrote:
> On Wed, May 2, 2012 at 5:25 AM, Georg Sch�nberger
> <gschoenberger@thomas-krenn.com> wrote:
>> Am 2012-05-02 14:15, schrieb Jens Axboe:
>>> On 05/02/2012 02:12 PM, Georg Sch�nberger wrote:
>>>> Same error after a clean compilation.
>>>> $ fio --version
>>>> fio-2.0.7-13-g3b7f
>>>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>>>> --rw=read --size=10M --bs=4k --numjobs=4
>>>> max value out of range: 4 (1 max)
>>>> max value out of range: 4 (1 max)
>>> Please don't top post, reply beneath what you are replying to.
>>>
>>> Parser looks confused, because you forgot to add the value for the
>>> create_only option. If you use --create_only=1 instead, it should work.
>>>
>>> I wonder why that isn't caught in the command line variant, if you put
>>> it in a job file it does catch it:
>>>
>>> axboe@nelson:/home/axboe/git/fio $ ./fio seq-null
>>> Option create_only requires an argument
>>> fio: failed parsing create_only
>>> fio: job global dropped
>>>
>>> I'll take a look at that separately.
>>>
>> That works now:
>> $ fio --create_only=1 --name=test --directory=./tests --rw=read
>> --size=10M --bs=4k --numjobs=4
>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> ...
>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>> fio-2.0.7-13-g3b7f
>> Starting 4 processes
>> test: Laying out IO file(s) (1 file(s) / 10MB)
>> test: Laying out IO file(s) (1 file(s) / 10MB)
>> test: Laying out IO file(s) (1 file(s) / 10MB)
>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>
>>
>> Run status group 0 (all jobs):
>> $ ls tests/
>> test.1.0  test.2.0  test.3.0  test.4.0
>>
>> Nice!
>> Thanks, Georg
>>
>> P.S.: Sorry for top posting
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Hi Jens,
> 
> Does this mean that if we create the files with appropriate names and
> size before starting fio, then fio will not lay them out explicitly?
> Just confirming .. this will be very useful (reduce runtime) when
> performing tests on large filesystems.

It does, yes, or you can just use this create_only option for the same
effect.

Fio uses repeatable file names, it always has.

-- 
Jens Axboe

Re: Tell fio to only lay out files

[Vikram Seth]

On Wed, May 9, 2012 at 1:53 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2012-05-09 22:52, Vikram Seth wrote:
>> On Wed, May 2, 2012 at 5:25 AM, Georg Schönberger
>> <gschoenberger@thomas-krenn.com> wrote:
>>> Am 2012-05-02 14:15, schrieb Jens Axboe:
>>>> On 05/02/2012 02:12 PM, Georg Schönberger wrote:
>>>>> Same error after a clean compilation.
>>>>> $ fio --version
>>>>> fio-2.0.7-13-g3b7f
>>>>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>>>>> --rw=read --size=10M --bs=4k --numjobs=4
>>>>> max value out of range: 4 (1 max)
>>>>> max value out of range: 4 (1 max)
>>>> Please don't top post, reply beneath what you are replying to.
>>>>
>>>> Parser looks confused, because you forgot to add the value for the
>>>> create_only option. If you use --create_only=1 instead, it should work.
>>>>
>>>> I wonder why that isn't caught in the command line variant, if you put
>>>> it in a job file it does catch it:
>>>>
>>>> axboe@nelson:/home/axboe/git/fio $ ./fio seq-null
>>>> Option create_only requires an argument
>>>> fio: failed parsing create_only
>>>> fio: job global dropped
>>>>
>>>> I'll take a look at that separately.
>>>>
>>> That works now:
>>> $ fio --create_only=1 --name=test --directory=./tests --rw=read
>>> --size=10M --bs=4k --numjobs=4
>>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>>> ...
>>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>>> fio-2.0.7-13-g3b7f
>>> Starting 4 processes
>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>>
>>>
>>> Run status group 0 (all jobs):
>>> $ ls tests/
>>> test.1.0  test.2.0  test.3.0  test.4.0
>>>
>>> Nice!
>>> Thanks, Georg
>>>
>>> P.S.: Sorry for top posting
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> Hi Jens,
>>
>> Does this mean that if we create the files with appropriate names and
>> size before starting fio, then fio will not lay them out explicitly?
>> Just confirming .. this will be very useful (reduce runtime) when
>> performing tests on large filesystems.
>
> It does, yes, or you can just use this create_only option for the same
> effect.
>
> Fio uses repeatable file names, it always has.
>
> --
> Jens Axboe
>

I guess my question was is that if I created the files using some
other method other than create_only option, will this still work ?
Will fio look for needed files and skip layout step if they already
exist?

Vikram

Re: Tell fio to only lay out files

[Jens Axboe]

On 10/05/2012, at 00.10, Vikram Seth <seth.vik@gmail.com> wrote:

> On Wed, May 9, 2012 at 1:53 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 2012-05-09 22:52, Vikram Seth wrote:
>>> On Wed, May 2, 2012 at 5:25 AM, Georg Schönberger
>>> <gschoenberger@thomas-krenn.com> wrote:
>>>> Am 2012-05-02 14:15, schrieb Jens Axboe:
>>>>> On 05/02/2012 02:12 PM, Georg Schönberger wrote:
>>>>>> Same error after a clean compilation.
>>>>>> $ fio --version
>>>>>> fio-2.0.7-13-g3b7f
>>>>>> $ fio --create_only --name=test-read-4k-4G-1-jobs --directory=./tests
>>>>>> --rw=read --size=10M --bs=4k --numjobs=4
>>>>>> max value out of range: 4 (1 max)
>>>>>> max value out of range: 4 (1 max)
>>>>> Please don't top post, reply beneath what you are replying to.
>>>>> 
>>>>> Parser looks confused, because you forgot to add the value for the
>>>>> create_only option. If you use --create_only=1 instead, it should work.
>>>>> 
>>>>> I wonder why that isn't caught in the command line variant, if you put
>>>>> it in a job file it does catch it:
>>>>> 
>>>>> axboe@nelson:/home/axboe/git/fio $ ./fio seq-null
>>>>> Option create_only requires an argument
>>>>> fio: failed parsing create_only
>>>>> fio: job global dropped
>>>>> 
>>>>> I'll take a look at that separately.
>>>>> 
>>>> That works now:
>>>> $ fio --create_only=1 --name=test --directory=./tests --rw=read
>>>> --size=10M --bs=4k --numjobs=4
>>>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>>>> ...
>>>> test: (g=0): rw=read, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
>>>> fio-2.0.7-13-g3b7f
>>>> Starting 4 processes
>>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>>> test: Laying out IO file(s) (1 file(s) / 10MB)
>>>> 
>>>> 
>>>> Run status group 0 (all jobs):
>>>> $ ls tests/
>>>> test.1.0  test.2.0  test.3.0  test.4.0
>>>> 
>>>> Nice!
>>>> Thanks, Georg
>>>> 
>>>> P.S.: Sorry for top posting
>>>> 
>>>> 
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> 
>>> Hi Jens,
>>> 
>>> Does this mean that if we create the files with appropriate names and
>>> size before starting fio, then fio will not lay them out explicitly?
>>> Just confirming .. this will be very useful (reduce runtime) when
>>> performing tests on large filesystems.
>> 
>> It does, yes, or you can just use this create_only option for the same
>> effect.
>> 
>> Fio uses repeatable file names, it always has.
>> 
>> --
>> Jens Axboe
>> 
> 
> I guess my question was is that if I created the files using some
> other method other than create_only option, will this still work ?
> Will fio look for needed files and skip layout step if they already
> exist?

Yes, fio doesn't care. If the files are already there and of the right size, it will skip the layout phase completely.

Jens