Unlink after each loop in job

Unlink after each loop in job

[Martin Dev]

I'm after the following data pattern from a job, which is 1 out of 4
jobs run at the same time:

Targeted at the filesystem level
Write Xgb of verifiable data to file
Verify file
Delete file
Loop

The idea is we want to test trim at the file system level, whilst also
having 3 other jobs that perform different workloads.

Unlink=1 will only trigger after all loops of a job are complete
exec_post_run will only trigger after all loops of a job are complete

I have implemented the functionality I'm after, by adding a
unlink_each_loop paremeter to options.c, and in backend.c under the
while(keep_running) I've added an unlink() to both the if block
checking verify is false (before the loop continue) and after the
verification, if the flag is set.

The problem is, I'm not a C programmer (in the slightest), and I don't
know FIO at all well. My code probably does not live up to the
standard, or understand the internals of FIO well enough to handle
errors in the correct way etc.

So I guess I have 2 questions,

1) Have I missed something which already does what I need (delete file
on each loop of a write job)
2) How can I get someone who actually knows what they're doing to
implement this feature :)

--Martin

Re: Unlink after each loop in job

[Jens Axboe]

On 08/02/2016 11:21 PM, Martin Dev wrote:
> I'm after the following data pattern from a job, which is 1 out of 4
> jobs run at the same time:
>
> Targeted at the filesystem level
> Write Xgb of verifiable data to file
> Verify file
> Delete file
> Loop
>
> The idea is we want to test trim at the file system level, whilst also
> having 3 other jobs that perform different workloads.
>
> Unlink=1 will only trigger after all loops of a job are complete
> exec_post_run will only trigger after all loops of a job are complete

Correct.

> I have implemented the functionality I'm after, by adding a
> unlink_each_loop paremeter to options.c, and in backend.c under the
> while(keep_running) I've added an unlink() to both the if block
> checking verify is false (before the loop continue) and after the
> verification, if the flag is set.
>
> The problem is, I'm not a C programmer (in the slightest), and I don't
> know FIO at all well. My code probably does not live up to the
> standard, or understand the internals of FIO well enough to handle
> errors in the correct way etc.

Don't worry about that! You've already done way more than most people
that would like changes to be made, or features to be added. Send out
what you have, and we can help you drive it to completion.

> So I guess I have 2 questions,
>
> 1) Have I missed something which already does what I need (delete file
> on each loop of a write job)

I don't think we currently have it.

> 2) How can I get someone who actually knows what they're doing to
> implement this feature :)

See above :-)

-- 
Jens Axboe

Re: Unlink after each loop in job

[Martin Dev]

Hey Jens,

I applied your patch, but it doesn't unlink the file as it continues
at the !fio_file_open(f) in unlink_files(), run for 2 loops

Clear State: 0
Clear State: 1
Doing unlink
Starting file loop
File is not open!

I assume it's OK to flip it and check if the file is open then
continue (so not to unlink files that are potentially still in use)

I'll update the readme/docs and fio.1 as suggested.



On Thu, Aug 4, 2016 at 6:02 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 08/04/2016 09:50 AM, Jens Axboe wrote:
>>
>> On 08/04/2016 02:42 AM, Martin Dev wrote:
>>>
>>> Ok so I cleaned up the code a bit (I found fio_each_file), but I don't
>>> know the usual protocol for this kind of thing. I forked FIO on github
>>> and made some changes, which can be seen here
>>> https://github.com/axboe/fio/compare/master...mrturtledev:master
>>>
>>> This produces the desired functionality, but as you can see from the
>>> comments I don't know how to handle errors in the context of FIO (or
>>> in C in general). Here is a quick validation showing the deletion
>>> after each loop
>>> tester@--------:/usr/src/git$ cat fio/testfile
>>> [global]
>>> ioengine=libaio
>>> bs=4k
>>> size=200m
>>> verify=meta
>>> direct=1
>>> directory=/tmp/
>>> filename=ssd.test.file
>>>
>>> [seq-write]
>>> rw=write
>>> unlink_each_loop=1
>>> do_verify=1
>>> loops=2
>>>
>>> tester@--------:/usr/src/git/fio$ ./fio testfile
>>>
>>> tester@---------:/tmp$ inotifywait -m -e delete -e create -e access /tmp/
>>> Setting up watches.
>>> Watches established.
>>> /tmp/ CREATE ssd.test.file
>>> /tmp/ DELETE ssd.test.file
>>> /tmp/ CREATE ssd.test.file
>>> /tmp/ DELETE ssd.test.file
>>>
>>> Thanks for the help
>>
>>
>> It looks pretty close. You need to call td_io_unlink_file(td, f) instead
>> of using unlink() directly, as some IO engines don't have normal regular
>> local files. Additionally, you want to check for f->filetype ==
>> FIO_TYPE_FILE as well before doing unlink.
>>
>> Error handling should be fixed up, as you noted. I can do that for you.
>> The options part looks fine, however you also need to update fio.1 and
>> HOWTO to add these new options. On top of that, you can to modify the
>> thread_options_pack structure as well, and add the conversion of this
>> option to cconv.c, otherwise it won't work for networked clients. And
>> since that structure is modified, server.h FIO_SERVER_VER needs to be
>> incremented as we know have a differently sized structure we pass.
>>
>> If you modify the documentation, I can take care of the rest of the
>> options work mentioned above.
>
>
> On top of your patch, below should take care of most of it.
>
>
> diff --git a/backend.c b/backend.c
> index cd58af97472e..5074654525b8 100644
> --- a/backend.c
> +++ b/backend.c
> @@ -574,19 +574,24 @@ static inline bool io_in_polling(struct thread_data
> *td)
>  /*
>   * Unlinks files from thread data fio_file structure
>   */
> -static void do_unlink(struct thread_data *td)
> +static int unlink_files(struct thread_data *td)
>  {
>         struct fio_file *f;
>         unsigned int i;
> +       int ret = 0;
>
>         for_each_file(td, f, i) {
> -
>                 if (!fio_file_open(f))
>                         continue;
> +               if (f->filetype != FIO_TYPE_FILE)
> +                       continue;
>
> -               // Don't know how FIO propagates errors through the
> framework
> -               int ret = unlink(f->file_name);
> +               ret = td_io_unlink_file(td, f);
> +               if (ret)
> +                       break;
>         }
> +
> +       return ret;
>  }
>  /*
>   * The main verify engine. Runs over the writes we previously submitted,
> @@ -1683,9 +1688,13 @@ static void *thread_main(void *data)
>                 fio_gettime(&td->start, NULL);
>                 memcpy(&td->tv_cache, &td->start, sizeof(td->start));
>
> -               if (clear_state)
> +               if (clear_state) {
>                         clear_io_state(td, 0);
>
> +                       if (o->unlink_each_loop)
> +                               unlink_files(td);
> +               }
> +
>                 prune_io_piece_log(td);
>
>                 if (td->o.verify_only && (td_write(td) || td_rw(td)))
> @@ -1730,13 +1739,8 @@ static void *thread_main(void *data)
>
>                 if (!o->do_verify ||
>                     o->verify == VERIFY_NONE ||
> -                   (td->io_ops->flags & FIO_UNIDIR)) {
> -
> -                       if (o->unlink_each_loop)
> -                               do_unlink(td);
> -
> +                   (td->io_ops->flags & FIO_UNIDIR))
>                         continue;
> -               }
>
>                 clear_io_state(td, 0);
>
> @@ -1744,8 +1748,6 @@ static void *thread_main(void *data)
>
>                 do_verify(td, verify_bytes);
>
> -               if (o->unlink_each_loop)
> -                       do_unlink(td);
>                 /*
>                  * See comment further up for why this is done here.
>                  */
> diff --git a/cconv.c b/cconv.c
> index ac826a3038c9..5aeced76ee82 100644
> --- a/cconv.c
> +++ b/cconv.c
> @@ -172,6 +172,7 @@ void convert_thread_options_to_cpu(struct thread_options
> *o,
>         o->verify_batch = le32_to_cpu(top->verify_batch);
>         o->use_thread = le32_to_cpu(top->use_thread);
>         o->unlink = le32_to_cpu(top->unlink);
> +       o->unlink_each_loop = le32_to_cpu(top->unlink_each_loop);
>         o->do_disk_util = le32_to_cpu(top->do_disk_util);
>         o->override_sync = le32_to_cpu(top->override_sync);
>         o->rand_repeatable = le32_to_cpu(top->rand_repeatable);
> @@ -362,6 +363,7 @@ void convert_thread_options_to_net(struct
> thread_options_pack *top,
>         top->verify_batch = cpu_to_le32(o->verify_batch);
>         top->use_thread = cpu_to_le32(o->use_thread);
>         top->unlink = cpu_to_le32(o->unlink);
> +       top->unlink_each_loop = cpu_to_le32(o->unlink_each_loop);
>         top->do_disk_util = cpu_to_le32(o->do_disk_util);
>         top->override_sync = cpu_to_le32(o->override_sync);
>         top->rand_repeatable = cpu_to_le32(o->rand_repeatable);
> diff --git a/server.h b/server.h
> index 79c751de2388..c17c3bb571ae 100644
> --- a/server.h
> +++ b/server.h
> @@ -38,7 +38,7 @@ struct fio_net_cmd_reply {
>  };
>
>  enum {
> -       FIO_SERVER_VER                  = 54,
> +       FIO_SERVER_VER                  = 55,
>
>         FIO_SERVER_MAX_FRAGMENT_PDU     = 1024,
>         FIO_SERVER_MAX_CMD_MB           = 2048,
> diff --git a/thread_options.h b/thread_options.h
> index 28817c87f56f..53173dfebe13 100644
> --- a/thread_options.h
> +++ b/thread_options.h
> @@ -376,6 +376,7 @@ struct thread_options_pack {
>         uint32_t verify_state_save;
>         uint32_t use_thread;
>         uint32_t unlink;
> +       uint32_t unlink_each_loop;
>         uint32_t do_disk_util;
>         uint32_t override_sync;
>         uint32_t rand_repeatable;
> @@ -392,7 +393,6 @@ struct thread_options_pack {
>         uint32_t bs_unaligned;
>         uint32_t fsync_on_close;
>         uint32_t bs_is_seq_rand;
> -       uint32_t pad1;
>
>         uint32_t random_distribution;
>         uint32_t exitall_error;
>
> --
> Jens Axboe
>

Re: Unlink after each loop in job

[Jens Axboe]

On 08/05/2016 01:31 AM, Martin Dev wrote:
> Hey Jens,
>
> I applied your patch, but it doesn't unlink the file as it continues
> at the !fio_file_open(f) in unlink_files(), run for 2 loops
>
> Clear State: 0
> Clear State: 1
> Doing unlink
> Starting file loop
> File is not open!
>
> I assume it's OK to flip it and check if the file is open then
> continue (so not to unlink files that are potentially still in use)
>
> I'll update the readme/docs and fio.1 as suggested.

The file_open() should just be removed, at best, or at least we should 
flip the check, it doesn't make a lot of sense to only unlink if it's 
open. In any case, it doesn't really matter if it's open or not, last 
user will drop it.

So I'd just drop that check.

-- 
Jens Axboe

Re: Unlink after each loop in job

[Erwan Velu]

Hey Jens,

I don't see where the patch is, did I missed something ?

----- Mail original -----
De: "Jens Axboe" <axboe@kernel.dk>
À: "Martin Dev" <mrturtledev@gmail.com>, fio@vger.kernel.org
Envoyé: Vendredi 5 Août 2016 15:50:53
Objet: Re: Unlink after each loop in job

On 08/05/2016 01:31 AM, Martin Dev wrote:
> Hey Jens,
>
> I applied your patch, but it doesn't unlink the file as it continues
> at the !fio_file_open(f) in unlink_files(), run for 2 loops
>
> Clear State: 0
> Clear State: 1
> Doing unlink
> Starting file loop
> File is not open!
>
> I assume it's OK to flip it and check if the file is open then
> continue (so not to unlink files that are potentially still in use)
>
> I'll update the readme/docs and fio.1 as suggested.

The file_open() should just be removed, at best, or at least we should 
flip the check, it doesn't make a lot of sense to only unlink if it's 
open. In any case, it doesn't really matter if it's open or not, last 
user will drop it.

So I'd just drop that check.

-- 
Jens Axboe

Re: Unlink after each loop in job

[Jens Axboe]

On 08/08/2016 01:19 AM, Erwan Velu wrote:
> Hey Jens,
>
> I don't see where the patch is, did I missed something ?

It's not applied yet, I think what I meant to say is that I reviewed the 
patch.

-- 
Jens Axboe

Re: Unlink after each loop in job

[Erwan Velu]

Sorry, I was'nt that clear in my message :/

The guy that offered the patch have passed through the #fio channel on OFTC and we spoke about the need of that patch.
I was just curious to read it and wasn't able to find where the patch that is under discussion is.

Cheers,

----- Mail original -----
De: "Jens Axboe" <axboe@kernel.dk>
À: "Erwan Velu" <evelu@redhat.com>
Cc: "Martin Dev" <mrturtledev@gmail.com>, fio@vger.kernel.org
Envoyé: Lundi 8 Août 2016 15:50:32
Objet: Re: Unlink after each loop in job

On 08/08/2016 01:19 AM, Erwan Velu wrote:
> Hey Jens,
>
> I don't see where the patch is, did I missed something ?

It's not applied yet, I think what I meant to say is that I reviewed the 
patch.

-- 
Jens Axboe

Re: Unlink after each loop in job

[Jens Axboe]

On 08/08/2016 08:14 AM, Erwan Velu wrote:
> Sorry, I was'nt that clear in my message :/
>
> The guy that offered the patch have passed through the #fio channel on
> OFTC and we spoke about the need of that patch.  > I was just curious
> to read it and wasn't able to find where the patch that is under
> discussion is.

It just needs a little polish, then it can go in.

-- 
Jens Axboe

Re: Unlink after each loop in job

[Jens Axboe]

On 08/08/2016 08:18 AM, Jens Axboe wrote:
> On 08/08/2016 08:14 AM, Erwan Velu wrote:
>> Sorry, I was'nt that clear in my message :/
>>
>> The guy that offered the patch have passed through the #fio channel on
>> OFTC and we spoke about the need of that patch.  > I was just curious
>> to read it and wasn't able to find where the patch that is under
>> discussion is.
>
> It just needs a little polish, then it can go in.

Martin, can you check if this one works as you expect? It's building on
top of yours, adding the missing bits, and fixing up td_io_unlink_file()
error propagation as well.


diff --git a/HOWTO b/HOWTO
index 0085b7471ff6..9d71a96cc844 100644
--- a/HOWTO
+++ b/HOWTO
@@ -1334,6 +1334,8 @@ unlink=bool	Unlink the job files when done. Not 
the default, as repeated
  		runs of that job would then waste time recreating the file
  		set again and again.

+unlink_each_loop=bool	Unlink job files after each iteration or loop.
+
  loops=int	Run the specified number of iterations of this job. Used
  		to repeat the same workload a given number of times. Defaults
  		to 1.
diff --git a/backend.c b/backend.c
index c3ad8312de67..58727114ab8f 100644
--- a/backend.c
+++ b/backend.c
@@ -571,6 +571,28 @@ static inline bool io_in_polling(struct thread_data 
*td)
  	return !td->o.iodepth_batch_complete_min &&
  		   !td->o.iodepth_batch_complete_max;
  }
+/*
+ * Unlinks files from thread data fio_file structure
+ */
+static int unlink_all_files(struct thread_data *td)
+{
+	struct fio_file *f;
+	unsigned int i;
+	int ret = 0;
+
+	for_each_file(td, f, i) {
+		if (f->filetype != FIO_TYPE_FILE)
+			continue;
+		ret = td_io_unlink_file(td, f);
+		if (ret)
+			break;
+	}
+
+	if (ret)
+		td_verror(td, ret, "unlink_all_files");
+
+	return ret;
+}

  /*
   * The main verify engine. Runs over the writes we previously submitted,
@@ -1667,9 +1689,13 @@ static void *thread_main(void *data)
  		fio_gettime(&td->start, NULL);
  		memcpy(&td->tv_cache, &td->start, sizeof(td->start));

-		if (clear_state)
+		if (clear_state) {
  			clear_io_state(td, 0);

+			if (o->unlink_each_loop && unlink_all_files(td))
+				break;
+		}
+
  		prune_io_piece_log(td);

  		if (td->o.verify_only && (td_write(td) || td_rw(td)))
diff --git a/cconv.c b/cconv.c
index 837963d37354..8d9a0a8e00e9 100644
--- a/cconv.c
+++ b/cconv.c
@@ -174,6 +174,7 @@ void convert_thread_options_to_cpu(struct 
thread_options *o,
  	o->verify_batch = le32_to_cpu(top->verify_batch);
  	o->use_thread = le32_to_cpu(top->use_thread);
  	o->unlink = le32_to_cpu(top->unlink);
+	o->unlink_each_loop = le32_to_cpu(top->unlink_each_loop);
  	o->do_disk_util = le32_to_cpu(top->do_disk_util);
  	o->override_sync = le32_to_cpu(top->override_sync);
  	o->rand_repeatable = le32_to_cpu(top->rand_repeatable);
@@ -367,6 +368,7 @@ void convert_thread_options_to_net(struct 
thread_options_pack *top,
  	top->verify_batch = cpu_to_le32(o->verify_batch);
  	top->use_thread = cpu_to_le32(o->use_thread);
  	top->unlink = cpu_to_le32(o->unlink);
+	top->unlink_each_loop = cpu_to_le32(o->unlink_each_loop);
  	top->do_disk_util = cpu_to_le32(o->do_disk_util);
  	top->override_sync = cpu_to_le32(o->override_sync);
  	top->rand_repeatable = cpu_to_le32(o->rand_repeatable);
diff --git a/engines/pmemblk.c b/engines/pmemblk.c
index 6d19864ae0db..ca7269781951 100644
--- a/engines/pmemblk.c
+++ b/engines/pmemblk.c
@@ -475,7 +475,7 @@ static int fio_pmemblk_unlink_file(struct 
thread_data *td, struct fio_file *f)

  	pmb_parse_path(f->file_name, &path, &bsize, &fsize);
  	if (!path)
-		return 1;
+		return ENOENT;

  	unlink(path);
  	free(path);
diff --git a/filesetup.c b/filesetup.c
index 1ecdda61c3f4..42a9f4155fbf 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -58,8 +58,12 @@ static int extend_file(struct thread_data *td, struct 
fio_file *f)
  		unlink_file = 1;

  	if (unlink_file || new_layout) {
+		int ret;
+
  		dprint(FD_FILE, "layout unlink %s\n", f->file_name);
-		if ((td_io_unlink_file(td, f) < 0) && (errno != ENOENT)) {
+
+		ret = td_io_unlink_file(td, f);
+		if (ret != 0 && ret != ENOENT) {
  			td_verror(td, errno, "unlink");
  			return 1;
  		}
diff --git a/fio.1 b/fio.1
index d1acebcdeef1..696664afb081 100644
--- a/fio.1
+++ b/fio.1
@@ -1246,6 +1246,9 @@ multiple times. Thus it will not work on eg 
network or splice IO.
  .BI unlink \fR=\fPbool
  Unlink job files when done.  Default: false.
  .TP
+.BI unlink_each_loop \fR=\fPbool
+Unlink job files after each iteration or loop.  Default: false.
+.TP
  .BI loops \fR=\fPint
  Specifies the number of iterations (runs of the same workload) of this 
job.
  Default: 1.
diff --git a/ioengines.c b/ioengines.c
index 4129ac2363b9..a06909e0ff9d 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -521,8 +521,15 @@ int td_io_unlink_file(struct thread_data *td, 
struct fio_file *f)
  {
  	if (td->io_ops->unlink_file)
  		return td->io_ops->unlink_file(td, f);
-	else
-		return unlink(f->file_name);
+	else {
+		int ret;
+
+		ret = unlink(f->file_name);
+		if (ret < 0)
+			return errno;
+
+		return 0;
+	}
  }

  int td_io_get_file_size(struct thread_data *td, struct fio_file *f)
diff --git a/options.c b/options.c
index 56d3e2b6027e..7cd5121988c0 100644
--- a/options.c
+++ b/options.c
@@ -3433,6 +3433,16 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
  		.group	= FIO_OPT_G_INVALID,
  	},
  	{
+		.name	= "unlink_each_loop",
+		.lname	= "Unlink file after each loop of a job",
+		.type	= FIO_OPT_BOOL,
+		.off1	= td_var_offset(unlink_each_loop),
+		.help	= "Unlink created files after each loop in a job has completed",
+		.def	= "0",
+		.category = FIO_OPT_C_FILE,
+		.group	= FIO_OPT_G_INVALID,
+	},
+	{
  		.name	= "exitall",
  		.lname	= "Exit-all on terminate",
  		.type	= FIO_OPT_STR_SET,
diff --git a/server.h b/server.h
index c17c3bb571ae..fb384fb15feb 100644
--- a/server.h
+++ b/server.h
@@ -38,7 +38,7 @@ struct fio_net_cmd_reply {
  };

  enum {
-	FIO_SERVER_VER			= 55,
+	FIO_SERVER_VER			= 56,

  	FIO_SERVER_MAX_FRAGMENT_PDU	= 1024,
  	FIO_SERVER_MAX_CMD_MB		= 2048,
diff --git a/thread_options.h b/thread_options.h
index 449c66f5b8af..d70fda3f9491 100644
--- a/thread_options.h
+++ b/thread_options.h
@@ -121,6 +121,7 @@ struct thread_options {
  	unsigned int verify_state_save;
  	unsigned int use_thread;
  	unsigned int unlink;
+	unsigned int unlink_each_loop;
  	unsigned int do_disk_util;
  	unsigned int override_sync;
  	unsigned int rand_repeatable;
@@ -378,6 +379,7 @@ struct thread_options_pack {
  	uint32_t verify_state_save;
  	uint32_t use_thread;
  	uint32_t unlink;
+	uint32_t unlink_each_loop;
  	uint32_t do_disk_util;
  	uint32_t override_sync;
  	uint32_t rand_repeatable;
@@ -396,7 +398,6 @@ struct thread_options_pack {
  	uint32_t bs_unaligned;
  	uint32_t fsync_on_close;
  	uint32_t bs_is_seq_rand;
-	uint32_t pad1;

  	uint32_t random_distribution;
  	uint32_t exitall_error;

-- 
Jens Axboe

Re: Unlink after each loop in job

[Jens Axboe]

On 08/08/2016 09:33 AM, Jens Axboe wrote:
> On 08/08/2016 08:18 AM, Jens Axboe wrote:
>> On 08/08/2016 08:14 AM, Erwan Velu wrote:
>>> Sorry, I was'nt that clear in my message :/
>>>
>>> The guy that offered the patch have passed through the #fio channel on
>>> OFTC and we spoke about the need of that patch.  > I was just curious
>>> to read it and wasn't able to find where the patch that is under
>>> discussion is.
>>
>> It just needs a little polish, then it can go in.
>
> Martin, can you check if this one works as you expect? It's building on
> top of yours, adding the missing bits, and fixing up td_io_unlink_file()
> error propagation as well.

And here's one against the current master, since I committed the unlink
error propagation as a separate fix.


diff --git a/HOWTO b/HOWTO
index 0085b7471ff6..9d71a96cc844 100644
--- a/HOWTO
+++ b/HOWTO
@@ -1334,6 +1334,8 @@ unlink=bool	Unlink the job files when done. Not 
the default, as repeated
  		runs of that job would then waste time recreating the file
  		set again and again.

+unlink_each_loop=bool	Unlink job files after each iteration or loop.
+
  loops=int	Run the specified number of iterations of this job. Used
  		to repeat the same workload a given number of times. Defaults
  		to 1.
diff --git a/backend.c b/backend.c
index c3ad8312de67..58727114ab8f 100644
--- a/backend.c
+++ b/backend.c
@@ -571,6 +571,28 @@ static inline bool io_in_polling(struct thread_data 
*td)
  	return !td->o.iodepth_batch_complete_min &&
  		   !td->o.iodepth_batch_complete_max;
  }
+/*
+ * Unlinks files from thread data fio_file structure
+ */
+static int unlink_all_files(struct thread_data *td)
+{
+	struct fio_file *f;
+	unsigned int i;
+	int ret = 0;
+
+	for_each_file(td, f, i) {
+		if (f->filetype != FIO_TYPE_FILE)
+			continue;
+		ret = td_io_unlink_file(td, f);
+		if (ret)
+			break;
+	}
+
+	if (ret)
+		td_verror(td, ret, "unlink_all_files");
+
+	return ret;
+}

  /*
   * The main verify engine. Runs over the writes we previously submitted,
@@ -1667,9 +1689,13 @@ static void *thread_main(void *data)
  		fio_gettime(&td->start, NULL);
  		memcpy(&td->tv_cache, &td->start, sizeof(td->start));

-		if (clear_state)
+		if (clear_state) {
  			clear_io_state(td, 0);

+			if (o->unlink_each_loop && unlink_all_files(td))
+				break;
+		}
+
  		prune_io_piece_log(td);

  		if (td->o.verify_only && (td_write(td) || td_rw(td)))
diff --git a/cconv.c b/cconv.c
index 837963d37354..8d9a0a8e00e9 100644
--- a/cconv.c
+++ b/cconv.c
@@ -174,6 +174,7 @@ void convert_thread_options_to_cpu(struct 
thread_options *o,
  	o->verify_batch = le32_to_cpu(top->verify_batch);
  	o->use_thread = le32_to_cpu(top->use_thread);
  	o->unlink = le32_to_cpu(top->unlink);
+	o->unlink_each_loop = le32_to_cpu(top->unlink_each_loop);
  	o->do_disk_util = le32_to_cpu(top->do_disk_util);
  	o->override_sync = le32_to_cpu(top->override_sync);
  	o->rand_repeatable = le32_to_cpu(top->rand_repeatable);
@@ -367,6 +368,7 @@ void convert_thread_options_to_net(struct 
thread_options_pack *top,
  	top->verify_batch = cpu_to_le32(o->verify_batch);
  	top->use_thread = cpu_to_le32(o->use_thread);
  	top->unlink = cpu_to_le32(o->unlink);
+	top->unlink_each_loop = cpu_to_le32(o->unlink_each_loop);
  	top->do_disk_util = cpu_to_le32(o->do_disk_util);
  	top->override_sync = cpu_to_le32(o->override_sync);
  	top->rand_repeatable = cpu_to_le32(o->rand_repeatable);
diff --git a/fio.1 b/fio.1
index d1acebcdeef1..696664afb081 100644
--- a/fio.1
+++ b/fio.1
@@ -1246,6 +1246,9 @@ multiple times. Thus it will not work on eg 
network or splice IO.
  .BI unlink \fR=\fPbool
  Unlink job files when done.  Default: false.
  .TP
+.BI unlink_each_loop \fR=\fPbool
+Unlink job files after each iteration or loop.  Default: false.
+.TP
  .BI loops \fR=\fPint
  Specifies the number of iterations (runs of the same workload) of this 
job.
  Default: 1.
diff --git a/options.c b/options.c
index 56d3e2b6027e..7cd5121988c0 100644
--- a/options.c
+++ b/options.c
@@ -3433,6 +3433,16 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
  		.group	= FIO_OPT_G_INVALID,
  	},
  	{
+		.name	= "unlink_each_loop",
+		.lname	= "Unlink file after each loop of a job",
+		.type	= FIO_OPT_BOOL,
+		.off1	= td_var_offset(unlink_each_loop),
+		.help	= "Unlink created files after each loop in a job has completed",
+		.def	= "0",
+		.category = FIO_OPT_C_FILE,
+		.group	= FIO_OPT_G_INVALID,
+	},
+	{
  		.name	= "exitall",
  		.lname	= "Exit-all on terminate",
  		.type	= FIO_OPT_STR_SET,
diff --git a/server.h b/server.h
index c17c3bb571ae..fb384fb15feb 100644
--- a/server.h
+++ b/server.h
@@ -38,7 +38,7 @@ struct fio_net_cmd_reply {
  };

  enum {
-	FIO_SERVER_VER			= 55,
+	FIO_SERVER_VER			= 56,

  	FIO_SERVER_MAX_FRAGMENT_PDU	= 1024,
  	FIO_SERVER_MAX_CMD_MB		= 2048,
diff --git a/thread_options.h b/thread_options.h
index 449c66f5b8af..d70fda3f9491 100644
--- a/thread_options.h
+++ b/thread_options.h
@@ -121,6 +121,7 @@ struct thread_options {
  	unsigned int verify_state_save;
  	unsigned int use_thread;
  	unsigned int unlink;
+	unsigned int unlink_each_loop;
  	unsigned int do_disk_util;
  	unsigned int override_sync;
  	unsigned int rand_repeatable;
@@ -378,6 +379,7 @@ struct thread_options_pack {
  	uint32_t verify_state_save;
  	uint32_t use_thread;
  	uint32_t unlink;
+	uint32_t unlink_each_loop;
  	uint32_t do_disk_util;
  	uint32_t override_sync;
  	uint32_t rand_repeatable;
@@ -396,7 +398,6 @@ struct thread_options_pack {
  	uint32_t bs_unaligned;
  	uint32_t fsync_on_close;
  	uint32_t bs_is_seq_rand;
-	uint32_t pad1;

  	uint32_t random_distribution;
  	uint32_t exitall_error;

-- 
Jens Axboe