* [patch 1/9] fio: fix job clone mem leak
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 2/9] fio: allow general repeatability Christian Ehrhardt
` (7 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
In the loop to create clones at the bottom of add_job the function
get_new_job
clones the thread_data, just to occaisonally get the allocated pointers for
filename and files overwritten a few lines later.
The dup files also duplicates the name strings so the references to
these are
lost by the setting to null.
This patch fixes takes care of that and frees the memory before
discarding the
pointers (found via valgrind).
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
init.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
[diff]
--- a/init.c
+++ b/init.c
@@ -1118,10 +1118,21 @@ static int add_job(struct thread_data *t
td_new->o.new_group = 0;
if (file_alloced) {
- td_new->o.filename = NULL;
td_new->files_index = 0;
td_new->files_size = 0;
- td_new->files = NULL;
+ if (td_new->files) {
+ struct fio_file *f;
+ for_each_file(td_new, f, i) {
+ if (f->file_name)
+ free(f->file_name);
+ free(f);
+ }
+ td_new->files = NULL;
+ }
+ if (td_new->o.filename) {
+ free(td_new->o.filename);
+ td_new->o.filename = NULL;
+ }
}
job_add_num = numjobs - 1;
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 2/9] fio: allow general repeatability
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
2014-02-19 15:12 ` [patch 1/9] fio: fix job clone mem leak Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 3/9] fio: allow milliseconds on all time specifiers Christian Ehrhardt
` (6 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
This patch adds a "allrandrepeat" option similar to the existing randrepeat.
But other than the former option it does not only affect the random I/O
pattern,
but also all remaining users of randomness as well.
By that e.g. testcases using nrfiles, filesize ranges, blocksizesplit ranges
and so on will stay repeatable across (euqally parametrized) runs as well.
To maintain compatibility the default is off.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
fio.1 | 8 ++++++--
init.c | 6 ++++++
options.c | 7 +++++++
3 files changed, 19 insertions(+), 2 deletions(-)
[diff]
--- a/fio.1
+++ b/fio.1
@@ -293,8 +293,12 @@ read, write, and trim are accounted and
set, the fio will sum the results and report them as "mixed" instead.
.TP
.BI randrepeat \fR=\fPbool
-Seed the random number generator in a predictable way so results are
repeatable
-across runs. Default: true.
+Seed the random number generator used for random I/O patterns in a
predictable
+way so the pattern is repeatable across runs. Default: true.
+.TP
+.BI allrandrepeat \fR=\fPbool
+Seed all random number generators in a predictable way so results are
+repeatable across runs. Default: false.
.TP
.BI randseed \fR=\fPint
Seed the random number generators based on this seed value, to be able to
--- a/init.c
+++ b/init.c
@@ -751,6 +751,12 @@ static void td_fill_rand_seeds_internal(
void td_fill_rand_seeds(struct thread_data *td)
{
+ if (td->o.allrand_repeatable) {
+ for (int i = 0; i < FIO_RAND_NR_OFFS; i++)
+ td->rand_seeds[i] = FIO_RANDSEED * td->thread_number
+ + i;
+ }
+
if (td->o.use_os_rand)
td_fill_rand_seeds_os(td);
else
--- a/options.c
+++ b/options.c
@@ -1784,6 +1784,13 @@ struct fio_option fio_options[FIO_MAX_OP
.group = FIO_OPT_G_RANDOM,
},
{
+ .name = "allrandrepeat",
+ .type = FIO_OPT_BOOL,
+ .off1 = td_var_offset(allrand_repeatable),
+ .help = "Use repeatable random numbers for everything",
+ .def = "0",
+ },
+ {
.name = "nrfiles",
.lname = "Number of files",
.alias = "nr_files",
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 3/9] fio: allow milliseconds on all time specifiers
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
2014-02-19 15:12 ` [patch 1/9] fio: fix job clone mem leak Christian Ehrhardt
2014-02-19 15:12 ` [patch 2/9] fio: allow general repeatability Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 4/9] fio: provide an option for a startdelay range Christian Ehrhardt
` (5 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
This patch allows all time specifiers to be specified down to milliseconds.
Default will stay seconds for compatibility with old configs.
It also adds documentation to the existing time units day, hour and minute.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
backend.c | 4 ++--
eta.c | 14 ++++++++------
fio.1 | 16 ++++++++++------
parse.c | 48 ++++++++++++++++++++++++++++++++++--------------
time.c | 2 +-
5 files changed, 55 insertions(+), 29 deletions(-)
[diff]
--- a/backend.c
+++ b/backend.c
@@ -346,7 +346,7 @@ static inline int runtime_exceeded(struc
return 0;
if (!td->o.timeout)
return 0;
- if (mtime_since(&td->epoch, t) >= td->o.timeout * 1000)
+ if (mtime_since(&td->epoch, t) >= td->o.timeout )
return 1;
return 0;
@@ -1783,7 +1783,7 @@ static void run_threads(void)
if (td->o.start_delay) {
spent = mtime_since_genesis();
- if (td->o.start_delay * 1000 > spent)
+ if (td->o.start_delay > spent)
continue;
}
--- a/eta.c
+++ b/eta.c
@@ -174,7 +174,8 @@ static int thread_eta(struct thread_data
perc = 1.0;
if (td->o.time_based) {
- perc_t = (double) elapsed / (double) td->o.timeout;
+ perc_t = (double) elapsed /
+ (double) (td->o.timeout / 1000);
if (perc_t < perc)
perc = perc_t;
}
@@ -182,8 +183,9 @@ static int thread_eta(struct thread_data
eta_sec = (unsigned long) (elapsed * (1.0 / perc)) - elapsed;
if (td->o.timeout &&
- eta_sec > (td->o.timeout + done_secs - elapsed))
- eta_sec = td->o.timeout + done_secs - elapsed;
+ eta_sec > ( (td->o.timeout / 1000) + done_secs - elapsed))
+ eta_sec = (td->o.timeout / 1000) + done_secs
+ - elapsed;
} else if (td->runstate == TD_NOT_CREATED || td->runstate == TD_CREATED
|| td->runstate == TD_INITIALIZED
|| td->runstate == TD_SETTING_UP
@@ -197,8 +199,8 @@ static int thread_eta(struct thread_data
* if given, otherwise assume it'll run at the specified rate.
*/
if (td->o.timeout) {
- t_eta = td->o.timeout + td->o.start_delay +
- td->o.ramp_time;
+ t_eta = (td->o.timeout + td->o.start_delay +
+ td->o.ramp_time ) / 1000;
if (in_ramp_time(td)) {
unsigned long ramp_left;
@@ -212,7 +214,7 @@ static int thread_eta(struct thread_data
rate_bytes = ddir_rw_sum(td->o.rate);
if (rate_bytes) {
r_eta = (bytes_total / 1024) / rate_bytes;
- r_eta += td->o.start_delay;
+ r_eta += td->o.start_delay / 1000;
}
if (r_eta && t_eta)
--- a/fio.1
+++ b/fio.1
@@ -121,12 +121,16 @@ String: a sequence of alphanumeric chara
SI integer: a whole number, possibly containing a suffix denoting the
base unit
of the value. Accepted suffixes are `k', 'M', 'G', 'T', and 'P', denoting
kilo (1024), mega (1024^2), giga (1024^3), tera (1024^4), and peta
(1024^5)
-respectively. The suffix is not case sensitive. If prefixed with '0x', the
-value is assumed to be base 16 (hexadecimal). A suffix may include a
trailing 'b',
-for instance 'kb' is identical to 'k'. You can specify a base 10 value
-by using 'KiB', 'MiB', 'GiB', etc. This is useful for disk drives where
-values are often given in base 10 values. Specifying '30GiB' will get you
-30*1000^3 bytes.
+respectively. If prefixed with '0x', the value is assumed to be base 16
+(hexadecimal). A suffix may include a trailing 'b', for instance 'kb' is
+identical to 'k'. You can specify a base 10 value by using 'KiB',
'MiB','GiB',
+etc. This is useful for disk drives where values are often given in base 10
+values. Specifying '30GiB' will get you 30*1000^3 bytes.
+When specifying times the default suffix meaning changes, still
denoting the
+base unit of the value, but accepted suffixes are 'D' (days), 'H'
(hours), 'M'
+(minutes), 'S' Seconds, 'ms' milli seconds. Time values without a unit
specify
+seconds.
+The suffixes are not case sensitive.
.TP
.I bool
Boolean: a true or false value. `0' denotes false, `1' denotes true.
--- a/parse.c
+++ b/parse.c
@@ -122,21 +122,41 @@ static void show_option_help(struct fio_
show_option_values(o);
}
-static unsigned long get_mult_time(char c)
+static unsigned long long get_mult_time(const char *str, int len)
{
- switch (c) {
- case 'm':
- case 'M':
- return 60;
- case 'h':
- case 'H':
- return 60 * 60;
- case 'd':
- case 'D':
- return 24 * 60 * 60;
- default:
- return 1;
+ const char *p = str;
+ char *c;
+ unsigned long long mult = 1000;
+
+ /*
+ * Go forward until we hit a non-digit, or +/- sign
+ */
+ while ((p - str) <= len) {
+ if (!isdigit((int) *p) && (*p != '+') && (*p != '-'))
+ break;
+ p++;
}
+
+ if (!isalpha((int) *p))
+ return 1000;
+
+ c = strdup(p);
+ for (int i = 0; i < strlen(c); i++)
+ c[i] = tolower(c[i]);
+
+ if (!strncmp("ms", c, 2))
+ mult = 1;
+ else if (!strcmp("s", c))
+ mult = 1000;
+ else if (!strcmp("m", c))
+ mult = 60 * 1000;
+ else if (!strcmp("h", c))
+ mult = 60 * 60 * 1000;
+ else if (!strcmp("d", c))
+ mult = 24 * 60 * 60 * 1000;
+
+ free(c);
+ return mult;
}
static int is_separator(char c)
@@ -275,7 +295,7 @@ int str_to_decimal(const char *str, long
else
*val *= mult;
} else
- *val *= get_mult_time(str[len - 1]);
+ *val *= get_mult_time(str, len);
return 0;
}
--- a/time.c
+++ b/time.c
@@ -71,7 +71,7 @@ int ramp_time_over(struct thread_data *t
return 1;
fio_gettime(&tv, NULL);
- if (mtime_since(&td->epoch, &tv) >= td->o.ramp_time * 1000) {
+ if (mtime_since(&td->epoch, &tv) >= td->o.ramp_time ) {
td->ramp_time_over = 1;
reset_all_stats(td);
td_set_runstate(td, TD_RAMP);
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 4/9] fio: provide an option for a startdelay range
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (2 preceding siblings ...)
2014-02-19 15:12 ` [patch 3/9] fio: allow milliseconds on all time specifiers Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 5/9] fio: add multi directory support Christian Ehrhardt
` (4 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
This patch allows the specification of start delay as range.
With a range each thread will chose an individual startdelay out of the
range.
That solves an issue of startdelay being the same for each numjob clone and
that way spreads all kind of activities e.g. that all clones with mixed r/w
jobs switch r/w at the same time.
Also all kind of other "thundering herd" issues can be softened by some
time spread due to this option.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
fio.1 | 8 ++++++--
fio.h | 5 +++++
init.c | 25 +++++++++++++++++++++++++
options.c | 1 +
4 files changed, 37 insertions(+), 2 deletions(-)
[diff]
--- a/fio.1
+++ b/fio.1
@@ -846,8 +846,12 @@ needed to be specified. For \fBprefer\fR
allowed. For \fBbind\fR and \fBinterleave\fR, \fBnodelist\fR allows
comma delimited list of numbers, A-B ranges, or 'all'.
.TP
-.BI startdelay \fR=\fPint
-Delay start of job for the specified number of seconds.
+.BI startdelay \fR=\fPirange
+Delay start of job for the specified number of seconds. Supports all time
+suffixes to allow specification of hours, minutes, seconds and
+milliseconds - seconds are the default if a unit is ommited.
+Can be given as a range which causes each thread to choose randomly out
of the
+range.
.TP
.BI runtime \fR=\fPint
Terminate processing after the specified number of seconds.
--- a/fio.h
+++ b/fio.h
@@ -85,6 +85,7 @@ enum {
FIO_RAND_SEQ_RAND_READ_OFF,
FIO_RAND_SEQ_RAND_WRITE_OFF,
FIO_RAND_SEQ_RAND_TRIM_OFF,
+ FIO_RAND_START_DELAY,
FIO_RAND_NR_OFFS,
};
@@ -164,6 +165,10 @@ struct thread_data {
os_random_state_t trim_state;
struct frand_state __trim_state;
};
+ union {
+ os_random_state_t delay_state;
+ struct frand_state __delay_state;
+ };
struct frand_state buf_state;
--- a/init.c
+++ b/init.c
@@ -412,6 +412,26 @@ static int fixed_block_size(struct threa
o->min_bs[DDIR_READ] == o->min_bs[DDIR_TRIM];
}
+
+static unsigned long long get_rand_start_delay(struct thread_data *td)
+{
+ unsigned long long delayrange;
+ unsigned long r;
+
+ delayrange = td->o.start_delay_high - td->o.start_delay;
+
+ if (td->o.use_os_rand) {
+ r = os_random_long(&td->delay_state);
+ delayrange = (unsigned long long) ((double) delayrange * (r /
(OS_RAND_MAX + 1.0)));
+ } else {
+ r = __rand(&td->__delay_state);
+ delayrange = (unsigned long long) ((double) delayrange * (r /
(FRAND_MAX + 1.0)));
+ }
+
+ delayrange += td->o.start_delay;
+ return delayrange;
+}
+
/*
* Lazy way of fixing up options that depend on each other. We could also
* define option callback handlers, but this is easier.
@@ -496,6 +516,9 @@ static int fixup_options(struct thread_d
if (!o->file_size_high)
o->file_size_high = o->file_size_low;
+ if (o->start_delay_high)
+ o->start_delay = get_rand_start_delay(td);
+
if (o->norandommap && o->verify != VERIFY_NONE
&& !fixed_block_size(o)) {
log_err("fio: norandommap given for variable block sizes, "
@@ -711,6 +734,7 @@ static void td_fill_rand_seeds_os(struct
os_random_seed(td->rand_seeds[FIO_RAND_FILE_SIZE_OFF],
&td->file_size_state);
os_random_seed(td->rand_seeds[FIO_RAND_TRIM_OFF], &td->trim_state);
+ os_random_seed(td->rand_seeds[FIO_RAND_START_DELAY], &td->delay_state);
if (!td_random(td))
return;
@@ -736,6 +760,7 @@ static void td_fill_rand_seeds_internal(
init_rand_seed(&td->__file_size_state,
td->rand_seeds[FIO_RAND_FILE_SIZE_OFF]);
init_rand_seed(&td->__trim_state, td->rand_seeds[FIO_RAND_TRIM_OFF]);
+ init_rand_seed(&td->__delay_state, td->rand_seeds[FIO_RAND_START_DELAY]);
if (!td_random(td))
return;
--- a/options.c
+++ b/options.c
@@ -2018,6 +2018,7 @@ struct fio_option fio_options[FIO_MAX_OP
.lname = "Start delay",
.type = FIO_OPT_STR_VAL_TIME,
.off1 = td_var_offset(start_delay),
+ .off2 = td_var_offset(start_delay_high),
.help = "Only start job when this period has passed",
.def = "0",
.category = FIO_OPT_C_GENERAL,
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 5/9] fio: add multi directory support
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (3 preceding siblings ...)
2014-02-19 15:12 ` [patch 4/9] fio: provide an option for a startdelay range Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 6/9] fio: allow to combine terse output with any selected output type Christian Ehrhardt
` (3 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
This patch adds support for ':' seperated multiple directories at the
directory
config statement in order to achieve an automatic distribution of job clones
(numjob) across directories.
That way people can distribute a load across these directories (usually
mount
points of disks) automatically - changing numjob will be sufficient to
get all
job clones evenly (optimal if dirs % numjobs = 0, otherwise as good as
possible) distributed at all times.
To avoid confused users old config Files will behave like they always
did, old
fio binaries using new config files won't abort but just use the first
specified dir. If one specifies an explcit (non generated) filename the
distribution to many directories is also deactivated.
It also fixes an issue of events seeming out of order like when running with
--debug=file seeing the "..." message meaning "I created the clones" prior
to the last clone activities.
Now the clones are called with index N-1 .. 1, zero being the base thread as
before.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
engines/net.c | 2 -
file.h | 8 +++++
filesetup.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++------
fio.1 | 6 ++++
init.c | 9 ++----
iolog.c | 2 -
options.c | 78
++++++++++++++++++++++++++++++++++++++++++++++------------
options.h | 2 +
8 files changed, 146 insertions(+), 31 deletions(-)
[diff]
--- a/engines/net.c
+++ b/engines/net.c
@@ -1194,7 +1194,7 @@ static int fio_netio_setup(struct thread
struct netio_data *nd;
if (!td->files_index) {
- add_file(td, td->o.filename ?: "net");
+ add_file(td, td->o.filename ?: "net", 0);
td->o.nr_files = td->o.nr_files ?: 1;
}
--- a/file.h
+++ b/file.h
@@ -125,6 +125,11 @@ struct fio_file {
struct disk_util *du;
};
+struct file_name {
+ struct flist_head list;
+ char *filename;
+};
+
#define FILE_FLAG_FNS(name) \
static inline void fio_file_set_##name(struct fio_file *f) \
{ \
@@ -162,7 +167,7 @@ extern int __must_check generic_close_fi
extern int __must_check generic_get_file_size(struct thread_data *,
struct fio_file *);
extern int __must_check file_lookup_open(struct fio_file *f, int flags);
extern int __must_check pre_read_files(struct thread_data *);
-extern int add_file(struct thread_data *, const char *);
+extern int add_file(struct thread_data *, const char *, int);
extern int add_file_exclusive(struct thread_data *, const char *);
extern void get_file(struct fio_file *);
extern int __must_check put_file(struct thread_data *, struct fio_file *);
@@ -175,6 +180,7 @@ extern int init_random_map(struct thread
extern void dup_files(struct thread_data *, struct thread_data *);
extern int get_fileno(struct thread_data *, const char *);
extern void free_release_files(struct thread_data *);
+extern void filesetup_mem_free(void);
void fio_file_reset(struct thread_data *, struct fio_file *);
int fio_files_done(struct thread_data *);
--- a/filesetup.c
+++ b/filesetup.c
@@ -11,6 +11,7 @@
#include "fio.h"
#include "smalloc.h"
#include "filehash.h"
+#include "options.h"
#include "os/os.h"
#include "hash.h"
#include "lib/axmap.h"
@@ -21,6 +22,8 @@
static int root_warn;
+static FLIST_HEAD(filename_list);
+
static inline void clear_error(struct thread_data *td)
{
td->error = 0;
@@ -1101,7 +1104,48 @@ static void get_file_type(struct fio_fil
}
}
-int add_file(struct thread_data *td, const char *fname)
+static void set_already_allocated(const char *fname) {
+ struct file_name *fn;
+
+ fn = malloc(sizeof(struct file_name));
+ fn->filename = strdup(fname);
+ flist_add_tail(&fn->list, &filename_list);
+}
+
+static int is_already_allocated(const char *fname)
+{
+ struct flist_head *entry;
+ char *filename;
+
+ if (!flist_empty(&filename_list))
+ {
+ flist_for_each(entry, &filename_list) {
+ filename = flist_entry(entry, struct file_name, list)->filename;
+
+ if (strcmp(filename, fname) == 0)
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+static void free_already_allocated() {
+ struct flist_head *entry, *tmp;
+ struct file_name *fn;
+
+ if (!flist_empty(&filename_list))
+ {
+ flist_for_each_safe(entry, tmp, &filename_list) {
+ fn = flist_entry(entry, struct file_name, list);
+ free(fn->filename);
+ flist_del(&fn->list);
+ free(fn);
+ }
+ }
+}
+
+int add_file(struct thread_data *td, const char *fname, int numjob)
{
int cur_files = td->files_index;
char file_name[PATH_MAX];
@@ -1110,6 +1154,15 @@ int add_file(struct thread_data *td, con
dprint(FD_FILE, "add file %s\n", fname);
+ if (td->o.directory)
+ len = set_name_idx(file_name, td->o.directory, numjob);
+
+ sprintf(file_name + len, "%s", fname);
+
+ /* clean cloned siblings using existing files */
+ if (numjob && is_already_allocated(file_name))
+ return 0;
+
f = smalloc(sizeof(*f));
if (!f) {
log_err("fio: smalloc OOM\n");
@@ -1149,10 +1202,6 @@ int add_file(struct thread_data *td, con
if (td->io_ops && (td->io_ops->flags & FIO_DISKLESSIO))
f->real_file_size = -1ULL;
- if (td->o.directory)
- len = sprintf(file_name, "%s/", td->o.directory);
-
- sprintf(file_name + len, "%s", fname);
f->file_name = smalloc_strdup(file_name);
if (!f->file_name) {
log_err("fio: smalloc OOM\n");
@@ -1179,6 +1228,8 @@ int add_file(struct thread_data *td, con
if (f->filetype == FIO_TYPE_FILE)
td->nr_normal_files++;
+ set_already_allocated(file_name);
+
dprint(FD_FILE, "file %p \"%s\" added at %d\n", f, f->file_name,
cur_files);
@@ -1195,7 +1246,7 @@ int add_file_exclusive(struct thread_dat
return i;
}
- return add_file(td, fname);
+ return add_file(td, fname, 0);
}
void get_file(struct fio_file *f)
@@ -1304,7 +1355,7 @@ static int recurse_dir(struct thread_dat
}
if (S_ISREG(sb.st_mode)) {
- add_file(td, full_path);
+ add_file(td, full_path, 0);
td->o.nr_files++;
continue;
}
@@ -1421,3 +1472,8 @@ int fio_files_done(struct thread_data *t
return 1;
}
+
+/* free memory used in initialization phase only */
+void filesetup_mem_free() {
+ free_already_allocated();
+}
--- a/fio.1
+++ b/fio.1
@@ -158,6 +158,12 @@ otherwise has no special purpose.
.BI directory \fR=\fPstr
Prefix filenames with this directory. Used to place files in a
location other
than `./'.
+You can specify a number of directories by separating the names with a ':'
+character. These directories will be assigned equally distributed to
job clones
+creates with \fInumjobs\fR as long as they are using generated filenames.
+If specific \fIfilename(s)\fR are set fio will use the first listed
directory,
+and thereby matching the \fIfilename\fR semantic which generates a
file each
+clone if not specified, but let all clones use the same if set.
.TP
.BI filename \fR=\fPstr
.B fio
--- a/init.c
+++ b/init.c
@@ -1020,10 +1020,10 @@ static int add_job(struct thread_data *t
file_alloced = 1;
if (o->nr_files == 1 && exists_and_not_file(jobname))
- add_file(td, jobname);
+ add_file(td, jobname, job_add_num);
else {
for (i = 0; i < o->nr_files; i++)
- add_file(td, make_filename(fname, o, jobname, td->thread_number, i));
+ add_file(td, make_filename(fname, o, jobname, job_add_num, i),
job_add_num);
}
}
@@ -1166,9 +1166,7 @@ static int add_job(struct thread_data *t
}
}
- job_add_num = numjobs - 1;
-
- if (add_job(td_new, jobname, job_add_num, 1, client_type))
+ if (add_job(td_new, jobname, numjobs, 1, client_type))
goto err;
}
@@ -2027,6 +2025,7 @@ int parse_options(int argc, char *argv[]
free(ini_file);
fio_options_free(&def_thread);
+ filesetup_mem_free();
if (!thread_number) {
if (parse_dryrun())
--- a/iolog.c
+++ b/iolog.c
@@ -324,7 +324,7 @@ static int read_iolog2(struct thread_dat
rw = DDIR_INVAL;
if (!strcmp(act, "add")) {
td->o.nr_files++;
- fileno = add_file(td, fname);
+ fileno = add_file(td, fname, 0);
file_action = FIO_LOG_ADD_FILE;
continue;
} else if (!strcmp(act, "open")) {
--- a/options.c
+++ b/options.c
@@ -729,11 +729,11 @@ static int str_random_distribution_cb(vo
}
/*
- * Return next file in the string. Files are separated with ':'. If the ':'
+ * Return next name in the string. Files are separated with ':'. If the ':'
* is escaped with a '\', then that ':' is part of the filename and
does not
* indicate a new file.
*/
-static char *get_next_file_name(char **ptr)
+static char *get_next_name(char **ptr)
{
char *str = *ptr;
char *p, *start;
@@ -774,6 +774,44 @@ static char *get_next_file_name(char **p
return start;
}
+
+static int get_max_name_idx(char *input)
+{
+ unsigned int cur_idx;
+ char *str, *p;
+
+ p = str = strdup(input);
+ for (cur_idx = 0; ; cur_idx++)
+ if (get_next_name(&str) == NULL)
+ break;
+
+ free(p);
+ return cur_idx;
+}
+
+/*
+ * Returns the directory at the index, indexes > entires will be
+ * assigned via modulo division of the index
+ */
+int set_name_idx(char *target, char *input, int index)
+{
+ unsigned int cur_idx;
+ int len;
+ char *fname, *str, *p;
+
+ p = str = strdup(input);
+
+ index %= get_max_name_idx(input);
+ for (cur_idx = 0; cur_idx <= index; cur_idx++) {
+ fname = get_next_name(&str);
+ }
+
+ len = sprintf(target, "%s/", fname);
+ free(p);
+
+ return len;
+}
+
static int str_filename_cb(void *data, const char *input)
{
struct thread_data *td = data;
@@ -787,10 +825,10 @@ static int str_filename_cb(void *data, c
if (!td->files_index)
td->o.nr_files = 0;
- while ((fname = get_next_file_name(&str)) != NULL) {
+ while ((fname = get_next_name(&str)) != NULL) {
if (!strlen(fname))
break;
- add_file(td, fname);
+ add_file(td, fname, 0);
td->o.nr_files++;
}
@@ -798,27 +836,35 @@ static int str_filename_cb(void *data, c
return 0;
}
-static int str_directory_cb(void *data, const char fio_unused *str)
+static int str_directory_cb(void *data, const char fio_unused *unused)
{
struct thread_data *td = data;
struct stat sb;
+ char *dirname, *str, *p;
+ int ret = 0;
if (parse_dryrun())
return 0;
- if (lstat(td->o.directory, &sb) < 0) {
- int ret = errno;
-
- log_err("fio: %s is not a directory\n", td->o.directory);
- td_verror(td, ret, "lstat");
- return 1;
- }
- if (!S_ISDIR(sb.st_mode)) {
- log_err("fio: %s is not a directory\n", td->o.directory);
- return 1;
+ p = str = strdup(td->o.directory);
+ while ((dirname = get_next_name(&str)) != NULL) {
+ if (lstat(dirname, &sb) < 0) {
+ ret = errno;
+
+ log_err("fio: %s is not a directory\n", dirname);
+ td_verror(td, ret, "lstat");
+ goto out;
+ }
+ if (!S_ISDIR(sb.st_mode)) {
+ log_err("fio: %s is not a directory\n", dirname);
+ ret = 1;
+ goto out;
+ }
}
- return 0;
+out:
+ free(p);
+ return ret;
}
static int str_lockfile_cb(void *data, const char fio_unused *str)
--- a/options.h
+++ b/options.h
@@ -17,6 +17,8 @@ void add_opt_posval(const char *, const
void del_opt_posval(const char *, const char *);
struct thread_data;
void fio_options_free(struct thread_data *);
+char *get_name_idx(char *, int);
+int set_name_idx(char *, char *, int);
extern struct fio_option fio_options[FIO_MAX_OPTS];
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 6/9] fio: allow to combine terse output with any selected output type
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (4 preceding siblings ...)
2014-02-19 15:12 ` [patch 5/9] fio: add multi directory support Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 7/9] fio: flush log files on test end Christian Ehrhardt
` (2 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
This patch adds the option --append-terse to be able to request a
combination
of any given selected output format AND terse output based on the same data.
This will help all users that want to parse the terse data for further use,
but need to look into the logs every now and then which then should be
readable.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
fio.1 | 10 +++++++---
fio.h | 1 +
init.c | 9 +++++++++
stat.c | 10 ++++++++++
4 files changed, 27 insertions(+), 3 deletions(-)
[diff]
--- a/fio.1
+++ b/fio.1
@@ -32,6 +32,9 @@ Generate per-job bandwidth logs.
.B \-\-minimal
Print statistics in a terse, semicolon-delimited format.
.TP
+.B \-\-append-terse
+Print statistics in selected mode AND terse, semicolon-delimited format.
+.TP
.B \-\-version
Display version information and exit.
.TP
@@ -1578,9 +1581,10 @@ It is also possible to get fio to dump t
running, without terminating the job. To do that, send fio the \fBUSR1\fR
signal.
.SH TERSE OUTPUT
-If the \fB\-\-minimal\fR option is given, the results will be printed in a
-semicolon-delimited format suitable for scripted use - a job description
-(if provided) follows on a new line. Note that the first
+If the \fB\-\-minimal\fR / \fB\-\-append-terse\fR options are given, the
+results will be printed/appended in a semicolon-delimited format
suitable for
+scripted use.
+A job description (if provided) follows on a new line. Note that the first
number in the line is the version number. If the output has to be changed
for some reason, this number will be incremented by 1 to signify that
change. The fields are:
--- a/fio.h
+++ b/fio.h
@@ -373,6 +373,7 @@ extern unsigned int stat_number;
extern int shm_id;
extern int groupid;
extern int output_format;
+extern int append_terse_output;
extern int temp_stall_ts;
extern uintptr_t page_mask, page_size;
extern int read_only;
--- a/init.c
+++ b/init.c
@@ -43,6 +43,7 @@ struct thread_data *threads = NULL;
int exitall_on_terminate = 0;
int output_format = FIO_OUTPUT_NORMAL;
+int append_terse_output = 0;
int eta_print = FIO_ETA_AUTO;
int eta_new_line = 0;
FILE *f_out = NULL;
@@ -109,6 +110,11 @@ static struct option l_opts[FIO_NR_OPTIO
.val = 'F' | FIO_CLIENT_FLAG,
},
{
+ .name = (char *) "append-terse",
+ .has_arg = optional_argument,
+ .val = 'f',
+ },
+ {
.name = (char *) "version",
.has_arg = no_argument,
.val = 'v' | FIO_CLIENT_FLAG,
@@ -1716,6 +1722,9 @@ int parse_cmd_line(int argc, char *argv[
else
output_format = FIO_OUTPUT_NORMAL;
break;
+ case 'f':
+ append_terse_output = 1;
+ break;
case 'h':
if (!cur_client) {
usage(argv[0]);
--- a/stat.c
+++ b/stat.c
@@ -1386,6 +1386,16 @@ static void __show_run_stats(void)
show_idle_prof_stats(FIO_OUTPUT_NORMAL, NULL);
}
+ if ( !(output_format == FIO_OUTPUT_TERSE) && append_terse_output) {
+ log_info("\nAdditional Terse Output:\n");
+
+ for (i = 0; i < nr_ts; i++) {
+ ts = &threadstats[i];
+ rs = &runstats[ts->groupid];
+ show_thread_status_terse(ts, rs);
+ }
+ }
+
log_info_flush();
free(runstats);
free(threadstats);
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 7/9] fio: flush log files on test end
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (5 preceding siblings ...)
2014-02-19 15:12 ` [patch 6/9] fio: allow to combine terse output with any selected output type Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 8/9] fio: fix last block never being touched by random offsets Christian Ehrhardt
2014-02-19 15:12 ` [patch 9/9] fio: allow 0 as compress percentage Christian Ehrhardt
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Ensure proper flushing of all logs at the end of tests.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
backend.c | 1 iolog.h | 1 stat.c | 73
++++++++++++++++++++++++++++++++++++++++----------------------
3 files changed, 50 insertions(+), 25 deletions(-)
--- a/backend.c
+++ b/backend.c
@@ -1461,6 +1461,7 @@ static void *thread_main(void *data)
fio_unpin_memory(td);
fio_mutex_down(writeout_mutex);
+ finalize_logs(td);
if (td->bw_log) {
if (o->bw_log_file) {
finish_log_named(td, td->bw_log,
--- a/iolog.h
+++ b/iolog.h
@@ -117,6 +117,7 @@ extern void write_iolog_close(struct thr
/*
* Logging
*/
+extern void finalize_logs(struct thread_data *td);
extern void add_lat_sample(struct thread_data *, enum fio_ddir,
unsigned long,
unsigned int);
extern void add_clat_sample(struct thread_data *, enum fio_ddir,
unsigned long,
--- a/stat.c
+++ b/stat.c
@@ -1579,6 +1579,37 @@ static inline void reset_io_stat(struct
ios->mean.u.f = ios->S.u.f = 0;
}
+static void _add_stat_to_log(struct io_log *iolog, unsigned long elapsed)
+{
+ /*
+ * Note an entry in the log. Use the mean from the logged samples,
+ * making sure to properly round up. Only write a log entry if we
+ * had actual samples done.
+ */
+ if (iolog->avg_window[DDIR_READ].samples) {
+ unsigned long mr;
+
+ mr = iolog->avg_window[DDIR_READ].mean.u.f + 0.50;
+ __add_log_sample(iolog, mr, DDIR_READ, 0, elapsed);
+ }
+ if (iolog->avg_window[DDIR_WRITE].samples) {
+ unsigned long mw;
+
+ mw = iolog->avg_window[DDIR_WRITE].mean.u.f + 0.50;
+ __add_log_sample(iolog, mw, DDIR_WRITE, 0, elapsed);
+ }
+ if (iolog->avg_window[DDIR_TRIM].samples) {
+ unsigned long mw;
+
+ mw = iolog->avg_window[DDIR_TRIM].mean.u.f + 0.50;
+ __add_log_sample(iolog, mw, DDIR_TRIM, 0, elapsed);
+ }
+
+ reset_io_stat(&iolog->avg_window[DDIR_READ]);
+ reset_io_stat(&iolog->avg_window[DDIR_WRITE]);
+ reset_io_stat(&iolog->avg_window[DDIR_TRIM]);
+}
+
static void add_log_sample(struct thread_data *td, struct io_log *iolog,
unsigned long val, enum fio_ddir ddir,
unsigned int bs)
@@ -1612,35 +1643,27 @@ static void add_log_sample(struct thread
if (this_window < iolog->avg_msec)
return;
- /*
- * Note an entry in the log. Use the mean from the logged samples,
- * making sure to properly round up. Only write a log entry if we
- * had actual samples done.
- */
- if (iolog->avg_window[DDIR_READ].samples) {
- unsigned long mr;
+ _add_stat_to_log(iolog, elapsed);
- mr = iolog->avg_window[DDIR_READ].mean.u.f + 0.50;
- __add_log_sample(iolog, mr, DDIR_READ, 0, elapsed);
- }
- if (iolog->avg_window[DDIR_WRITE].samples) {
- unsigned long mw;
-
- mw = iolog->avg_window[DDIR_WRITE].mean.u.f + 0.50;
- __add_log_sample(iolog, mw, DDIR_WRITE, 0, elapsed);
- }
- if (iolog->avg_window[DDIR_TRIM].samples) {
- unsigned long mw;
+ iolog->avg_last = elapsed;
+}
- mw = iolog->avg_window[DDIR_TRIM].mean.u.f + 0.50;
- __add_log_sample(iolog, mw, DDIR_TRIM, 0, elapsed);
- }
+void finalize_logs(struct thread_data *td)
+{
+ unsigned long elapsed;
+ elapsed = mtime_since_now(&td->epoch);
- reset_io_stat(&iolog->avg_window[DDIR_READ]);
- reset_io_stat(&iolog->avg_window[DDIR_WRITE]);
- reset_io_stat(&iolog->avg_window[DDIR_TRIM]);
- iolog->avg_last = elapsed;
+ if (td->clat_log)
+ _add_stat_to_log(td->clat_log, elapsed);
+ if (td->slat_log)
+ _add_stat_to_log(td->slat_log, elapsed);
+ if (td->lat_log)
+ _add_stat_to_log(td->lat_log, elapsed);
+ if (td->bw_log)
+ _add_stat_to_log(td->bw_log, elapsed);
+ if (td->iops_log)
+ _add_stat_to_log(td->iops_log, elapsed);
}
void add_agg_sample(unsigned long val, enum fio_ddir ddir, unsigned
int bs)
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 8/9] fio: fix last block never being touched by random offsets
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (6 preceding siblings ...)
2014-02-19 15:12 ` [patch 7/9] fio: flush log files on test end Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
2014-02-19 15:12 ` [patch 9/9] fio: allow 0 as compress percentage Christian Ehrhardt
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Fix the available range for random offsets which never touched the last
block.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
io_u.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/io_u.c
+++ b/io_u.c
@@ -104,7 +104,7 @@ static int __get_next_rand_offset(struct
dprint(FD_RANDOM, "off rand %llu\n", (unsigned long long) r);
- *b = (lastb - 1) * (r / ((uint64_t) rmax + 1.0));
+ *b = lastb * (r / ((uint64_t) rmax + 1.0));
} else {
uint64_t off = 0;
^ permalink raw reply [flat|nested] 13+ messages in thread* [patch 9/9] fio: allow 0 as compress percentage
[not found] <20140219143639.168501090@linux.vnet.ibm.com>
` (7 preceding siblings ...)
2014-02-19 15:12 ` [patch 8/9] fio: fix last block never being touched by random offsets Christian Ehrhardt
@ 2014-02-19 15:12 ` Christian Ehrhardt
8 siblings, 0 replies; 13+ messages in thread
From: Christian Ehrhardt @ 2014-02-19 15:12 UTC (permalink / raw)
To: fio; +Cc: oberpar, Christian Ehrhardt
From: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Allow 0 as value for option compress_percentage which can be useful for
certain deduplication and compression based storage back ends.
Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
---
[diffstat]
options.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/options.c
+++ b/options.c
@@ -3072,7 +3072,7 @@ struct fio_option fio_options[FIO_MAX_OP
.type = FIO_OPT_INT,
.off1 = td_var_offset(compress_percentage),
.maxval = 100,
- .minval = 1,
+ .minval = 0,
.help = "How compressible the buffer is (approximately)",
.interval = 5,
.category = FIO_OPT_C_IO,
^ permalink raw reply [flat|nested] 13+ messages in thread