[PATCH] Add environment-variable substitution to config options

[PATCH] Add environment-variable substitution to config options

[Aaron Carroll]

Hi,

I have a nasty bash script that performs environment variable substitution for running
automated benchmarks.  This seems like a reasonable feature for FIO... so here it is.

The code is quite perilous, and hasn't been particularly well tested.

Thanks,
   -- Aaron


---

Strings of the form ${VARNAME} in config option strings are substituted with
the value of the environment variable VARNAME.  Only the right hand side of
an option assignment undergoes substitution.  If VARNAME is empty or
undefined, the empty string is substituted.

Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au>
---
 parse.c |   52 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 parse.h |    1 +
 2 files changed, 52 insertions(+), 1 deletions(-)

diff --git a/parse.c b/parse.c
index b946690..456e3ba 100644
--- a/parse.c
+++ b/parse.c
@@ -8,6 +8,7 @@
 #include <string.h>
 #include <errno.h>
 #include <limits.h>
+#include <stdlib.h>
 
 #include "parse.h"
 #include "debug.h"
@@ -529,12 +530,61 @@ int parse_cmd_option(const char *opt, const char *val,
 	return 1;
 }
 
+/*
+ * Return a copy of the input string with substrings of the form ${VARNAME}
+ * substituted with the value of the environment variable VARNAME.  The
+ * substitution always occurs, even if VARNAME is empty or the corresponding
+ * environment variable undefined.
+ */
+static char *option_dup_subs(const char *opt)
+{
+	char out[OPT_LEN_MAX+1];
+	char in[OPT_LEN_MAX+1];
+	char *outptr = out;
+	char *inptr = in;
+	char *ch1, *ch2, *env;
+	ssize_t nchr = OPT_LEN_MAX;
+	size_t envlen;
+
+	in[OPT_LEN_MAX] = '\0';
+	strncpy(in, opt, OPT_LEN_MAX);
+
+	while (*inptr && nchr > 0) {
+		if (inptr[0] == '$' && inptr[1] == '{') {
+			ch2 = strchr(inptr, '}');
+			if (ch2 && inptr+1 < ch2) {
+				ch1 = inptr+2;
+				inptr = ch2+1;
+				*ch2 = '\0';
+
+				env = getenv(ch1);
+				if (env) {
+					envlen = strlen(env);
+					if (envlen <= nchr) {
+						memcpy(outptr, env, envlen);
+						outptr += envlen;
+						nchr -= envlen;
+					}
+				}
+
+				continue;
+			}
+		}
+
+		*outptr++ = *inptr++;
+		--nchr;
+	}
+
+	*outptr = '\0';
+	return strdup(out);
+}
+
 int parse_option(const char *opt, struct fio_option *options, void *data)
 {
 	struct fio_option *o;
 	char *post, *tmp;
 
-	tmp = strdup(opt);
+	tmp = option_dup_subs(opt);
 
 	o = get_option(tmp, options, &post);
 	if (!o) {
diff --git a/parse.h b/parse.h
index 4f3f94d..4546259 100644
--- a/parse.h
+++ b/parse.h
@@ -26,6 +26,7 @@ struct value_pair {
 	const char *help;		/* help text for sub option */
 };
 
+#define OPT_LEN_MAX 	1024
 #define PARSE_MAX_VP	16
 
 /*

Re: [PATCH] Add environment-variable substitution to config options

[Jens Axboe]

On Sat, Oct 04 2008, Aaron Carroll wrote:
> Hi,
> 
> I have a nasty bash script that performs environment variable substitution 
> for running
> automated benchmarks.  This seems like a reasonable feature for FIO... so 
> here it is.
> 
> The code is quite perilous, and hasn't been particularly well tested.

I'm fine with including this, but could you also please provide an
example (preferably in HOWTO/README)? Thanks :)

-- 
Jens Axboe

[PATCH] Document environment variable expansion

[Aaron Carroll]

Add environment variable expansion documentation to HOWTO, and fix
a few nearby typo's while we're at it.

Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au>
---
 HOWTO |   31 +++++++++++++++++++++++++++++--
 1 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/HOWTO b/HOWTO
index 178bc98..129926b 100644
--- a/HOWTO
+++ b/HOWTO
@@ -111,7 +111,7 @@ several global sections if so desired. A job is only affected by a global
 section residing above it. If the first character in a line is a ';' or a
 '#', the entire line is discarded as a comment.
 
-So lets look at a really simple job file that define to threads, each
+So let's look at a really simple job file that defines two processes, each
 randomly reading from a 128MiB file.
 
 ; -- start job file --
@@ -133,7 +133,7 @@ line, this job would look as follows:
 $ fio --name=global --rw=randread --size=128m --name=job1 --name=job2
 
 
-Lets look at an example that have a number of processes writing randomly
+Let's look at an example that has a number of processes writing randomly
 to files.
 
 ; -- start job file --
@@ -158,6 +158,33 @@ specify:
 
 $ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4
 
+fio also supports environment variable expansion in job files. Any
+substring of the form "${VARNAME}" as part of an option value (in other
+words, on the right of the `='), will be expanded to the value of the
+environment variable called VARNAME.  If no such environment variable
+is defined, or VARNAME is the empty string, the empty string will be
+substituted.
+
+As an example, let's look at a sample fio invocation and job file:
+
+$ SIZE=64m NUMJOBS=4 fio jobfile.fio
+
+; -- start job file --
+[random-writers]
+rw=randwrite
+size=${SIZE}
+numjobs=${NUMJOBS}
+; -- end job file --
+
+This will expand to the following equivalent job file at runtime:
+
+; -- start job file --
+[random-writers]
+rw=randwrite
+size=64m
+numjobs=4
+; -- end job file --
+
 fio ships with a few example job files, you can also look there for
 inspiration.
 
-- 
1.6.0.2

Re: [PATCH] Document environment variable expansion

[Jens Axboe]

On Tue, Oct 07 2008, Aaron Carroll wrote:
> Add environment variable expansion documentation to HOWTO, and fix
> a few nearby typo's while we're at it.

Super, thanks! I've applied this and the original patch.

-- 
Jens Axboe