FIO - Client and Server - Suggestion

FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

Hi All,

This is neto from Brazil

How are you?

One small suggestion:

Running Client and Server architecture on FIO, we need to have all config
files "locally" to run it.


Nossa Senhora:tools neto$ ls
151 152 fio



Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
10.61.109.152 152
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
<s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
iodepth=1
ioengine=libaio, iodepth=1

If we are doing a test in an HPC environment with hundred of files,
wouldn't be easier to try to point to a single location for the file and
access it remotely (we won't need to copy it locally).


For example:

./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
10.61.109.152 --remote-config /root/fio/152


Thoughts?

Thank you,

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
> Hi All,
> 
> This is neto from Brazil
> 
> How are you?
> 
> One small suggestion:
> 
> Running Client and Server architecture on FIO, we need to have all config
> files "locally" to run it.
> 
> 
> Nossa Senhora:tools neto$ ls
> 151 152 fio
> 
> 
> 
> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
> 10.61.109.152 152
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
> iodepth=1
> ioengine=libaio, iodepth=1
> 
> If we are doing a test in an HPC environment with hundred of files,
> wouldn't be easier to try to point to a single location for the file and
> access it remotely (we won't need to copy it locally).
> 
> 
> For example:
> 
> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
> 10.61.109.152 --remote-config /root/fio/152
> 
> 
> Thoughts?

That's a good idea, would not be hard to insert that step of having the
remote fio server load a local config file. I'll look into that.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

On 10/07/2014 03:29 PM, Jens Axboe wrote:
> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>> Hi All,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> One small suggestion:
>>
>> Running Client and Server architecture on FIO, we need to have all config
>> files "locally" to run it.
>>
>>
>> Nossa Senhora:tools neto$ ls
>> 151 152 fio
>>
>>
>>
>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>> 10.61.109.152 152
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>> flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>> flags=1
>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>> iodepth=1
>> ioengine=libaio, iodepth=1
>>
>> If we are doing a test in an HPC environment with hundred of files,
>> wouldn't be easier to try to point to a single location for the file and
>> access it remotely (we won't need to copy it locally).
>>
>>
>> For example:
>>
>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>> 10.61.109.152 --remote-config /root/fio/152
>>
>>
>> Thoughts?
> 
> That's a good idea, would not be hard to insert that step of having the
> remote fio server load a local config file. I'll look into that.

Totally untested, but the below is a start. On the client side, you'd do:

fio --client=server-hostname --remote-config /some/path/to/file

and then the server should attempt to open that. Needs a bit of error
handling, but the concept should be there.

-- 
Jens Axboe


[-- Attachment #2: net-remote-config.patch --]
[-- Type: text/x-patch, Size: 8531 bytes --]

diff --git a/client.c b/client.c
index 1dded0966cf9..757ead356c39 100644
--- a/client.c
+++ b/client.c
@@ -262,17 +262,24 @@ err:
 	return NULL;
 }
 
-void fio_client_add_ini_file(void *cookie, const char *ini_file)
+void fio_client_add_ini_file(void *cookie, const char *ini_file, int remote)
 {
 	struct fio_client *client = cookie;
 	size_t new_size;
 
 	dprint(FD_NET, "client <%s>: add ini %s\n", client->hostname, ini_file);
 
-	new_size = (client->nr_ini_file + 1) * sizeof(char *);
-	client->ini_file = realloc(client->ini_file, new_size);
-	client->ini_file[client->nr_ini_file] = strdup(ini_file);
-	client->nr_ini_file++;
+	if (!remote) {
+		new_size = (client->nr_ini_file + 1) * sizeof(char *);
+		client->ini_file = realloc(client->ini_file, new_size);
+		client->ini_file[client->nr_ini_file] = strdup(ini_file);
+		client->nr_ini_file++;
+	} else {
+		new_size = (client->nr_remote_file + 1) * sizeof(char *);
+		client->remote_file = realloc(client->remote_file, new_size);
+		client->remote_file[client->nr_remote_file] = strdup(ini_file);
+		client->nr_remote_file++;
+	}
 }
 
 int fio_client_add(struct client_ops *ops, const char *hostname, void **cookie)
@@ -599,11 +606,33 @@ int fio_start_all_clients(void)
 	return flist_empty(&client_list);
 }
 
+static int __fio_client_send_remote_ini(struct fio_client *client,
+					const char *filename)
+{
+	struct cmd_load_file_pdu *pdu;
+	size_t p_size;
+	int ret;
+
+	dprint(FD_NET, "send remote ini %s to %s\n", filename, client->hostname);
+
+	p_size = sizeof(*pdu) + strlen(filename);
+	pdu = malloc(p_size);
+	pdu->name_len = strlen(filename);
+	strcpy((char *) pdu->file, filename);
+	pdu->client_type = cpu_to_le16((uint16_t) client->type);
+
+	client->sent_job = 1;
+	ret = fio_net_send_cmd(client->fd, FIO_NET_CMD_LOAD_FILE, pdu, p_size,NULL, NULL);
+	free(pdu);
+	return ret;
+}
+
 /*
  * Send file contents to server backend. We could use sendfile(), but to remain
  * more portable lets just read/write the darn thing.
  */
-static int __fio_client_send_ini(struct fio_client *client, const char *filename)
+static int __fio_client_send_local_ini(struct fio_client *client,
+				       const char *filename)
 {
 	struct cmd_job_pdu *pdu;
 	size_t p_size;
@@ -668,11 +697,16 @@ static int __fio_client_send_ini(struct fio_client *client, const char *filename
 	return ret;
 }
 
-int fio_client_send_ini(struct fio_client *client, const char *filename)
+int fio_client_send_ini(struct fio_client *client, const char *filename,
+			int remote)
 {
 	int ret;
 
-	ret = __fio_client_send_ini(client, filename);
+	if (!remote)
+		ret = __fio_client_send_local_ini(client, filename);
+	else
+		ret = __fio_client_send_remote_ini(client, filename);
+
 	if (!ret)
 		client->sent_job = 1;
 
@@ -685,20 +719,29 @@ int fio_clients_send_ini(const char *filename)
 	struct flist_head *entry, *tmp;
 
 	flist_for_each_safe(entry, tmp, &client_list) {
+		int i;
+
 		client = flist_entry(entry, struct fio_client, list);
 
 		if (client->nr_ini_file) {
-			int i;
-
 			for (i = 0; i < client->nr_ini_file; i++) {
 				const char *ini = client->ini_file[i];
 
-				if (fio_client_send_ini(client, ini)) {
+				if (fio_client_send_ini(client, ini, 0)) {
+					remove_client(client);
+					break;
+				}
+			}
+		} else if (client->nr_remote_file) {
+			for (i = 0; i < client->nr_remote_file; i++) {
+				const char *ini = client->remote_file[i];
+
+				if (fio_client_send_ini(client, ini, 1)) {
 					remove_client(client);
 					break;
 				}
 			}
-		} else if (!filename || fio_client_send_ini(client, filename))
+		} else if (!filename || fio_client_send_ini(client, filename, 0))
 			remove_client(client);
 	}
 
diff --git a/client.h b/client.h
index c8ff23ecdd68..89a0783d8d57 100644
--- a/client.h
+++ b/client.h
@@ -66,6 +66,9 @@ struct fio_client {
 
 	char **ini_file;
 	unsigned int nr_ini_file;
+
+	char **remote_file;
+	unsigned int nr_remote_file;
 };
 
 struct cmd_iolog_pdu;
@@ -119,13 +122,13 @@ extern int fio_client_connect(struct fio_client *);
 extern int fio_clients_connect(void);
 extern int fio_start_client(struct fio_client *);
 extern int fio_start_all_clients(void);
-extern int fio_client_send_ini(struct fio_client *, const char *);
 extern int fio_clients_send_ini(const char *);
+extern int fio_client_send_ini(struct fio_client *, const char *, int);
 extern int fio_handle_clients(struct client_ops *);
 extern int fio_client_add(struct client_ops *, const char *, void **);
 extern struct fio_client *fio_client_add_explicit(struct client_ops *, const char *, int, int);
 extern void fio_client_add_cmd_option(void *, const char *);
-extern void fio_client_add_ini_file(void *, const char *);
+extern void fio_client_add_ini_file(void *, const char *, int);
 extern int fio_client_terminate(struct fio_client *);
 extern void fio_clients_terminate(void);
 extern struct fio_client *fio_get_client(struct fio_client *);
diff --git a/init.c b/init.c
index 861b1f5fd0f4..e3ce4702c743 100644
--- a/init.c
+++ b/init.c
@@ -217,6 +217,11 @@ static struct option l_opts[FIO_NR_OPTIONS] = {
 		.val		= 'C',
 	},
 	{
+		.name		= (char *) "remote-config",
+		.has_arg	= required_argument,
+		.val		= 'R',
+	},
+	{
 		.name		= (char *) "cpuclock-test",
 		.has_arg	= no_argument,
 		.val		= 'T',
@@ -2174,10 +2179,14 @@ int parse_cmd_line(int argc, char *argv[], int client_type)
 				    !strncmp(argv[optind], "-", 1))
 					break;
 
-				fio_client_add_ini_file(cur_client, argv[optind]);
+				fio_client_add_ini_file(cur_client, argv[optind], 0);
 				optind++;
 			}
 			break;
+		case 'R':
+			fio_client_add_ini_file(cur_client, optarg, 1);
+			did_arg = 1;
+			break;
 		case 'T':
 			did_arg = 1;
 			do_exit++;
diff --git a/server.c b/server.c
index 36713ee57160..fa029ca3bc38 100644
--- a/server.c
+++ b/server.c
@@ -69,6 +69,8 @@ static const char *fio_server_ops[FIO_NET_CMD_NR] = {
 	"ADD_JOB",
 	"CMD_RUN",
 	"CMD_IOLOG",
+	"CMD_UPDATE_JOB",
+	"CMD_LOAD_FILE",
 };
 
 const char *fio_server_op(unsigned int op)
@@ -550,6 +552,28 @@ static void fio_server_check_conns(struct flist_head *conn_list)
 	fio_server_check_fork_items(conn_list);
 }
 
+static int handle_load_file_cmd(struct fio_net_cmd *cmd)
+{
+	struct cmd_load_file_pdu *pdu = (struct cmd_load_file_pdu *) cmd->payload;
+	void *file_name = pdu->file;
+	struct cmd_start_pdu spdu;
+
+	dprint(FD_NET, "server: loading local file %s\n", (char *) file_name);
+
+	pdu->name_len = le16_to_cpu(pdu->name_len);
+	pdu->client_type = le16_to_cpu(pdu->client_type);
+
+	if (parse_jobs_ini(file_name, 0, 0, pdu->client_type)) {
+		fio_net_send_quit(server_fd);
+		return -1;
+	}
+
+	spdu.jobs = cpu_to_le32(thread_number);
+	spdu.stat_outputs = cpu_to_le32(stat_number);
+	fio_net_send_cmd(server_fd, FIO_NET_CMD_START, &spdu, sizeof(spdu), NULL, NULL);
+	return 0;
+}
+
 static int handle_run_cmd(struct flist_head *job_list, struct fio_net_cmd *cmd)
 {
 	pid_t pid;
@@ -747,6 +771,9 @@ static int handle_command(struct flist_head *job_list, struct fio_net_cmd *cmd)
 	case FIO_NET_CMD_EXIT:
 		exit_backend = 1;
 		return -1;
+	case FIO_NET_CMD_LOAD_FILE:
+		ret = handle_load_file_cmd(cmd);
+		break;
 	case FIO_NET_CMD_JOB:
 		ret = handle_job_cmd(cmd);
 		break;
diff --git a/server.h b/server.h
index 1b131b92f08a..67ba38d216d9 100644
--- a/server.h
+++ b/server.h
@@ -38,7 +38,7 @@ struct fio_net_cmd_reply {
 };
 
 enum {
-	FIO_SERVER_VER			= 36,
+	FIO_SERVER_VER			= 37,
 
 	FIO_SERVER_MAX_FRAGMENT_PDU	= 1024,
 	FIO_SERVER_MAX_CMD_MB		= 2048,
@@ -61,7 +61,8 @@ enum {
 	FIO_NET_CMD_RUN			= 16,
 	FIO_NET_CMD_IOLOG		= 17,
 	FIO_NET_CMD_UPDATE_JOB		= 18,
-	FIO_NET_CMD_NR			= 19,
+	FIO_NET_CMD_LOAD_FILE		= 19,
+	FIO_NET_CMD_NR			= 20,
 
 	FIO_NET_CMD_F_MORE		= 1UL << 0,
 
@@ -76,6 +77,12 @@ enum {
 	FIO_PROBE_FLAG_ZLIB		= 1UL << 0,
 };
 
+struct cmd_load_file_pdu {
+	uint16_t name_len;
+	uint16_t client_type;
+	uint8_t file[];
+};
+
 struct cmd_ts_pdu {
 	struct thread_stat ts;
 	struct group_run_stats rs;
@@ -169,11 +176,6 @@ extern void fio_server_send_gs(struct group_run_stats *);
 extern void fio_server_send_du(void);
 extern void fio_server_idle_loop(void);
 
-extern int fio_clients_connect(void);
-extern int fio_clients_send_ini(const char *);
-extern void fio_client_add_cmd_option(void *, const char *);
-extern void fio_client_add_ini_file(void *, const char *);
-
 extern int fio_recv_data(int sk, void *p, unsigned int len);
 extern int fio_send_data(int sk, const void *p, unsigned int len);
 extern void fio_net_cmd_crc(struct fio_net_cmd *);

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 10/07/2014 03:29 PM, Jens Axboe wrote:
>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>> Hi All,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> One small suggestion:
>>>
>>> Running Client and Server architecture on FIO, we need to have all
>>>config
>>> files "locally" to run it.
>>>
>>>
>>> Nossa Senhora:tools neto$ ls
>>> 151 152 fio
>>>
>>>
>>>
>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>>> 10.61.109.152 152
>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>ioengine=libaio,
>>> iodepth=1
>>> ioengine=libaio, iodepth=1
>>>
>>> If we are doing a test in an HPC environment with hundred of files,
>>> wouldn't be easier to try to point to a single location for the file
>>>and
>>> access it remotely (we won't need to copy it locally).
>>>
>>>
>>> For example:
>>>
>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>>> 10.61.109.152 --remote-config /root/fio/152
>>>
>>>
>>> Thoughts?
>> 
>> That's a good idea, would not be hard to insert that step of having the
>> remote fio server load a local config file. I'll look into that.
>
>Totally untested, but the below is a start. On the client side, you'd do:
>
>fio --client=server-hostname --remote-config /some/path/to/file
>
>and then the server should attempt to open that. Needs a bit of error
>handling, but the concept should be there.
>
>-- 
>Jens Axboe


Hi Jens,

This is neto from Brazil

How are you?

I think it is not working.... Please see below:

Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
/root/fio.patch/model
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1

Thank you,


neto

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:

>
>
>On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>>On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>> Hi All,
>>>>
>>>> This is neto from Brazil
>>>>
>>>> How are you?
>>>>
>>>> One small suggestion:
>>>>
>>>> Running Client and Server architecture on FIO, we need to have all
>>>>config
>>>> files "locally" to run it.
>>>>
>>>>
>>>> Nossa Senhora:tools neto$ ls
>>>> 151 152 fio
>>>>
>>>>
>>>>
>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>>>> 10.61.109.152 152
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>ioengine=libaio,
>>>> iodepth=1
>>>> ioengine=libaio, iodepth=1
>>>>
>>>> If we are doing a test in an HPC environment with hundred of files,
>>>> wouldn't be easier to try to point to a single location for the file
>>>>and
>>>> access it remotely (we won't need to copy it locally).
>>>>
>>>>
>>>> For example:
>>>>
>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>
>>>>
>>>> Thoughts?
>>> 
>>> That's a good idea, would not be hard to insert that step of having the
>>> remote fio server load a local config file. I'll look into that.
>>
>>Totally untested, but the below is a start. On the client side, you'd do:
>>
>>fio --client=server-hostname --remote-config /some/path/to/file
>>
>>and then the server should attempt to open that. Needs a bit of error
>>handling, but the concept should be there.
>>
>>-- 
>>Jens Axboe
>
>
>Hi Jens,
>
>This is neto from Brazil
>
>How are you?
>
>I think it is not working.... Please see below:
>
>Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>/root/fio.patch/model
>hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>flags=1
>
>Thank you,
>
>
>neto

On server side, I've got:

[root@s1 fio.patch]# ./fio --server
fio: server listening on 0.0.0.0,8765
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory
fopen job file: No such file or directory

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
> 
> 
> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
> 
>>
>>
>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>
>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>> Hi All,
>>>>>
>>>>> This is neto from Brazil
>>>>>
>>>>> How are you?
>>>>>
>>>>> One small suggestion:
>>>>>
>>>>> Running Client and Server architecture on FIO, we need to have all
>>>>> config
>>>>> files "locally" to run it.
>>>>>
>>>>>
>>>>> Nossa Senhora:tools neto$ ls
>>>>> 151 152 fio
>>>>>
>>>>>
>>>>>
>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>>>>> 10.61.109.152 152
>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>> ioengine=libaio,
>>>>> iodepth=1
>>>>> ioengine=libaio, iodepth=1
>>>>>
>>>>> If we are doing a test in an HPC environment with hundred of files,
>>>>> wouldn't be easier to try to point to a single location for the file
>>>>> and
>>>>> access it remotely (we won't need to copy it locally).
>>>>>
>>>>>
>>>>> For example:
>>>>>
>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>
>>>>>
>>>>> Thoughts?
>>>>
>>>> That's a good idea, would not be hard to insert that step of having the
>>>> remote fio server load a local config file. I'll look into that.
>>>
>>> Totally untested, but the below is a start. On the client side, you'd do:
>>>
>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>
>>> and then the server should attempt to open that. Needs a bit of error
>>> handling, but the concept should be there.
>>>
>>> -- 
>>> Jens Axboe
>>
>>
>> Hi Jens,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> I think it is not working.... Please see below:
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fio.patch/model
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>> flags=1
>>
>> Thank you,
>>
>>
>> neto
> 
> On server side, I've got:
> 
> [root@s1 fio.patch]# ./fio --server
> fio: server listening on 0.0.0.0,8765
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory
> fopen job file: No such file or directory

Just to be sure, when you use --remote-config /root/fio.patch/model,
then it will open /root/fio.patch/model on the s1 machine running fio
--server.

Ran a quick test here, and it seems to work for me. The fio --server
opens the file at the given path, not the client. If you wanted the
client to open it, you'd just do

fio --client 10.61.109.151 /root/fio.patch/model

and it'd do the opposite - load the file on the client side, and send it
to the server.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

> On Oct 7, 2014, at 6:33 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
>> 
>> 
>> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>> 
>>> 
>>> 
>>>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>> 
>>>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>> Hi All,
>>>>>> 
>>>>>> This is neto from Brazil
>>>>>> 
>>>>>> How are you?
>>>>>> 
>>>>>> One small suggestion:
>>>>>> 
>>>>>> Running Client and Server architecture on FIO, we need to have all
>>>>>> config
>>>>>> files "locally" to run it.
>>>>>> 
>>>>>> 
>>>>>> Nossa Senhora:tools neto$ ls
>>>>>> 151 152 fio
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>>>>>> 10.61.109.152 152
>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>> ioengine=libaio,
>>>>>> iodepth=1
>>>>>> ioengine=libaio, iodepth=1
>>>>>> 
>>>>>> If we are doing a test in an HPC environment with hundred of files,
>>>>>> wouldn't be easier to try to point to a single location for the file
>>>>>> and
>>>>>> access it remotely (we won't need to copy it locally).
>>>>>> 
>>>>>> 
>>>>>> For example:
>>>>>> 
>>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>> 
>>>>>> 
>>>>>> Thoughts?
>>>>> 
>>>>> That's a good idea, would not be hard to insert that step of having the
>>>>> remote fio server load a local config file. I'll look into that.
>>>> 
>>>> Totally untested, but the below is a start. On the client side, you'd do:
>>>> 
>>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>> 
>>>> and then the server should attempt to open that. Needs a bit of error
>>>> handling, but the concept should be there.
>>>> 
>>>> -- 
>>>> Jens Axboe
>>> 
>>> 
>>> Hi Jens,
>>> 
>>> This is neto from Brazil
>>> 
>>> How are you?
>>> 
>>> I think it is not working.... Please see below:
>>> 
>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fio.patch/model
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> 
>>> Thank you,
>>> 
>>> 
>>> neto
>> 
>> On server side, I've got:
>> 
>> [root@s1 fio.patch]# ./fio --server
>> fio: server listening on 0.0.0.0,8765
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
>> fopen job file: No such file or directory
> 
> Just to be sure, when you use --remote-config /root/fio.patch/model,
> then it will open /root/fio.patch/model on the s1 machine running fio
> --server.
> 
> Ran a quick test here, and it seems to work for me. The fio --server
> opens the file at the given path, not the client. If you wanted the
> client to open it, you'd just do
> 
> fio --client 10.61.109.151 /root/fio.patch/model
> 
> and it'd do the opposite - load the file on the client side, and send it
> to the server.
> 
> -- 
> Jens Axboe
> 

I will try again tonight but it did not work

My client is on mac and servers are on linux

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 6:44 PM, "Neto, Antonio Jose Rodrigues"
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:

>
>
>
>
>> On Oct 7, 2014, at 6:33 PM, Jens Axboe <axboe@kernel.dk> wrote:
>> 
>>> On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
>>> 
>>> 
>>> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
>>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>> 
>>>> 
>>>> 
>>>>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>> 
>>>>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> This is neto from Brazil
>>>>>>> 
>>>>>>> How are you?
>>>>>>> 
>>>>>>> One small suggestion:
>>>>>>> 
>>>>>>> Running Client and Server architecture on FIO, we need to have all
>>>>>>> config
>>>>>>> files "locally" to run it.
>>>>>>> 
>>>>>>> 
>>>>>>> Nossa Senhora:tools neto$ ls
>>>>>>> 151 152 fio
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151 --client
>>>>>>> 10.61.109.152 152
>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>> flags=1
>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>> flags=1
>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>> ioengine=libaio,
>>>>>>> iodepth=1
>>>>>>> ioengine=libaio, iodepth=1
>>>>>>> 
>>>>>>> If we are doing a test in an HPC environment with hundred of files,
>>>>>>> wouldn't be easier to try to point to a single location for the
>>>>>>>file
>>>>>>> and
>>>>>>> access it remotely (we won't need to copy it locally).
>>>>>>> 
>>>>>>> 
>>>>>>> For example:
>>>>>>> 
>>>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151 --client
>>>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>>> 
>>>>>>> 
>>>>>>> Thoughts?
>>>>>> 
>>>>>> That's a good idea, would not be hard to insert that step of having
>>>>>>the
>>>>>> remote fio server load a local config file. I'll look into that.
>>>>> 
>>>>> Totally untested, but the below is a start. On the client side,
>>>>>you'd do:
>>>>> 
>>>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>>> 
>>>>> and then the server should attempt to open that. Needs a bit of error
>>>>> handling, but the concept should be there.
>>>>> 
>>>>> -- 
>>>>> Jens Axboe
>>>> 
>>>> 
>>>> Hi Jens,
>>>> 
>>>> This is neto from Brazil
>>>> 
>>>> How are you?
>>>> 
>>>> I think it is not working.... Please see below:
>>>> 
>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>>--remote-config
>>>> /root/fio.patch/model
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> 
>>>> Thank you,
>>>> 
>>>> 
>>>> neto
>>> 
>>> On server side, I've got:
>>> 
>>> [root@s1 fio.patch]# ./fio --server
>>> fio: server listening on 0.0.0.0,8765
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>>> fopen job file: No such file or directory
>> 
>> Just to be sure, when you use --remote-config /root/fio.patch/model,
>> then it will open /root/fio.patch/model on the s1 machine running fio
>> --server.
>> 
>> Ran a quick test here, and it seems to work for me. The fio --server
>> opens the file at the given path, not the client. If you wanted the
>> client to open it, you'd just do
>> 
>> fio --client 10.61.109.151 /root/fio.patch/model
>> 
>> and it'd do the opposite - load the file on the client side, and send it
>> to the server.
>> 
>> -- 
>> Jens Axboe
>> 
>
>I will try again tonight but it did not work
>
>My client is on mac and servers are on linux



Hi Jens,

This is neto from Brazil

How are you?

I found what is happening

If I use a non absolute path it works: the remote config should be on the
same directory with fio


Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
conf3



If I specify an absolute path, it does not work

Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
/root/fio.patch/conf3
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1


Do you think we could fix it to make sure we can use --remote-config with
an absolute path?

Thanks

neto

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 9:07 PM, "Neto, Antonio Jose Rodrigues"
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:

>
>
>On 10/7/14, 6:44 PM, "Neto, Antonio Jose Rodrigues"
><Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>
>>
>>
>>
>>
>>> On Oct 7, 2014, at 6:33 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>> 
>>>> On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
>>>> 
>>>> 
>>>> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
>>>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>>> 
>>>>> 
>>>>> 
>>>>>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>> 
>>>>>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>>>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> This is neto from Brazil
>>>>>>>> 
>>>>>>>> How are you?
>>>>>>>> 
>>>>>>>> One small suggestion:
>>>>>>>> 
>>>>>>>> Running Client and Server architecture on FIO, we need to have all
>>>>>>>> config
>>>>>>>> files "locally" to run it.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Nossa Senhora:tools neto$ ls
>>>>>>>> 151 152 fio
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151
>>>>>>>>--client
>>>>>>>> 10.61.109.152 152
>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>> flags=1
>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>> flags=1
>>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>>> ioengine=libaio,
>>>>>>>> iodepth=1
>>>>>>>> ioengine=libaio, iodepth=1
>>>>>>>> 
>>>>>>>> If we are doing a test in an HPC environment with hundred of
>>>>>>>>files,
>>>>>>>> wouldn't be easier to try to point to a single location for the
>>>>>>>>file
>>>>>>>> and
>>>>>>>> access it remotely (we won't need to copy it locally).
>>>>>>>> 
>>>>>>>> 
>>>>>>>> For example:
>>>>>>>> 
>>>>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151
>>>>>>>>--client
>>>>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thoughts?
>>>>>>> 
>>>>>>> That's a good idea, would not be hard to insert that step of having
>>>>>>>the
>>>>>>> remote fio server load a local config file. I'll look into that.
>>>>>> 
>>>>>> Totally untested, but the below is a start. On the client side,
>>>>>>you'd do:
>>>>>> 
>>>>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>>>> 
>>>>>> and then the server should attempt to open that. Needs a bit of
>>>>>>error
>>>>>> handling, but the concept should be there.
>>>>>> 
>>>>>> -- 
>>>>>> Jens Axboe
>>>>> 
>>>>> 
>>>>> Hi Jens,
>>>>> 
>>>>> This is neto from Brazil
>>>>> 
>>>>> How are you?
>>>>> 
>>>>> I think it is not working.... Please see below:
>>>>> 
>>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>>>--remote-config
>>>>> /root/fio.patch/model
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> 
>>>>> Thank you,
>>>>> 
>>>>> 
>>>>> neto
>>>> 
>>>> On server side, I've got:
>>>> 
>>>> [root@s1 fio.patch]# ./fio --server
>>>> fio: server listening on 0.0.0.0,8765
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>>> fopen job file: No such file or directory
>>> 
>>> Just to be sure, when you use --remote-config /root/fio.patch/model,
>>> then it will open /root/fio.patch/model on the s1 machine running fio
>>> --server.
>>> 
>>> Ran a quick test here, and it seems to work for me. The fio --server
>>> opens the file at the given path, not the client. If you wanted the
>>> client to open it, you'd just do
>>> 
>>> fio --client 10.61.109.151 /root/fio.patch/model
>>> 
>>> and it'd do the opposite - load the file on the client side, and send
>>>it
>>> to the server.
>>> 
>>> -- 
>>> Jens Axboe
>>> 
>>
>>I will try again tonight but it did not work
>>
>>My client is on mac and servers are on linux
>
>
>
>Hi Jens,
>
>This is neto from Brazil
>
>How are you?
>
>I found what is happening
>
>If I use a non absolute path it works: the remote config should be on the
>same directory with fio
>
>
>Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>conf3
>
>
>
>If I specify an absolute path, it does not work
>
>Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>/root/fio.patch/conf3
>hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>flags=1
>
>
>Do you think we could fix it to make sure we can use --remote-config with
>an absolute path?
>
>Thanks
>
>neto
>


Also, another thing that I have figured out is (not sure if this was by
design or not)


Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
model --client 10.61.109.152 --remote-config model
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
<s2> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, <s1> workload:
(g=0): rw=read, ioengine=libaio, iodepth=1
bs=64K-64K/64K-64K/64K-64K, <s2> ...
ioengine=libaio, iodepth=1
<s1> ...
<s2> Starting <s1> Starting 128 threads
128 threads
Jobs: 0 (f=0)
Jobs: 0 (f=0)


Jobs: 0 (f=0)
Jobs: 0 (f=0)





Running client and server, we do not have the progress (%) and IOPS and
KB/s and the time....

Is this expected?

Thank you,

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-07 19:19, Neto, Antonio Jose Rodrigues wrote:
>
>
> On 10/7/14, 9:07 PM, "Neto, Antonio Jose Rodrigues"
> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>
>>
>>
>> On 10/7/14, 6:44 PM, "Neto, Antonio Jose Rodrigues"
>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>
>>>
>>>
>>>
>>>
>>>> On Oct 7, 2014, at 6:33 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>
>>>>> On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
>>>>>
>>>>>
>>>>> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
>>>>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>>
>>>>>>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>>>>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> This is neto from Brazil
>>>>>>>>>
>>>>>>>>> How are you?
>>>>>>>>>
>>>>>>>>> One small suggestion:
>>>>>>>>>
>>>>>>>>> Running Client and Server architecture on FIO, we need to have all
>>>>>>>>> config
>>>>>>>>> files "locally" to run it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nossa Senhora:tools neto$ ls
>>>>>>>>> 151 152 fio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151
>>>>>>>>> --client
>>>>>>>>> 10.61.109.152 152
>>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>>> flags=1
>>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>>> flags=1
>>>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>>>> ioengine=libaio,
>>>>>>>>> iodepth=1
>>>>>>>>> ioengine=libaio, iodepth=1
>>>>>>>>>
>>>>>>>>> If we are doing a test in an HPC environment with hundred of
>>>>>>>>> files,
>>>>>>>>> wouldn't be easier to try to point to a single location for the
>>>>>>>>> file
>>>>>>>>> and
>>>>>>>>> access it remotely (we won't need to copy it locally).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For example:
>>>>>>>>>
>>>>>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151
>>>>>>>>> --client
>>>>>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> That's a good idea, would not be hard to insert that step of having
>>>>>>>> the
>>>>>>>> remote fio server load a local config file. I'll look into that.
>>>>>>>
>>>>>>> Totally untested, but the below is a start. On the client side,
>>>>>>> you'd do:
>>>>>>>
>>>>>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>>>>>
>>>>>>> and then the server should attempt to open that. Needs a bit of
>>>>>>> error
>>>>>>> handling, but the concept should be there.
>>>>>>>
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>
>>>>>>
>>>>>> Hi Jens,
>>>>>>
>>>>>> This is neto from Brazil
>>>>>>
>>>>>> How are you?
>>>>>>
>>>>>> I think it is not working.... Please see below:
>>>>>>
>>>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>>>> --remote-config
>>>>>> /root/fio.patch/model
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>>
>>>>>> neto
>>>>>
>>>>> On server side, I've got:
>>>>>
>>>>> [root@s1 fio.patch]# ./fio --server
>>>>> fio: server listening on 0.0.0.0,8765
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>> fopen job file: No such file or directory
>>>>
>>>> Just to be sure, when you use --remote-config /root/fio.patch/model,
>>>> then it will open /root/fio.patch/model on the s1 machine running fio
>>>> --server.
>>>>
>>>> Ran a quick test here, and it seems to work for me. The fio --server
>>>> opens the file at the given path, not the client. If you wanted the
>>>> client to open it, you'd just do
>>>>
>>>> fio --client 10.61.109.151 /root/fio.patch/model
>>>>
>>>> and it'd do the opposite - load the file on the client side, and send
>>>> it
>>>> to the server.
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>>
>>> I will try again tonight but it did not work
>>>
>>> My client is on mac and servers are on linux
>>
>>
>>
>> Hi Jens,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> I found what is happening
>>
>> If I use a non absolute path it works: the remote config should be on the
>> same directory with fio
>>
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>> conf3
>>
>>
>>
>> If I specify an absolute path, it does not work
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fio.patch/conf3
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
>> flags=1
>>
>>
>> Do you think we could fix it to make sure we can use --remote-config with
>> an absolute path?
>>
>> Thanks
>>
>> neto
>>
>
>
> Also, another thing that I have figured out is (not sure if this was by
> design or not)
>
>
> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
> model --client 10.61.109.152 --remote-config model
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> <s2> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, <s1> workload:
> (g=0): rw=read, ioengine=libaio, iodepth=1
> bs=64K-64K/64K-64K/64K-64K, <s2> ...
> ioengine=libaio, iodepth=1
> <s1> ...
> <s2> Starting <s1> Starting 128 threads
> 128 threads
> Jobs: 0 (f=0)
> Jobs: 0 (f=0)
>
>
> Jobs: 0 (f=0)
> Jobs: 0 (f=0)
>
>
>
>
>
> Running client and server, we do not have the progress (%) and IOPS and
> KB/s and the time....
>
> Is this expected?

Nope, that should work as non client/server. It does for me. You might 
want to try a full make clean and remake, the patch touches some 
structures and fio isn't always consistent in rebuilding perfectly (this 
is something that should be fixed...).

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 9:58 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-07 19:19, Neto, Antonio Jose Rodrigues wrote:
>>
>>
>> On 10/7/14, 9:07 PM, "Neto, Antonio Jose Rodrigues"
>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>
>>>
>>>
>>> On 10/7/14, 6:44 PM, "Neto, Antonio Jose Rodrigues"
>>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Oct 7, 2014, at 6:33 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 10/07/2014 04:11 PM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>
>>>>>>
>>>>>> On 10/7/14, 6:09 PM, "Neto, Antonio Jose Rodrigues"
>>>>>> <Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On 10/7/14, 6:01 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>>>
>>>>>>>>> On 10/07/2014 03:29 PM, Jens Axboe wrote:
>>>>>>>>>> On 10/07/2014 11:38 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> This is neto from Brazil
>>>>>>>>>>
>>>>>>>>>> How are you?
>>>>>>>>>>
>>>>>>>>>> One small suggestion:
>>>>>>>>>>
>>>>>>>>>> Running Client and Server architecture on FIO, we need to have
>>>>>>>>>>all
>>>>>>>>>> config
>>>>>>>>>> files "locally" to run it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nossa Senhora:tools neto$ ls
>>>>>>>>>> 151 152 fio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nossa Senhora:tools neto$ ./fio --client 10.61.109.151 151
>>>>>>>>>> --client
>>>>>>>>>> 10.61.109.152 152
>>>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>>>> flags=1
>>>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>>>> flags=1
>>>>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>>>>> ioengine=libaio,
>>>>>>>>>> iodepth=1
>>>>>>>>>> ioengine=libaio, iodepth=1
>>>>>>>>>>
>>>>>>>>>> If we are doing a test in an HPC environment with hundred of
>>>>>>>>>> files,
>>>>>>>>>> wouldn't be easier to try to point to a single location for the
>>>>>>>>>> file
>>>>>>>>>> and
>>>>>>>>>> access it remotely (we won't need to copy it locally).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For example:
>>>>>>>>>>
>>>>>>>>>> ./fio --client 10.61.109.151 --remote-config /root/fio/151
>>>>>>>>>> --client
>>>>>>>>>> 10.61.109.152 --remote-config /root/fio/152
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>
>>>>>>>>> That's a good idea, would not be hard to insert that step of
>>>>>>>>>having
>>>>>>>>> the
>>>>>>>>> remote fio server load a local config file. I'll look into that.
>>>>>>>>
>>>>>>>> Totally untested, but the below is a start. On the client side,
>>>>>>>> you'd do:
>>>>>>>>
>>>>>>>> fio --client=server-hostname --remote-config /some/path/to/file
>>>>>>>>
>>>>>>>> and then the server should attempt to open that. Needs a bit of
>>>>>>>> error
>>>>>>>> handling, but the concept should be there.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jens Axboe
>>>>>>>
>>>>>>>
>>>>>>> Hi Jens,
>>>>>>>
>>>>>>> This is neto from Brazil
>>>>>>>
>>>>>>> How are you?
>>>>>>>
>>>>>>> I think it is not working.... Please see below:
>>>>>>>
>>>>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>>>>> --remote-config
>>>>>>> /root/fio.patch/model
>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>> flags=1
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>>
>>>>>>> neto
>>>>>>
>>>>>> On server side, I've got:
>>>>>>
>>>>>> [root@s1 fio.patch]# ./fio --server
>>>>>> fio: server listening on 0.0.0.0,8765
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>> fopen job file: No such file or directory
>>>>>
>>>>> Just to be sure, when you use --remote-config /root/fio.patch/model,
>>>>> then it will open /root/fio.patch/model on the s1 machine running fio
>>>>> --server.
>>>>>
>>>>> Ran a quick test here, and it seems to work for me. The fio --server
>>>>> opens the file at the given path, not the client. If you wanted the
>>>>> client to open it, you'd just do
>>>>>
>>>>> fio --client 10.61.109.151 /root/fio.patch/model
>>>>>
>>>>> and it'd do the opposite - load the file on the client side, and send
>>>>> it
>>>>> to the server.
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>>
>>>>
>>>> I will try again tonight but it did not work
>>>>
>>>> My client is on mac and servers are on linux
>>>
>>>
>>>
>>> Hi Jens,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> I found what is happening
>>>
>>> If I use a non absolute path it works: the remote config should be on
>>>the
>>> same directory with fio
>>>
>>>
>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>--remote-config
>>> conf3
>>>
>>>
>>>
>>> If I specify an absolute path, it does not work
>>>
>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>--remote-config
>>> /root/fio.patch/conf3
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>>
>>>
>>> Do you think we could fix it to make sure we can use --remote-config
>>>with
>>> an absolute path?
>>>
>>> Thanks
>>>
>>> neto
>>>
>>
>>
>> Also, another thing that I have figured out is (not sure if this was by
>> design or not)
>>
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>--remote-config
>> model --client 10.61.109.152 --remote-config model
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> <s2> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, <s1>
>>workload:
>> (g=0): rw=read, ioengine=libaio, iodepth=1
>> bs=64K-64K/64K-64K/64K-64K, <s2> ...
>> ioengine=libaio, iodepth=1
>> <s1> ...
>> <s2> Starting <s1> Starting 128 threads
>> 128 threads
>> Jobs: 0 (f=0)
>> Jobs: 0 (f=0)
>>
>>
>> Jobs: 0 (f=0)
>> Jobs: 0 (f=0)
>>
>>
>>
>>
>>
>> Running client and server, we do not have the progress (%) and IOPS and
>> KB/s and the time....
>>
>> Is this expected?
>
>Nope, that should work as non client/server. It does for me. You might
>want to try a full make clean and remake, the patch touches some
>structures and fio isn't always consistent in rebuilding perfectly (this
>is something that should be fixed...).
>
>-- 
>Jens Axboe


Hi Jens,

This is neto from Brazil

How are you?

Download from latest git (branch remote-login)

The name of the config file is model

Please see below (please note model:70?)


Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
/root/fio.patch/fio/model
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-42-g3232,
flags=1
<s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
client: host=10.61.109.151 disconnected

Any ideas?

Thanks a lot

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fio.patch/fio/model
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-42-g3232,
> flags=1
> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
> client: host=10.61.109.151 disconnected
>
> Any ideas?

Looks like I just forgot to zero terminate that string. It was never 
absolute or relative path, just luck and what was in memory. Try and 
pull again, I committed a fix for that.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fio.patch/fio/model
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-42-g3232,
>> flags=1
>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>> client: host=10.61.109.151 disconnected
>>
>> Any ideas?
>
>Looks like I just forgot to zero terminate that string. It was never
>absolute or relative path, just luck and what was in memory. Try and
>pull again, I committed a fix for that.
>
>-- 
>Jens Axboe
>
>--


Hi Jens,

This is neto from Brazil

How are you?

Seems to me it's working with absolute path now with the latest commit to
remote-config branch.

But, running the workload from my mac (connected to 2 Linux clients) I do
not see the progress.

Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13, flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
<s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
iodepth=1
ioengine=libaio, iodepth=1
<s2> ...
<s1> ...
<s1> Starting <s2> Starting 128 threads
128 threads
Jobs: 0 (f=0)

Any idea why?


Thanks

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
> 
> 
> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
> 
>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fio.patch/fio/model
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-42-g3232,
>>> flags=1
>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>> client: host=10.61.109.151 disconnected
>>>
>>> Any ideas?
>>
>> Looks like I just forgot to zero terminate that string. It was never
>> absolute or relative path, just luck and what was in memory. Try and
>> pull again, I committed a fix for that.
>>
>> -- 
>> Jens Axboe
>>
>> --
> 
> 
> Hi Jens,
> 
> This is neto from Brazil
> 
> How are you?
> 
> Seems to me it's working with absolute path now with the latest commit to
> remote-config branch.

Great, I verified this morning that it was an issue, we'd be looking at
unitialized/allocated memory without it.

> But, running the workload from my mac (connected to 2 Linux clients) I do
> not see the progress.
> 
> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13, flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
> iodepth=1
> ioengine=libaio, iodepth=1
> <s2> ...
> <s1> ...
> <s1> Starting <s2> Starting 128 threads
> 128 threads
> Jobs: 0 (f=0)
> 
> Any idea why?

Works for me, just tried it from an OSX client. I notice that you don't
seem to have updated the 's2' fio version, however. So I'd suggest you
ensure you are running the same thing on all of them.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>> 
>> 
>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> 
>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fio.patch/fio/model
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-42-g3232,
>>>> flags=1
>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>> client: host=10.61.109.151 disconnected
>>>>
>>>> Any ideas?
>>>
>>> Looks like I just forgot to zero terminate that string. It was never
>>> absolute or relative path, just luck and what was in memory. Try and
>>> pull again, I committed a fix for that.
>>>
>>> -- 
>>> Jens Axboe
>>>
>>> --
>> 
>> 
>> Hi Jens,
>> 
>> This is neto from Brazil
>> 
>> How are you?
>> 
>> Seems to me it's working with absolute path now with the latest commit
>>to
>> remote-config branch.
>
>Great, I verified this morning that it was an issue, we'd be looking at
>unitialized/allocated memory without it.
>
>> But, running the workload from my mac (connected to 2 Linux clients) I
>>do
>> not see the progress.
>> 
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>> iodepth=1
>> ioengine=libaio, iodepth=1
>> <s2> ...
>> <s1> ...
>> <s1> Starting <s2> Starting 128 threads
>> 128 threads
>> Jobs: 0 (f=0)
>> 
>> Any idea why?
>
>Works for me, just tried it from an OSX client. I notice that you don't
>seem to have updated the 's2' fio version, however. So I'd suggest you
>ensure you are running the same thing on all of them.
>
>-- 
>Jens Axboe
>


Hi Jens,

This is neto from Brazil

How are you?

With one client and one server it works

Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/model 
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
<s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
ioengine=libaio, iodepth=1
<s1> ...
<s1> Starting 128 threads
Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0 iops]
[eta 09m:45s]





But with one client and 2 servers it does not work (the progress)


Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
flags=1
<s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
iodepth=1
ioengine=libaio, iodepth=1
<s2> ...
<s1> ...
<s2> Starting <s1> Starting 128 threads128 threads

Jobs: 0 (f=0)
Jobs: 0 (f=0)




Jobs: 0 (f=0)

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
> 
> 
> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
> 
>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>
>>>
>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>
>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fio.patch/fio/model
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-42-g3232,
>>>>> flags=1
>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>> client: host=10.61.109.151 disconnected
>>>>>
>>>>> Any ideas?
>>>>
>>>> Looks like I just forgot to zero terminate that string. It was never
>>>> absolute or relative path, just luck and what was in memory. Try and
>>>> pull again, I committed a fix for that.
>>>>
>>>> -- 
>>>> Jens Axboe
>>>>
>>>> --
>>>
>>>
>>> Hi Jens,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> Seems to me it's working with absolute path now with the latest commit
>>> to
>>> remote-config branch.
>>
>> Great, I verified this morning that it was an issue, we'd be looking at
>> unitialized/allocated memory without it.
>>
>>> But, running the workload from my mac (connected to 2 Linux clients) I
>>> do
>>> not see the progress.
>>>
>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>> flags=1
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>>> iodepth=1
>>> ioengine=libaio, iodepth=1
>>> <s2> ...
>>> <s1> ...
>>> <s1> Starting <s2> Starting 128 threads
>>> 128 threads
>>> Jobs: 0 (f=0)
>>>
>>> Any idea why?
>>
>> Works for me, just tried it from an OSX client. I notice that you don't
>> seem to have updated the 's2' fio version, however. So I'd suggest you
>> ensure you are running the same thing on all of them.
>>
>> -- 
>> Jens Axboe
>>
> 
> 
> Hi Jens,
> 
> This is neto from Brazil
> 
> How are you?
> 
> With one client and one server it works
> 
> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fiop/model 
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
> ioengine=libaio, iodepth=1
> <s1> ...
> <s1> Starting 128 threads
> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0 iops]
> [eta 09m:45s]
> 
> 
> 
> 
> 
> But with one client and 2 servers it does not work (the progress)
> 
> 
> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
> iodepth=1
> ioengine=libaio, iodepth=1
> <s2> ...
> <s1> ...
> <s2> Starting <s1> Starting 128 threads128 threads
> 
> Jobs: 0 (f=0)
> Jobs: 0 (f=0)

Weird, tested two here, running different jobs, and it summed them up
fine and reported the ETA line. I will take a look, when time permits.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>> 
>> 
>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>> 
>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>
>>>>
>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>
>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>> /root/fio.patch/fio/model
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>> flags=1
>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>>> client: host=10.61.109.151 disconnected
>>>>>>
>>>>>> Any ideas?
>>>>>
>>>>> Looks like I just forgot to zero terminate that string. It was never
>>>>> absolute or relative path, just luck and what was in memory. Try and
>>>>> pull again, I committed a fix for that.
>>>>>
>>>>> -- 
>>>>> Jens Axboe
>>>>>
>>>>> --
>>>>
>>>>
>>>> Hi Jens,
>>>>
>>>> This is neto from Brazil
>>>>
>>>> How are you?
>>>>
>>>> Seems to me it's working with absolute path now with the latest commit
>>>> to
>>>> remote-config branch.
>>>
>>> Great, I verified this morning that it was an issue, we'd be looking at
>>> unitialized/allocated memory without it.
>>>
>>>> But, running the workload from my mac (connected to 2 Linux clients) I
>>>> do
>>>> not see the progress.
>>>>
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>/root/fiop/model
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>> flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>ioengine=libaio,
>>>> iodepth=1
>>>> ioengine=libaio, iodepth=1
>>>> <s2> ...
>>>> <s1> ...
>>>> <s1> Starting <s2> Starting 128 threads
>>>> 128 threads
>>>> Jobs: 0 (f=0)
>>>>
>>>> Any idea why?
>>>
>>> Works for me, just tried it from an OSX client. I notice that you don't
>>> seem to have updated the 's2' fio version, however. So I'd suggest you
>>> ensure you are running the same thing on all of them.
>>>
>>> -- 
>>> Jens Axboe
>>>
>> 
>> 
>> Hi Jens,
>> 
>> This is neto from Brazil
>> 
>> How are you?
>> 
>> With one client and one server it works
>> 
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/model
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>> ioengine=libaio, iodepth=1
>> <s1> ...
>> <s1> Starting 128 threads
>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>iops]
>> [eta 09m:45s]
>> 
>> 
>> 
>> 
>> 
>> But with one client and 2 servers it does not work (the progress)
>> 
>> 
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>> iodepth=1
>> ioengine=libaio, iodepth=1
>> <s2> ...
>> <s1> ...
>> <s2> Starting <s1> Starting 128 threads128 threads
>> 
>> Jobs: 0 (f=0)
>> Jobs: 0 (f=0)
>
>Weird, tested two here, running different jobs, and it summed them up
>fine and reported the ETA line. I will take a look, when time permits.
>
>-- 
>Jens Axboe


Hi Jens,

This is neto from Brazil

How are you?

Just a quick update to help you with the troubleshooting.

From latest commit ...

When I start the fio (from my mac to use 2 Linux servers)

When the job starts, I do not see anything on the screen only this:

Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
/root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
flags=1
<s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
iodepth=4
ioengine=libaio, iodepth=4
<s2> ...
<s1> ...
<s2> Starting <s1> Starting 64 threads
64 threads
Jobs: 0 (f=0)




After 60 seconds.... (on my config file)

I have this:

workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50 2014
  mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
    slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
    clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
     lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
    clat percentiles (usec):
     |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
3024],
     | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
8096],
     | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
100th=[197632],
     | 100th=[211968], 100th=[254976]
    bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
stdev=4463.44
    lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
    lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
    lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
  cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s, maxb=1223.2MB/s,
mint=30015msec, maxt=30015msec
Jobs: 0 (f=0)



After 60 seconds.... ( I have this)...


<s1>  0 (f=0)
workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04 2014
  mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
    slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
    clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
     lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
    clat percentiles (usec):
     |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
3120],
     | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
7328],
     | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
100th=[181248],
     | 100th=[207872], 100th=[244736]
    bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
stdev=4885.09
    lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
    lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
    lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
  cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s, maxb=1338.6MB/s,
mint=45006msec, maxt=45006msec



Thanks,

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-10 07:32, Neto, Antonio Jose Rodrigues wrote:
>
>
> On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>> On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>>>
>>>
>>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>
>>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>
>>>>>
>>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>>> /root/fio.patch/fio/model
>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>>> flags=1
>>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>>>> client: host=10.61.109.151 disconnected
>>>>>>>
>>>>>>> Any ideas?
>>>>>>
>>>>>> Looks like I just forgot to zero terminate that string. It was never
>>>>>> absolute or relative path, just luck and what was in memory. Try and
>>>>>> pull again, I committed a fix for that.
>>>>>>
>>>>>> --
>>>>>> Jens Axboe
>>>>>>
>>>>>> --
>>>>>
>>>>>
>>>>> Hi Jens,
>>>>>
>>>>> This is neto from Brazil
>>>>>
>>>>> How are you?
>>>>>
>>>>> Seems to me it's working with absolute path now with the latest commit
>>>>> to
>>>>> remote-config branch.
>>>>
>>>> Great, I verified this morning that it was an issue, we'd be looking at
>>>> unitialized/allocated memory without it.
>>>>
>>>>> But, running the workload from my mac (connected to 2 Linux clients) I
>>>>> do
>>>>> not see the progress.
>>>>>
>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>> /root/fiop/model
>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>>> flags=1
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>> ioengine=libaio,
>>>>> iodepth=1
>>>>> ioengine=libaio, iodepth=1
>>>>> <s2> ...
>>>>> <s1> ...
>>>>> <s1> Starting <s2> Starting 128 threads
>>>>> 128 threads
>>>>> Jobs: 0 (f=0)
>>>>>
>>>>> Any idea why?
>>>>
>>>> Works for me, just tried it from an OSX client. I notice that you don't
>>>> seem to have updated the 's2' fio version, however. So I'd suggest you
>>>> ensure you are running the same thing on all of them.
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>>
>>>
>>> Hi Jens,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> With one client and one server it works
>>>
>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fiop/model
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>>> ioengine=libaio, iodepth=1
>>> <s1> ...
>>> <s1> Starting 128 threads
>>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>> iops]
>>> [eta 09m:45s]
>>>
>>>
>>>
>>>
>>>
>>> But with one client and 2 servers it does not work (the progress)
>>>
>>>
>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>>> iodepth=1
>>> ioengine=libaio, iodepth=1
>>> <s2> ...
>>> <s1> ...
>>> <s2> Starting <s1> Starting 128 threads128 threads
>>>
>>> Jobs: 0 (f=0)
>>> Jobs: 0 (f=0)
>>
>> Weird, tested two here, running different jobs, and it summed them up
>> fine and reported the ETA line. I will take a look, when time permits.
>>
>> --
>> Jens Axboe
>
>
> Hi Jens,
>
> This is neto from Brazil
>
> How are you?
>
> Just a quick update to help you with the troubleshooting.
>
>  From latest commit ...
>
> When I start the fio (from my mac to use 2 Linux servers)
>
> When the job starts, I do not see anything on the screen only this:
>
> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
> flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
> flags=1
> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
> iodepth=4
> ioengine=libaio, iodepth=4
> <s2> ...
> <s1> ...
> <s2> Starting <s1> Starting 64 threads
> 64 threads
> Jobs: 0 (f=0)
>
>
>
>
> After 60 seconds.... (on my config file)
>
> I have this:
>
> workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50 2014
>    mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
>      slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
>      clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
>       lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
>      clat percentiles (usec):
>       |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
> 3024],
>       | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
> 8096],
>       | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
> 100th=[197632],
>       | 100th=[211968], 100th=[254976]
>      bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
> stdev=4463.44
>      lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
>      lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
>      lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
>    cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>> =64=0.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
>       latency   : target=0, window=0, percentile=100.00%, depth=4
>
> Run status group 0 (all jobs):
>    MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s, maxb=1223.2MB/s,
> mint=30015msec, maxt=30015msec
> Jobs: 0 (f=0)
>
>
>
> After 60 seconds.... ( I have this)...
>
>
> <s1>  0 (f=0)
> workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04 2014
>    mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
>      slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
>      clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
>       lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
>      clat percentiles (usec):
>       |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
> 3120],
>       | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
> 7328],
>       | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
> 100th=[181248],
>       | 100th=[207872], 100th=[244736]
>      bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
> stdev=4885.09
>      lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
>      lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
>      lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
>    cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>> =64=0.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
>       latency   : target=0, window=0, percentile=100.00%, depth=4
>
> Run status group 0 (all jobs):
>    MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s, maxb=1338.6MB/s,
> mint=45006msec, maxt=45006msec

It's weird, like there's some clock source issue. What happens if you 
add --eta=always as an option to fio?

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

> On Oct 11, 2014, at 12:27 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 2014-10-10 07:32, Neto, Antonio Jose Rodrigues wrote:
>> 
>> 
>>> On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>> 
>>>> On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>>>> 
>>>> 
>>>>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>> 
>>>>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>> 
>>>>>> 
>>>>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>> 
>>>>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>>>> /root/fio.patch/fio/model
>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>>>> flags=1
>>>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>>>>> client: host=10.61.109.151 disconnected
>>>>>>>> 
>>>>>>>> Any ideas?
>>>>>>> 
>>>>>>> Looks like I just forgot to zero terminate that string. It was never
>>>>>>> absolute or relative path, just luck and what was in memory. Try and
>>>>>>> pull again, I committed a fix for that.
>>>>>>> 
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>> 
>>>>>>> --
>>>>>> 
>>>>>> 
>>>>>> Hi Jens,
>>>>>> 
>>>>>> This is neto from Brazil
>>>>>> 
>>>>>> How are you?
>>>>>> 
>>>>>> Seems to me it's working with absolute path now with the latest commit
>>>>>> to
>>>>>> remote-config branch.
>>>>> 
>>>>> Great, I verified this morning that it was an issue, we'd be looking at
>>>>> unitialized/allocated memory without it.
>>>>> 
>>>>>> But, running the workload from my mac (connected to 2 Linux clients) I
>>>>>> do
>>>>>> not see the progress.
>>>>>> 
>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>>> /root/fiop/model
>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>>>> flags=1
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>> ioengine=libaio,
>>>>>> iodepth=1
>>>>>> ioengine=libaio, iodepth=1
>>>>>> <s2> ...
>>>>>> <s1> ...
>>>>>> <s1> Starting <s2> Starting 128 threads
>>>>>> 128 threads
>>>>>> Jobs: 0 (f=0)
>>>>>> 
>>>>>> Any idea why?
>>>>> 
>>>>> Works for me, just tried it from an OSX client. I notice that you don't
>>>>> seem to have updated the 's2' fio version, however. So I'd suggest you
>>>>> ensure you are running the same thing on all of them.
>>>>> 
>>>>> --
>>>>> Jens Axboe
>>>> 
>>>> 
>>>> Hi Jens,
>>>> 
>>>> This is neto from Brazil
>>>> 
>>>> How are you?
>>>> 
>>>> With one client and one server it works
>>>> 
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/model
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>>>> ioengine=libaio, iodepth=1
>>>> <s1> ...
>>>> <s1> Starting 128 threads
>>>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>>> iops]
>>>> [eta 09m:45s]
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> But with one client and 2 servers it does not work (the progress)
>>>> 
>>>> 
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/model --client 10.61.109.152 --remote-config /root/fiop/model
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio,
>>>> iodepth=1
>>>> ioengine=libaio, iodepth=1
>>>> <s2> ...
>>>> <s1> ...
>>>> <s2> Starting <s1> Starting 128 threads128 threads
>>>> 
>>>> Jobs: 0 (f=0)
>>>> Jobs: 0 (f=0)
>>> 
>>> Weird, tested two here, running different jobs, and it summed them up
>>> fine and reported the ETA line. I will take a look, when time permits.
>>> 
>>> --
>>> Jens Axboe
>> 
>> 
>> Hi Jens,
>> 
>> This is neto from Brazil
>> 
>> How are you?
>> 
>> Just a quick update to help you with the troubleshooting.
>> 
>> From latest commit ...
>> 
>> When I start the fio (from my mac to use 2 Linux servers)
>> 
>> When the job starts, I do not see anything on the screen only this:
>> 
>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
>> flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-58-g3441,
>> flags=1
>> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
>> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
>> iodepth=4
>> ioengine=libaio, iodepth=4
>> <s2> ...
>> <s1> ...
>> <s2> Starting <s1> Starting 64 threads
>> 64 threads
>> Jobs: 0 (f=0)
>> 
>> 
>> 
>> 
>> After 60 seconds.... (on my config file)
>> 
>> I have this:
>> 
>> workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50 2014
>>   mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
>>     slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
>>     clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
>>      lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
>>     clat percentiles (usec):
>>      |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
>> 3024],
>>      | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
>> 8096],
>>      | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
>> 100th=[197632],
>>      | 100th=[211968], 100th=[254976]
>>     bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
>> stdev=4463.44
>>     lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
>>     lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
>>     lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
>>   cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
>>   IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> =64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>      issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
>> drop=r=0/w=0/d=0
>>      latency   : target=0, window=0, percentile=100.00%, depth=4
>> 
>> Run status group 0 (all jobs):
>>   MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s, maxb=1223.2MB/s,
>> mint=30015msec, maxt=30015msec
>> Jobs: 0 (f=0)
>> 
>> 
>> 
>> After 60 seconds.... ( I have this)...
>> 
>> 
>> <s1>  0 (f=0)
>> workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04 2014
>>   mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
>>     slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
>>     clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
>>      lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
>>     clat percentiles (usec):
>>      |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
>> 3120],
>>      | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
>> 7328],
>>      | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
>> 100th=[181248],
>>      | 100th=[207872], 100th=[244736]
>>     bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
>> stdev=4885.09
>>     lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
>>     lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
>>     lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
>>   cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
>>   IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> =64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>      issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
>> drop=r=0/w=0/d=0
>>      latency   : target=0, window=0, percentile=100.00%, depth=4
>> 
>> Run status group 0 (all jobs):
>>   MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s, maxb=1338.6MB/s,
>> mint=45006msec, maxt=45006msec
> 
> It's weird, like there's some clock source issue. What happens if you add --eta=always as an option to fio?
> 
> -- 
> Jens Axboe

Hi Jens

I will try

Thanks

neto

> 

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 8611 bytes --]

On 2014-10-11 10:28, Jens Axboe wrote:
> On 2014-10-10 07:32, Neto, Antonio Jose Rodrigues wrote:
>>
>>
>> On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>
>>> On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>
>>>>
>>>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>
>>>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>
>>>>>>
>>>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>
>>>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151
>>>>>>>> --remote-config
>>>>>>>> /root/fio.patch/fio/model
>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>>>> flags=1
>>>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>>>>> client: host=10.61.109.151 disconnected
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Looks like I just forgot to zero terminate that string. It was never
>>>>>>> absolute or relative path, just luck and what was in memory. Try and
>>>>>>> pull again, I committed a fix for that.
>>>>>>>
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>>
>>>>>> Hi Jens,
>>>>>>
>>>>>> This is neto from Brazil
>>>>>>
>>>>>> How are you?
>>>>>>
>>>>>> Seems to me it's working with absolute path now with the latest
>>>>>> commit
>>>>>> to
>>>>>> remote-config branch.
>>>>>
>>>>> Great, I verified this morning that it was an issue, we'd be
>>>>> looking at
>>>>> unitialized/allocated memory without it.
>>>>>
>>>>>> But, running the workload from my mac (connected to 2 Linux
>>>>>> clients) I
>>>>>> do
>>>>>> not see the progress.
>>>>>>
>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>>> /root/fiop/model
>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>>>> flags=1
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>> ioengine=libaio,
>>>>>> iodepth=1
>>>>>> ioengine=libaio, iodepth=1
>>>>>> <s2> ...
>>>>>> <s1> ...
>>>>>> <s1> Starting <s2> Starting 128 threads
>>>>>> 128 threads
>>>>>> Jobs: 0 (f=0)
>>>>>>
>>>>>> Any idea why?
>>>>>
>>>>> Works for me, just tried it from an OSX client. I notice that you
>>>>> don't
>>>>> seem to have updated the 's2' fio version, however. So I'd suggest you
>>>>> ensure you are running the same thing on all of them.
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>>
>>>>
>>>>
>>>> Hi Jens,
>>>>
>>>> This is neto from Brazil
>>>>
>>>> How are you?
>>>>
>>>> With one client and one server it works
>>>>
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/model
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>>>> ioengine=libaio, iodepth=1
>>>> <s1> ...
>>>> <s1> Starting 128 threads
>>>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>>> iops]
>>>> [eta 09m:45s]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> But with one client and 2 servers it does not work (the progress)
>>>>
>>>>
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>> /root/fiop/model
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>> ioengine=libaio,
>>>> iodepth=1
>>>> ioengine=libaio, iodepth=1
>>>> <s2> ...
>>>> <s1> ...
>>>> <s2> Starting <s1> Starting 128 threads128 threads
>>>>
>>>> Jobs: 0 (f=0)
>>>> Jobs: 0 (f=0)
>>>
>>> Weird, tested two here, running different jobs, and it summed them up
>>> fine and reported the ETA line. I will take a look, when time permits.
>>>
>>> --
>>> Jens Axboe
>>
>>
>> Hi Jens,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> Just a quick update to help you with the troubleshooting.
>>
>>  From latest commit ...
>>
>> When I start the fio (from my mac to use 2 Linux servers)
>>
>> When the job starts, I do not see anything on the screen only this:
>>
>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-58-g3441,
>> flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-58-g3441,
>> flags=1
>> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
>> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
>> iodepth=4
>> ioengine=libaio, iodepth=4
>> <s2> ...
>> <s1> ...
>> <s2> Starting <s1> Starting 64 threads
>> 64 threads
>> Jobs: 0 (f=0)
>>
>>
>>
>>
>> After 60 seconds.... (on my config file)
>>
>> I have this:
>>
>> workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50
>> 2014
>>    mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
>>      slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
>>      clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
>>       lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
>>      clat percentiles (usec):
>>       |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
>> 3024],
>>       | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
>> 8096],
>>       | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
>> 100th=[197632],
>>       | 100th=[211968], 100th=[254976]
>>      bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
>> stdev=4463.44
>>      lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
>>      lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
>>      lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
>>    cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> =64=0.0%
>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>       issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
>> drop=r=0/w=0/d=0
>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>
>> Run status group 0 (all jobs):
>>    MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s, maxb=1223.2MB/s,
>> mint=30015msec, maxt=30015msec
>> Jobs: 0 (f=0)
>>
>>
>>
>> After 60 seconds.... ( I have this)...
>>
>>
>> <s1>  0 (f=0)
>> workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04
>> 2014
>>    mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
>>      slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
>>      clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
>>       lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
>>      clat percentiles (usec):
>>       |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
>> 3120],
>>       | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
>> 7328],
>>       | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
>> 100th=[181248],
>>       | 100th=[207872], 100th=[244736]
>>      bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
>> stdev=4885.09
>>      lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
>>      lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
>>      lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
>>    cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> =64=0.0%
>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> =64=0.0%
>>       issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
>> drop=r=0/w=0/d=0
>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>
>> Run status group 0 (all jobs):
>>    MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s, maxb=1338.6MB/s,
>> mint=45006msec, maxt=45006msec
>
> It's weird, like there's some clock source issue. What happens if you
> add --eta=always as an option to fio?

Or perhaps try this attached patch.


-- 
Jens Axboe


[-- Attachment #2: genesis.patch --]
[-- Type: text/x-patch, Size: 315 bytes --]

diff --git a/fio.c b/fio.c
index 7e6b06d3793f..9adc29aee47b 100644
--- a/fio.c
+++ b/fio.c
@@ -43,6 +43,8 @@ int main(int argc, char *argv[], char *envp[])
 	fio_time_init();
 
 	if (nr_clients) {
+		set_genesis_time();
+
 		if (fio_start_all_clients())
 			return 1;
 		return fio_handle_clients(&fio_client_ops);

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/11/14, 12:30 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-11 10:28, Jens Axboe wrote:
>> On 2014-10-10 07:32, Neto, Antonio Jose Rodrigues wrote:
>>>
>>>
>>> On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>
>>>> On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>
>>>>>
>>>>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>>
>>>>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151
>>>>>>>>> --remote-config
>>>>>>>>> /root/fio.patch/fio/model
>>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>>>>> flags=1
>>>>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job file
>>>>>>>>> client: host=10.61.109.151 disconnected
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Looks like I just forgot to zero terminate that string. It was
>>>>>>>>never
>>>>>>>> absolute or relative path, just luck and what was in memory. Try
>>>>>>>>and
>>>>>>>> pull again, I committed a fix for that.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jens Axboe
>>>>>>>>
>>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>> Hi Jens,
>>>>>>>
>>>>>>> This is neto from Brazil
>>>>>>>
>>>>>>> How are you?
>>>>>>>
>>>>>>> Seems to me it's working with absolute path now with the latest
>>>>>>> commit
>>>>>>> to
>>>>>>> remote-config branch.
>>>>>>
>>>>>> Great, I verified this morning that it was an issue, we'd be
>>>>>> looking at
>>>>>> unitialized/allocated memory without it.
>>>>>>
>>>>>>> But, running the workload from my mac (connected to 2 Linux
>>>>>>> clients) I
>>>>>>> do
>>>>>>> not see the progress.
>>>>>>>
>>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>>--remote-config
>>>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>>>> /root/fiop/model
>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>>>>> flags=1
>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>> flags=1
>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>> ioengine=libaio,
>>>>>>> iodepth=1
>>>>>>> ioengine=libaio, iodepth=1
>>>>>>> <s2> ...
>>>>>>> <s1> ...
>>>>>>> <s1> Starting <s2> Starting 128 threads
>>>>>>> 128 threads
>>>>>>> Jobs: 0 (f=0)
>>>>>>>
>>>>>>> Any idea why?
>>>>>>
>>>>>> Works for me, just tried it from an OSX client. I notice that you
>>>>>> don't
>>>>>> seem to have updated the 's2' fio version, however. So I'd suggest
>>>>>>you
>>>>>> ensure you are running the same thing on all of them.
>>>>>>
>>>>>> --
>>>>>> Jens Axboe
>>>>>>
>>>>>
>>>>>
>>>>> Hi Jens,
>>>>>
>>>>> This is neto from Brazil
>>>>>
>>>>> How are you?
>>>>>
>>>>> With one client and one server it works
>>>>>
>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fiop/model
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>>>>> ioengine=libaio, iodepth=1
>>>>> <s1> ...
>>>>> <s1> Starting 128 threads
>>>>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>>>> iops]
>>>>> [eta 09m:45s]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> But with one client and 2 servers it does not work (the progress)
>>>>>
>>>>>
>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>> /root/fiop/model
>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-31-g15e3,
>>>>> flags=1
>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>> ioengine=libaio,
>>>>> iodepth=1
>>>>> ioengine=libaio, iodepth=1
>>>>> <s2> ...
>>>>> <s1> ...
>>>>> <s2> Starting <s1> Starting 128 threads128 threads
>>>>>
>>>>> Jobs: 0 (f=0)
>>>>> Jobs: 0 (f=0)
>>>>
>>>> Weird, tested two here, running different jobs, and it summed them up
>>>> fine and reported the ETA line. I will take a look, when time permits.
>>>>
>>>> --
>>>> Jens Axboe
>>>
>>>
>>> Hi Jens,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> Just a quick update to help you with the troubleshooting.
>>>
>>>  From latest commit ...
>>>
>>> When I start the fio (from my mac to use 2 Linux servers)
>>>
>>> When the job starts, I do not see anything on the screen only this:
>>>
>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-58-g3441,
>>> flags=1
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-58-g3441,
>>> flags=1
>>> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
>>> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K,
>>>ioengine=libaio,
>>> iodepth=4
>>> ioengine=libaio, iodepth=4
>>> <s2> ...
>>> <s1> ...
>>> <s2> Starting <s1> Starting 64 threads
>>> 64 threads
>>> Jobs: 0 (f=0)
>>>
>>>
>>>
>>>
>>> After 60 seconds.... (on my config file)
>>>
>>> I have this:
>>>
>>> workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50
>>> 2014
>>>    mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
>>>      slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
>>>      clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
>>>       lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
>>>      clat percentiles (usec):
>>>       |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
>>> 3024],
>>>       | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
>>> 8096],
>>>       | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
>>> 100th=[197632],
>>>       | 100th=[211968], 100th=[254976]
>>>      bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
>>> stdev=4463.44
>>>      lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
>>>      lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
>>>      lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
>>>    cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
>>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>> =64=0.0%
>>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>>       issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
>>> drop=r=0/w=0/d=0
>>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>>
>>> Run status group 0 (all jobs):
>>>    MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s,
>>>maxb=1223.2MB/s,
>>> mint=30015msec, maxt=30015msec
>>> Jobs: 0 (f=0)
>>>
>>>
>>>
>>> After 60 seconds.... ( I have this)...
>>>
>>>
>>> <s1>  0 (f=0)
>>> workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04
>>> 2014
>>>    mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
>>>      slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
>>>      clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
>>>       lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
>>>      clat percentiles (usec):
>>>       |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
>>> 3120],
>>>       | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
>>> 7328],
>>>       | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
>>> 100th=[181248],
>>>       | 100th=[207872], 100th=[244736]
>>>      bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
>>> stdev=4885.09
>>>      lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
>>>      lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
>>>      lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
>>>    cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
>>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>> =64=0.0%
>>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>> =64=0.0%
>>>       issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
>>> drop=r=0/w=0/d=0
>>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>>
>>> Run status group 0 (all jobs):
>>>    MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s,
>>>maxb=1338.6MB/s,
>>> mint=45006msec, maxt=45006msec
>>
>> It's weird, like there's some clock source issue. What happens if you
>> add --eta=always as an option to fio?
>
>Or perhaps try this attached patch.
>
>
>-- 
>Jens Axboe


Hi Jens,

This is neto from Brazil

How are you?

Just applied the patch and it's perfect.

Please see below:

Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
<s2> fio: unable to open '/root/fio/iotest' job file
<s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
ioengine=libaio, iodepth=4
<s1> ...
<s1> Starting 64 threads
Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0 iops]
[eta 00m:34s]

Thank you very much,


neto

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/12/14, 11:26 AM, "Neto, Antonio Jose Rodrigues"
<Antonio.Jose.Rodrigues.Neto@netapp.com> wrote:

>
>
>On 10/11/14, 12:30 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>>On 2014-10-11 10:28, Jens Axboe wrote:
>>> On 2014-10-10 07:32, Neto, Antonio Jose Rodrigues wrote:
>>>>
>>>>
>>>> On 10/8/14, 10:52 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>
>>>>> On 10/08/2014 08:47 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>
>>>>>>
>>>>>> On 10/8/14, 10:33 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>
>>>>>>> On 10/08/2014 08:13 AM, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/8/14, 12:03 AM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>>>>>
>>>>>>>>> On 2014-10-07 21:24, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151
>>>>>>>>>> --remote-config
>>>>>>>>>> /root/fio.patch/fio/model
>>>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>>> fio=fio-2.1.13-42-g3232,
>>>>>>>>>> flags=1
>>>>>>>>>> <s1> fio: unable to open '/root/fio.patch/fio/model:70?' job
>>>>>>>>>>file
>>>>>>>>>> client: host=10.61.109.151 disconnected
>>>>>>>>>>
>>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Looks like I just forgot to zero terminate that string. It was
>>>>>>>>>never
>>>>>>>>> absolute or relative path, just luck and what was in memory. Try
>>>>>>>>>and
>>>>>>>>> pull again, I committed a fix for that.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jens Axboe
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Jens,
>>>>>>>>
>>>>>>>> This is neto from Brazil
>>>>>>>>
>>>>>>>> How are you?
>>>>>>>>
>>>>>>>> Seems to me it's working with absolute path now with the latest
>>>>>>>> commit
>>>>>>>> to
>>>>>>>> remote-config branch.
>>>>>>>
>>>>>>> Great, I verified this morning that it was an issue, we'd be
>>>>>>> looking at
>>>>>>> unitialized/allocated memory without it.
>>>>>>>
>>>>>>>> But, running the workload from my mac (connected to 2 Linux
>>>>>>>> clients) I
>>>>>>>> do
>>>>>>>> not see the progress.
>>>>>>>>
>>>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>>>--remote-config
>>>>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>>>>> /root/fiop/model
>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13,
>>>>>>>> flags=1
>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>>>> flags=1
>>>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>>>> ioengine=libaio,
>>>>>>>> iodepth=1
>>>>>>>> ioengine=libaio, iodepth=1
>>>>>>>> <s2> ...
>>>>>>>> <s1> ...
>>>>>>>> <s1> Starting <s2> Starting 128 threads
>>>>>>>> 128 threads
>>>>>>>> Jobs: 0 (f=0)
>>>>>>>>
>>>>>>>> Any idea why?
>>>>>>>
>>>>>>> Works for me, just tried it from an OSX client. I notice that you
>>>>>>> don't
>>>>>>> seem to have updated the 's2' fio version, however. So I'd suggest
>>>>>>>you
>>>>>>> ensure you are running the same thing on all of them.
>>>>>>>
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Jens,
>>>>>>
>>>>>> This is neto from Brazil
>>>>>>
>>>>>> How are you?
>>>>>>
>>>>>> With one client and one server it works
>>>>>>
>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>--remote-config
>>>>>> /root/fiop/model
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> <s1> workload: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K,
>>>>>> ioengine=libaio, iodepth=1
>>>>>> <s1> ...
>>>>>> <s1> Starting 128 threads
>>>>>> Jobs: 128 (f=2048): [R(128)] [4.4% done] [1770M/0K/0K /s] [27.7K/0/0
>>>>>> iops]
>>>>>> [eta 09m:45s]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> But with one client and 2 servers it does not work (the progress)
>>>>>>
>>>>>>
>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>--remote-config
>>>>>> /root/fiop/model --client 10.61.109.152 --remote-config
>>>>>> /root/fiop/model
>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-31-g15e3,
>>>>>> flags=1
>>>>>> <s2> workload: (g=0): rw=read, <s1> workload: (g=0): rw=read,
>>>>>> bs=64K-64K/64K-64K/64K-64K, bs=64K-64K/64K-64K/64K-64K,
>>>>>> ioengine=libaio,
>>>>>> iodepth=1
>>>>>> ioengine=libaio, iodepth=1
>>>>>> <s2> ...
>>>>>> <s1> ...
>>>>>> <s2> Starting <s1> Starting 128 threads128 threads
>>>>>>
>>>>>> Jobs: 0 (f=0)
>>>>>> Jobs: 0 (f=0)
>>>>>
>>>>> Weird, tested two here, running different jobs, and it summed them up
>>>>> fine and reported the ETA line. I will take a look, when time
>>>>>permits.
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>
>>>>
>>>> Hi Jens,
>>>>
>>>> This is neto from Brazil
>>>>
>>>> How are you?
>>>>
>>>> Just a quick update to help you with the troubleshooting.
>>>>
>>>>  From latest commit ...
>>>>
>>>> When I start the fio (from my mac to use 2 Linux servers)
>>>>
>>>> When the job starts, I do not see anything on the screen only this:
>>>>
>>>> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-58-g3441,
>>>> flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-58-g3441,
>>>> flags=1
>>>> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
>>>> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K,
>>>>ioengine=libaio,
>>>> iodepth=4
>>>> ioengine=libaio, iodepth=4
>>>> <s2> ...
>>>> <s1> ...
>>>> <s2> Starting <s1> Starting 64 threads
>>>> 64 threads
>>>> Jobs: 0 (f=0)
>>>>
>>>>
>>>>
>>>>
>>>> After 60 seconds.... (on my config file)
>>>>
>>>> I have this:
>>>>
>>>> workload: (groupid=0, jobs=64): err= 0: pid=3644: Fri Oct 10 09:31:50
>>>> 2014
>>>>    mixed: io=36714MB, bw=1223.2MB/s, iops=39141, runt= 30015msec
>>>>      slat (usec): min=10, max=308, avg=20.57, stdev= 5.24
>>>>      clat (usec): min=265, max=295734, avg=6496.79, stdev=12165.17
>>>>       lat (usec): min=280, max=295748, avg=6517.62, stdev=12165.19
>>>>      clat percentiles (usec):
>>>>       |  1th=[  868],  5th=[ 1336], 10th=[ 1720], 20th=[ 2384], 30th=[
>>>> 3024],
>>>>       | 40th=[ 3696], 50th=[ 4448], 60th=[ 5344], 70th=[ 6496], 80th=[
>>>> 8096],
>>>>       | 90th=[11968], 95th=[15808], 99th=[22912], 100th=[69120],
>>>> 100th=[197632],
>>>>       | 100th=[211968], 100th=[254976]
>>>>      bw (KB  /s): min= 3648, max=58112, per=1.57%, avg=19618.75,
>>>> stdev=4463.44
>>>>      lat (usec) : 500=0.06%, 750=0.45%, 1000=1.38%
>>>>      lat (msec) : 2=12.27%, 4=29.89%, 10=41.98%, 20=12.22%, 50=1.23%
>>>>      lat (msec) : 100=0.06%, 250=0.45%, 500=0.01%
>>>>    cpu          : usr=0.72%, sys=1.15%, ctx=1146612, majf=0, minf=223
>>>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>>> =64=0.0%
>>>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>>       issued    : total=r=1174844/w=0/d=0, short=r=0/w=0/d=0,
>>>> drop=r=0/w=0/d=0
>>>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>>>
>>>> Run status group 0 (all jobs):
>>>>    MIXED: io=36714MB, aggrb=1223.2MB/s, minb=1223.2MB/s,
>>>>maxb=1223.2MB/s,
>>>> mint=30015msec, maxt=30015msec
>>>> Jobs: 0 (f=0)
>>>>
>>>>
>>>>
>>>> After 60 seconds.... ( I have this)...
>>>>
>>>>
>>>> <s1>  0 (f=0)
>>>> workload: (groupid=0, jobs=64): err= 0: pid=3607: Fri Oct 10 09:32:04
>>>> 2014
>>>>    mixed: io=60243MB, bw=1338.6MB/s, iops=42833, runt= 45006msec
>>>>      slat (usec): min=10, max=1097, avg=21.47, stdev= 7.11
>>>>      clat (usec): min=256, max=302936, avg=5944.33, stdev=9841.20
>>>>       lat (usec): min=275, max=302957, avg=5966.03, stdev=9841.14
>>>>      clat percentiles (usec):
>>>>       |  1th=[  820],  5th=[ 1240], 10th=[ 1656], 20th=[ 2416], 30th=[
>>>> 3120],
>>>>       | 40th=[ 4048], 50th=[ 4704], 60th=[ 5152], 70th=[ 6048], 80th=[
>>>> 7328],
>>>>       | 90th=[10304], 95th=[14912], 99th=[21888], 100th=[25728],
>>>> 100th=[181248],
>>>>       | 100th=[207872], 100th=[244736]
>>>>      bw (KB  /s): min= 8256, max=53824, per=1.57%, avg=21476.50,
>>>> stdev=4885.09
>>>>      lat (usec) : 500=0.07%, 750=0.59%, 1000=1.81%
>>>>      lat (msec) : 2=11.88%, 4=25.20%, 10=49.90%, 20=8.94%, 50=1.29%
>>>>      lat (msec) : 100=0.04%, 250=0.26%, 500=0.01%
>>>>    cpu          : usr=0.78%, sys=1.29%, ctx=1952463, majf=0, minf=219
>>>>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>>>> =64=0.0%
>>>>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>>>> =64=0.0%
>>>>       issued    : total=r=1927764/w=0/d=0, short=r=0/w=0/d=0,
>>>> drop=r=0/w=0/d=0
>>>>       latency   : target=0, window=0, percentile=100.00%, depth=4
>>>>
>>>> Run status group 0 (all jobs):
>>>>    MIXED: io=60243MB, aggrb=1338.6MB/s, minb=1338.6MB/s,
>>>>maxb=1338.6MB/s,
>>>> mint=45006msec, maxt=45006msec
>>>
>>> It's weird, like there's some clock source issue. What happens if you
>>> add --eta=always as an option to fio?
>>
>>Or perhaps try this attached patch.
>>
>>
>>-- 
>>Jens Axboe
>
>
>Hi Jens,
>
>This is neto from Brazil
>
>How are you?
>
>Just applied the patch and it's perfect.
>
>Please see below:
>
>Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>/root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>fio=fio-2.1.13-59-gaa7bc, flags=1
>hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>fio=fio-2.1.13-59-gaa7bc, flags=1
><s2> fio: unable to open '/root/fio/iotest' job file
><s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>ioengine=libaio, iodepth=4
><s1> ...
><s1> Starting 64 threads
>Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0 iops]
>[eta 00m:34s]
>
>Thank you very much,
>
>
>neto



Hi Jens,

This is neto from Brazil

How are you?

This is the correct output (I forgot to add a "p" (patch) on the path for
the fio)

The only thing, I have realized when the job finishes, I have the results
for <s2> but I do not have the results for <s1>.

Please see below

Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/iotest --client 10.61.109.152 --remote-config /root/fiop/iotest
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
<s2> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K, <s1> workload:
(g=0): rw=write, ioengine=libaio, iodepth=4
bs=32K-32K/32K-32K/32K-32K, <s2> ...
ioengine=libaio, iodepth=4
<s1> ...
<s2> Starting 64 threads
<s1> Starting 64 threads
Jobs: 128 (f=2048): [W(64)] [35.6% done] [1290M/0K/0K /s] [40.4K/0/0 iops]
[eta 00m:47s] 
<s2>  128 (f=2048): [W(64)] [100.0% done] [1421M/0K/0K /s] [44.5K/0/0
iops] [eta 00m:00s]
workload: (groupid=0, jobs=64): err= 0: pid=3664: Sun Oct 12 11:32:39 2014
  mixed: io=37125MB, bw=633320KB/s, iops=19791, runt= 60027msec
    slat (usec): min=9, max=675, avg=20.72, stdev= 8.64
    clat (usec): min=2, max=393854, avg=12859.99, stdev=15404.54
     lat (usec): min=272, max=393873, avg=12880.96, stdev=15404.53
    clat percentiles (usec):
     |  1th=[  564],  5th=[  844], 10th=[ 1096], 20th=[ 2160], 30th=[
6432],
     | 40th=[ 7264], 50th=[ 8256], 60th=[10816], 70th=[14144],
80th=[20608],
     | 90th=[29824], 95th=[36096], 99th=[50432], 100th=[80384],
100th=[197632],
     | 100th=[222208], 100th=[272384]
    bw (KB  /s): min=  454, max=54976, per=1.57%, avg=9929.23,
stdev=3057.87
    lat (usec) : 4=0.01%, 250=0.01%, 500=0.39%, 750=3.05%, 1000=5.00%
    lat (msec) : 2=10.81%, 4=5.39%, 10=33.56%, 20=21.23%, 50=19.53%
    lat (msec) : 100=0.67%, 250=0.36%, 500=0.02%
  cpu          : usr=0.40%, sys=0.61%, ctx=1289065, majf=0, minf=238
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1188010/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=37125MB, aggrb=633320KB/s, minb=633320KB/s, maxb=633320KB/s,
mint=60027msec, maxt=60027msec
<s1> 
workload: (groupid=0, jobs=64): err= 0: pid=13542: Sun Oct 12 11:32:39 2014
  mixed: io=39244MB, bw=669649KB/s, iops=20926, runt= 60011msec
    slat (usec): min=10, max=605, avg=19.55, stdev= 8.80
    clat (usec): min=38, max=533252, avg=12159.86, stdev=15260.62
     lat (usec): min=284, max=533274, avg=12179.67, stdev=15260.68
    clat percentiles (usec):
     |  1th=[  580],  5th=[  852], 10th=[ 1080], 20th=[ 1800], 30th=[
6048],
     | 40th=[ 6816], 50th=[ 7712], 60th=[10048], 70th=[13248],
80th=[19072],
     | 90th=[28800], 95th=[34560], 99th=[49408], 100th=[78336],
100th=[199680],
     | 100th=[226304], 100th=[284672]
    bw (KB  /s): min=  522, max=59712, per=1.57%, avg=10504.86,
stdev=2763.83
    lat (usec) : 50=0.01%, 250=0.01%, 500=0.32%, 750=2.84%, 1000=5.21%
    lat (msec) : 2=13.09%, 4=5.60%, 10=32.84%, 20=21.09%, 50=18.05%
    lat (msec) : 100=0.60%, 250=0.33%, 500=0.03%, 750=0.01%
  cpu          : usr=0.41%, sys=0.60%, ctx=1347527, majf=0, minf=219
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1255823/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=39244MB, aggrb=669649KB/s, minb=669649KB/s, maxb=669649KB/s,
mint=60011msec, maxt=60011msec








>

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
> Just applied the patch and it's perfect.
>
> Please see below:
>
> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
> fio=fio-2.1.13-59-gaa7bc, flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
> fio=fio-2.1.13-59-gaa7bc, flags=1
> <s2> fio: unable to open '/root/fio/iotest' job file
> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
> ioengine=libaio, iodepth=4
> <s1> ...
> <s1> Starting 64 threads
> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0 iops]
> [eta 00m:34s]

Great, at least that took care of that issue. As to missing output from 
one client, I've seen that here before, I will look into that. It's a 
separate issue.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

> On Oct 12, 2014, at 3:11 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>> Just applied the patch and it's perfect.
>> 
>> Please see below:
>> 
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-59-gaa7bc, flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-59-gaa7bc, flags=1
>> <s2> fio: unable to open '/root/fio/iotest' job file
>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>> ioengine=libaio, iodepth=4
>> <s1> ...
>> <s1> Starting 64 threads
>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0 iops]
>> [eta 00m:34s]
> 
> Great, at least that took care of that issue. As to missing output from one client, I've seen that here before, I will look into that. It's a separate issue.
> 
> -- 
> Jens Axboe
> 

Thank you so much, Jens.

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-12 13:12, Jens Axboe wrote:
> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>> Just applied the patch and it's perfect.
>>
>> Please see below:
>>
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-59-gaa7bc, flags=1
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-59-gaa7bc, flags=1
>> <s2> fio: unable to open '/root/fio/iotest' job file
>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>> ioengine=libaio, iodepth=4
>> <s1> ...
>> <s1> Starting 64 threads
>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>> iops]
>> [eta 00m:34s]
>
> Great, at least that took care of that issue. As to missing output from
> one client, I've seen that here before, I will look into that. It's a
> separate issue.

For the above one, s2 never started since it could not find the config 
file you gave it. Have you seen missing final output for cases where the 
jobs did all start? This particular one does not look valid.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 2014-10-12 13:12, Jens Axboe wrote:
>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>> Just applied the patch and it's perfect.
>>> 
>>> Please see below:
>>> 
>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>> ioengine=libaio, iodepth=4
>>> <s1> ...
>>> <s1> Starting 64 threads
>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>> iops]
>>> [eta 00m:34s]
>> 
>> Great, at least that took care of that issue. As to missing output from
>> one client, I've seen that here before, I will look into that. It's a
>> separate issue.
> 
> For the above one, s2 never started since it could not find the config file you gave it. Have you seen missing final output for cases where the jobs did all start? This particular one does not look valid.
> 
> -- 
> Jens Axboe
> 


Yes. I did.

I ran using both servers but the output was showing the latest client - s2

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>
>
>
>
>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>
>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>> Just applied the patch and it's perfect.
>>>>
>>>> Please see below:
>>>>
>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>> ioengine=libaio, iodepth=4
>>>> <s1> ...
>>>> <s1> Starting 64 threads
>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>>> iops]
>>>> [eta 00m:34s]
>>>
>>> Great, at least that took care of that issue. As to missing output from
>>> one client, I've seen that here before, I will look into that. It's a
>>> separate issue.
>>
>> For the above one, s2 never started since it could not find the config file you gave it. Have you seen missing final output for cases where the jobs did all start? This particular one does not look valid.
>>
>> --
>> Jens Axboe
>>
>
>
> Yes. I did.
>
> I ran using both servers but the output was showing the latest client - s2

Odd. Can you reproduce and send the output of such a run?

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

> On Oct 12, 2014, at 8:27 PM, Jens Axboe <axboe@kernel.dk> wrote:
> 
>> On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>> 
>> 
>> 
>> 
>>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>> 
>>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>> Just applied the patch and it's perfect.
>>>>> 
>>>>> Please see below:
>>>>> 
>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config /root/fio/iotest
>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>> ioengine=libaio, iodepth=4
>>>>> <s1> ...
>>>>> <s1> Starting 64 threads
>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>>>> iops]
>>>>> [eta 00m:34s]
>>>> 
>>>> Great, at least that took care of that issue. As to missing output from
>>>> one client, I've seen that here before, I will look into that. It's a
>>>> separate issue.
>>> 
>>> For the above one, s2 never started since it could not find the config file you gave it. Have you seen missing final output for cases where the jobs did all start? This particular one does not look valid.
>>> 
>>> --
>>> Jens Axboe
>> 
>> 
>> Yes. I did.
>> 
>> I ran using both servers but the output was showing the latest client - s2
> 
> Odd. Can you reproduce and send the output of such a run?
> 
> -- 
> Jens Axboe
> 

Yes, tomorrow morning I will send it over.

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/12/14, 8:27 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>>
>>
>>
>>
>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>> Just applied the patch and it's perfect.
>>>>>
>>>>> Please see below:
>>>>>
>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config
>>>>>/root/fio/iotest
>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>> ioengine=libaio, iodepth=4
>>>>> <s1> ...
>>>>> <s1> Starting 64 threads
>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>>>> iops]
>>>>> [eta 00m:34s]
>>>>
>>>> Great, at least that took care of that issue. As to missing output
>>>>from
>>>> one client, I've seen that here before, I will look into that. It's a
>>>> separate issue.
>>>
>>> For the above one, s2 never started since it could not find the config
>>>file you gave it. Have you seen missing final output for cases where
>>>the jobs did all start? This particular one does not look valid.
>>>
>>> --
>>> Jens Axboe
>>>
>>
>>
>> Yes. I did.
>>
>> I ran using both servers but the output was showing the latest client -
>>s2
>
>Odd. Can you reproduce and send the output of such a run?
>
>-- 
>Jens Axboe


Hi Jens,

This is neto from Brazil

How are you?

I believe the issue could be formatting....

Also, when running from both clients, seem to me unified_rw_reporting is
not working... I do not have the total for all clients....

Please see below:




Config file for s1:

[workload]
bs=32k
ioengine=libaio
iodepth=4
size=160g
numjobs=64
direct=1
runtime=60
file_service_type=random
filename=/n1_11/f1:/n1_11/f2:/n1_11/f3:/n1_11/f4:/n1_11/f5:/n1_11/f6:/n1_11
/f7:/n1_11/f8:/n1_11/f9:/n1_11/f10:/n1_11/f11:/n1_11/f12:/n1_11/f13:/n1_11/
f14:/n1_11/f15:/n1_11/f16
rw=write
thread
unified_rw_reporting=1
group_reporting=1




Config file for s2:

[workload]
bs=32k
ioengine=libaio
iodepth=4
size=160g
numjobs=64
direct=1
runtime=60
file_service_type=random
filename=/n1_21/g1:/n1_21/g2:/n1_21/g3:/n1_21/g4:/n1_21/g5:/n1_21/g6:/n1_21
/g7:/n1_21/g8:/n1_21/g9:/n1_21/g10:/n1_21/g11:/n1_21/g12:/n1_21/g13:/n1_21/
g14:/n1_21/g15:/n1_21/g16
rw=write
thread
unified_rw_reporting=1
group_reporting=1



Client: --output (did not work)

./fio --client 10.61.109.151 --remote-config /root/fiop/iotest --client
10.61.109.152 --remote-config /root/fiop/iotest > multiple-clients



Nossa Senhora:fiop neto$ cat multiple-clients
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.13-59-gaa7bc, flags=1
<s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
iodepth=4
ioengine=libaio, iodepth=4
<s2> ...
<s1> ...
<s2> Starting 64 threads
<s1> Starting 64 threads
<s2>  128 (f=2048): [W(64)] [100.0% done] [1481M/0K/0K /s] [46.3K/0/0
iops] [eta 00m:00s]
workload: (groupid=0, jobs=64): err= 0: pid=13568: Mon Oct 13 09:26:32 2014
  mixed: io=37370MB, bw=637546KB/s, iops=19923, runt= 60022msec
    slat (usec): min=9, max=811, avg=20.85, stdev= 9.34
    clat (usec): min=2, max=585971, avg=12759.21, stdev=16114.41
     lat (usec): min=303, max=585986, avg=12780.31, stdev=16114.37
    clat percentiles (usec):
     |  1th=[  572],  5th=[  852], 10th=[ 1096], 20th=[ 2096], 30th=[
6304],
     | 40th=[ 7136], 50th=[ 8160], 60th=[10816], 70th=[14016],
80th=[20096],
     | 90th=[29568], 95th=[36096], 99th=[50432], 100th=[71168],
100th=[218112],
     | 100th=[248832], 100th=[317440]
    bw (KB  /s): min=  387, max=58816, per=1.57%, avg=10013.38,
stdev=3302.95
    lat (usec) : 4=0.01%, 250=0.01%, 500=0.35%, 750=2.88%, 1000=4.98%
    lat (msec) : 2=11.36%, 4=5.94%, 10=32.81%, 20=21.54%, 50=19.10%
    lat (msec) : 100=0.66%, 250=0.32%, 500=0.05%, 750=0.01%
  cpu          : usr=0.41%, sys=0.61%, ctx=1322346, majf=0, minf=175
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1195837/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=37370MB, aggrb=637545KB/s, minb=637545KB/s, maxb=637545KB/s,
mint=60022msec, maxt=60022msec
<s1> 
workload: (groupid=0, jobs=64): err= 0: pid=23378: Mon Oct 13 09:26:32 2014
  mixed: io=39328MB, bw=670996KB/s, iops=20968, runt= 60018msec
    slat (usec): min=9, max=865, avg=19.48, stdev= 8.78
    clat (usec): min=3, max=834759, avg=12122.18, stdev=16022.95
     lat (usec): min=288, max=834776, avg=12141.92, stdev=16022.99
    clat percentiles (usec):
     |  1th=[  588],  5th=[  860], 10th=[ 1112], 20th=[ 1848], 30th=[
5920],
     | 40th=[ 6816], 50th=[ 7648], 60th=[10048], 70th=[13248],
80th=[18816],
     | 90th=[28544], 95th=[34560], 99th=[49408], 100th=[67072],
100th=[214016],
     | 100th=[246784], 100th=[374784]
    bw (KB  /s): min=   62, max=53205, per=1.57%, avg=10546.28,
stdev=2892.35
    lat (usec) : 4=0.01%, 100=0.01%, 250=0.01%, 500=0.28%, 750=2.70%
    lat (usec) : 1000=5.02%
    lat (msec) : 2=13.21%, 4=6.35%, 10=32.46%, 20=21.33%, 50=17.70%
    lat (msec) : 100=0.60%, 250=0.30%, 500=0.05%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.39%, sys=0.61%, ctx=1347629, majf=0, minf=205
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=1258494/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=39328MB, aggrb=670995KB/s, minb=670995KB/s, maxb=670995KB/s,
mint=60018msec, maxt=60018msec




My suggestion for format ....

128 (f=2048): [W(64)] [100.0% done] [1481M/0K/0K /s] [46.3K/0/0 iops] [eta
00m:00s]


Client <s2>
workload: (groupid=0, jobs=64): err= 0: pid=13568: Mon Oct 13 09:26:32 2014



....


Client <s1>
workload: (groupid=0, jobs=64): err= 0: pid=23378: Mon Oct 13 09:26:32 2014




.....


Total Clients = 2

Aggregate Workload: xxx MB/s   yyyy IOPS  zzzz latency

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-13 07:37, Neto, Antonio Jose Rodrigues wrote:
>
>
> On 10/12/14, 8:27 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>> On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>>>
>>>
>>>
>>>
>>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>
>>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>>> Just applied the patch and it's perfect.
>>>>>>
>>>>>> Please see below:
>>>>>>
>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config
>>>>>> /root/fio/iotest
>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>>> ioengine=libaio, iodepth=4
>>>>>> <s1> ...
>>>>>> <s1> Starting 64 threads
>>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>>>>> iops]
>>>>>> [eta 00m:34s]
>>>>>
>>>>> Great, at least that took care of that issue. As to missing output
>>>>> from
>>>>> one client, I've seen that here before, I will look into that. It's a
>>>>> separate issue.
>>>>
>>>> For the above one, s2 never started since it could not find the config
>>>> file you gave it. Have you seen missing final output for cases where
>>>> the jobs did all start? This particular one does not look valid.
>>>>
>>>> --
>>>> Jens Axboe
>>>>
>>>
>>>
>>> Yes. I did.
>>>
>>> I ran using both servers but the output was showing the latest client -
>>> s2
>>
>> Odd. Can you reproduce and send the output of such a run?
>>
>> --
>> Jens Axboe
>
>
> Hi Jens,
>
> This is neto from Brazil
>
> How are you?
>
> I believe the issue could be formatting....
>
> Also, when running from both clients, seem to me unified_rw_reporting is
> not working... I do not have the total for all clients....

unified_rw_reporting groups reads, writes, and discards into the same 
reporting bucket. I'm assuming you mean that group_reporting doesn't 
work for multiple connections? That's the option that groups multiple 
jobs together for reporting. And yes, that's not supported right now for 
multiple connections. But it could be, it's not that different from the 
ETA which is grouped as it would be on a local run.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-13 08:37, Jens Axboe wrote:
> On 2014-10-13 07:37, Neto, Antonio Jose Rodrigues wrote:
>>
>>
>> On 10/12/14, 8:27 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>
>>> On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>>>> Just applied the patch and it's perfect.
>>>>>>>
>>>>>>> Please see below:
>>>>>>>
>>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>> --remote-config
>>>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config
>>>>>>> /root/fio/iotest
>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>>>> ioengine=libaio, iodepth=4
>>>>>>> <s1> ...
>>>>>>> <s1> Starting 64 threads
>>>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s] [27.6K/0/0
>>>>>>> iops]
>>>>>>> [eta 00m:34s]
>>>>>>
>>>>>> Great, at least that took care of that issue. As to missing output
>>>>>> from
>>>>>> one client, I've seen that here before, I will look into that. It's a
>>>>>> separate issue.
>>>>>
>>>>> For the above one, s2 never started since it could not find the config
>>>>> file you gave it. Have you seen missing final output for cases where
>>>>> the jobs did all start? This particular one does not look valid.
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>>
>>>>
>>>>
>>>> Yes. I did.
>>>>
>>>> I ran using both servers but the output was showing the latest client -
>>>> s2
>>>
>>> Odd. Can you reproduce and send the output of such a run?
>>>
>>> --
>>> Jens Axboe
>>
>>
>> Hi Jens,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> I believe the issue could be formatting....
>>
>> Also, when running from both clients, seem to me unified_rw_reporting is
>> not working... I do not have the total for all clients....
>
> unified_rw_reporting groups reads, writes, and discards into the same
> reporting bucket. I'm assuming you mean that group_reporting doesn't
> work for multiple connections? That's the option that groups multiple
> jobs together for reporting. And yes, that's not supported right now for
> multiple connections. But it could be, it's not that different from the
> ETA which is grouped as it would be on a local run.

Try newest -git. It now outputs an "All clients" summed section, if you 
have more than 1 client.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/13/14, 12:10 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-13 08:37, Jens Axboe wrote:
>> On 2014-10-13 07:37, Neto, Antonio Jose Rodrigues wrote:
>>>
>>>
>>> On 10/12/14, 8:27 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>
>>>> On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>
>>>>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>> Just applied the patch and it's perfect.
>>>>>>>>
>>>>>>>> Please see below:
>>>>>>>>
>>>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>>> --remote-config
>>>>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config
>>>>>>>> /root/fio/iotest
>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>>>>> ioengine=libaio, iodepth=4
>>>>>>>> <s1> ...
>>>>>>>> <s1> Starting 64 threads
>>>>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s]
>>>>>>>>[27.6K/0/0
>>>>>>>> iops]
>>>>>>>> [eta 00m:34s]
>>>>>>>
>>>>>>> Great, at least that took care of that issue. As to missing output
>>>>>>> from
>>>>>>> one client, I've seen that here before, I will look into that.
>>>>>>>It's a
>>>>>>> separate issue.
>>>>>>
>>>>>> For the above one, s2 never started since it could not find the
>>>>>>config
>>>>>> file you gave it. Have you seen missing final output for cases where
>>>>>> the jobs did all start? This particular one does not look valid.
>>>>>>
>>>>>> --
>>>>>> Jens Axboe
>>>>>>
>>>>>
>>>>>
>>>>> Yes. I did.
>>>>>
>>>>> I ran using both servers but the output was showing the latest
>>>>>client -
>>>>> s2
>>>>
>>>> Odd. Can you reproduce and send the output of such a run?
>>>>
>>>> --
>>>> Jens Axboe
>>>
>>>
>>> Hi Jens,
>>>
>>> This is neto from Brazil
>>>
>>> How are you?
>>>
>>> I believe the issue could be formatting....
>>>
>>> Also, when running from both clients, seem to me unified_rw_reporting
>>>is
>>> not working... I do not have the total for all clients....
>>
>> unified_rw_reporting groups reads, writes, and discards into the same
>> reporting bucket. I'm assuming you mean that group_reporting doesn't
>> work for multiple connections? That's the option that groups multiple
>> jobs together for reporting. And yes, that's not supported right now for
>> multiple connections. But it could be, it's not that different from the
>> ETA which is grouped as it would be on a local run.
>
>Try newest -git. It now outputs an "All clients" summed section, if you
>have more than 1 client.
>
>-- 
>Jens Axboe

Hi Jens,

This is neto from Brazil

How are you?

The All Clients session is very nice on the report, but ... Please look
below (the progress has been splitting in two sessions) why?


Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
/root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-64-ga89d,
flags=1
hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-64-ga89d,
flags=1
<s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
iodepth=4
ioengine=libaio, iodepth=4
<s2> ...
<s1> ...
<s2> Starting <s1> Starting 64 threads
64 threads
>>>>>>>>>>>>>>>>>>      <s2>  128 (f=2048): [W(64)] [66.7% done]
>>>>>>>>>>>>>>>>>>[1265M/0K/0K /s] [39.6K/0/0 iops] [eta 00m:30s]
workload: (groupid=0, jobs=64): err= 0: pid=17874: Mon Oct 13 14:27:31 2014
  mixed: io=17775MB, bw=606116KB/s, iops=18941, runt= 30030msec
    slat (usec): min=10, max=700, avg=19.76, stdev= 8.97
    clat (usec): min=2, max=368477, avg=13382.30, stdev=17037.71
     lat (usec): min=303, max=368498, avg=13402.32, stdev=17037.69
    clat percentiles (usec):
     |  1th=[  532],  5th=[  780], 10th=[  996], 20th=[ 1992], 30th=[
6432],
     | 40th=[ 7392], 50th=[ 8384], 60th=[10816], 70th=[14400],
80th=[21120],
     | 90th=[31360], 95th=[37120], 99th=[58112], 100th=[134144],
100th=[197632],
     | 100th=[218112], 100th=[296960]
    bw (KB  /s): min=  479, max=71040, per=1.57%, avg=9508.74,
stdev=3605.67
    lat (usec) : 4=0.01%, 250=0.01%, 500=0.59%, 750=3.93%, 1000=5.57%
    lat (msec) : 2=9.98%, 4=4.78%, 10=33.23%, 20=20.74%, 50=19.80%
    lat (msec) : 100=0.75%, 250=0.61%, 500=0.02%
  cpu          : usr=0.37%, sys=0.57%, ctx=624447, majf=0, minf=164
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=568802/w=0/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  MIXED: io=17775MB, aggrb=606116KB/s, minb=606116KB/s, maxb=606116KB/s,
mint=30030msec, maxt=30030msec
>>>>>>>>>>>>>>>>>>>>>>    Jobs: 64 (f=1024): [W(64)] [90.0% done] [879.7M/0K/0K
/s] [27.5K/0/0 iops] [eta 00m:06s]

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-13 12:29, Neto, Antonio Jose Rodrigues wrote:
>
>
> On 10/13/14, 12:10 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>> On 2014-10-13 08:37, Jens Axboe wrote:
>>> On 2014-10-13 07:37, Neto, Antonio Jose Rodrigues wrote:
>>>>
>>>>
>>>> On 10/12/14, 8:27 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>>>>
>>>>> On 2014-10-12 14:28, Neto, Antonio Jose Rodrigues wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 12, 2014, at 3:26 PM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>>
>>>>>>>> On 2014-10-12 13:12, Jens Axboe wrote:
>>>>>>>>> On 2014-10-12 09:26, Neto, Antonio Jose Rodrigues wrote:
>>>>>>>>> Just applied the patch and it's perfect.
>>>>>>>>>
>>>>>>>>> Please see below:
>>>>>>>>>
>>>>>>>>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151
>>>>>>>>> --remote-config
>>>>>>>>> /root/fiop/iotest --client 10.61.109.152 --remote-config
>>>>>>>>> /root/fio/iotest
>>>>>>>>> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>>>>>>> fio=fio-2.1.13-59-gaa7bc, flags=1
>>>>>>>>> <s2> fio: unable to open '/root/fio/iotest' job file
>>>>>>>>> <s1> workload: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K,
>>>>>>>>> ioengine=libaio, iodepth=4
>>>>>>>>> <s1> ...
>>>>>>>>> <s1> Starting 64 threads
>>>>>>>>> Jobs: 64 (f=1024): [W(64)] [43.3% done] [882.5M/0K/0K /s]
>>>>>>>>> [27.6K/0/0
>>>>>>>>> iops]
>>>>>>>>> [eta 00m:34s]
>>>>>>>>
>>>>>>>> Great, at least that took care of that issue. As to missing output
>>>>>>>> from
>>>>>>>> one client, I've seen that here before, I will look into that.
>>>>>>>> It's a
>>>>>>>> separate issue.
>>>>>>>
>>>>>>> For the above one, s2 never started since it could not find the
>>>>>>> config
>>>>>>> file you gave it. Have you seen missing final output for cases where
>>>>>>> the jobs did all start? This particular one does not look valid.
>>>>>>>
>>>>>>> --
>>>>>>> Jens Axboe
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes. I did.
>>>>>>
>>>>>> I ran using both servers but the output was showing the latest
>>>>>> client -
>>>>>> s2
>>>>>
>>>>> Odd. Can you reproduce and send the output of such a run?
>>>>>
>>>>> --
>>>>> Jens Axboe
>>>>
>>>>
>>>> Hi Jens,
>>>>
>>>> This is neto from Brazil
>>>>
>>>> How are you?
>>>>
>>>> I believe the issue could be formatting....
>>>>
>>>> Also, when running from both clients, seem to me unified_rw_reporting
>>>> is
>>>> not working... I do not have the total for all clients....
>>>
>>> unified_rw_reporting groups reads, writes, and discards into the same
>>> reporting bucket. I'm assuming you mean that group_reporting doesn't
>>> work for multiple connections? That's the option that groups multiple
>>> jobs together for reporting. And yes, that's not supported right now for
>>> multiple connections. But it could be, it's not that different from the
>>> ETA which is grouped as it would be on a local run.
>>
>> Try newest -git. It now outputs an "All clients" summed section, if you
>> have more than 1 client.
>>
>> --
>> Jens Axboe
>
> Hi Jens,
>
> This is neto from Brazil
>
> How are you?
>
> The All Clients session is very nice on the report, but ... Please look
> below (the progress has been splitting in two sessions) why?
>
>
> Nossa Senhora:fio neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fio/write --client 10.61.109.152 --remote-config /root/fio/write
> hostname=s2, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-64-ga89d,
> flags=1
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-64-ga89d,
> flags=1
> <s2> workload: (g=0): rw=write, <s1> workload: (g=0): rw=write,
> bs=32K-32K/32K-32K/32K-32K, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio,
> iodepth=4
> ioengine=libaio, iodepth=4
> <s2> ...
> <s1> ...
> <s2> Starting <s1> Starting 64 threads
> 64 threads
>>>>>>>>>>>>>>>>>>>       <s2>  128 (f=2048): [W(64)] [66.7% done]
>>>>>>>>>>>>>>>>>>> [1265M/0K/0K /s] [39.6K/0/0 iops] [eta 00m:30s]
> workload: (groupid=0, jobs=64): err= 0: pid=17874: Mon Oct 13 14:27:31 2014
>    mixed: io=17775MB, bw=606116KB/s, iops=18941, runt= 30030msec
>      slat (usec): min=10, max=700, avg=19.76, stdev= 8.97
>      clat (usec): min=2, max=368477, avg=13382.30, stdev=17037.71
>       lat (usec): min=303, max=368498, avg=13402.32, stdev=17037.69
>      clat percentiles (usec):
>       |  1th=[  532],  5th=[  780], 10th=[  996], 20th=[ 1992], 30th=[
> 6432],
>       | 40th=[ 7392], 50th=[ 8384], 60th=[10816], 70th=[14400],
> 80th=[21120],
>       | 90th=[31360], 95th=[37120], 99th=[58112], 100th=[134144],
> 100th=[197632],
>       | 100th=[218112], 100th=[296960]
>      bw (KB  /s): min=  479, max=71040, per=1.57%, avg=9508.74,
> stdev=3605.67
>      lat (usec) : 4=0.01%, 250=0.01%, 500=0.59%, 750=3.93%, 1000=5.57%
>      lat (msec) : 2=9.98%, 4=4.78%, 10=33.23%, 20=20.74%, 50=19.80%
>      lat (msec) : 100=0.75%, 250=0.61%, 500=0.02%
>    cpu          : usr=0.37%, sys=0.57%, ctx=624447, majf=0, minf=164
>    IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>> =64=0.0%
>       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> =64=0.0%
>       issued    : total=r=568802/w=0/d=0, short=r=0/w=0/d=0,
> drop=r=0/w=0/d=0
>       latency   : target=0, window=0, percentile=100.00%, depth=4
>
> Run status group 0 (all jobs):
>    MIXED: io=17775MB, aggrb=606116KB/s, minb=606116KB/s, maxb=606116KB/s,
> mint=30030msec, maxt=30030msec
>>>>>>>>>>>>>>>>>>>>>>>     Jobs: 64 (f=1024): [W(64)] [90.0% done] [879.7M/0K/0K
> /s] [27.5K/0/0 iops] [eta 00m:06s]

Should probably halt the ETA for the final output, it's just 
intermingled atm. I'll update that.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-07 19:07, Neto, Antonio Jose Rodrigues wrote:
> I found what is happening
>
> If I use a non absolute path it works: the remote config should be on the
> same directory with fio
>
>
> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
> conf3
>
>
>
> If I specify an absolute path, it does not work
>
> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fio.patch/conf3
> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64, fio=fio-2.1.13-31-g15e3,
> flags=1
>
>
> Do you think we could fix it to make sure we can use --remote-config with
> an absolute path?

That's very odd, there should be nothing special about it. If I do this:

axboe@lenny:/home/axboe/git/fio $ stat /root/x.fio
stat: cannot stat �/root/x.fio�: Permission denied
$ ./fio --client=localhost --remote-config /root/x.fio

and run

axboe@lenny:/home/axboe/git/fio $ sudo strace -o x -f ./fio --server
fio: server listening on 0.0.0.0,8765

and then look at the x strace output file, it all looks fine:

28632 recvfrom(4, "\v\0\1\0/root/x.fio", 15, MSG_WAITALL, NULL, NULL) = 
15
28632 open("/root/x.fio", O_RDONLY)     = 5

which is all find and dandy. Fio just passes the string of the filename, 
and the server just does an fopen on that passed in file. So please run 
the same experiment on your end with an strace of the server and the 
arguments you used on the client, I'd like to see that.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-07 19:53, Jens Axboe wrote:
> On 2014-10-07 19:07, Neto, Antonio Jose Rodrigues wrote:
>> I found what is happening
>>
>> If I use a non absolute path it works: the remote config should be on the
>> same directory with fio
>>
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>> --remote-config
>> conf3
>>
>>
>>
>> If I specify an absolute path, it does not work
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>> --remote-config
>> /root/fio.patch/conf3
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.1.13-31-g15e3,
>> flags=1
>>
>>
>> Do you think we could fix it to make sure we can use --remote-config with
>> an absolute path?
>
> That's very odd, there should be nothing special about it. If I do this:
>
> axboe@lenny:/home/axboe/git/fio $ stat /root/x.fio
> stat: cannot stat �/root/x.fio�: Permission denied
> $ ./fio --client=localhost --remote-config /root/x.fio
>
> and run
>
> axboe@lenny:/home/axboe/git/fio $ sudo strace -o x -f ./fio --server
> fio: server listening on 0.0.0.0,8765
>
> and then look at the x strace output file, it all looks fine:
>
> 28632 recvfrom(4, "\v\0\1\0/root/x.fio", 15, MSG_WAITALL, NULL, NULL) = 15
> 28632 open("/root/x.fio", O_RDONLY)     = 5
>
> which is all find and dandy. Fio just passes the string of the filename,
> and the server just does an fopen on that passed in file. So please run
> the same experiment on your end with an strace of the server and the
> arguments you used on the client, I'd like to see that.

Actually, you should be able to just run the server with --debug=net 
added, and it'll print what it will attempt to open.

-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

On 10/7/14, 9:53 PM, "Jens Axboe" <axboe@kernel.dk> wrote:

>On 2014-10-07 19:07, Neto, Antonio Jose Rodrigues wrote:
>> I found what is happening
>>
>> If I use a non absolute path it works: the remote config should be on
>>the
>> same directory with fio
>>
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>--remote-config
>> conf3
>>
>>
>>
>> If I specify an absolute path, it does not work
>>
>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>--remote-config
>> /root/fio.patch/conf3
>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>fio=fio-2.1.13-31-g15e3,
>> flags=1
>>
>>
>> Do you think we could fix it to make sure we can use --remote-config
>>with
>> an absolute path?
>
>That's very odd, there should be nothing special about it. If I do this:
>
>axboe@lenny:/home/axboe/git/fio $ stat /root/x.fio
>stat: cannot stat Œ/root/x.fio¹: Permission denied
>$ ./fio --client=localhost --remote-config /root/x.fio
>
>and run
>
>axboe@lenny:/home/axboe/git/fio $ sudo strace -o x -f ./fio --server
>fio: server listening on 0.0.0.0,8765
>
>and then look at the x strace output file, it all looks fine:
>
>28632 recvfrom(4, "\v\0\1\0/root/x.fio", 15, MSG_WAITALL, NULL, NULL) =
>15
>28632 open("/root/x.fio", O_RDONLY)     = 5
>
>which is all find and dandy. Fio just passes the string of the filename,
>and the server just does an fopen on that passed in file. So please run
>the same experiment on your end with an strace of the server and the
>arguments you used on the client, I'd like to see that.
>
>-- 
>Jens Axboe
>

Hi Jens,

This is neto from Brazil

How are you?

It's very strange....

I have applied the patch again... Make clean, make and it's working now :-(

On linux it's working

But I applied the patch on my mac, but it's not working

Nossa Senhora:fiop neto$ patch < net-remote-config.patch
(Stripping trailing CRs from patch.)
patching file client.c
(Stripping trailing CRs from patch.)
patching file client.h
(Stripping trailing CRs from patch.)
patching file init.c
(Stripping trailing CRs from patch.)
patching file server.c
(Stripping trailing CRs from patch.)
patching file server.h



Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
/root/fiop/model
Nossa Senhora:fiop neto$



Could you please make the changes to github and I will download form
there. Something is not right here with the patch.

Thanks

neto

Re: FIO - Client and Server - Suggestion

[Jens Axboe]

On 2014-10-07 20:54, Neto, Antonio Jose Rodrigues wrote:
>
>
> On 10/7/14, 9:53 PM, "Jens Axboe" <axboe@kernel.dk> wrote:
>
>> On 2014-10-07 19:07, Neto, Antonio Jose Rodrigues wrote:
>>> I found what is happening
>>>
>>> If I use a non absolute path it works: the remote config should be on
>>> the
>>> same directory with fio
>>>
>>>
>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>> --remote-config
>>> conf3
>>>
>>>
>>>
>>> If I specify an absolute path, it does not work
>>>
>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>> --remote-config
>>> /root/fio.patch/conf3
>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.1.13-31-g15e3,
>>> flags=1
>>>
>>>
>>> Do you think we could fix it to make sure we can use --remote-config
>>> with
>>> an absolute path?
>>
>> That's very odd, there should be nothing special about it. If I do this:
>>
>> axboe@lenny:/home/axboe/git/fio $ stat /root/x.fio
>> stat: cannot stat �/root/x.fio�: Permission denied
>> $ ./fio --client=localhost --remote-config /root/x.fio
>>
>> and run
>>
>> axboe@lenny:/home/axboe/git/fio $ sudo strace -o x -f ./fio --server
>> fio: server listening on 0.0.0.0,8765
>>
>> and then look at the x strace output file, it all looks fine:
>>
>> 28632 recvfrom(4, "\v\0\1\0/root/x.fio", 15, MSG_WAITALL, NULL, NULL) =
>> 15
>> 28632 open("/root/x.fio", O_RDONLY)     = 5
>>
>> which is all find and dandy. Fio just passes the string of the filename,
>> and the server just does an fopen on that passed in file. So please run
>> the same experiment on your end with an strace of the server and the
>> arguments you used on the client, I'd like to see that.
>>
>> --
>> Jens Axboe
>>
>
> Hi Jens,
>
> This is neto from Brazil
>
> How are you?
>
> It's very strange....
>
> I have applied the patch again... Make clean, make and it's working now :-(
>
> On linux it's working
>
> But I applied the patch on my mac, but it's not working
>
> Nossa Senhora:fiop neto$ patch < net-remote-config.patch
> (Stripping trailing CRs from patch.)
> patching file client.c
> (Stripping trailing CRs from patch.)
> patching file client.h
> (Stripping trailing CRs from patch.)
> patching file init.c
> (Stripping trailing CRs from patch.)
> patching file server.c
> (Stripping trailing CRs from patch.)
> patching file server.h
>
>
>
> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
> /root/fiop/model
> Nossa Senhora:fiop neto$
>
>
>
> Could you please make the changes to github and I will download form
> there. Something is not right here with the patch.

Always ensure that you make clean, might be the same on the mac. In any 
case, it's committed, but it a separate branch. So when pulling, you 
need to checkout the remote-config branch. Once it's fully done, I will 
pull it into the master branch.


-- 
Jens Axboe

Re: FIO - Client and Server - Suggestion

[Neto, Antonio Jose Rodrigues]

>>>> I found what is happening
>>>>
>>>> If I use a non absolute path it works: the remote config should be on
>>>> the
>>>> same directory with fio
>>>>
>>>>
>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>> --remote-config
>>>> conf3
>>>>
>>>>
>>>>
>>>> If I specify an absolute path, it does not work
>>>>
>>>> Nossa Senhora:fio.patch neto$ ./fio --client 10.61.109.151
>>>> --remote-config
>>>> /root/fio.patch/conf3
>>>> hostname=s1, be=0, 64-bit, os=Linux, arch=x86-64,
>>>> fio=fio-2.1.13-31-g15e3,
>>>> flags=1
>>>>
>>>>
>>>> Do you think we could fix it to make sure we can use --remote-config
>>>> with
>>>> an absolute path?
>>>
>>> That's very odd, there should be nothing special about it. If I do
>>>this:
>>>
>>> axboe@lenny:/home/axboe/git/fio $ stat /root/x.fio
>>> stat: cannot stat Œ/root/x.fio¹: Permission denied
>>> $ ./fio --client=localhost --remote-config /root/x.fio
>>>
>>> and run
>>>
>>> axboe@lenny:/home/axboe/git/fio $ sudo strace -o x -f ./fio --server
>>> fio: server listening on 0.0.0.0,8765
>>>
>>> and then look at the x strace output file, it all looks fine:
>>>
>>> 28632 recvfrom(4, "\v\0\1\0/root/x.fio", 15, MSG_WAITALL, NULL, NULL) =
>>> 15
>>> 28632 open("/root/x.fio", O_RDONLY)     = 5
>>>
>>> which is all find and dandy. Fio just passes the string of the
>>>filename,
>>> and the server just does an fopen on that passed in file. So please run
>>> the same experiment on your end with an strace of the server and the
>>> arguments you used on the client, I'd like to see that.
>>>
>>> --
>>> Jens Axboe
>>>
>>
>> Hi Jens,
>>
>> This is neto from Brazil
>>
>> How are you?
>>
>> It's very strange....
>>
>> I have applied the patch again... Make clean, make and it's working now
>>:-(
>>
>> On linux it's working
>>
>> But I applied the patch on my mac, but it's not working
>>
>> Nossa Senhora:fiop neto$ patch < net-remote-config.patch
>> (Stripping trailing CRs from patch.)
>> patching file client.c
>> (Stripping trailing CRs from patch.)
>> patching file client.h
>> (Stripping trailing CRs from patch.)
>> patching file init.c
>> (Stripping trailing CRs from patch.)
>> patching file server.c
>> (Stripping trailing CRs from patch.)
>> patching file server.h
>>
>>
>>
>> Nossa Senhora:fiop neto$ ./fio --client 10.61.109.151 --remote-config
>> /root/fiop/model
>> Nossa Senhora:fiop neto$
>>
>>
>>
>> Could you please make the changes to github and I will download form
>> there. Something is not right here with the patch.
>
>Always ensure that you make clean, might be the same on the mac. In any
>case, it's committed, but it a separate branch. So when pulling, you
>need to checkout the remote-config branch. Once it's fully done, I will
>pull it into the master branch.
>
>
>-- 
>Jens Axboe
>


Hi Jens,

This is neto from Brazil

How are you?

Git clone the remote-config branch.... But get an error when trying to
compile

Nossa Senhora:fio.patch neto$ make V=1
gcc  -std=gnu99 -Wwrite-strings -Wall -Wdeclaration-after-statement -O3 -g
-ffast-math  -D_GNU_SOURCE -include config-host.h -DBITS_PER_LONG=64
-DFIO_VERSION='"fio-2.1.13-41-gc05da"' -o fio gettime.o ioengines.o init.o
stat.o log.o time.o filesetup.o eta.o verify.o memory.o io_u.o parse.o
mutex.o options.o lib/rbtree.o smalloc.o filehash.o profile.o debug.o
lib/rand.o lib/num2str.o lib/ieee754.o crc/crc16.o crc/crc32.o
crc/crc32c-intel.o crc/crc32c.o crc/crc64.o crc/crc7.o crc/fnv.o crc/md5.o
crc/murmur3.o crc/sha1.o crc/sha256.o crc/sha512.o crc/test.o crc/xxhash.o
engines/cpu.o engines/mmap.o engines/sync.o engines/null.o engines/net.o
memalign.o server.o client.o iolog.o backend.o libfio.o flow.o cconv.o
lib/prio_tree.o json.o lib/zipf.o lib/axmap.o lib/lfsr.o gettime-thread.o
helpers.o lib/flist_sort.o lib/hweight.o lib/getrusage.o idletime.o
td_error.o profiles/tiobench.o profiles/act.o io_u_queue.o filelock.o
lib/tp.o lib/bloom.o engines/posixaio.o fio.o lex.yy.o y.tab.o -ll -lz
-lm  -lpthread -ldl
ld: can't open output file for writing: fio, errno=21 for architecture
x86_64
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make: *** [fio] Error 1






>