Testing write master and multiple read clients on NFS share

Testing write master and multiple read clients on NFS share

[Michael Weinrich]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

Hi,

in my company we have some sort of database that operates on bit lists 
read directly from the files on the hard disk. While this is ok if you 
have an SSD in your server it gets tricky to keep performance up when 
using an NFS share.

While I was trying to assess different solutions I stumbled upon fio and 
was happy to see that it can also run in client-server-mode. So I was 
trying to emulate the following scenario:

- 1 node that takes care of updating the bit lists in the db files,
   nothing else
- 5 nodes that only read from those files and respond to queries, 
hopefully leveraging the local filesystem or NFS client cache

I read through the HOWTO and tried to pick the options that should 
simulate the scenario as best as possible. This is what I came up with:

[global]
ioengine=sync
iodepth=1
size=100%
io_size=4g
filesize=4g
blocksize=64k
direct=0
numjobs=4
directory=/mnt/nfs
filename_format=index.$filenum
fadvise_hint=0
lockfile=readwrite
randrepeat=1
nrfiles=8
openfiles=4
runtime=120
ramp_time=10
blocksize=64k
file_service_type=random:8
overwrite=1
fsync_on_close=1
random_distribution=zipf:0.6
norandommap
file_append=0
rate=1m

[reader]
rw=randread

[writer]
rw=randrw
rwmixwrite=80


Is this a viable approach or is this scenario not really testable with fio?


Thanks,
Michael


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4213 bytes --]

Re: Testing write master and multiple read clients on NFS share

[Sitsofe Wheeler]

On 14 May 2015 at 23:09, Michael Weinrich <micxer@micxer.de> wrote:
> Hi,
>
> in my company we have some sort of database that operates on bit lists read
> directly from the files on the hard disk. While this is ok if you have an
> SSD in your server it gets tricky to keep performance up when using an NFS
> share.
>
> While I was trying to assess different solutions I stumbled upon fio and was
> happy to see that it can also run in client-server-mode. So I was trying to
> emulate the following scenario:
>
> - 1 node that takes care of updating the bit lists in the db files,
>   nothing else
> - 5 nodes that only read from those files and respond to queries, hopefully
> leveraging the local filesystem or NFS client cache
>
> I read through the HOWTO and tried to pick the options that should simulate
> the scenario as best as possible. This is what I came up with:
>
> [global]
> ioengine=sync
> iodepth=1
> size=100%
> io_size=4g
> filesize=4g
> blocksize=64k
> direct=0
> numjobs=4
> directory=/mnt/nfs
> filename_format=index.$filenum
> fadvise_hint=0
> lockfile=readwrite
> randrepeat=1
> nrfiles=8
> openfiles=4
> runtime=120
> ramp_time=10
> blocksize=64k
> file_service_type=random:8
> overwrite=1
> fsync_on_close=1
> random_distribution=zipf:0.6
> norandommap
> file_append=0
> rate=1m
>
> [reader]
> rw=randread
>
> [writer]
> rw=randrw
> rwmixwrite=80
>
>
> Is this a viable approach or is this scenario not really testable with fio?

I could be wrong but won't this make 4 readers and 4 writers. If
that's not what you wanted perhaps numjobs could be moved to the
reader entry only?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Testing write master and multiple read clients on NFS share

[Michael Weinrich]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

Thanks for the hint. I also found another mistake in limiting the reads 
and writes where I only wanted to limit the writes and have the reads as 
fast as possible. But in general this is a viable approach to my problem?

Am 16.05.15 um 13:46 schrieb Sitsofe Wheeler:
> On 14 May 2015 at 23:09, Michael Weinrich <micxer@micxer.de> wrote:
>> Hi,
>>
>> in my company we have some sort of database that operates on bit lists read
>> directly from the files on the hard disk. While this is ok if you have an
>> SSD in your server it gets tricky to keep performance up when using an NFS
>> share.
>>
>> While I was trying to assess different solutions I stumbled upon fio and was
>> happy to see that it can also run in client-server-mode. So I was trying to
>> emulate the following scenario:
>>
>> - 1 node that takes care of updating the bit lists in the db files,
>>    nothing else
>> - 5 nodes that only read from those files and respond to queries, hopefully
>> leveraging the local filesystem or NFS client cache
>>
>> I read through the HOWTO and tried to pick the options that should simulate
>> the scenario as best as possible. This is what I came up with:
>>
>> [global]
>> ioengine=sync
>> iodepth=1
>> size=100%
>> io_size=4g
>> filesize=4g
>> blocksize=64k
>> direct=0
>> numjobs=4
>> directory=/mnt/nfs
>> filename_format=index.$filenum
>> fadvise_hint=0
>> lockfile=readwrite
>> randrepeat=1
>> nrfiles=8
>> openfiles=4
>> runtime=120
>> ramp_time=10
>> blocksize=64k
>> file_service_type=random:8
>> overwrite=1
>> fsync_on_close=1
>> random_distribution=zipf:0.6
>> norandommap
>> file_append=0
>> rate=1m
>>
>> [reader]
>> rw=randread
>>
>> [writer]
>> rw=randrw
>> rwmixwrite=80
>>
>>
>> Is this a viable approach or is this scenario not really testable with fio?
>
> I could be wrong but won't this make 4 readers and 4 writers. If
> that's not what you wanted perhaps numjobs could be moved to the
> reader entry only?
>


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4213 bytes --]

Re: Testing write master and multiple read clients on NFS share

[Sitsofe Wheeler]

On 17 May 2015 at 21:55, Michael Weinrich <micxer@micxer.de> wrote:
> Thanks for the hint. I also found another mistake in limiting the reads and
> writes where I only wanted to limit the writes and have the reads as fast as
> possible. But in general this is a viable approach to my problem?

I can't say for sure whether it models your workload (your example
made use of options I had to look up!) but it does look like an
interesting approach...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Testing write master and multiple read clients on NFS share

[Michael Weinrich]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

Yeah, well, I needed a process that writes to the same files the others 
are reading from to somehow simulate expiring local caches (and to see, 
if NFS handles that the right way). And yes, it took me quite some time 
to dig through most of the options to tailor the tests to my needs

But I'm also glad to hear, that it doesn't look completely wrong for a 
first try :-)

Am 20.05.2015 um 11:03 schrieb Sitsofe Wheeler:
> On 17 May 2015 at 21:55, Michael Weinrich <micxer@micxer.de> wrote:
>> Thanks for the hint. I also found another mistake in limiting the reads and
>> writes where I only wanted to limit the writes and have the reads as fast as
>> possible. But in general this is a viable approach to my problem?
> I can't say for sure whether it models your workload (your example
> made use of options I had to look up!) but it does look like an
> interesting approach...
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4213 bytes --]

Re: Testing write master and multiple read clients on NFS share

[danielabuggie .]

I'd add the time_based option in there.  May not be needed now but if
you start tweaking things it could come up later.

Daniel

On Wed, May 20, 2015 at 4:51 AM, Michael Weinrich <micxer@micxer.de> wrote:
> Yeah, well, I needed a process that writes to the same files the others are
> reading from to somehow simulate expiring local caches (and to see, if NFS
> handles that the right way). And yes, it took me quite some time to dig
> through most of the options to tailor the tests to my needs
>
> But I'm also glad to hear, that it doesn't look completely wrong for a first
> try :-)
>
>
> Am 20.05.2015 um 11:03 schrieb Sitsofe Wheeler:
>>
>> On 17 May 2015 at 21:55, Michael Weinrich <micxer@micxer.de> wrote:
>>>
>>> Thanks for the hint. I also found another mistake in limiting the reads
>>> and
>>> writes where I only wanted to limit the writes and have the reads as fast
>>> as
>>> possible. But in general this is a viable approach to my problem?
>>
>> I can't say for sure whether it models your workload (your example
>> made use of options I had to look up!) but it does look like an
>> interesting approach...
>>
>
>

Re: Testing write master and multiple read clients on NFS share

[Jens Axboe]

On 05/20/2015 05:51 AM, Michael Weinrich wrote:
> Yeah, well, I needed a process that writes to the same files the others
> are reading from to somehow simulate expiring local caches (and to see,
> if NFS handles that the right way). And yes, it took me quite some time
> to dig through most of the options to tailor the tests to my needs
>
> But I'm also glad to hear, that it doesn't look completely wrong for a
> first try :-)

I think it looks very reasonable. I'd like to include your job file in 
the examples/ directory, if that's OK with you.

In general, I'd love to have a richer repository of job files included 
(or available, at least). It's much easier for new users to find 
something roughly like what they want and then modify it, than it is to 
start from scratch.

-- 
Jens Axboe

Re: Testing write master and multiple read clients on NFS share

[Michael Weinrich]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

Well, as I wrote it was a first try but since it seems to have no 
obvious flaws and I was satisfied with the results, I'm happy to have 
provided a useful example. Please go ahead and add it if you like. 
However I would add the time_based option as Daniel Abuggie suggested. 
It made more sense.

I also like the idea to have a bunch of example files since the HOWTO is 
very detailed but also very heavy for a beginner to work through.

-- Michael

Am 24.05.15 um 03:15 schrieb Jens Axboe:
> On 05/20/2015 05:51 AM, Michael Weinrich wrote:
>> Yeah, well, I needed a process that writes to the same files the others
>> are reading from to somehow simulate expiring local caches (and to see,
>> if NFS handles that the right way). And yes, it took me quite some time
>> to dig through most of the options to tailor the tests to my needs
>>
>> But I'm also glad to hear, that it doesn't look completely wrong for a
>> first try :-)
>
> I think it looks very reasonable. I'd like to include your job file in
> the examples/ directory, if that's OK with you.
>
> In general, I'd love to have a richer repository of job files included
> (or available, at least). It's much easier for new users to find
> something roughly like what they want and then modify it, than it is to
> start from scratch.
>


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4213 bytes --]