[NFS] Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[NFS] Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Brandon Simmons]

I am using tiobench to test performance of an NFS mounted volume, and
notice that Sequential Reads are much slower than Random Reads. This
isn't the behavior when I run the same test on the disk mounted
locally.

For random reads I'm getting:

    50 MB/s  over NFS

v.s

    384 MB/s  when mounted locally

This is in comparison to the benchmark for _Random Reads_, in which I get:

    288 MB/s both over NFS _and_ when directly mounted

The other benchmarks seem to be in line with what I would expect, but
I'm fairly new to NFS. Why would sequential reads over NFS be sooo
much slower than random reads over NFS?

I am exporting the volume on the server like this

/export *.internal(no_subtree_check,rw,no_root_squash)

and mounting with this:

mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs

Additionally I am doing all this in amazon EC2, exporting an EBS
volume with the XFS file system (redundant, I know).

I have tried using jumbo frames and various other mount options, but
none seem to have much effect.

Thanks for any clues.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Brandon Simmons]

I am using tiobench to test performance of an NFS mounted volume, and
notice that Sequential Reads are much slower than Random Reads. This
isn't the behavior when I run the same test on the disk mounted
locally.

For random reads I'm getting:

   50 MB/s  over NFS

v.s

   384 MB/s  when mounted locally

This is in comparison to the benchmark for _Random Reads_, in which I get:

   288 MB/s both over NFS _and_ when directly mounted

The other benchmarks seem to be in line with what I would expect, but
I'm fairly new to NFS. Why would sequential reads over NFS be sooo
much slower than random reads over NFS?

I am exporting the volume on the server like this

/export *.internal(no_subtree_check,rw,no_root_squash)

and mounting with this:

mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs

Additionally I am doing all this in amazon EC2, exporting an EBS
volume with the XFS file system (redundant, I know).

I have tried using jumbo frames and various other mount options, but
none seem to have much effect.

Thanks for any clues.

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Ric Wheeler]

On 03/12/2010 01:22 PM, Brandon Simmons wrote:
> I am using tiobench to test performance of an NFS mounted volume, and
> notice that Sequential Reads are much slower than Random Reads. This
> isn't the behavior when I run the same test on the disk mounted
> locally.
>
> For random reads I'm getting:
>
>     50 MB/s  over NFS
>
> v.s
>
>     384 MB/s  when mounted locally
>
> This is in comparison to the benchmark for _Random Reads_, in which I get:
>
>     288 MB/s both over NFS _and_ when directly mounted
>
> The other benchmarks seem to be in line with what I would expect, but
> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
> much slower than random reads over NFS?
>
> I am exporting the volume on the server like this
>
> /export *.internal(no_subtree_check,rw,no_root_squash)
>
> and mounting with this:
>
> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
>
> Additionally I am doing all this in amazon EC2, exporting an EBS
> volume with the XFS file system (redundant, I know).
>
> I have tried using jumbo frames and various other mount options, but
> none seem to have much effect.
>
> Thanks for any clues.
>    

Not sure what kind of network you are running the NFS test over so it is 
quite hard to figure out why your performance varies so wildly.

Normal NFS testing with a gigabit network between the client and server 
would be much closer to 50MB/sec than your 288MB/sec.

Can you try to reproduce this locally with known client and server hardware?

ric

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Trond Myklebust]

On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote: 
> I am using tiobench to test performance of an NFS mounted volume, and
> notice that Sequential Reads are much slower than Random Reads. This
> isn't the behavior when I run the same test on the disk mounted
> locally.
> 
> For random reads I'm getting:
> 
>    50 MB/s  over NFS
> 
> v.s
> 
>    384 MB/s  when mounted locally
> 
> This is in comparison to the benchmark for _Random Reads_, in which I get:
> 
>    288 MB/s both over NFS _and_ when directly mounted
> 
> The other benchmarks seem to be in line with what I would expect, but
> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
> much slower than random reads over NFS?

They're not usually. My guess is that this is an artifact of your test.
What is tiobench doing prior to the sequential read?

Trond

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Brandon Simmons]

On Fri, Mar 12, 2010 at 1:30 PM, Ric Wheeler <ricwheeler@gmail.com> wro=
te:
> On 03/12/2010 01:22 PM, Brandon Simmons wrote:
>>
>> I am using tiobench to test performance of an NFS mounted volume, an=
d
>> notice that Sequential Reads are much slower than Random Reads. This
>> isn't the behavior when I run the same test on the disk mounted
>> locally.
>>
>> For random reads I'm getting:
>>
>> =A0 =A050 MB/s =A0over NFS
>>
>> v.s
>>
>> =A0 =A0384 MB/s =A0when mounted locally
>>
>> This is in comparison to the benchmark for _Random Reads_, in which =
I get:
>>
>> =A0 =A0288 MB/s both over NFS _and_ when directly mounted
>>
>> The other benchmarks seem to be in line with what I would expect, bu=
t
>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>> much slower than random reads over NFS?
>>
>> I am exporting the volume on the server like this
>>
>> /export *.internal(no_subtree_check,rw,no_root_squash)
>>
>> and mounting with this:
>>
>> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/expor=
t /nfs
>>
>> Additionally I am doing all this in amazon EC2, exporting an EBS
>> volume with the XFS file system (redundant, I know).
>>
>> I have tried using jumbo frames and various other mount options, but
>> none seem to have much effect.
>>
>> Thanks for any clues.
>>
>
> Not sure what kind of network you are running the NFS test over so it=
 is
> quite hard to figure out why your performance varies so wildly.
>
> Normal NFS testing with a gigabit network between the client and serv=
er
> would be much closer to 50MB/sec than your 288MB/sec.
>
> Can you try to reproduce this locally with known client and server ha=
rdware?
>
> ric
>
>

I'm not sure. My servers are EC2 instances in Amazon's cloud computing
service.I am doing the test from an EBS which is a virtual disk
mounted locally on an instance and exported via NFS.

So I don't think I can do any relevant tests locally.

Thanks,
Brandon

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Ric Wheeler]

On 03/12/2010 02:09 PM, Brandon Simmons wrote:
> On Fri, Mar 12, 2010 at 1:30 PM, Ric Wheeler<ricwheeler@gmail.com>  wrote:
>    
>> On 03/12/2010 01:22 PM, Brandon Simmons wrote:
>>      
>>> I am using tiobench to test performance of an NFS mounted volume, and
>>> notice that Sequential Reads are much slower than Random Reads. This
>>> isn't the behavior when I run the same test on the disk mounted
>>> locally.
>>>
>>> For random reads I'm getting:
>>>
>>>     50 MB/s  over NFS
>>>
>>> v.s
>>>
>>>     384 MB/s  when mounted locally
>>>
>>> This is in comparison to the benchmark for _Random Reads_, in which I get:
>>>
>>>     288 MB/s both over NFS _and_ when directly mounted
>>>
>>> The other benchmarks seem to be in line with what I would expect, but
>>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>>> much slower than random reads over NFS?
>>>
>>> I am exporting the volume on the server like this
>>>
>>> /export *.internal(no_subtree_check,rw,no_root_squash)
>>>
>>> and mounting with this:
>>>
>>> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
>>>
>>> Additionally I am doing all this in amazon EC2, exporting an EBS
>>> volume with the XFS file system (redundant, I know).
>>>
>>> I have tried using jumbo frames and various other mount options, but
>>> none seem to have much effect.
>>>
>>> Thanks for any clues.
>>>
>>>        
>> Not sure what kind of network you are running the NFS test over so it is
>> quite hard to figure out why your performance varies so wildly.
>>
>> Normal NFS testing with a gigabit network between the client and server
>> would be much closer to 50MB/sec than your 288MB/sec.
>>
>> Can you try to reproduce this locally with known client and server hardware?
>>
>> ric
>>
>>
>>      
> I'm not sure. My servers are EC2 instances in Amazon's cloud computing
> service.I am doing the test from an EBS which is a virtual disk
> mounted locally on an instance and exported via NFS.
>
> So I don't think I can do any relevant tests locally.
>
> Thanks,
> Brandon
>    

The joys of working in the cloud :-)

One possible reason could be that one test is actually going to an NFS 
server that is remote, one might be going to one locally (not leaving 
the box). I think that you will have to escalate with the Amazon 
support/technical people to try and peek under the covers a bit.

ric

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Brandon Simmons]

On Fri, Mar 12, 2010 at 1:33 PM, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
> On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote:
>> I am using tiobench to test performance of an NFS mounted volume, an=
d
>> notice that Sequential Reads are much slower than Random Reads. This
>> isn't the behavior when I run the same test on the disk mounted
>> locally.
>>
>> For random reads I'm getting:
>>
>> =A0 =A050 MB/s =A0over NFS
>>
>> v.s
>>
>> =A0 =A0384 MB/s =A0when mounted locally
>>
>> This is in comparison to the benchmark for _Random Reads_, in which =
I get:
>>
>> =A0 =A0288 MB/s both over NFS _and_ when directly mounted
>>
>> The other benchmarks seem to be in line with what I would expect, bu=
t
>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>> much slower than random reads over NFS?
>
> They're not usually. My guess is that this is an artifact of your tes=
t.
> What is tiobench doing prior to the sequential read?
>
> Trond
>
>

I'm wondering if this is caused by caching. my benchmark does
Sequential Reads first, then it does Random Reads. Is it possible that
the random reads could be working on cached data?

That would mean my sequential reads aren't too slow, rather the random
reads are extraordinarily fast because of caching. Looking at running
some different benchmarks.

Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Wendy Cheng]

Brandon Simmons wrote:
> On Fri, Mar 12, 2010 at 1:33 PM, Trond Myklebust
> <trond.myklebust@fys.uio.no> wrote:
>   
>> On Fri, 2010-03-12 at 13:22 -0500, Brandon Simmons wrote:
>>     
>>> I am using tiobench to test performance of an NFS mounted volume, and
>>> notice that Sequential Reads are much slower than Random Reads. This
>>> isn't the behavior when I run the same test on the disk mounted
>>> locally.
>>>
>>> For random reads I'm getting:
>>>
>>>    50 MB/s  over NFS
>>>
>>> v.s
>>>
>>>    384 MB/s  when mounted locally
>>>
>>> This is in comparison to the benchmark for _Random Reads_, in which I get:
>>>
>>>    288 MB/s both over NFS _and_ when directly mounted
>>>
>>> The other benchmarks seem to be in line with what I would expect, but
>>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>>> much slower than random reads over NFS?
>>>       
>> They're not usually. My guess is that this is an artifact of your test.
>> What is tiobench doing prior to the sequential read?
>>
>> Trond
>>     
>
> I'm wondering if this is caused by caching. my benchmark does
> Sequential Reads first, then it does Random Reads. Is it possible that
> the random reads could be working on cached data?
>
> That would mean my sequential reads aren't too slow, rather the random
> reads are extraordinarily fast because of caching. Looking at running
> some different benchmarks.
>
>   

I think so too - since your throughput are way too good (when compared 
with other I/O-bound numbers). Using simple benchmark(s) such as 
tiobench can generate very misleading results if you don't know how to 
steer away from cache effects. For NFS workload, SPECsfs is a much 
better benchmark but it is cumbersome to run (and you have to pay for 
it, I think).

In general, however, I find sequential read performance hard to predict 
in a virtual environment. This is because the data can be (physically) 
scattered around. It takes non-trivial efforts for system software to 
get its pre-fetch logic right.

-- Wendy