Re: Very Slow Sequential Reads over NFS from an XFS disk in Amazon EC2

[Ric Wheeler <ricwheeler@gmail.com>]

On 03/12/2010 02:09 PM, Brandon Simmons wrote:
> On Fri, Mar 12, 2010 at 1:30 PM, Ric Wheeler<ricwheeler@gmail.com>  wrote:
>    
>> On 03/12/2010 01:22 PM, Brandon Simmons wrote:
>>      
>>> I am using tiobench to test performance of an NFS mounted volume, and
>>> notice that Sequential Reads are much slower than Random Reads. This
>>> isn't the behavior when I run the same test on the disk mounted
>>> locally.
>>>
>>> For random reads I'm getting:
>>>
>>>     50 MB/s  over NFS
>>>
>>> v.s
>>>
>>>     384 MB/s  when mounted locally
>>>
>>> This is in comparison to the benchmark for _Random Reads_, in which I get:
>>>
>>>     288 MB/s both over NFS _and_ when directly mounted
>>>
>>> The other benchmarks seem to be in line with what I would expect, but
>>> I'm fairly new to NFS. Why would sequential reads over NFS be sooo
>>> much slower than random reads over NFS?
>>>
>>> I am exporting the volume on the server like this
>>>
>>> /export *.internal(no_subtree_check,rw,no_root_squash)
>>>
>>> and mounting with this:
>>>
>>> mount -o hard,intr,async,noatime,nodiratime,noacl $NFS_SERVER:/export /nfs
>>>
>>> Additionally I am doing all this in amazon EC2, exporting an EBS
>>> volume with the XFS file system (redundant, I know).
>>>
>>> I have tried using jumbo frames and various other mount options, but
>>> none seem to have much effect.
>>>
>>> Thanks for any clues.
>>>
>>>        
>> Not sure what kind of network you are running the NFS test over so it is
>> quite hard to figure out why your performance varies so wildly.
>>
>> Normal NFS testing with a gigabit network between the client and server
>> would be much closer to 50MB/sec than your 288MB/sec.
>>
>> Can you try to reproduce this locally with known client and server hardware?
>>
>> ric
>>
>>
>>      
> I'm not sure. My servers are EC2 instances in Amazon's cloud computing
> service.I am doing the test from an EBS which is a virtual disk
> mounted locally on an instance and exported via NFS.
>
> So I don't think I can do any relevant tests locally.
>
> Thanks,
> Brandon
>    

The joys of working in the cloud :-)

One possible reason could be that one test is actually going to an NFS 
server that is remote, one might be going to one locally (not leaving 
the box). I think that you will have to escalate with the Amazon 
support/technical people to try and peek under the covers a bit.

ric