Should NFS over loopback be over 10x slower compared to local fs

Should NFS over loopback be over 10x slower compared to local fs

[Yusuf Goolamabbas]

Hi, I did a postmark run on a local ext3 filesystem (data=writeback) and
then compared it with the same filesystem mounted over loopback and then
over a 100Mbit switch, I see a dramatic slowdown with postmark numbers
of loopback NFS compared to local fs. 

I am not sure if the numbers are appropiate. The ext3fs is exported 

/ext3 192.168.2.0/24(sync,rw,no_root_squash) 127.0.0.1/8(sync,rw,no_root_squash)

I am preparing a FreeBSD box to compare with it. Just wanted to share my
methodology and numbers with others to see if they could reproduce and
confirm/deny these numbers. All tests were run by 'root'


NFS server
kernel version 2.6.5-rc2-mm4, P3-500/384 MB 2x9GB SCSI, aic7xxx driver
global_tag_depth = 4
using anticipatory io scheduler

Postmark: 
        http://www.netapp.com/tech_library/3022.print
postmark config

set size 1024 20480
set subdirectories 100
set number 1000
set transactions 10000
set location /loopmount/nfsexport
set read 8192
set write 8192
set bias read 10
set bias create 5

local FS ext3 (data=writeback), htree disabled

Time:
        5 seconds total
        5 seconds of transactions (2000 per second)

Files:
        5982 created (1196 per second)
                Creation alone: 1000 files (1000 per second)
                Mixed with transactions: 4982 files (996 per second)
        10000 read (2000 per second)
        0 appended (0 per second)
        5982 deleted (1196 per second)
                Deletion alone: 964 files (964 per second)
                Mixed with transactions: 5018 files (1003 per second)

Data:
        102.62 megabytes read (20.52 megabytes per second)
        61.01 megabytes written (12.20 megabytes per second)

NFSv3 over loopback

mount -t nfs -o nfsvers=3,noatime,rsize=16384,wsize=16384 127.0.0.1:/ext3 /loopmount

Time:
        85 seconds total
        71 seconds of transactions (140 per second)

Files:
        5982 created (70 per second)
                Creation alone: 1000 files (71 per second)
                Mixed with transactions: 4982 files (70 per second)
        10000 read (140 per second)
        0 appended (0 per second)
        5982 deleted (70 per second)
                Deletion alone: 964 files (964 per second)
                Mixed with transactions: 5018 files (70 per second)

Data:
        102.62 megabytes read (1.21 megabytes per second)
        61.01 megabytes written (735.01 kilobytes per second)

NFS client (2.6.5-rc2-mm3)

NFSv3 over 100Mbps (TCP)
mount -t nfs -o tcp,nfsvers=3,noatime,bg,soft,rsize=32768,wsize=32768
192.168.2.214:/ext3 /nfsmount

Time:
        120 seconds total
        101 seconds of transactions (99 per second)

Files:
        5982 created (49 per second)
                Creation alone: 1000 files (66 per second)
                Mixed with transactions: 4982 files (49 per second)
        10000 read (99 per second)
        0 appended (0 per second)
        5982 deleted (49 per second)
                Deletion alone: 964 files (241 per second)
                Mixed with transactions: 5018 files (49 per second)

Data:
        102.62 megabytes read (875.68 kilobytes per second)
        61.01 megabytes written (520.64 kilobytes per second)

NFSv3 over 100Mbps (UDP)

mount -t nfs -o nfsvers=3,noatime,bg,soft,rsize=8192,wsize=8192 192.168.2.214:/ext3 /nfsmount

Time:
        208 seconds total
        174 seconds of transactions (57 per second)

Files:
        5982 created (28 per second)
                Creation alone: 1000 files (66 per second)
                Mixed with transactions: 4982 files (28 per second)
        10000 read (57 per second)
        0 appended (0 per second)
        5982 deleted (28 per second)
                Deletion alone: 964 files (50 per second)
                Mixed with transactions: 5018 files (28 per second)

Data:
        102.62 megabytes read (505.20 kilobytes per second)
        61.01 megabytes written (300.37 kilobytes per second)


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Should NFS over loopback be over 10x slower compared to local fs

[Yusuf Goolamabbas]

> > Hi, I did a postmark run on a local ext3 filesystem (data=writeback) and
> > then compared it with the same filesystem mounted over loopback and then
> > over a 100Mbit switch, I see a dramatic slowdown with postmark numbers
> > of loopback NFS compared to local fs.
> 
> Depending on drive speed and write clustering, expect a significant
> slowdown I'm not sure 10x is OK, but it's certainly realistic.

I expected a slowdown, it's just that I felt that 10x was inappropiate.
Curious as to what others get with the same test run on different
OS/hardware

On FreeBSD 4.9-RELEASE on similar hardware, the slowdown is only half
compared to postmark over local softupdate enabled fs.

Also on 2.6.x, NFS over loopback wasn't that much faster than NFS over
100Mbit 

> There's an important difference between NFS and local file system, in
> semantics: Once the NFS server has acknowledged the _write_ request, the
> data is on disk. You don't get that sort of guarantee with a local file
> system unless you open with O_SYNC (and I wonder how many bugs
> "not-quite-synchronous" need to be shaken out WRT local file systems
> given the recent fsync() discussion).

Hmm, That's plausible. I did a tiobench run over a local ext3 fs
(data=writeback) and the same run over loopback NFS and the slowdown was
in sequential/random writes though sequential writes was much slower
than random writes



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Should NFS over loopback be over 10x slower compared to local fs

[Matthias Andree]

[re-sending after it hasn't been approved in almost a week]

On Tue, 30 Mar 2004, Yusuf Goolamabbas wrote:

> I expected a slowdown, it's just that I felt that 10x was inappropiate.
> Curious as to what others get with the same test run on different
> OS/hardware
> 
> On FreeBSD 4.9-RELEASE on similar hardware, the slowdown is only half
> compared to postmark over local softupdate enabled fs.

You are comparing apples to oranges. softupdate enabled file systems are
an allegedly "on-disk-image-always-consistent" version of saying
"asynchronous operation", but I've recently seen corruption so one or
another bug may still linger in softdep.

To compare apples to apples, you need to mount the UFS file with -o sync
on FreeBSD.

> Hmm, That's plausible. I did a tiobench run over a local ext3 fs
> (data=writeback) and the same run over loopback NFS and the slowdown was
> in sequential/random writes though sequential writes was much slower
> than random writes

If "slower" describes the ratio between local and NFS, then that's sane,
sequential writes don't suffer the head positioning latency.

-- 
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs