From: "dE ." <de.techno@gmail.com>
To: linux-nfs@vger.kernel.org
Subject: Re: Slow NFS loop performance.
Date: Mon, 26 Nov 2012 23:15:14 +0530 [thread overview]
Message-ID: <50B3AAAA.9070601@gmail.com> (raw)
In-Reply-To: <20121126160853.GB11860@fieldses.org>
On 11/26/12 21:38, J. Bruce Fields wrote:
> On Mon, Nov 26, 2012 at 09:28:06PM +0530, dE . wrote:
>> On 11/23/12 22:53, J. Bruce Fields wrote:
>>> On Fri, Nov 23, 2012 at 04:31:55PM +0530, dE . wrote:
>>>> Humm... The last thing I expected was no response even in the mailing list.
>>>>
>>>> So I'm filing a bug on this.
>>>>
>>>> On Oct 23, 2012 2:19 PM, "dE ."<de.techno@gmail.com> wrote:
>>>>> Hi!
>>>>>
>>>>> Great job with NFS server, it surely is fast, but not on loop devices.
>>>>>
>>>>> If I loop mount a file and share the mount point over NFS3 or NFS4, the
>>>>> write performance of the client on the loop mounted share is pretty bad.
>>>>>
>>>>> On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps
>>>>> speeds, whereas on the loop mounted device, I get at best 6MBps.
>>> What exactly is your test?
>>>
>>> --b.
>> Sorry for the late response. I'd 200+ unread mails.
>>
>> I'm writing a large file to the mounted loop device.
> How large, and do you have the exact command you're using for that?
>
> Also, what are the client and server versions?
>
> I don't have any good idea off the top of my head. I doubt anyone's
> worked on optimizing exports of loop devices. I guess the first step
> would be to collect some statistics in the two cases (loop device and
> non-loop device), compare them, and see if you can see any patterns.
> /proc/self/mountstats on the client, and /proc/fs/nfsd/pool_stats, on
> the server, would be starting points. Maybe perf on the server could
> also show up something. Just running "top" on the server might be
> interesting. (E.g. is the CPU obviously busier in the slow case?)
>
> --b.
It's clear --
On the loop device --
dd if=/dev/zero of=/mnt/shares/mount_point/xfs_large_files/test.zero
bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1788.51 s, 600 kB/s
And otherwise --
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 157.447 s, 6.8 MB/s
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 128.628 s, 8.3 MB/s
I ran out of patience to do any tests on the loop device.
cat /proc/self/mountstats --
device rootfs mounted on / with fstype rootfs
device /dev/root mounted on / with fstype reiserfs
..
..
device 192.168.1.3:/ mounted on /mnt/shares with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 4026
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 14 54 0 8523 6 5 47 611072 0 0 611072 3040 12 6 12 12 0
6 0 5 611072 0 0 0 0 0 0
bytes: 0 2502950912 0 0 0 2502950912 0 611072
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 10035 10035 0 2505279032 1324620 53001291 3776691
56778324
COMMIT: 48 48 0 10176 5952 151346 14030 165378
OPEN: 6 6 0 1784 2352 0 43 43
OPEN_CONFIRM: 2 2 0 408 136 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 6 6 0 1320 792 0 1 1
SETATTR: 6 6 0 1432 1440 0 3478 3479
FSINFO: 2 2 0 336 216 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 13 13 0 2568 1488 15581 8484 28896
GETATTR: 15 15 0 2860 3300 0 7 8
LOOKUP: 6 7 0 1472 516 19755 11318 31074
LOOKUP_ROOT: 1 1 0 160 240 0 26 27
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 164 72 0 0 0
STATFS: 6 6 0 1128 696 0 1 1
READLINK: 0 0 0 0 0 0 0 0
READDIR: 3 3 0 652 3380 0 42 42
SERVER_CAPS: 3 3 0 492 276 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0
device 192.168.1.3:/mount_point/xfs_large_files/ mounted on
/mnt/shares/mount_point/xfs_large_files with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,port=0,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 2344
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 0 0 0 6464 0 0 2 262144 0 0 262144 10131 0 1 2 2 0 1 0
0 262144 0 0 0 0 0 0
bytes: 0 1073741824 0 0 0 1073741824 0 262144
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 7821 12504 5 1360145244 1032372 461231706
30244294 493689738
COMMIT: 23 28 0 5600 2852 981824 61062 1049646
OPEN: 1 1 0 336 404 0 122 122
OPEN_CONFIRM: 1 1 0 216 68 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 1 1 0 232 132 0 0 0
SETATTR: 1 1 0 244 240 0 0 0
FSINFO: 1 1 0 200 108 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 1 1 0 208 124 0 0 0
GETATTR: 1 1 0 200 220 0 0 0
LOOKUP: 0 0 0 0 0 0 0 0
LOOKUP_ROOT: 0 0 0 0 0 0 0 0
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 196 72 0 0 0
STATFS: 0 0 0 0 0 0 0 0
READLINK: 0 0 0 0 0 0 0 0
READDIR: 0 0 0 0 0 0 0 0
SERVER_CAPS: 2 2 0 392 184 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0
And /proc/fs/nfsd/pool_stats
cat /proc/fs/nfsd/pool_stats
# pool packets-arrived sockets-enqueued threads-woken overloads-avoided
threads-timedout
0 4637559 21028 1689208 269 0
Server --
uname -r
2.6.32-5-amd64
modinfo nfs
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfs/nfs.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: sunrpc,fscache,lockd,auth_rpcgss,nfs_acl
vermagic: 2.6.32-5-amd64 SMP mod_unload modversions
parm: callback_tcpport:portnr
parm: cache_getent:Path to the client cache upcall program
(string)
parm: cache_getent_timeout:Timeout (in seconds) after which
the cache upcall is assumed to have failed (ulong)
parm: enable_ino64:bool
modinfo nfsd
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfsd/nfsd.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: auth_rpcgss,sunrpc,lockd,exportfs,nfs_acl
vermagic: 2.6.32-5-amd64 SMP mod_unload modversions
Client --
uname -r
3.4.2-gentoo-r1
modinfo nfs
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfs/nfs.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: sunrpc,lockd,auth_rpcgss,nfs_acl
intree: Y
vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions
parm: callback_tcpport:portnr
parm: nfs_idmap_cache_timeout:int
parm: send_implementation_id:Send implementation ID with
NFSv4.1 exchange_id (ushort)
parm: max_session_slots:Maximum number of outstanding NFSv4.1
requests the client will negotiate (ushort)
parm: cache_getent:Path to the client cache upcall program
(string)
parm: cache_getent_timeout:Timeout (in seconds) after which
the cache upcall is assumed to have failed (ulong)
parm: enable_ino64:bool
parm: nfs4_disable_idmapping:Turn off NFSv4 idmapping when
using 'sec=sys' (bool)
modinfo nfsd
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfsd/nfsd.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: auth_rpcgss,sunrpc,lockd,nfs_acl
intree: Y
vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions
parm: nfs4_disable_idmapping:Turn off server's NFSv4 idmapping
when using 'sec=sys' (bool)
net-fs/nfs-utils version 1.2.3-r1
This problem was reproducible from day 1, even during the 2.6 days. This
also happen(ed/s) when I loop mount some file (on the server) in the client.
I also had an old Debian testing system which (Wheezy) which had 3.0
kernel and shared the same problem.
next prev parent reply other threads:[~2012-11-26 17:45 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-23 11:01 Slow NFS loop performance dE .
2012-11-23 17:23 ` J. Bruce Fields
2012-11-26 15:58 ` dE .
2012-11-26 16:08 ` J. Bruce Fields
2012-11-26 16:15 ` dE .
2012-11-26 17:45 ` dE . [this message]
2012-11-23 19:04 ` Myklebust, Trond
2012-11-26 16:00 ` dE .
2012-11-26 16:39 ` Myklebust, Trond
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50B3AAAA.9070601@gmail.com \
--to=de.techno@gmail.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox