From: "dE ." <de.techno@gmail.com>
To: linux-nfs@vger.kernel.org
Subject: Re: Slow NFS loop performance.
Date: Mon, 26 Nov 2012 23:15:14 +0530 [thread overview]
Message-ID: <50B3AAAA.9070601@gmail.com> (raw)
In-Reply-To: <20121126160853.GB11860@fieldses.org>
On 11/26/12 21:38, J. Bruce Fields wrote:
> On Mon, Nov 26, 2012 at 09:28:06PM +0530, dE . wrote:
>> On 11/23/12 22:53, J. Bruce Fields wrote:
>>> On Fri, Nov 23, 2012 at 04:31:55PM +0530, dE . wrote:
>>>> Humm... The last thing I expected was no response even in the mailing list.
>>>>
>>>> So I'm filing a bug on this.
>>>>
>>>> On Oct 23, 2012 2:19 PM, "dE ."<de.techno@gmail.com> wrote:
>>>>> Hi!
>>>>>
>>>>> Great job with NFS server, it surely is fast, but not on loop devices.
>>>>>
>>>>> If I loop mount a file and share the mount point over NFS3 or NFS4, the
>>>>> write performance of the client on the loop mounted share is pretty bad.
>>>>>
>>>>> On a 100 Mbps (or 12.5MBps) full duplex Ethernet link, I get ~8MBps
>>>>> speeds, whereas on the loop mounted device, I get at best 6MBps.
>>> What exactly is your test?
>>>
>>> --b.
>> Sorry for the late response. I'd 200+ unread mails.
>>
>> I'm writing a large file to the mounted loop device.
> How large, and do you have the exact command you're using for that?
>
> Also, what are the client and server versions?
>
> I don't have any good idea off the top of my head. I doubt anyone's
> worked on optimizing exports of loop devices. I guess the first step
> would be to collect some statistics in the two cases (loop device and
> non-loop device), compare them, and see if you can see any patterns.
> /proc/self/mountstats on the client, and /proc/fs/nfsd/pool_stats, on
> the server, would be starting points. Maybe perf on the server could
> also show up something. Just running "top" on the server might be
> interesting. (E.g. is the CPU obviously busier in the slow case?)
>
> --b.
It's clear --
On the loop device --
dd if=/dev/zero of=/mnt/shares/mount_point/xfs_large_files/test.zero
bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1788.51 s, 600 kB/s
And otherwise --
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 157.447 s, 6.8 MB/s
dd if=/dev/zero of=/mnt/shares/test.zero bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 128.628 s, 8.3 MB/s
I ran out of patience to do any tests on the loop device.
cat /proc/self/mountstats --
device rootfs mounted on / with fstype rootfs
device /dev/root mounted on / with fstype reiserfs
..
..
device 192.168.1.3:/ mounted on /mnt/shares with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 4026
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 14 54 0 8523 6 5 47 611072 0 0 611072 3040 12 6 12 12 0
6 0 5 611072 0 0 0 0 0 0
bytes: 0 2502950912 0 0 0 2502950912 0 611072
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 10035 10035 0 2505279032 1324620 53001291 3776691
56778324
COMMIT: 48 48 0 10176 5952 151346 14030 165378
OPEN: 6 6 0 1784 2352 0 43 43
OPEN_CONFIRM: 2 2 0 408 136 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 6 6 0 1320 792 0 1 1
SETATTR: 6 6 0 1432 1440 0 3478 3479
FSINFO: 2 2 0 336 216 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 13 13 0 2568 1488 15581 8484 28896
GETATTR: 15 15 0 2860 3300 0 7 8
LOOKUP: 6 7 0 1472 516 19755 11318 31074
LOOKUP_ROOT: 1 1 0 160 240 0 26 27
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 164 72 0 0 0
STATFS: 6 6 0 1128 696 0 1 1
READLINK: 0 0 0 0 0 0 0 0
READDIR: 3 3 0 652 3380 0 42 42
SERVER_CAPS: 3 3 0 492 276 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0
device 192.168.1.3:/mount_point/xfs_large_files/ mounted on
/mnt/shares/mount_point/xfs_large_files with fstype nfs4 statvers=1.1
opts:
rw,vers=4.0,rsize=262144,wsize=262144,namlen=255,acregmin=900,acregmax=900,acdirmin=900,acdirmax=900,hard,proto=tcp,port=0,timeo=60,retrans=2,sec=sys,clientaddr=192.168.1.2,lookupcache=pos,local_lock=none
age: 2344
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
nfsv4: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
sec: flavor=1,pseudoflavor=1
events: 0 0 0 6464 0 0 2 262144 0 0 262144 10131 0 1 2 2 0 1 0
0 262144 0 0 0 0 0 0
bytes: 0 1073741824 0 0 0 1073741824 0 262144
RPC iostats version: 1.0 p/v: 100003/4 (nfs)
xprt: tcp 763 0 348 341795 14 30200 26072 7 42669773 0 174
6344270 677055
per-op statistics
NULL: 0 0 0 0 0 0 0 0
READ: 0 0 0 0 0 0 0 0
WRITE: 7821 12504 5 1360145244 1032372 461231706
30244294 493689738
COMMIT: 23 28 0 5600 2852 981824 61062 1049646
OPEN: 1 1 0 336 404 0 122 122
OPEN_CONFIRM: 1 1 0 216 68 0 0 0
OPEN_NOATTR: 0 0 0 0 0 0 0 0
OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
CLOSE: 1 1 0 232 132 0 0 0
SETATTR: 1 1 0 244 240 0 0 0
FSINFO: 1 1 0 200 108 0 0 0
RENEW: 0 0 0 0 0 0 0 0
SETCLIENTID: 0 0 0 0 0 0 0 0
SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
LOCK: 0 0 0 0 0 0 0 0
LOCKT: 0 0 0 0 0 0 0 0
LOCKU: 0 0 0 0 0 0 0 0
ACCESS: 1 1 0 208 124 0 0 0
GETATTR: 1 1 0 200 220 0 0 0
LOOKUP: 0 0 0 0 0 0 0 0
LOOKUP_ROOT: 0 0 0 0 0 0 0 0
REMOVE: 0 0 0 0 0 0 0 0
RENAME: 0 0 0 0 0 0 0 0
LINK: 0 0 0 0 0 0 0 0
SYMLINK: 0 0 0 0 0 0 0 0
CREATE: 0 0 0 0 0 0 0 0
PATHCONF: 1 1 0 196 72 0 0 0
STATFS: 0 0 0 0 0 0 0 0
READLINK: 0 0 0 0 0 0 0 0
READDIR: 0 0 0 0 0 0 0 0
SERVER_CAPS: 2 2 0 392 184 0 0 0
DELEGRETURN: 0 0 0 0 0 0 0 0
GETACL: 0 0 0 0 0 0 0 0
SETACL: 0 0 0 0 0 0 0 0
FS_LOCATIONS: 0 0 0 0 0 0 0 0
RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
SECINFO: 0 0 0 0 0 0 0 0
EXCHANGE_ID: 0 0 0 0 0 0 0 0
CREATE_SESSION: 0 0 0 0 0 0 0 0
DESTROY_SESSION: 0 0 0 0 0 0 0 0
SEQUENCE: 0 0 0 0 0 0 0 0
GET_LEASE_TIME: 0 0 0 0 0 0 0 0
RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
LAYOUTGET: 0 0 0 0 0 0 0 0
GETDEVICEINFO: 0 0 0 0 0 0 0 0
LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
LAYOUTRETURN: 0 0 0 0 0 0 0 0
SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
TEST_STATEID: 0 0 0 0 0 0 0 0
FREE_STATEID: 0 0 0 0 0 0 0 0
GETDEVICELIST: 0 0 0 0 0 0 0 0
And /proc/fs/nfsd/pool_stats
cat /proc/fs/nfsd/pool_stats
# pool packets-arrived sockets-enqueued threads-woken overloads-avoided
threads-timedout
0 4637559 21028 1689208 269 0
Server --
uname -r
2.6.32-5-amd64
modinfo nfs
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfs/nfs.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: sunrpc,fscache,lockd,auth_rpcgss,nfs_acl
vermagic: 2.6.32-5-amd64 SMP mod_unload modversions
parm: callback_tcpport:portnr
parm: cache_getent:Path to the client cache upcall program
(string)
parm: cache_getent_timeout:Timeout (in seconds) after which
the cache upcall is assumed to have failed (ulong)
parm: enable_ino64:bool
modinfo nfsd
filename: /lib/modules/2.6.32-5-amd64/kernel/fs/nfsd/nfsd.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: auth_rpcgss,sunrpc,lockd,exportfs,nfs_acl
vermagic: 2.6.32-5-amd64 SMP mod_unload modversions
Client --
uname -r
3.4.2-gentoo-r1
modinfo nfs
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfs/nfs.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: sunrpc,lockd,auth_rpcgss,nfs_acl
intree: Y
vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions
parm: callback_tcpport:portnr
parm: nfs_idmap_cache_timeout:int
parm: send_implementation_id:Send implementation ID with
NFSv4.1 exchange_id (ushort)
parm: max_session_slots:Maximum number of outstanding NFSv4.1
requests the client will negotiate (ushort)
parm: cache_getent:Path to the client cache upcall program
(string)
parm: cache_getent_timeout:Timeout (in seconds) after which
the cache upcall is assumed to have failed (ulong)
parm: enable_ino64:bool
parm: nfs4_disable_idmapping:Turn off NFSv4 idmapping when
using 'sec=sys' (bool)
modinfo nfsd
filename: /lib/modules/3.4.2-gentoo-r1/kernel/fs/nfsd/nfsd.ko
license: GPL
author: Olaf Kirch <okir@monad.swb.de>
depends: auth_rpcgss,sunrpc,lockd,nfs_acl
intree: Y
vermagic: 3.4.2-gentoo-r1 SMP preempt mod_unload modversions
parm: nfs4_disable_idmapping:Turn off server's NFSv4 idmapping
when using 'sec=sys' (bool)
net-fs/nfs-utils version 1.2.3-r1
This problem was reproducible from day 1, even during the 2.6 days. This
also happen(ed/s) when I loop mount some file (on the server) in the client.
I also had an old Debian testing system which (Wheezy) which had 3.0
kernel and shared the same problem.
next prev parent reply other threads:[~2012-11-26 17:45 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-23 11:01 Slow NFS loop performance dE .
2012-11-23 17:23 ` J. Bruce Fields
2012-11-26 15:58 ` dE .
2012-11-26 16:08 ` J. Bruce Fields
2012-11-26 16:15 ` dE .
2012-11-26 17:45 ` dE . [this message]
2012-11-23 19:04 ` Myklebust, Trond
2012-11-26 16:00 ` dE .
2012-11-26 16:39 ` Myklebust, Trond
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50B3AAAA.9070601@gmail.com \
--to=de.techno@gmail.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.