NFS3 + kerberos: performance issues

NFS3 + kerberos: performance issues

[Vladimir Elisseev]

Hello,

After using nfs3 with kerberos for a while I discovered very poor
performance for some file operations like creation of files. There are
two observations:
- the overal nfs3 performance is good: for instance copying large files
works fine
- this poor performance is definitely related to kerberos as the same
volume exported without sec=krb5 works ~10 times faster with creating
files
I'd appreciate if somebody can explain this behaviour?

Below are some details about NFS server and client:
server:
- kernel 2.6.36
- export options rw,sec=krb5,no_root_squash,no_subtree_check
- nfs-utils 1.2.3

client:
- kernel 2.6.38 or 2.6.39 
- mout options (from /proc/mounts):
rw,nosuid,nodev,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219

when the same volume exported without sec=krb5, the only option changes
in /proc/mounts is sec=sys

Regards,
Vlad.

Re: NFS3 + kerberos: performance issues

[Trond Myklebust]

On Fri, 2011-06-03 at 09:29 +0200, Vladimir Elisseev wrote: 
> Hello,
> 
> After using nfs3 with kerberos for a while I discovered very poor
> performance for some file operations like creation of files. There are
> two observations:
> - the overal nfs3 performance is good: for instance copying large files
> works fine
> - this poor performance is definitely related to kerberos as the same
> volume exported without sec=krb5 works ~10 times faster with creating
> files
> I'd appreciate if somebody can explain this behaviour?
> 
> Below are some details about NFS server and client:
> server:
> - kernel 2.6.36
> - export options rw,sec=krb5,no_root_squash,no_subtree_check
> - nfs-utils 1.2.3
> 
> client:
> - kernel 2.6.38 or 2.6.39 
> - mout options (from /proc/mounts):
> rw,nosuid,nodev,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219
> 
> when the same volume exported without sec=krb5, the only option changes
> in /proc/mounts is sec=sys

Can you please tell us a bit more about how you are measuring this? I
might expect a x10 performance difference if the workload is dominated
by the actual setting up of the RPCSEC_GSS session (i.e. you are timing
mount+create one file or something like that). If we're talking about a
workload where the RPCSEC_GSS negotiation can be neglected, however
(mount+create 100000 files), I wouldn't expect any measurable
performance impact from 'sec=krb'.


-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFS3 + kerberos: performance issues

[Vladimir Elisseev]

There's no mount command involved in timing. It's simply copy of the
same directory (with many small files to NFS share with and without
sec=krb5 mount option.

Vlad.

On Fri, 2011-06-03 at 08:32 -0400, Trond Myklebust wrote:
> On Fri, 2011-06-03 at 09:29 +0200, Vladimir Elisseev wrote: 
> > Hello,
> > 
> > After using nfs3 with kerberos for a while I discovered very poor
> > performance for some file operations like creation of files. There are
> > two observations:
> > - the overal nfs3 performance is good: for instance copying large files
> > works fine
> > - this poor performance is definitely related to kerberos as the same
> > volume exported without sec=krb5 works ~10 times faster with creating
> > files
> > I'd appreciate if somebody can explain this behaviour?
> > 
> > Below are some details about NFS server and client:
> > server:
> > - kernel 2.6.36
> > - export options rw,sec=krb5,no_root_squash,no_subtree_check
> > - nfs-utils 1.2.3
> > 
> > client:
> > - kernel 2.6.38 or 2.6.39 
> > - mout options (from /proc/mounts):
> > rw,nosuid,nodev,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219
> > 
> > when the same volume exported without sec=krb5, the only option changes
> > in /proc/mounts is sec=sys
> 
> Can you please tell us a bit more about how you are measuring this? I
> might expect a x10 performance difference if the workload is dominated
> by the actual setting up of the RPCSEC_GSS session (i.e. you are timing
> mount+create one file or something like that). If we're talking about a
> workload where the RPCSEC_GSS negotiation can be neglected, however
> (mount+create 100000 files), I wouldn't expect any measurable
> performance impact from 'sec=krb'.
> 
> 

Re: NFS3 + kerberos: performance issues

[J. Bruce Fields]

On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> There's no mount command involved in timing. It's simply copy of the
> same directory (with many small files to NFS share with and without
> sec=krb5 mount option.

Weird.

I wonder if the krb5 principal is being mapped to a different user on
the server side, and that's making some difference.

Still, could you give us the full details?  (Exactly what commands are
you running, what results do you see?)

--b.

Re: NFS3 + kerberos: performance issues

[Vladimir Elisseev]

All of the accounts, except very limited amount, are in LDAP, and as I
can see user information is identical on server and client. As for my
tests, below are all the details. Difference in speed while copying a
lot of small files is _huge_... I'd appreciate if somebody more familiar
with NFS/kerberos combination can provide a kind of explanation. 

Regards,
Vlad.

* Local directory is /mnt/data/tmp/coreboot/src
# find x86/ | wc -l
296
# du -csh x86
1.5M	x86
1.5M	total
* First try without kerberos: grep /mnt/tmp/ /proc/mounts
nfs:/mnt/tmp /mnt/tmp nfs
rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
# time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 0.01s user
0.04s system 10% cpu 0.476 total

* Second try with kerberos: grep /mnt/tmp/ /proc/mounts
nfs:/mnt/tmp /mnt/tmp nfs
rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
# time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86  0.01s user
0.07s system 0% cpu 23.294 total

At the same time test with a single file (621M):
* without kerberos: 
cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
user 0.69s system 5% cpu 13.521 total
* with kerberos:
cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
user 0.55s system 2% cpu 26.299 total

With the same file, but with "dd
if=/mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso
of=/mnt/tmp/test.iso bs=32M"
gives:
* without kerberos: 651165696 bytes (651 MB) copied, 10.7168 s, 60.8
MB/s
* with kerberos: 651165696 bytes (651 MB) copied, 24.6176 s, 26.5 MB/s


On Tue, 2011-06-07 at 19:07 -0400, J. Bruce Fields wrote:
> On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> > There's no mount command involved in timing. It's simply copy of the
> > same directory (with many small files to NFS share with and without
> > sec=krb5 mount option.
> 
> Weird.
> 
> I wonder if the krb5 principal is being mapped to a different user on
> the server side, and that's making some difference.
> 
> Still, could you give us the full details?  (Exactly what commands are
> you running, what results do you see?)
> 
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS3 + kerberos: performance issues

[J. Bruce Fields]

On Wed, Jun 08, 2011 at 08:26:17AM +0200, Vladimir Elisseev wrote:
> All of the accounts, except very limited amount, are in LDAP, and as I
> can see user information is identical on server and client. As for my
> tests, below are all the details. Difference in speed while copying a
> lot of small files is _huge_... I'd appreciate if somebody more familiar
> with NFS/kerberos combination can provide a kind of explanation. 
> 
> Regards,
> Vlad.
> 
> * Local directory is /mnt/data/tmp/coreboot/src
> # find x86/ | wc -l
> 296
> # du -csh x86
> 1.5M	x86
> 1.5M	total
> * First try without kerberos: grep /mnt/tmp/ /proc/mounts
> nfs:/mnt/tmp /mnt/tmp nfs
> rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 0.01s user
> 0.04s system 10% cpu 0.476 total
> 
> * Second try with kerberos: grep /mnt/tmp/ /proc/mounts
> nfs:/mnt/tmp /mnt/tmp nfs
> rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86  0.01s user
> 0.07s system 0% cpu 23.294 total

So if I'm reading that right, the client isn't doing any more work, it's
just taking longer, so presumably it's spending more time waiting for
the server.

Might be worth looking at /proc/self/mountstats to see which rpc's are
taking longer.

(Chuck, what's up with the mountstats script?  On a fedora 15 machine
with mountstats installed, it gives me "Statistics for mount point /mnt
not found", but

	# grep mnt /proc/self/mountstats 
	pip1:/exports/ mounted on /mnt with fstype nfs4 statvers=1.0

On a ubuntu machine running the mountstats out of
nfs-utils/tools/mountstats/mountstats.py, it runs without any output.)

Might also be worth looking at vmstat or something on the server.

--b.


> 
> At the same time test with a single file (621M):
> * without kerberos: 
> cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> user 0.69s system 5% cpu 13.521 total
> * with kerberos:
> cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> user 0.55s system 2% cpu 26.299 total
> 
> With the same file, but with "dd
> if=/mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso
> of=/mnt/tmp/test.iso bs=32M"
> gives:
> * without kerberos: 651165696 bytes (651 MB) copied, 10.7168 s, 60.8
> MB/s
> * with kerberos: 651165696 bytes (651 MB) copied, 24.6176 s, 26.5 MB/s
> 
> 
> On Tue, 2011-06-07 at 19:07 -0400, J. Bruce Fields wrote:
> > On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> > > There's no mount command involved in timing. It's simply copy of the
> > > same directory (with many small files to NFS share with and without
> > > sec=krb5 mount option.
> > 
> > Weird.
> > 
> > I wonder if the krb5 principal is being mapped to a different user on
> > the server side, and that's making some difference.
> > 
> > Still, could you give us the full details?  (Exactly what commands are
> > you running, what results do you see?)
> > 
> > --b.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: NFS3 + kerberos: performance issues

[Vladimir Elisseev]

Thanks for pointing me to the right to look at. However, the main point
is huge performance degradation while using sec=krb5 option instead of
sec=sys. With sec=sys everything works as expected. Personally I simply
can't find any reasonable explanation. BTW, I'm using Gentoo and getting
the same error while using mountstats script.

Regards,
Vlad.

On Wed, 2011-06-08 at 11:39 -0400, J. Bruce Fields wrote:
> On Wed, Jun 08, 2011 at 08:26:17AM +0200, Vladimir Elisseev wrote:
> > All of the accounts, except very limited amount, are in LDAP, and as I
> > can see user information is identical on server and client. As for my
> > tests, below are all the details. Difference in speed while copying a
> > lot of small files is _huge_... I'd appreciate if somebody more familiar
> > with NFS/kerberos combination can provide a kind of explanation. 
> > 
> > Regards,
> > Vlad.
> > 
> > * Local directory is /mnt/data/tmp/coreboot/src
> > # find x86/ | wc -l
> > 296
> > # du -csh x86
> > 1.5M	x86
> > 1.5M	total
> > * First try without kerberos: grep /mnt/tmp/ /proc/mounts
> > nfs:/mnt/tmp /mnt/tmp nfs
> > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 0.01s user
> > 0.04s system 10% cpu 0.476 total
> > 
> > * Second try with kerberos: grep /mnt/tmp/ /proc/mounts
> > nfs:/mnt/tmp /mnt/tmp nfs
> > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86  0.01s user
> > 0.07s system 0% cpu 23.294 total
> 
> So if I'm reading that right, the client isn't doing any more work, it's
> just taking longer, so presumably it's spending more time waiting for
> the server.
> 
> Might be worth looking at /proc/self/mountstats to see which rpc's are
> taking longer.
> 
> (Chuck, what's up with the mountstats script?  On a fedora 15 machine
> with mountstats installed, it gives me "Statistics for mount point /mnt
> not found", but
> 
> 	# grep mnt /proc/self/mountstats 
> 	pip1:/exports/ mounted on /mnt with fstype nfs4 statvers=1.0
> 
> On a ubuntu machine running the mountstats out of
> nfs-utils/tools/mountstats/mountstats.py, it runs without any output.)
> 
> Might also be worth looking at vmstat or something on the server.
> 
> --b.
> 
> 
> > 
> > At the same time test with a single file (621M):
> > * without kerberos: 
> > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > user 0.69s system 5% cpu 13.521 total
> > * with kerberos:
> > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > user 0.55s system 2% cpu 26.299 total
> > 
> > With the same file, but with "dd
> > if=/mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso
> > of=/mnt/tmp/test.iso bs=32M"
> > gives:
> > * without kerberos: 651165696 bytes (651 MB) copied, 10.7168 s, 60.8
> > MB/s
> > * with kerberos: 651165696 bytes (651 MB) copied, 24.6176 s, 26.5 MB/s
> > 
> > 
> > On Tue, 2011-06-07 at 19:07 -0400, J. Bruce Fields wrote:
> > > On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> > > > There's no mount command involved in timing. It's simply copy of the
> > > > same directory (with many small files to NFS share with and without
> > > > sec=krb5 mount option.
> > > 
> > > Weird.
> > > 
> > > I wonder if the krb5 principal is being mapped to a different user on
> > > the server side, and that's making some difference.
> > > 
> > > Still, could you give us the full details?  (Exactly what commands are
> > > you running, what results do you see?)
> > > 
> > > --b.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 

Re: NFS3 + kerberos: performance issues

[J. Bruce Fields]

On Wed, Jun 08, 2011 at 06:00:36PM +0200, Vladimir Elisseev wrote:
> Thanks for pointing me to the right to look at. However, the main point
> is huge performance degradation while using sec=krb5 option instead of
> sec=sys. With sec=sys everything works as expected. Personally I simply
> can't find any reasonable explanation.

Right, me neither.  That's why I suggest a) looking at mountstats, to
figure out which rpc operations are taking longer in the sec=krb5 case
than they are in the sec=sys case, and b) looking at server-side
statistics (mainly cpu usage, I guess) to start looking for bottlenecks
on the server.

--b.

> BTW, I'm using Gentoo and getting
> the same error while using mountstats script.
> 
> Regards,
> Vlad.
> 
> On Wed, 2011-06-08 at 11:39 -0400, J. Bruce Fields wrote:
> > On Wed, Jun 08, 2011 at 08:26:17AM +0200, Vladimir Elisseev wrote:
> > > All of the accounts, except very limited amount, are in LDAP, and as I
> > > can see user information is identical on server and client. As for my
> > > tests, below are all the details. Difference in speed while copying a
> > > lot of small files is _huge_... I'd appreciate if somebody more familiar
> > > with NFS/kerberos combination can provide a kind of explanation. 
> > > 
> > > Regards,
> > > Vlad.
> > > 
> > > * Local directory is /mnt/data/tmp/coreboot/src
> > > # find x86/ | wc -l
> > > 296
> > > # du -csh x86
> > > 1.5M	x86
> > > 1.5M	total
> > > * First try without kerberos: grep /mnt/tmp/ /proc/mounts
> > > nfs:/mnt/tmp /mnt/tmp nfs
> > > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 0.01s user
> > > 0.04s system 10% cpu 0.476 total
> > > 
> > > * Second try with kerberos: grep /mnt/tmp/ /proc/mounts
> > > nfs:/mnt/tmp /mnt/tmp nfs
> > > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86  0.01s user
> > > 0.07s system 0% cpu 23.294 total
> > 
> > So if I'm reading that right, the client isn't doing any more work, it's
> > just taking longer, so presumably it's spending more time waiting for
> > the server.
> > 
> > Might be worth looking at /proc/self/mountstats to see which rpc's are
> > taking longer.
> > 
> > (Chuck, what's up with the mountstats script?  On a fedora 15 machine
> > with mountstats installed, it gives me "Statistics for mount point /mnt
> > not found", but
> > 
> > 	# grep mnt /proc/self/mountstats 
> > 	pip1:/exports/ mounted on /mnt with fstype nfs4 statvers=1.0
> > 
> > On a ubuntu machine running the mountstats out of
> > nfs-utils/tools/mountstats/mountstats.py, it runs without any output.)
> > 
> > Might also be worth looking at vmstat or something on the server.
> > 
> > --b.
> > 
> > 
> > > 
> > > At the same time test with a single file (621M):
> > > * without kerberos: 
> > > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > > user 0.69s system 5% cpu 13.521 total
> > > * with kerberos:
> > > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > > user 0.55s system 2% cpu 26.299 total
> > > 
> > > With the same file, but with "dd
> > > if=/mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso
> > > of=/mnt/tmp/test.iso bs=32M"
> > > gives:
> > > * without kerberos: 651165696 bytes (651 MB) copied, 10.7168 s, 60.8
> > > MB/s
> > > * with kerberos: 651165696 bytes (651 MB) copied, 24.6176 s, 26.5 MB/s
> > > 
> > > 
> > > On Tue, 2011-06-07 at 19:07 -0400, J. Bruce Fields wrote:
> > > > On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> > > > > There's no mount command involved in timing. It's simply copy of the
> > > > > same directory (with many small files to NFS share with and without
> > > > > sec=krb5 mount option.
> > > > 
> > > > Weird.
> > > > 
> > > > I wonder if the krb5 principal is being mapped to a different user on
> > > > the server side, and that's making some difference.
> > > > 
> > > > Still, could you give us the full details?  (Exactly what commands are
> > > > you running, what results do you see?)
> > > > 
> > > > --b.
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> 

Re: NFS3 + kerberos: performance issues

[Vladimir Elisseev]

This morning I've decided to redo all the tests and include some more
statistics ( mountstat and vmstat ). However, when I exported the same
mount point with different options I missed one _very_ important option.
This option was presented for the export with sec=sys, but omitted with
sec=krb5. The option is "async"... I promise to be more careful before
posting any results. Thanks all of you for the help and I'm sorry to
trouble you.

Regards,
Vlad.


On Wed, 2011-06-08 at 12:24 -0400, J. Bruce Fields wrote:
> On Wed, Jun 08, 2011 at 06:00:36PM +0200, Vladimir Elisseev wrote:
> > Thanks for pointing me to the right to look at. However, the main point
> > is huge performance degradation while using sec=krb5 option instead of
> > sec=sys. With sec=sys everything works as expected. Personally I simply
> > can't find any reasonable explanation.
> 
> Right, me neither.  That's why I suggest a) looking at mountstats, to
> figure out which rpc operations are taking longer in the sec=krb5 case
> than they are in the sec=sys case, and b) looking at server-side
> statistics (mainly cpu usage, I guess) to start looking for bottlenecks
> on the server.
> 
> --b.
> 
> > BTW, I'm using Gentoo and getting
> > the same error while using mountstats script.
> > 
> > Regards,
> > Vlad.
> > 
> > On Wed, 2011-06-08 at 11:39 -0400, J. Bruce Fields wrote:
> > > On Wed, Jun 08, 2011 at 08:26:17AM +0200, Vladimir Elisseev wrote:
> > > > All of the accounts, except very limited amount, are in LDAP, and as I
> > > > can see user information is identical on server and client. As for my
> > > > tests, below are all the details. Difference in speed while copying a
> > > > lot of small files is _huge_... I'd appreciate if somebody more familiar
> > > > with NFS/kerberos combination can provide a kind of explanation. 
> > > > 
> > > > Regards,
> > > > Vlad.
> > > > 
> > > > * Local directory is /mnt/data/tmp/coreboot/src
> > > > # find x86/ | wc -l
> > > > 296
> > > > # du -csh x86
> > > > 1.5M	x86
> > > > 1.5M	total
> > > > * First try without kerberos: grep /mnt/tmp/ /proc/mounts
> > > > nfs:/mnt/tmp /mnt/tmp nfs
> > > > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > > > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > > > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 0.01s user
> > > > 0.04s system 10% cpu 0.476 total
> > > > 
> > > > * Second try with kerberos: grep /mnt/tmp/ /proc/mounts
> > > > nfs:/mnt/tmp /mnt/tmp nfs
> > > > rw,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,mountaddr=192.168.1.219,mountvers=3,mountport=32767,mountproto=tcp,local_lock=none,addr=192.168.1.219 0 0
> > > > # time cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86 
> > > > cp -i -arv /mnt/data/tmp/coreboot/src/arch/x86 /mnt/tmp/x86  0.01s user
> > > > 0.07s system 0% cpu 23.294 total
> > > 
> > > So if I'm reading that right, the client isn't doing any more work, it's
> > > just taking longer, so presumably it's spending more time waiting for
> > > the server.
> > > 
> > > Might be worth looking at /proc/self/mountstats to see which rpc's are
> > > taking longer.
> > > 
> > > (Chuck, what's up with the mountstats script?  On a fedora 15 machine
> > > with mountstats installed, it gives me "Statistics for mount point /mnt
> > > not found", but
> > > 
> > > 	# grep mnt /proc/self/mountstats 
> > > 	pip1:/exports/ mounted on /mnt with fstype nfs4 statvers=1.0
> > > 
> > > On a ubuntu machine running the mountstats out of
> > > nfs-utils/tools/mountstats/mountstats.py, it runs without any output.)
> > > 
> > > Might also be worth looking at vmstat or something on the server.
> > > 
> > > --b.
> > > 
> > > 
> > > > 
> > > > At the same time test with a single file (621M):
> > > > * without kerberos: 
> > > > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > > > user 0.69s system 5% cpu 13.521 total
> > > > * with kerberos:
> > > > cp -i -av /mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso   0.00s
> > > > user 0.55s system 2% cpu 26.299 total
> > > > 
> > > > With the same file, but with "dd
> > > > if=/mnt/media/images/GNOME_3.x86_64-0.2.0-Build1.1.iso
> > > > of=/mnt/tmp/test.iso bs=32M"
> > > > gives:
> > > > * without kerberos: 651165696 bytes (651 MB) copied, 10.7168 s, 60.8
> > > > MB/s
> > > > * with kerberos: 651165696 bytes (651 MB) copied, 24.6176 s, 26.5 MB/s
> > > > 
> > > > 
> > > > On Tue, 2011-06-07 at 19:07 -0400, J. Bruce Fields wrote:
> > > > > On Fri, Jun 03, 2011 at 07:57:16PM +0200, Vladimir Elisseev wrote:
> > > > > > There's no mount command involved in timing. It's simply copy of the
> > > > > > same directory (with many small files to NFS share with and without
> > > > > > sec=krb5 mount option.
> > > > > 
> > > > > Weird.
> > > > > 
> > > > > I wonder if the krb5 principal is being mapped to a different user on
> > > > > the server side, and that's making some difference.
> > > > > 
> > > > > Still, could you give us the full details?  (Exactly what commands are
> > > > > you running, what results do you see?)
> > > > > 
> > > > > --b.
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > > > the body of a message to majordomo@vger.kernel.org
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > 

Re: NFS3 + kerberos: performance issues

[J. Bruce Fields]

On Thu, Jun 09, 2011 at 07:02:15AM +0200, Vladimir Elisseev wrote:
> This morning I've decided to redo all the tests and include some more
> statistics ( mountstat and vmstat ). However, when I exported the same
> mount point with different options I missed one _very_ important option.
> This option was presented for the export with sec=sys, but omitted with
> sec=krb5. The option is "async"... I promise to be more careful before
> posting any results.

Yep, that would explain it--thanks for letting us know.

--b.