Re: [dm-crypt] Poor performances with nfs and Kernel 3.x

[Arno Wagner <arno@wagner.name>]

On Mon, Feb 06, 2012 at 10:28:24PM +0000, Mickael wrote:
> Hello,
>
> I'm writing about poor performances accessing a dm-crypt/LUKS partition
> using nfs.
> 
> I was running a fileserver Ubuntu_Server_11.04 (Kernel 2.6.38-13#52) with
> no problem.  But after an upgrade to Ubuntu_Server_11.10 (Kernel
> 3.0.0-15#26) reading and writing a crypted partition with nfs is very
> slow.

Interesting. I think have no issues here with Debian stable and Kernel
3.2.2 on the server and 3.1.6 on the client. 

What is the kernel version on your client?

3.0.0 is not a very good kernel though with a number of issues. 
There is also a local privilege excalation in 3.x that has been 
fixed only in 3.2.2 and later. (Unless Ubuntu did it.)

> 
> I've made different tests with a 2GB file to show the difference when the
> server is running a 2.6 or 3.0 kernel.  All others parameters are always
> the same.
> 
> First, a test using nfs to access a crypted partition (dm-crypt/LUKS/ext4)
> on the server:
>
> *Svr Kernel 3.0.0-15#26? : NFS (R/W) = 30 MB/s / 33 MB/s?? <-------- The problem is here !
> *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 81 MB/s / 53 MB/s
>
> As you can see, with the Kernel 3.0, performances are very bad. The flow
> is always constant at ~30MB; there's clearly a limitation

What did you use to do the benchmark? 

> For comparison, the same test, but this time, using nfs to read and write
> a non-crypted partition (ext4) (same HD of the crypted partition):
>
> *Svr Kernel 3.0.0-15#26? : NFS (R/W) = 92 MB/s / 77 MB/s
> *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 111 MB/s / 74 MB/s
> Using nfs without dm_crypt, performances are good with the 2 Kernels

Interesting.

I will do a benchmark here and see whether I have the same.

Arno


> 
> Is dm_crypt module the problem ?  Following, a test without nfs, copying
> the 2GB file using a 2nd HD on the server from/to the crypted partition:
>
> *Svr Kernel 3.0.0-15#26? : cp (from/to) the crypted partition = 64 MB/s / 57 MB/s
> *Svr Kernel 2.6.38-13#52 : cp (from/to) the crypted partition = 61 MB/s / 60 MB/s
> Using dm_crypt without nfs, there's no difference between the 2 Kernels
> 
> Apparently, there is a bad interaction between nfs and dm_crypt only with
> a 3.0 Kernel ?
>
> What king of change has been made in the Kernel 3.x branch ?
>
> Do you think that the dm_crypt module is involved ? Or perhaps it's a
> mapper/buffer problem ?  nfs seems to work correctly: no paquets are lost
> during transfers (I'm using default parameters, except for exportfs: async
> option instead of sync) Unfortunately, I'm just an user and my knowledge
> is limited, especially with dm_crypt and device_mapper.  Perhaps you can
> help me with this part ?
>
> Feel free to ask more informations / tests.
> 
> Regards,
> Mickael
> 
> 
> Note:
> - I've build other servers with a debian Wheezy (Kernel 3.1.0-1) and a Fedora 16 (Kernel 3.2.1-3).
> Both obtained the same results.
> 
> - Following, hardware/software informations about my system:
> 
> * Kernel version (from /proc/version):
> Linux version 3.0.0-15-server (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012
> 
> 
> * Environment (Server)
> CPU AMD Athlon64 3500+ (1core)
> 1GB RAM
> mobo Asus A8N nForce4
> eth 1GB nForce4
> HD WD 2To Green (sata_nv)
> 
> 
> :~$ cryptsetup luksDump /dev/sda1
> Version:?????? ??? 1
> Cipher name:?? ??? aes
> Cipher mode:?? ??? cbc-essiv:sha256
> Hash spec:???? ??? sha1
> Payload offset:??? 2056
> MK bits:?????? ??? 256
> 
> 
> :~$ mkfs.ext4 /dev/mapper/crypted -m 0.2
> 
> 
> :~$ exportfs -v
> /mnt/crypted???? ??? 192.168.0.1(rw,async,wdelay,no_root_squash,no_subtree_check)
> 
> 
> :~$ cat /proc/mount (server)
> /dev/mapper/crypted /mnt/crypted ext4 rw,relatime,user_xattr,acl,barrier=1,data=ordered 0 0
> 
> :~$ cat /proc/mount (client)
> nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
> 192.168.0.10:/mnt/crypted/ /mnt/crypted nfs4 rw,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.1,minorversion=0,local_lock=none,addr=192.168.0.10 0 0
> 
> 
> :~$ nfsiostat (after reading and writing a 2GB file)
> 192.168.0.10:/mnt/crypted/ mounted on /mnt/crypted:
> ?? op/s??? ??? rpc bklog
> 1432.16 ??? ?? 0.88
> read:???????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms)
> ??? ??? ?32.037 ??? 4108.500 ??? 128.242??????? 0 (0.0%) ??? ?66.974 ??? ?94.895
> write:??????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms)
> ??? ??? ?47.713 ??? 4116.645 ??? ?86.280??????? 0 (0.0%) ??? ?27.833 ??? 1814.244
> 
> 192.168.0.10:/mnt/sda2/ mounted on /mnt/sda2:? (sda2 -> ext4 not crypted partition)
> ?? op/s??? ??? rpc bklog
> 9417.74 ??? ?? 5.37
> read:???????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms)
> ??? ??? 199.295 ??? 25502.509 ??? 127.964??????? 0 (0.0%) ??? ?16.920 ??? ?22.618
> write:??????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms)
> ??? ??? 329.141 ??? 25555.419 ??? ?77.643??????? 0 (0.0%) ??? ? 3.888 ??? 507.813
> 
> 
> :~$ cat /proc/fs/nfsd
> export_features:??? 0x17e3f 0xf
> exports:??? /mnt/crypted??? 192.168.0.1(rw,no_root_squash,async,wdelay,no_subtree_check)
> max_block_size:??? 131072
> nfsv4gracetime:??? 90
> nfsv4leasetime:??? ??? 90
> nfsv4recoverydir:??? /var/lib/nfs/v4recovery
> pool_stats:??? 0 350070842 335513464 11979013 24
> pool_threads:??? 8
> portlist:??? udp/tcp=2049
> supported_krb5_enctypes:??? 18,17,16,23,3,1,2
> threads:??? 8
> versions:??? +2 +3 +4 +4.1
> 
> 
> :~$ cat /sys/kernel/slab/dm_crypt_io/
> aliases : [0]
> align : [8]
> alloc_calls : []
> cache_dma : [0]
> cpu_slabs : [1 N0=1]
> ctor : []
> destroy_by_rcu : [0]
> free_calls : []
> hwcache_align : [0]
> min_partial : [7]
> objects : [26 N0=26]
> object_size : [152]
> objects_partial : [0]
> objs_per_slab : [26]
> order : [0]
> partial : [0]
> poison : [0]
> reclaim_account : [0]
> red_zone : [0]
> remote_node_defrag_ratio : [100]
> reserved : [0]
> sanity_checks : [0]
> shrink : []
> slabs : [1 N0=1]
> slab_size : [152]
> store_user : [0]
> total_objects : [26 N0=26]
> trace : [0]
> validate : []
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell