[dm-crypt] Poor performances with nfs and Kernel 3.x

* [dm-crypt] Poor performances with nfs and Kernel 3.x
@ 2012-02-06 22:28 Mickael
  2012-02-07  8:11 ` Arno Wagner
  0 siblings, 1 reply; 13+ messages in thread
From: Mickael @ 2012-02-06 22:28 UTC (permalink / raw)
  To: dm-crypt@saout.de

Hello,
I'm writing about poor performances accessing a dm-crypt/LUKS partition using nfs.

I was running a fileserver Ubuntu_Server_11.04 (Kernel 2.6.38-13#52) with no problem.
But after an upgrade to Ubuntu_Server_11.10 (Kernel 3.0.0-15#26) reading and writing a crypted partition with nfs is very slow.

I've made different tests with a 2GB file to show the difference when the server is running a 2.6 or 3.0 kernel.
All others parameters are always the same.

First, a test using nfs to access a crypted partition (dm-crypt/LUKS/ext4) on the server:
*Svr Kernel 3.0.0-15#26  : NFS (R/W) = 30 MB/s / 33 MB/s   <-------- The problem is here !
*Svr Kernel 2.6.38-13#52 : NFS (R/W) = 81 MB/s / 53 MB/s

As you can see, with the Kernel 3.0, performances are very bad. The flow is always constant at ~30MB; there's clearly a limitation

For comparison, the same test, but this time, using nfs to read and write a non-crypted partition (ext4) (same HD of the crypted partition):
*Svr Kernel 3.0.0-15#26  : NFS (R/W) = 92 MB/s / 77 MB/s
*Svr Kernel 2.6.38-13#52 : NFS (R/W) = 111 MB/s / 74 MB/s
Using nfs without dm_crypt, performances are good with the 2 Kernels

Is dm_crypt module the problem ?
Following, a test without nfs, copying the 2GB file using a 2nd HD on the server from/to the crypted partition:
*Svr Kernel 3.0.0-15#26  : cp (from/to) the crypted partition = 64 MB/s / 57 MB/s
*Svr Kernel 2.6.38-13#52 : cp (from/to) the crypted partition = 61 MB/s / 60 MB/s
Using dm_crypt without nfs, there's no difference between the 2 Kernels

Apparently, there is a bad interaction between nfs and dm_crypt only with a 3.0 Kernel ?
What king of change has been made in the Kernel 3.x branch ?
Do you think that the dm_crypt module is involved ? Or perhaps it's a mapper/buffer problem ?
nfs seems to work correctly: no paquets are lost during transfers (I'm using default parameters, except for exportfs: async option instead of sync)
Unfortunately, I'm just an user and my knowledge is limited, especially with dm_crypt and device_mapper.
Perhaps you can help me with this part ?

Feel free to ask more informations / tests.

Regards,
Mickael

Note:
- I've build other servers with a debian Wheezy (Kernel 3.1.0-1) and a Fedora 16 (Kernel 3.2.1-3).
Both obtained the same results.

- Following, hardware/software informations about my system:

* Kernel version (from /proc/version):
Linux version 3.0.0-15-server (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012

* Environment (Server)
CPU AMD Athlon64 3500+ (1core)
1GB RAM
mobo Asus A8N nForce4
eth 1GB nForce4
HD WD 2To Green (sata_nv)

:~$ cryptsetup luksDump /dev/sda1
Version:           1
Cipher name:       aes
Cipher mode:       cbc-essiv:sha256
Hash spec:         sha1
Payload offset:    2056
MK bits:           256

:~$ mkfs.ext4 /dev/mapper/crypted -m 0.2

:~$ exportfs -v
/mnt/crypted         192.168.0.1(rw,async,wdelay,no_root_squash,no_subtree_check)

:~$ cat /proc/mount (server)
/dev/mapper/crypted /mnt/crypted ext4 rw,relatime,user_xattr,acl,barrier=1,data=ordered 0 0

:~$ cat /proc/mount (client)
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
192.168.0.10:/mnt/crypted/ /mnt/crypted nfs4 rw,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.1,minorversion=0,local_lock=none,addr=192.168.0.10 0 0

:~$ nfsiostat (after reading and writing a 2GB file)
192.168.0.10:/mnt/crypted/ mounted on /mnt/crypted:
   op/s        rpc bklog
1432.16        0.88
read:             ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
         32.037     4108.500     128.242        0 (0.0%)      66.974      94.895
write:            ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
         47.713     4116.645      86.280        0 (0.0%)      27.833     1814.244

192.168.0.10:/mnt/sda2/ mounted on /mnt/sda2:  (sda2 -> ext4 not crypted partition)
   op/s        rpc bklog
9417.74        5.37
read:             ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
        199.295     25502.509     127.964        0 (0.0%)      16.920      22.618
write:            ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
        329.141     25555.419      77.643        0 (0.0%)       3.888     507.813

:~$ cat /proc/fs/nfsd
export_features:    0x17e3f 0xf
exports:    /mnt/crypted    192.168.0.1(rw,no_root_squash,async,wdelay,no_subtree_check)
max_block_size:    131072
nfsv4gracetime:    90
nfsv4leasetime:        90
nfsv4recoverydir:    /var/lib/nfs/v4recovery
pool_stats:    0 350070842 335513464 11979013 24
pool_threads:    8
portlist:    udp/tcp=2049
supported_krb5_enctypes:    18,17,16,23,3,1,2
threads:    8
versions:    +2 +3 +4 +4.1

:~$ cat /sys/kernel/slab/dm_crypt_io/
aliases : [0]
align : [8]
alloc_calls : []
cache_dma : [0]
cpu_slabs : [1 N0=1]
ctor : []
destroy_by_rcu : [0]
free_calls : []
hwcache_align : [0]
min_partial : [7]
objects : [26 N0=26]
object_size : [152]
objects_partial : [0]
objs_per_slab : [26]
order : [0]
partial : [0]
poison : [0]
reclaim_account : [0]
red_zone : [0]
remote_node_defrag_ratio : [100]
reserved : [0]
sanity_checks : [0]
shrink : []
slabs : [1 N0=1]
slab_size : [152]
store_user : [0]
total_objects : [26 N0=26]
trace : [0]
validate : []

^ permalink raw reply	[flat|nested] 13+ messages in thread