Re: [dm-crypt] Re : Poor performances with nfs and Kernel 3.x

[Arno Wagner <arno@wagner.name>]

Have you looked at CPU load on the server? Maybe the difference is 
just the CPU-power that NFS consumes and decryption is already 
CPU-Limited before. The low non-encrypted performance,
compared to local performance, is also a bit fishy. 

Pure speculation, but maybe something is consuming CPU
on the server or the CPU is slowed-down in some way.
If this not a CPU-power issue, then the CPU should have
significant idle percentage during encrypted accesses.

Arno


On Wed, Feb 08, 2012 at 03:04:57PM +0000, Mickael wrote:
> >De?: Arno Wagner <arno@wagner.name>
> >??: dm-crypt@saout.de
> >Cc?: 
> >Envoy? le : Mardi 7 f?vrier 2012 9h33
> >Objet?: Re: [dm-crypt] Poor performances with nfs and Kernel 3.x
> >
> >O.k., no such obervation with 3.2.2 on client and server. I got
> >
> >root /gate/tmp>cat ttt | wcs > /dev/null
> >read:?  2.147 GB [?  2147483648 B]? ? avg:? 69.274 MB/sec [? ? 31 sec]
> >
> >(wcs, a.k.a. wc-stream is a small tool I wrote to do real-time
> >monitoring of pipeline throughput and byte count, sources
> >below.) This is with some ramp-up and almost 100% CPU load
> >on the (slower) server. 
> >
> >I noticed though that nfsiostat is not the right tool to measure, as
> >it gives you performance over the whole time the device has been 
> >mounted, i.e. the throughput number keeps being updated in
> >real-time. 
> >
> >Maybe re-run with wcs as shown above (compile instructions are in
> >the header of wcs.c) and also monitor CPU usage on the server
> >while this is rrunning, e.g. with "top". I am not saying you 
> >imagine the issue, but lets be sure the measurement is good.
> >
> >If you do not want to compile anything, you could also use
> >something like "time cat ttt > /dev/null" and calculate
> >throughput manually. 
> >
> >Arno
> >
> >----- CUTed -----
> >
> 
> 
> Hello Arno,
> Here are the new tests with wcs: results are still the same.
> 
> /tmp is the tmp directory on the client (7200 rpm HD)
> /CRYPT is a crypted partition on the server (Green HD ~5600 rpm)
> /SAV is a non-crypted partition on the server (Green HD ~5600 rpm)
> CPU monitored with htop
> 
> 
> *** Server with Kernel 3.0.0-15-server #26
> 
> * From client, using NFS on crypted partition:
> 
> cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 41.476 MB/sec [??? 49 sec]
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 29.454 MB/sec [??? 69 sec]??? <---------------
> 
> -----------
> 
> * From client, using NFS on non-crypted partition
> 
> cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> 
> cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 65.559 MB/sec [??? 31 sec]
> 
> -----------
> 
> 
> * From Server:
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> 
> cat /mnt/SAV/testfile.avi | wcs > /dev/null
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]
> 
> -----------
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 56.454 MB/sec [??? 36 sec]
> 
> cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]
> 
> 
> 
> ===============================
> 
> *** Server with Kernel 2.6.38-13-server #52
> 
> * From client, using NFS on crypted partition:
> 
> cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]??? CPU=100%
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec]??? CPU=98%??? ??? <-------------------
> 
> -----------
> 
> * From client, using NFS on non-crypted partition
> 
> cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=50%
> 
> cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=25%
> 
> 
> 
> * From Server:
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 84.680 MB/sec [??? 24 sec]??? CPU=98%
> 
> cat /mnt/SAV/testfile.avi | wcs > /dev/null
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]??? CPU=25%
> 
> -----------
> 
> cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 52.111 MB/sec [??? 39 sec]??? CPU=50%-100%
> 
> cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi 
> ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 47.264 MB/sec [??? 43 sec]??? CPU=75%-100%
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell