From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail1.tehlulz.com (www.tehlulz.com [67.18.187.246]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Sun, 1 May 2011 01:45:06 +0200 (CEST) Received: from [166.70.195.196] (heidegger.mj.utho1.B8.BZ [166.70.195.196]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mdye@divisive.info) by mail1.tehlulz.com (Postfix) with ESMTPSA id 1618274330 for ; Sat, 30 Apr 2011 18:38:33 -0500 (CDT) Message-ID: <4DBC9D78.8070203@divisive.info> Date: Sat, 30 Apr 2011 17:38:32 -0600 From: michael dye MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [dm-crypt] LUKS performance penalty Reply-To: m-dm-crypt@divisive.info List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dm-crypt@saout.de I find that using LUKS results in a tremendous performance penalty: 156MB/s writes with LUKS v. 613MB/s without on a PCI-E SSD using ext4. Am I failing to set up LUKS correctly? What can I do to improve performance? I am using kernel 2.6.36, 64bit userspace, and LUKS via cryptsetup 1.30 on a Q6600 w/ 4GB of RAM. I'm using bonnie++ for testing while the machine is at idle. I made a volume group and logical volume (using lvm2 v.2.02) on an 80gb OCZ revodrive, put LUKS on top of that, and then did a naive ext4 format. Here is a record of my setup steps: pvcreate /dev/mapper/sil_bgbiejcbajdj3 vgcreate Storage /dev/mapper/sil_bgbiejcbajdj3; lvcreate -l100%FREE -n STORAGE Storage cryptsetup luksFormat /dev/mapper/Storage-STORAGE --key-file=/root/storage_tmp.key cryptsetup luksOpen /dev/mapper/Storage-STORAGE local_storage --key-file=/root/storage_tmp.key mkfs.ext4 /dev/mapper/Storage-STORAGE mount /dev/mapper/Storage-STORAGE /mnt/local_storage/; mkdir /mnt/local_storage/file_perf; chown 87:87 /mnt/local_storage/file_perf Here is the output from bonnie++ (bonnie++ -d /mnt/local_storage/file_perf -f -r 4096 -u 87:87): Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP h_ 4G 82203 11 46437 6 156632 9 +++++ +++ Latency 3175ms 2090ms 5362us 4344us Version 1.96 ------Sequential Create------ --------Random Create-------- h_ -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 106us 483us 625us 165us 78us 87us ... and the results with ext4 directly on the lv: Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP h_ 4G 457214 66 202553 32 613262 34 +++++ +++ Latency 226ms 157ms 5418us 941us Version 1.96 ------Sequential Create------ --------Random Create-------- h_ -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 726us 530us 586us 113us 11us 46us I notice similar degradation on a different storage system on the same box. There I put 2 SAS 15K cheetahs in RAID0, created a vg and lv on that, added LUKS, and formatted with XFS. With LUKS I got 127MB/s writes, and without I got 215MB/s. Am I misunderstanding something or somehow failing to configure this? Do others experience this kind of performance hit? I thought at first it could be explained by the overhead of performing crypto calculations, but my processor's cores are quite underutilized during my tests. Any help that can be offered is greatly appreciated: I use LUKS on 4 boxes and all seem to suffer similarly. Finding a solution would be a big help to me. -mike dye