[dm-crypt] LUKS performance penalty

[michael dye <m-dm-crypt@divisive.info>]

I find that using LUKS results in a tremendous performance penalty: 
156MB/s writes with LUKS v. 613MB/s without on a PCI-E SSD using ext4.  
Am I failing to set up LUKS correctly?  What can I do to improve 
performance?

I am using kernel 2.6.36, 64bit userspace, and LUKS via cryptsetup 1.30 
on a Q6600 w/ 4GB of RAM.  I'm using bonnie++ for testing while the 
machine is at idle.  I made a volume group and logical volume (using 
lvm2 v.2.02) on an 80gb OCZ revodrive, put LUKS on top of that, and then 
did a naive ext4 format.  Here is a record of my setup steps:

pvcreate /dev/mapper/sil_bgbiejcbajdj3
vgcreate Storage /dev/mapper/sil_bgbiejcbajdj3; lvcreate -l100%FREE -n 
STORAGE Storage
cryptsetup luksFormat /dev/mapper/Storage-STORAGE 
--key-file=/root/storage_tmp.key
cryptsetup luksOpen /dev/mapper/Storage-STORAGE local_storage 
--key-file=/root/storage_tmp.key
mkfs.ext4 /dev/mapper/Storage-STORAGE
mount /dev/mapper/Storage-STORAGE /mnt/local_storage/; mkdir 
/mnt/local_storage/file_perf; chown 87:87 /mnt/local_storage/file_perf

Here is the output from bonnie++ (bonnie++ -d 
/mnt/local_storage/file_perf -f -r 4096 -u 87:87):

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
h_               4G           82203  11 46437   6           156632   9 
+++++ +++
Latency                        3175ms    2090ms              5362us    
4344us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
h_                  -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
+++++ +++
Latency               106us     483us     625us     165us      78us      
87us

... and the results with ext4 directly on the lv:

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
h_               4G           457214  66 202553  32           613262  34 
+++++ +++
Latency                         226ms     157ms              5418us     
941us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
h_                  -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
+++++ +++
Latency               726us     530us     586us     113us      11us      
46us

I notice similar degradation on a different storage system on the same 
box.  There I put 2 SAS 15K cheetahs in RAID0, created a vg and lv on 
that, added LUKS, and formatted with XFS.  With LUKS I got 127MB/s 
writes, and without I got 215MB/s.

Am I misunderstanding something or somehow failing to configure this?  
Do others experience this kind of performance hit?  I thought at first 
it could be explained by the overhead of performing crypto calculations, 
but my processor's cores are quite underutilized during my tests.  Any 
help that can be offered is greatly appreciated: I use LUKS on 4 boxes 
and all seem to suffer similarly.  Finding a solution would be a big 
help to me.

-mike dye