Re: [dm-crypt] Encrypting with larger packet size (+some experimental patch)

[Michael Stapelberg <michael+dmcrypt@stapelberg.de>]

Hi Milan,

On Mon, 28 Jan 2013 13:00:07 +0100
gmazyland at gmail.com (Milan Broz) wrote:
> I need some real reason before submitting such patch.
I am running a qnap TS-119P II NAS with Debian GNU/Linux. The device
has a CESA crypto engine, which is supported by the in-mainline-kernel
mv_cesa module. Unfortunately, this crypto engine exhibits the same
symptoms as the one the author of this thread was describing:

With small block sizes, the crypto engine is really slow.

I have done a series of tests with a WD WD3000FYYZ SATA hard disk drive
that achieves a raw read/write rate of about 177 MiB/s:

dd if=/dev/sda2 of=/dev/null bs=1M count=4096 iflag=direct
4294967296 bytes (4.3 GB) copied, 24.2781 s, 177 MB/s

dd if=/dev/zero of=/dev/sda2 bs=1M count=768 oflag=direct
805306368 bytes (805 MB) copied, 4.5337 s, 178 MB/s

Using mv_cesa with aes-cbc-essiv:sha256 (one of the accelerated
ciphers, as cryptsetup benchmark confirms), I get a rate of about 12.9
MiB/s:

dd if=/dev/zero of=/dev/sda2_crypt bs=1M count=4096 oflag=direct
4294967296 bytes (4.3 GB) copied, 332.425 s, 12.9 MB/s

There is a patch by Phil Sutter which makes mv_cesa use the TDMA engine
of that SoC, and it improves performance quite a bit:

dd if=/dev/zero of=/dev/mapper/sda2_crypt bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 50.5507 s, 21.2 MB/s

This is just to let you know where I am coming from :).

> That said... here is the patch which can use larger block size
> option. It is not complete (mainly IV generators need changes and I
> am afraid there should be morech changes than basic io hints) but it
> should work for basic performance testing if you want to play with it
> (you have to use dmsetup).
> 
> So show your crypto performance numbers now :-)
Thanks for posting this patch. I have set up my device as follows:

echo "0 5858578432 crypt aes-cbc-essiv:sha256
0123456789abcdef0123456789abcdef 0 /dev/sda2 0 2 block_size 4096" |
dmsetup create sda2_crypt

The encryption rate improves significantly:

dd if=/dev/zero of=/dev/mapper/sda2_crypt bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 26.4864 s, 40.5 MB/s

I hope you realize that this is a real-world usecase for me — I cannot
switch to a full-blown Intel system with AES-NI due to space and power
efficiency reasons. Nevertheless, I need to encrypt my hard disk(s).

I would appreciate it very much if you could implement the FIXME and
commit that patch to dmcrypt.

BTW, when using a block_size of more than 4096 (e.g. 8192),
I get a kernel oops:

[   36.601704] device-mapper: ioctl: 4.23.1-ioctl (2012-12-18) initialised: dm-devel@redhat.com
[   36.658525] bio: create slab <bio-1> at 1
[   36.672332] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[   36.680496] pgd = decf8000
[   36.683212] [00000000] *pgd=1eeed831, *pte=00000000, *ppte=00000000
[   36.689537] Internal error: Oops: 17 [#1] ARM
[   36.693911] Modules linked in: sha256_generic dm_crypt dm_mod nfsd auth_rpcgss nfs_acl nfs lockd dns_resolver fscache sunrpc hmac sha1_generic ehci_hcd mv_cesa mv643xx_eth usbcore inet_lro libphy usb_common mv_dma evdev loop gpio_keys autofs4 ext4 jbd2 mbcache sd_mod crc_t10dif sata_mv libata scsi_mod
[   36.720988] CPU: 0    Not tainted  (3.8-trunk-kirkwood #1 Debian 3.8-1~experimental.2)
[   36.728940] PC is at create_empty_buffers+0x24/0x11c
[   36.733921] LR is at create_empty_buffers+0x14/0x11c
[   36.738899] pc : [<c00f44ec>]    lr : [<c00f44dc>]    psr: 80000013
[   36.738899] sp : de8e5d10  ip : c04eb02c  fp : c093d280
[   36.750416] r10: 00100100  r9 : df6370b8  r8 : c093d280
[   36.755658] r7 : dea39c80  r6 : 00000000  r5 : 00000000  r4 : c093d280
[   36.762206] r3 : 00000000  r2 : 00000001  r1 : 00002000  r0 : 00000000
[   36.768756] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   36.775914] Control: 0005397f  Table: 1ecf8000  DAC: 00000015
[   36.781678] Process blkid (pid: 931, stack limit = 0xde8e41b8)
[   36.787530] Stack: (0xde8e5d10 to 0xde8e6000)
[   36.796608] 5d00:                                     c093d280 df63717c 00000000 c00f4ad8
[   36.806338] 5d20: 00000001 c00f4b14 c0502024 00000040 00000000 c055af38 00000010 000213d0
[   36.816046] 5d40: 00000000 c0501a24 df637180 c00f9850 c093d280 df63717c 00000000 2ba657f0
[   36.825732] 5d60: 000000d0 00200200 00100100 c0092b4c 60000013 00000001 df63717c 00000000
[   36.833978] 5d80: dea39c80 c093d280 00200200 00100100 de8e5da0 c009b434 2ba657ff 2ba657f0
[   36.844821] 5da0: de8e5da0 de8e5da0 91827364 de8e5dac de8e5dac de8e5db4 de8e5db4 00000000
[   36.853045] 5dc0: c057bee0 00000000 00000001 2ba657f0 00000001 df63717c 000001ff dea39c80
[   36.863832] 5de0: 00000000 c009b6c4 00000000 dea39c80 2ba657f0 00000000 00000001 00000000
[   36.873483] 5e00: 00000000 2ba657f0 df63717c dea39c80 ffffffff c0093d08 00000001 00000000
[   36.883145] 5e20: dea39c80 de8e5ec0 00000101 de8e5f00 657f0000 000002ba 00000fff b6f03000
[   36.892789] 5e40: 000005b7 de8e5eb8 def41520 def6a1d8 decfadb8 00000028 00000000 de8e5ef8
[   36.902532] 5e60: df6370b8 dea39cc0 2ba657f1 00000001 b6f031f0 00000000 00000000 00000040
[   36.912233] 5e80: 0096ca78 00000000 c00f99e4 de8e5ec0 fffffdee de8e5f80 00000040 dea39c80
[   36.921889] 5ea0: de8e4000 de8e5ec0 00000000 c00cd8b0 657f0000 000002ba 0096ca78 00000040
[   36.931551] 5ec0: 00000000 00000000 00000000 00000001 ffffffff dea39c80 00000000 00000000
[   36.941233] 5ee0: 00000000 00000000 dee603c0 00000000 00000000 00000000 657f0000 000002ba
[   36.950920] 5f00: 00000000 00000000 00000040 00000000 00000040 00000000 00000000 00000000
[   36.960574] 5f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   36.970268] 5f40: 0096ca78 dea39c80 0096ca78 de8e5f80 00000000 00000040 00000040 c00cdfb4
[   36.979934] 5f60: dea39c80 0096ca78 657f0000 000002ba dea39c80 00000000 0096ca78 c00ce0c4
[   36.989635] 5f80: 657f0000 000002ba 00000040 0096ca70 00000040 0096c018 00000003 c000e508
[   36.999297] 5fa0: 00000000 c000e380 0096ca70 00000040 00000003 0096ca78 00000040 00008000
[   37.008957] 5fc0: 0096ca70 00000040 0096c018 00000003 657f0000 b6fc7000 0096ca58 00000000
[   37.018632] 5fe0: 0096c064 bee91370 b6fac760 b6f0320c 60000010 00000003 00000000 00000000
[   37.026882] [<c00f44ec>] (create_empty_buffers+0x24/0x11c) from [<c00f4ad8>] (create_page_buffers+0x34/0x4c)
[   37.036764] [<c00f4ad8>] (create_page_buffers+0x34/0x4c) from [<c00f4b14>] (block_read_full_page+0x24/0x358)
[   37.046646] [<c00f4b14>] (block_read_full_page+0x24/0x358) from [<c009b434>] (__do_page_cache_readahead+0x194/0x1ec)
[   37.057230] [<c009b434>] (__do_page_cache_readahead+0x194/0x1ec) from [<c009b6c4>] (force_page_cache_readahead+0x6c/0xa4)
[   37.068240] [<c009b6c4>] (force_page_cache_readahead+0x6c/0xa4) from [<c0093d08>] (generic_file_aio_read+0x290/0x6dc)
[   37.078906] [<c0093d08>] (generic_file_aio_read+0x290/0x6dc) from [<c00cd8b0>] (do_sync_read+0x90/0xcc)
[   37.088348] [<c00cd8b0>] (do_sync_read+0x90/0xcc) from [<c00cdfb4>] (vfs_read+0xa4/0x17c)
[   37.096569] [<c00cdfb4>] (vfs_read+0xa4/0x17c) from [<c00ce0c4>] (sys_read+0x38/0x64)
[   37.104447] [<c00ce0c4>] (sys_read+0x38/0x64) from [<c000e380>] (ret_fast_syscall+0x0/0x2c)
[   37.112924] Code: e1a05000 e1a03000 ea000000 e1a03001 (e5932000) 
[   37.119055] ---[ end trace c844ca55a4ae58f6 ]---


-- 
Best regards,
Michael