Re: Some impossible benchmark results with LUKS - what am I missing?

[Christian Robert <christian.robert@polymtl.ca>]

Only thing I can see is Lurk is not passing the barrier (do that writes in sequence then ack) to the devices.
so it act as a caching device (dangerous for btrfs point of view).

I'm *no* expert in that matter at all.

I may be wrong.

Xtian.

On 2014-03-26 21:56, Evan Powell wrote:
> Order did not matter - I have run a few tests showing this same disparity in both orders.
>
> I reran this evening using the drop_caches command you provided, no change in throughput results. The improvement I've seen has been present through multiple reboots so far, and clearing RAM using dd on different test files. Doesn't look like it's caching causing this, so I'm still stumped.
>
> Output of cryptsetup luksDump is similar on all 4 devices:
>
> Version:       	1
> Cipher name:   	aes
> Cipher mode:   	xts-plain64
> Hash spec:     	sha1
> Payload offset:	4102
> MK bits:       	256
> ...
> (followed by different iterations and salts)
>
> (Offtopic: migrating between encrypted and unencrypted drives is really painless on btrfs: rebalance to raid1, delete two devices, remove their encrypted volumes, readd the devices, remove and readd the 2 remaining encrypted disks without encryption, rebalance to raid10. All live, without losing data! This is awesome!)
>
> Evan Powell | Technical Lead
> epowell@zenoss.com
>
> ----- Original Message -----
>
> From: "Matt" <jackdachef@gmail.com>
> To: "Evan Powell" <epowell@zenoss.com>
> Cc: "linux-btrfs" <linux-btrfs@vger.kernel.org>
> Sent: Wednesday, March 26, 2014 4:47:19 PM
> Subject: Some impossible benchmark results with LUKS - what am I missing?
>
>> Hey folks,
>
>> I have been experimenting with btrfs on a home NAS box, and have some benchmark \
>> results that I don't believe. I'm hoping someone here has some insight on what I've \
>> missed that would cause such a result.
>
>> The short version: I'm seeing roughly a 36% (on write) to 55% (on read) performance \
>> *improvement* when using btrfs on LUKS containers over btrfs on raw disks. This \
>> should not be the case!
>
>> The test setup:
>> My test file is a large file that I previously generated from /dev/urandom. I saw \
>> similar results using /dev/zero as input, as well as a repeated copy of the whole \
>> Ubuntu 14.04 ISO (i.e., real-ish data).
>
>> My calculated MB/s numbers are based on the 'real' time in the output.
>
>> Live-booted Ubuntu 14.04 (nightly from 3/11/14, kernel 3.13.5)
>> 4x 4TB WD Red drives in standard (no RAID) configuration
>> i3-4130 CPU (has AES-NI for accelerated encryption)
>> Default BTRFS options from the disk gui, always raid10.
>
>> Tested configurations:
>> Raw: btrfs raid10 on 4x raw drives
>> Encrypted: btrfs raid10 on 4 separate LUKS containers on 4x raw drives (default LUKS \
>> options)
>
>
>> Read command: $ time sh -c "dd if=test.out of=/dev/null bs=4k"
>> Raw:
>> 5120000+0 records in
>> 5120000+0 records out
>> 20971520000 bytes (21 GB) copied, 149.841 s, 140 MB/s
>
>> real 2m29.849s
>> user 0m2.764s
>> sys 0m7.064s
>
>> = 133.467690809 MB/s
>
>> Encrypted:
>> $ time sh -c "dd if=test2.out of=/dev/null bs=4k"
>> 5120000+0 records in
>> 5120000+0 records out
>> 20971520000 bytes (21 GB) copied, 96.6127 s, 217 MB/s
>
>> real 1m36.627s
>> user 0m3.331s
>> sys 0m9.518s
>
>> = 206.981485506 MB/s
>
>
>> Read+Write: $ time sh -c "dd if=test2.out of=test20grand.out bs=4k && sync"
>> Raw:
>> 5120000+0 records in
>> 5120000+0 records out
>> 20971520000 bytes (21 GB) copied, 227.069 s, 92.4 MB/s
>
>> real 3m49.701s
>> user 0m2.854s
>> sys 0m15.936s
>
>> = 87.069712365 MB/s
>
>> Encrypted:
>> 5120000+0 records in
>> 5120000+0 records out
>> 20971520000 bytes (21 GB) copied, 167.823 s, 125 MB/s
>
>> real 2m48.784s
>> user 0m2.955s
>> sys 0m17.956s
>
>> = 118.494644042 MB/s
>
>
>> Any ideas what could explain this result?
>
>> One coworker suggested that perhaps the LUKS container was returning from 'sync' \
>> early, before actually finishing the write to disk. This would seem to violate the \
>> assumptions of the 'sync' primitive, so I have my doubts.
>
>> I'm also interested in learning how I can reliably benchmark the real cost of running \
>> full-disk encryption under btrfs on my system.
>
>> Thanks!
>
>
>> Evan Powell | Technical Lead
>> epowell@zenoss.com
>
>
> Hi Evan,
>
> just to be sure:
>
> did you do a
>
> echo 3 > /proc/sys/vm/drop_caches
>
> before *each* test ?
>
> also try reversing the order of tests like so:
>
> Encrypted
> RAW
>
> whether that makes a difference
>
>
> It would also be interesting to see the output of
>
> cryptsetup luksDump
>
> and
>
> Version: *
> Cipher name: *
> Cipher mode: *
> Hash spec: *
>
>
>
>
> Interesting find indeed ! Thanks for sharing the finding
>
> I'm currently using Btrfs on an encrypted system partition (without
> AES-NI supported hardware) and things already feel and are faster than
> with ext4
>
> we need to find out what this magic is =)
>
>
> Kind Regards
>
>
> Thanks
>
> Matt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>