[dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Karol Babioch]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

Hi,

I'm currently reencrypting my (hardware) RAID setup, which is quite big
(12 TB) in order to change the cipher from aes-xts-plain to aes-xts-plain64.

While playing around with various options, I found out that a block size
of 64 MB and the use of direct I/O results in the highest speed for me.

Now, I admit that at least the "optimal" block size depends very much on
your setup and the caches and/or buffers involved.

However I would argue that enabling direct I/O should be faster on most
systems considering that usual block devices are probably quite big - at
least compared to "normal" files and reencrypting the device is a purely
consecutive process.

Running some tests I could confirm my assumption. In my case where I've
got a hardware RAID with read ahead enabled, it makes at least a
difference of about 10%.

Is there a reason why direct I/O is not enabled by default? Maybe I'm
missing something obvious here? Personally I think that 10% can be quite
much, especially when the process takes 24 hours and more.

That said I think the bigger influence is the choice of the right block
size. By choosing the right block size I could double the speed. As
already said above I think it probably is quite hard to get this one
right automatically for each and everyone, because it depends upon
various caches involved. From a theoretical stand point it probably
should be about half the size of the smallest cache involved. I'm not
sure whether it makes any sense (or is even possible) to probe for these
things, but considering the speed enhancement of - at least in my case -
100%, there should probably be something done about it.

Now, before implementing any of this, I would like to know whether
you've got similar and/or even contradicting experiences and whether
there are specific reasons for the default values of the options
mentioned above. Maybe I'm just generalizing too much here when trying
to come up with some conclusions based on my hardware.

Best regards,
Karol Babioch


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Arno Wagner]

On Fri, Nov 23, 2012 at 05:12:35AM +0100, Karol Babioch wrote:
> Hi,
> 
> I'm currently reencrypting my (hardware) RAID setup, which is quite big
> (12 TB) in order to change the cipher from aes-xts-plain to aes-xts-plain64.
> 
> While playing around with various options, I found out that a block size
> of 64 MB and the use of direct I/O results in the highest speed for me.

This seems to be RAID options (i.e. out-of-scope for cryptsetup), and 
very specific ones as you have some hardware RAID solution. Also 64MB 
block size is very, very large and will be a problem for some workloads.
 
Also, your sped gain of 10% is negligible, as it is likely not robust 
and very much dependent on the workload.

[...]
> Maybe I'm just generalizing too much here when trying
> to come up with some conclusions based on my hardware.

I think you do. I also think that HW raid is a bit exotic on Linux
these days, as SW RAID gives better features and often better
performance.

Arno

--
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Milan Broz]

On 11/23/2012 07:07 AM, Arno Wagner wrote:

> I think you do. I also think that HW raid is a bit exotic on Linux
> these days, as SW RAID gives better features and often better
> performance.

If you work in enterprise environment, this is definitely not true :)

And for some strange reasons, cloud environments are more
and more popular where "elastic" storage means usually some
virtualized HW storage array.

Also there are SSD drives which uses internally RAID0 to speed
up operations (in reality it is more independent SSD drives inside).

Milan

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Arno Wagner]

On Fri, Nov 23, 2012 at 10:00:47AM +0100, Milan Broz wrote:
> On 11/23/2012 07:07 AM, Arno Wagner wrote:
> 
> > I think you do. I also think that HW raid is a bit exotic on Linux
> > these days, as SW RAID gives better features and often better
> > performance.
> 
> If you work in enterprise environment, this is definitely not true :)

It depends. I have used a 12-way SAS RAID controller as 
dumb controller with software-RAID and have gotten better 
results. But that was one with Linux support that sucked.
With good Linux support, that may be entirely different.

On the exotic, I agree that in enterprise environments
HW controllers are the norm, but I think by now that is
mainly because they are simpler to use.
 
> And for some strange reasons, cloud environments are more
> and more popular where "elastic" storage means usually some
> virtualized HW storage array.

Ah, yes. The "cloud" stuff, where all your data turns into
hot vapor. I really do not get why people think using
even more complex environments that they do not even fully
control is a good idea when we obviously have not even mastered
simpler systems...
 
> Also there are SSD drives which uses internally RAID0 to speed
> up operations (in reality it is more independent SSD drives inside).

Ok, but I think _that_ does not count, unless you know of some
that expose the controller. ;-)

Arno
--
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Milan Broz]

On 11/23/2012 10:27 AM, Arno Wagner wrote:
>> Also there are SSD drives which uses internally RAID0 to speed
>> up operations (in reality it is more independent SSD drives inside).
> 
> Ok, but I think _that_ does not count, unless you know of some
> that expose the controller. ;-)

just FYI
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=99a700bcc75429ba84a672d04f0b650dcc5b3042

All these drives in manline linux (with patch above) are visible as separate devices
(separate controllers) while with proprietary driver you see one big device
(which is raid0 internally in fact)

I was able to mount it with mdadm with 64k chunk size later (with separate drives).

Its fw seems to be optimized for the raid0 case though...

Milan

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Sven Eschenberg]

On Fri, November 23, 2012 07:07, Arno Wagner wrote:
> On Fri, Nov 23, 2012 at 05:12:35AM +0100, Karol Babioch wrote:
>> Hi,
>>
>> I'm currently reencrypting my (hardware) RAID setup, which is quite big
>> (12 TB) in order to change the cipher from aes-xts-plain to
>> aes-xts-plain64.
>>
>> While playing around with various options, I found out that a block size
>> of 64 MB and the use of direct I/O results in the highest speed for me.
>
> This seems to be RAID options (i.e. out-of-scope for cryptsetup), and
> very specific ones as you have some hardware RAID solution. Also 64MB
> block size is very, very large and will be a problem for some workloads.
>
>

On a side note, most HW Raids do not provide IO hinting, even worse, as a
result the IO hinting is wrong, which has an impact on alignment etc.,
esp. when it boils down to something special like 4k sector drives. If the
RAID stripes are not aligned to the 4k boundaries of the drive, and
dmcrypt cannot align to the 4k boundary as well as the stripe size, you
will suffer a strong degradation in performance.

BTW, what exactly are you referring to, when you talk about 64 MB blocksize?


>
> [...]
>
> Arno
>
> --
> Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
> GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D
> 9718
> ----
> One of the painful things about our time is that those who feel certainty
> are stupid, and those with any imagination and understanding are filled
> with doubt and indecision. -- Bertrand Russell
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

-Sven

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Milan Broz]

On 11/24/2012 06:01 PM, Sven Eschenberg wrote:

> BTW, what exactly are you referring to, when you talk about 64 MB blocksize?

Here cryptsetup-reencrypt is in principle simple program, it reads a "block"
and write it back to device with new encryption parameters (and optionally with
some different offset). So block here is meant as an unit which is handled in one
reencryption step.
(But the real atomic unit of encryption is still 512B block of course.)

There is no requirement this block to be aligned to underlying hw alignment
(but if it is misaligned, the same performance degradation problems apply of course).

Milan

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[DarKRaveR]

Sorry for the late reply,

just a small followup. What you are saying Milan, is absolutely true,
but for misaligned HW Raid setups, a huge block size together with
O_DIRECT will bypass the OS'es caching strategy and the controller
(knowing the actual layout) has a way better chance of compensating
unnecessary IO with it's optimized write back caching (in the linear
read-modify-write case of reenrypt). For rather small block sizes (near
the size of a RAID stripe) in a misaligned case, such a
compensation/elimination of unneeded IO is much harder.

So small blocksize, lack of topology information, misalignment -> approx
50% loss in performance (as surveilled by opener)

Regards

-Sven


On Sat, 2012-11-24 at 21:59 +0100, Milan Broz wrote:
> On 11/24/2012 06:01 PM, Sven Eschenberg wrote:
> 
> > BTW, what exactly are you referring to, when you talk about 64 MB blocksize?
> 
> Here cryptsetup-reencrypt is in principle simple program, it reads a "block"
> and write it back to device with new encryption parameters (and optionally with
> some different offset). So block here is meant as an unit which is handled in one
> reencryption step.
> (But the real atomic unit of encryption is still 512B block of course.)
> 
> There is no requirement this block to be aligned to underlying hw alignment
> (but if it is misaligned, the same performance degradation problems apply of course).
> 
> Milan
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] Reconsidering default options for cryptsetup-reencrypt

[Milan Broz]

On 11/23/2012 05:12 AM, Karol Babioch wrote:
> However I would argue that enabling direct I/O should be faster on most
> systems considering that usual block devices are probably quite big - at
> least compared to "normal" files and reencrypting the device is a purely
> consecutive process.

But unfortunately your assumption is not correct. Not only direct hw is used,
many times you are running container from file (namely for virtual machines).

I have no problem with changing defaults but I selected current defaults
because of testing quite a lot of systems. But everything is configurable
(and you can even restart reencryption with changed options.)

> Running some tests I could confirm my assumption. In my case where I've
> got a hardware RAID with read ahead enabled, it makes at least a
> difference of about 10%.

Differences can be even 30-40% but both directions, depends on system.
(For different block sizes even more.)

So not only direct io, also block size and sometimes enabling fsync helps too.

> That said I think the bigger influence is the choice of the right block
> size. By choosing the right block size I could double the speed. As
> already said above I think it probably is quite hard to get this one
> right automatically for each and everyone, because it depends upon
> various caches involved. From a theoretical stand point it probably
> should be about half the size of the smallest cache involved. I'm not
> sure whether it makes any sense (or is even possible) to probe for these
> things, but considering the speed enhancement of - at least in my case -
> 100%, there should probably be something done about it.

Just few use cases I tried
- notebooks with SSD or rotational disks
- high speed PCIe SSD array
- direct rotational disk
- MD arrays of rotational disks
- external storage (with FC multipath betwen it)
- iSCSI
- VM with file backend (VMware and KVM)
...

Try it, you will see that common default doesn't work for some of these.

Anyway, defaults can be changed, we can add some heuristic or benchmark
but I would like to see that tool tested more first.

Milan