RAID + LUKS + LVM performance

RAID + LUKS + LVM performance

[Mathias Buren]

Hi,

(please cc me as I'm not subscribed)

I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
http://www.areca.com.tw/products/pcie.htm ) etc.
The plan is create a RAID5 array spanning all the disks, then create 4
partitions. These 4 partitions would be encrypted using LUKS (Twofish or
AES256).
These 4 encrypted partition would be set up in RAID0 using Linux' software
(mdadm), then LVM would be used on top of that (one big PV, one big VG and
a big LV or so).

The reason for this is that kcryptd is not multithreaded (afaik). By having
4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
processes to run on all four cpu cores whenever something is written to the
disks, which should improve (encryption) performance.

Is this a good way of doing it, or is there a smarter way?


Regards,

Mathias,

Re: RAID + LUKS + LVM performance

[Matthias Schniedermeyer]

On 11.03.2010 13:08, Mathias Buren wrote:
> 
> Hi,
> 
> (please cc me as I'm not subscribed)
> 
> I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
> HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
> http://www.areca.com.tw/products/pcie.htm ) etc.
> The plan is create a RAID5 array spanning all the disks, then create 4
> partitions. These 4 partitions would be encrypted using LUKS (Twofish or
> AES256).
> These 4 encrypted partition would be set up in RAID0 using Linux' software
> (mdadm), then LVM would be used on top of that (one big PV, one big VG and
> a big LV or so).
> 
> The reason for this is that kcryptd is not multithreaded (afaik). By having
> 4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
> processes to run on all four cpu cores whenever something is written to the
> disks, which should improve (encryption) performance.
> 
> Is this a good way of doing it, or is there a smarter way?

The setup you describe would only work with SSDs. HDDs would seek 
themselves to death.

The problem is the RAID-0 over the 4 partitions. At that point you would 
need, instead of the 4 partitions, something that is round-robin. So 
that the mapping of the (physical) blocks from the upper to the lower 
would be effectivly linear/unchanged.

AFAIK something like that is (currently) not possible.





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: RAID + LUKS + LVM performance

[david]

On Thu, 11 Mar 2010, Matthias Schniedermeyer wrote:

> On 11.03.2010 13:08, Mathias Buren wrote:
>>
>> Hi,
>>
>> (please cc me as I'm not subscribed)
>>
>> I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
>> HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
>> http://www.areca.com.tw/products/pcie.htm ) etc.
>> The plan is create a RAID5 array spanning all the disks, then create 4
>> partitions. These 4 partitions would be encrypted using LUKS (Twofish or
>> AES256).
>> These 4 encrypted partition would be set up in RAID0 using Linux' software
>> (mdadm), then LVM would be used on top of that (one big PV, one big VG and
>> a big LV or so).
>>
>> The reason for this is that kcryptd is not multithreaded (afaik). By having
>> 4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
>> processes to run on all four cpu cores whenever something is written to the
>> disks, which should improve (encryption) performance.
>>
>> Is this a good way of doing it, or is there a smarter way?
>
> The setup you describe would only work with SSDs. HDDs would seek
> themselves to death.
>
> The problem is the RAID-0 over the 4 partitions. At that point you would
> need, instead of the 4 partitions, something that is round-robin. So
> that the mapping of the (physical) blocks from the upper to the lower
> would be effectivly linear/unchanged.
>
> AFAIK something like that is (currently) not possible.

linux software raid (the md tools) support linear or striped modes for 
raid0, so what you are looking for is available.

however I think that defeats part of the OPs purpose, which was to try and 
spread the I/O across all 4 partitions to be able to use multiple cores 
for the encryption.

David Lang

Re: RAID + LUKS + LVM performance

[Matthias Schniedermeyer]

On 11.03.2010 09:51, david@lang.hm wrote:
> On Thu, 11 Mar 2010, Matthias Schniedermeyer wrote:
>
>> On 11.03.2010 13:08, Mathias Buren wrote:
>>>
>>> Hi,
>>>
>>> (please cc me as I'm not subscribed)
>>>
>>> I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
>>> HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
>>> http://www.areca.com.tw/products/pcie.htm ) etc.
>>> The plan is create a RAID5 array spanning all the disks, then create 4
>>> partitions. These 4 partitions would be encrypted using LUKS (Twofish or
>>> AES256).
>>> These 4 encrypted partition would be set up in RAID0 using Linux' software
>>> (mdadm), then LVM would be used on top of that (one big PV, one big VG and
>>> a big LV or so).
>>>
>>> The reason for this is that kcryptd is not multithreaded (afaik). By having
>>> 4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
>>> processes to run on all four cpu cores whenever something is written to the
>>> disks, which should improve (encryption) performance.
>>>
>>> Is this a good way of doing it, or is there a smarter way?
>>
>> The setup you describe would only work with SSDs. HDDs would seek
>> themselves to death.
>>
>> The problem is the RAID-0 over the 4 partitions. At that point you would
>> need, instead of the 4 partitions, something that is round-robin. So
>> that the mapping of the (physical) blocks from the upper to the lower
>> would be effectivly linear/unchanged.
>>
>> AFAIK something like that is (currently) not possible.
>
> linux software raid (the md tools) support linear or striped modes for  
> raid0, so what you are looking for is available.

Nope. What i meant is:

Let say you had a block-device which has 16 blocks:
0-15
With the OPs description the blocks would be distributed like this:
Part 0: 00 01 02 03
Part 1: 04 05 06 07
Part 2: 08 09 10 11
Part 3: 12 13 14 15

What you need is a distribution like this:
Device 0: 01 05 09 13 
Device 1: 02 06 10 14
Device 2: 03 07 11 15
Device 3: 04 08 12 16

IOW:
Blocks % 4 == 0 on device 0
Blocks % 4 == 1 on device 1
Blocks % 4 == 2 on device 2
Blocks % 4 == 3 on device 3

I still other words:
You don't want a cake in exactly 4 same size parts. You want a cake in a 
million parts and then every 4th starting from the first piece in one 
set, every 4th starting from the second in the next and so on.

> however I think that defeats part of the OPs purpose, which was to try 
> and spread the I/O across all 4 partitions to be able to use multiple 
> cores for the encryption.

I think i just didn't make clear enough what i meant.




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: RAID + LUKS + LVM performance

[Mathias Buren]

Matthias Schniedermeyer <ms@citd.de> wrote on 2010-03-11 17:36:04:

> Re: RAID + LUKS + LVM performance
>
> Matthias Schniedermeyer
>
> to:
>
> Mathias Buren
>
> 2010-03-11 17:39
>
> Cc:
>
> linux-kernel
>
> On 11.03.2010 13:08, Mathias Buren wrote:
> >
> > Hi,
> >
> > (please cc me as I'm not subscribed)
> >
> > I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
> > HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
> > http://www.areca.com.tw/products/pcie.htm ) etc.
> > The plan is create a RAID5 array spanning all the disks, then create 4
> > partitions. These 4 partitions would be encrypted using LUKS (Twofish
or
> > AES256).
> > These 4 encrypted partition would be set up in RAID0 using Linux'
software
> > (mdadm), then LVM would be used on top of that (one big PV, one big VG
and
> > a big LV or so).
> >
> > The reason for this is that kcryptd is not multithreaded (afaik). By
having
> > 4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
> > processes to run on all four cpu cores whenever something is written to
the
> > disks, which should improve (encryption) performance.
> >
> > Is this a good way of doing it, or is there a smarter way?
>
> The setup you describe would only work with SSDs. HDDs would seek
> themselves to death.
>
> The problem is the RAID-0 over the 4 partitions. At that point you would
> need, instead of the 4 partitions, something that is round-robin. So
> that the mapping of the (physical) blocks from the upper to the lower
> would be effectivly linear/unchanged.
>
> AFAIK something like that is (currently) not possible.
>
>
>
>
>
> Bis denn
>
> --
> Real Programmers consider "what you see is what you get" to be just as
> bad a concept in Text Editors as it is in women. No, the Real Programmer
> wants a "you asked for it, you got it" text editor -- complicated,
> cryptic, powerful, unforgiving, dangerous.
>

Hm. But I thought, since the hw RAID card does its own RAID5 thing on the
harddrives, that they wouldn't seek themselves do death. Perhaps they
would, anyway...

What's the best way to set this up then? Or will kcryptd be able to
encrypt/decrypt everything fast enough anyway (~>5-600MB/s I'd say)?

Mathias

Re: RAID + LUKS + LVM performance

[Matthias Schniedermeyer]

On 12.03.2010 08:47, Mathias Buren wrote:
> Matthias Schniedermeyer <ms@citd.de> wrote on 2010-03-11 17:36:04:
> 
> > Re: RAID + LUKS + LVM performance
> >
> > Matthias Schniedermeyer
> >
> > to:
> >
> > Mathias Buren
> >
> > 2010-03-11 17:39
> >
> > Cc:
> >
> > linux-kernel
> >
> > On 11.03.2010 13:08, Mathias Buren wrote:
> > >
> > > Hi,
> > >
> > > (please cc me as I'm not subscribed)
> > >
> > > I've a friend who's going to set up a fileserver consisting of 8x 1.5TB
> > > HDDs, an 8-port PCI-E RAID card (Areca ARC-1220 @
> > > http://www.areca.com.tw/products/pcie.htm ) etc.
> > > The plan is create a RAID5 array spanning all the disks, then create 4
> > > partitions. These 4 partitions would be encrypted using LUKS (Twofish
> or
> > > AES256).
> > > These 4 encrypted partition would be set up in RAID0 using Linux'
> software
> > > (mdadm), then LVM would be used on top of that (one big PV, one big VG
> and
> > > a big LV or so).
> > >
> > > The reason for this is that kcryptd is not multithreaded (afaik). By
> having
> > > 4 encrypted partitions, then md0 on top of them, I'm forcing 4 kcryptd
> > > processes to run on all four cpu cores whenever something is written to
> the
> > > disks, which should improve (encryption) performance.
> > >
> > > Is this a good way of doing it, or is there a smarter way?
> >
> > The setup you describe would only work with SSDs. HDDs would seek
> > themselves to death.
> >
> > The problem is the RAID-0 over the 4 partitions. At that point you would
> > need, instead of the 4 partitions, something that is round-robin. So
> > that the mapping of the (physical) blocks from the upper to the lower
> > would be effectivly linear/unchanged.
> >
> > AFAIK something like that is (currently) not possible.
> 
> Hm. But I thought, since the hw RAID card does its own RAID5 thing on the
> harddrives, that they wouldn't seek themselves do death. Perhaps they
> would, anyway...
> 
> What's the best way to set this up then? Or will kcryptd be able to
> encrypt/decrypt everything fast enough anyway (~>5-600MB/s I'd say)?

Personally i have only experience with loop-aes (which has the exact 
same problem) and with AES128 i reach a top-speed of about 130MB/s on my 
Core i7-860 for a single thread (tested with a SSD).

Using one of the Westmere Core i5-6XX, which support AES-NI, should get 
you better performance (if supported by krcyptd), altough you loose 2 
cores as onyl have 2. But i don't know what kind of performance 
improvement (if any) AES-NI provides.
Or you wait a few weeks and buy on Core I7-980X with 6 core and AES-NI.





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: RAID + LUKS + LVM performance

[Milan Broz]

On 03/12/2010 01:06 PM, Matthias Schniedermeyer wrote:

> Using one of the Westmere Core i5-6XX, which support AES-NI, should get 
> you better performance (if supported by krcyptd), altough you loose 2 
> cores as onyl have 2. But i don't know what kind of performance 
> improvement (if any) AES-NI provides.
> Or you wait a few weeks and buy on Core I7-980X with 6 core and AES-NI.

dm-crypt uses whatever is implemented in cryptoAPI, AES-NI of course works
when you have proper modules loaded.

The main reason why dm-crypt is single-threaded is that all these new
crypto accelerators uses asynchronous crypto API, so if hw allows
parallel operation it can use it already.
(dm-crypt submits all sectors in bio as separate asynchronous crypto requests).

But note: single threaded per volume - if you have LVM with multiple
encrypted LVs - every LV have its own thread.

If we build another multi-core(thread) processing in dm-crypt, it can
help on multi-core CPU but complicate things otherwise.
(I tried simple implementation and abandoned it, I would prefer
if cryptoAPI can do some parallel processing itself here.
That question was discussed several times on dm-crypt mailing list.)

Anyway, only two real-world examples, where one thread 
in dm-crypt can cause real problems with speed

 - dm-crypt over fast RAID(5)
 - dm-crypt over fast SSD.

And many users of these configurations have now AES-NI acceleration.

Milan
--
mbroz@redhat.com

Re: RAID + LUKS + LVM performance

[Matthias Schniedermeyer]

On 12.03.2010 14:12, Milan Broz wrote:
> On 03/12/2010 01:06 PM, Matthias Schniedermeyer wrote:
> 
> And many users of these configurations have now AES-NI acceleration.

Do you have any numbers on the kind of performance gain you get from 
using AES-NI?





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.