SMP-aware kcryptd?

SMP-aware kcryptd?

[Tomasz Chmielewski]

Currently, the trend with CPUs is to add more cores rather than increase 
the speed of a single core.

This does not scale very well with certain things in the Linux kernel. 
One of them is kcryptd.


A system able to deliver data with a speed of ~200 MB/s from a RAID 
array, will be only able to deliver a fraction of it (i.e. ~40 MB/s in 
my case) when reads are being done from a dm-crypt device.

This is because kcryptd is not SMP-aware: it performs all calculations 
on a single logical CPU only.

Are there any plans to change it?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: SMP-aware kcryptd?

[Milan Broz]

Tomasz Chmielewski wrote:
> Currently, the trend with CPUs is to add more cores rather than increase 
> the speed of a single core.
> 
> This does not scale very well with certain things in the Linux kernel. 
> One of them is kcryptd.
> 
> 
> A system able to deliver data with a speed of ~200 MB/s from a RAID 
> array, will be only able to deliver a fraction of it (i.e. ~40 MB/s in 
> my case) when reads are being done from a dm-crypt device.
> 
> This is because kcryptd is not SMP-aware: it performs all calculations 
> on a single logical CPU only.
> 
> Are there any plans to change it?

Yes, I think I already commented this in some thread,
there are basically 2 possible approaches:

1) create several "encryption" threads in dm-crypt (e.g. per CPU core) or
2) parallelize async crypto requests processing (in crypto layer)

I have some patches for 1) but the result is not what I expected
- performance highly depends on write pattern for example.
In short, the code need more thought.
(I tried to parallelize sector encryption _vs_ whole bio request encryption,
and there are still some problem with both. Also barriers and reordering
of requests can be problem here.)

Another problem with 1) is what happens when there is hw acceleration -
multiple threads are useful only if main cpu runs encryption,
for hw & async crypto requests it can cause problems.

The crypto driver is completely transparent - dm-crypt have no idea
if encryption runs on main cpu or in an accelerator hw
(=> solution 2) is more generic).

Anyway, it is not highest priority currently... but still in slow progress.
(but I guess high speed SSDs will increase priority for this too,
currently it is real problem only for RAID arrays:-)

Milan
--
mbroz@redhat.com

Re: SMP-aware kcryptd?

[Tomasz Chmielewski]

Milan Broz schrieb:
> Tomasz Chmielewski wrote:
>> Currently, the trend with CPUs is to add more cores rather than increase 
>> the speed of a single core.
>>
>> This does not scale very well with certain things in the Linux kernel. 
>> One of them is kcryptd.
>>
>>
>> A system able to deliver data with a speed of ~200 MB/s from a RAID 
>> array, will be only able to deliver a fraction of it (i.e. ~40 MB/s in 
>> my case) when reads are being done from a dm-crypt device.

(...)

> Anyway, it is not highest priority currently... but still in slow progress.
> (but I guess high speed SSDs will increase priority for this too,
> currently it is real problem only for RAID arrays:-)

And, laptops/netbooks with (slow) multi-core CPUs.

Encryption is getting popular there as well.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: SMP-aware kcryptd?

[Larry Dickson]

[-- Attachment #1.1: Type: text/plain, Size: 2392 bytes --]

On 4/13/09, Milan Broz <mbroz@redhat.com> wrote:
>
> Tomasz Chmielewski wrote:
> > Currently, the trend with CPUs is to add more cores rather than increase
> > the speed of a single core.
> >
> > This does not scale very well with certain things in the Linux kernel.
> > One of them is kcryptd.
> >
> >
> > A system able to deliver data with a speed of ~200 MB/s from a RAID
> > array, will be only able to deliver a fraction of it (i.e. ~40 MB/s in
> > my case) when reads are being done from a dm-crypt device.
> >
> > This is because kcryptd is not SMP-aware: it performs all calculations
> > on a single logical CPU only.
> >
> > Are there any plans to change it?
>
> Yes, I think I already commented this in some thread,
> there are basically 2 possible approaches:
>
> 1) create several "encryption" threads in dm-crypt (e.g. per CPU core) or
> 2) parallelize async crypto requests processing (in crypto layer)
>
> I have some patches for 1) but the result is not what I expected
> - performance highly depends on write pattern for example.
> In short, the code need more thought.


This is a golden opportunity. So many times, code (e.g. barriers) that still
may "need more thought" has been rushed into production, leaving headaches
that apparently last forever. I think this is a chance to get everyone in at
the design stage. Can you publish a short sketch of the proposed design
approaches for 1) and 2), with their advantages and drawbacks?

Larry Dickson
Cutting Edge Networked Storage

(I tried to parallelize sector encryption _vs_ whole bio request encryption,
> and there are still some problem with both. Also barriers and reordering
> of requests can be problem here.)
>
> Another problem with 1) is what happens when there is hw acceleration -
> multiple threads are useful only if main cpu runs encryption,
> for hw & async crypto requests it can cause problems.
>
> The crypto driver is completely transparent - dm-crypt have no idea
> if encryption runs on main cpu or in an accelerator hw
> (=> solution 2) is more generic).
>
> Anyway, it is not highest priority currently... but still in slow progress.
> (but I guess high speed SSDs will increase priority for this too,
> currently it is real problem only for RAID arrays:-)
>
> Milan
> --
> mbroz@redhat.com
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 3135 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]