[dm-crypt] multiple crypt devices not using multiple CPUs

[dm-crypt] multiple crypt devices not using multiple CPUs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1672 bytes --]

Hi,

I guess I must be missing something.  I had understood that if one had
multiple block devices being crypted by dm-crypt, if they were being
operated on at the same time, their parallel crypto operations would
utilize different CPUs.

So I just did the following:

# lvcreate -L10G -n t1 vg
# cryptsetup --verbose --verify-passphrase luksFormat /dev/vg/t1
# cryptsetup luksOpen /dev/vg/t1 t1-use
# dd of=/dev/null if=/dev/mapper/t1-use bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.1207 s, 86.5 MB/s

observe (a series of) "kworker(s)" each using near 100% of a core to
carry out the crypto.  I say a series of kworkers since it appears to
almost be a "relay" of them, each handing the work of to another and
going away.  But all in all there is only every near 1 full core's worth
running.

So now...

# lvcreate -L10G -n t2 vg
# cryptsetup --verbose --verify-passphrase luksFormat /dev/vg/t2
# cryptsetup luksOpen /dev/vg/t2 t2-use
# for ((x=1; x<3; x++)); do
>     dd of=/dev/null if=/dev/mapper/t$x-use bs=1M count=1000 &
> done
[1] 7369
[2] 7370
# 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 27.62 s, 38.0 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 27.8015 s, 37.7 MB/s

Here I observe two kworker threads each near 50% of a core.  Clearly
they are both using the same core.  But why?  As as an aside, aggregate
throughput of the two threads is quite a bit less than a single thread.

But what I really want to understand is why crypto kworkers are not
being scheduled to different, idle cores.

Cheers,
b.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [dm-crypt] multiple crypt devices not using multiple CPUs

[Milan Broz]

On 12/16/2012 10:04 PM, Brian J. Murrell wrote:
> Hi,
> 
> I guess I must be missing something.  I had understood that if one had
> multiple block devices being crypted by dm-crypt, if they were being
> operated on at the same time, their parallel crypto operations would
> utilize different CPUs.

In current kernels, work keeps on cpu which submitted it.

Please read
http://www.saout.de/pipermail/dm-crypt/2012-July/002582.html

(and try to run the second case with direct-io, iow add oflag=direct for dd)

m.

Re: [dm-crypt] multiple crypt devices not using multiple CPUs

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

On 12-12-16 04:25 PM, Milan Broz wrote:
> 
> In current kernels, work keeps on cpu which submitted it.

So a single process doing parallel reads from multiple devices (I'm sure
you could guess where I'm going with this) will never see more
throughput than it would from a single device?  If so, all the more
reason to get multi-core support (for a single process of course)
involved.  :-/

> Please read
> http://www.saout.de/pipermail/dm-crypt/2012-July/002582.html

Yeah.  I'm trying to see this "more processes" behavior but even the two
dds each running in different shells are still attacking the same core.

> (and try to run the second case with direct-io, iow add oflag=direct for dd)

oflag=direct?

# dd oflag=direct of=/dev/null if=/dev/mapper/t1-use bs=1M count=1000
dd: opening ‘/dev/null’: Invalid argument

I guess O_DIRECT it not valid for /dev/null.

Cheers,
b.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]