From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <schmidt@digadd.de>
Received: from mail.saout.de ([127.0.0.1])
 by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id 7qxp_31uE9aO for <dm-crypt@saout.de>;
 Mon, 25 Mar 2013 11:56:27 +0100 (CET)
Received: from mx1.digadd.de (unknown [IPv6:2a01:130:11::241])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by mail.saout.de (Postfix) with ESMTPS
 for <dm-crypt@saout.de>; Mon, 25 Mar 2013 11:56:26 +0100 (CET)
Message-ID: <51502D27.8080606@digadd.de>
Date: Mon, 25 Mar 2013 11:55:35 +0100
From: Christian Schmidt <schmidt@digadd.de>
MIME-Version: 1.0
References: <51501C0A.5060406@digadd.de> <m2obe7ixck.fsf@firstfloor.org>
In-Reply-To: <m2obe7ixck.fsf@firstfloor.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [dm-crypt] kworkers for dm-crypt locked to CPU core 0?
List-Id: <dm-crypt.saout.de>
List-Unsubscribe: <http://www.saout.de/mailman/options/dm-crypt>,
 <mailto:dm-crypt-request@saout.de?subject=unsubscribe>
List-Archive: <http://www.saout.de/pipermail/dm-crypt/>
List-Post: <mailto:dm-crypt@saout.de>
List-Help: <mailto:dm-crypt-request@saout.de?subject=help>
List-Subscribe: <http://www.saout.de/mailman/listinfo/dm-crypt>,
 <mailto:dm-crypt-request@saout.de?subject=subscribe>
To: Andi Kleen <andi@firstfloor.org>
Cc: dm-crypt@saout.de, linux-kernel@vger.kernel.org

Hi Andi,

On 25/03/13 11:32, Andi Kleen wrote:
> Christian Schmidt <schmidt@digadd.de> writes:
>>
>> Is there a way I can make the scheduler put those on multiple cores?
> 
> Submit the IO from multiple cores. Don't use dd.

The dd processes run on multiple cores. I do understand that you were
saying "multiple requests to the same device from multiple cores",
however I only see kworkers from core 0 active, even with corelocked dd
processes (via numactl). Since I just skimmed to the pcrypt.c I kind of
realized that was the answer - do I understand correctly that the core
for the kworker follows the core of the requestor?

Basically that means that I can not get the aggregate throughput for all
devices higher than that for a single device by changing the stacking,
e.g. dm-crypt on top of md-raid / below the md-raid, or below zfs/btrfs
vdevs, since all user space requests will always appear to the crypto
layer to originate from core 0, irrespective of the core the application
runs on, unless the file system runs multithreaded? I just tested this
with ZFS, and all kworkers run on core 0 during file I/O or when
resilvering a raidZ. I don't have a btrfs managing its own vdevs at hand
to compare though.

Thanks for your input,
Christian