* Re: [PATCH] crypto: use timespec64 for jent_get_nstime
From: John Stultz @ 2016-06-21 16:22 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Herbert Xu, Alexander Kuleshov, Stephan Mueller, lkml,
y2038 Mailman List, linux-crypto, David S. Miller, Kees Cook
In-Reply-To: <3776557.os2uYrMn6M@wuerfel>
On Tue, Jun 21, 2016 at 1:32 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Tuesday, June 21, 2016 8:20:10 AM CEST Stephan Mueller wrote:
>> Am Freitag, 17. Juni 2016, 17:59:41 schrieb Arnd Bergmann:
>>
> Compared to the previous __getnstimeofday(), the difference is
>
> - using "monotonic" timebase instead of "real", so the zero time
> is when the system booted rather than Jan 1 1970
I haven't looked at the details of the calling code, but I'd worry for
crypto uses, especially if its being used for entropy collection,
using the monotonic clock instead of the realtime clock might be
problematic.
> - "raw" means we don't honor updates for the rate based on ntp,
> which is probably better as the ntp state might be observable
> over the net (it probably doesn't matter, but it can't hurt)
So... this feels like a very vague explanation, and the lack of
frequency correction here probably need a really good comment. Keeping
multiple time domains is usually asking for trouble, but we added the
MONOTONIC_RAW clock to address a few cases where people really wanted
an abstract hardware counter, which was unaffected by frequency
corrections. I'd really make sure its clear why this is what you want
vs the standard system time domain so we don't run into problems
understanding it later.
> - "fast" means that in very rare cases, the time might appear
> to go backwards (it probably can't happen here because you are not
> called in an NMI).
"fast" really means "safe-for-nmi wrt to locking". The tradeoff being
that when frequency adjustments occur, and if your code is delayed,
you might see time go backwards by a small amount. This allows
tracing/sched code (or other code called from NMI) to not have to
duplicate the timekeeping infrastructure.
I think without a much better explanation, using the "fast" method
isn't really warranted here.
thanks
-john
_______________________________________________
Y2038 mailing list
Y2038@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/y2038
^ permalink raw reply
* Re: [PATCH v5 0/7] /dev/random - a new approach
From: Austin S. Hemmelgarn @ 2016-06-21 16:03 UTC (permalink / raw)
To: Stephan Mueller, Nikos Mavrogiannopoulos
Cc: Theodore Ts'o, Pavel Machek, Herbert Xu, Andi Kleen,
Sandy Harris, Jason Cooper, John Denker, H. Peter Anvin,
Joe Perches, George Spelvin, Linux Crypto Mailing List,
Linux Kernel Mailing List
In-Reply-To: <24059874.5WizEqNrfz@tauon.atsec.com>
On 2016-06-21 03:32, Stephan Mueller wrote:
> Am Dienstag, 21. Juni 2016, 09:12:07 schrieb Nikos Mavrogiannopoulos:
>
> Hi Nikos,
>
>> On Mon, Jun 20, 2016 at 5:43 PM, Stephan Mueller <smueller@chronox.de>
> wrote:
>>>> Personally, I don't really use /dev/random, nor would I recommend it
>>>> for most application programmers. At this point, getrandom(2) really
>>>> is the preferred interface unless you have some very specialized
>>>> needs.
>>>
>>> I fully agree. But there are use cases for /dev/random, notably as a seed
>>> source for other DRNG.
>>
>> Is that really the case? I believe all DRNG's use /dev/urandom anyway
>> for seeding since they cannot afford indeterminate blocking. It would
>> be a gain for everyone if /dev/random was the same as /dev/urandom in
>> Linux.
>
> For standard approaches, this is true. But there are regulations, notably in
> the German realm, /dev/random shall be used, at least partially (see AIS
> 20/31).
Which just goes to show how utterly stupid some people who write laws
and regulations are. Saying specifically that '/dev/random shall be
used' does not enforce any improvement of entrophic value in the data at
all, it just coincidentally improves the theoretical quality of the data
because of how Linux and some legacy UNIX systems are designed. This
'regulation' already provides zero benefit other than a placebo effect
on at least OpenBSD, FreeBSD, and I'm pretty certain most other BSD
derivatives, as /dev/random and /dev/urandom point to the same thing
there (on OpenBSD it's an arcfour based drbg, FreeBSD does similar but
uses a CSPRNG called Fortuna), and while I'm not certain, I believe AIX
does likewise (although they use a design based on yarrow).
On top of that though, just because some poorly thought out standard
requires it's usage doesn't mean we have to work based on that.
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: Pavel Machek @ 2016-06-21 13:42 UTC (permalink / raw)
To: Austin S. Hemmelgarn
Cc: Stephan Mueller, Theodore Ts'o, David Jaša, Andi Kleen,
sandyinchina, Jason Cooper, John Denker, H. Peter Anvin,
Joe Perches, George Spelvin, linux-crypto, linux-kernel
In-Reply-To: <6b8c8f6a-862a-3e7c-e950-75cd93cdc1f7@gmail.com>
Hi!
> 6. You have a significant lack of data regarding embedded systems, which is
> one of the two biggest segments of Linux's market share. You list no
> results for any pre-ARMv6 systems (Linux still runs on and is regularly used
> on ARMv4 CPU's, and it's worth also pointing out that the values on
> the
Feel free to contribute more test results.
I mean... you can't expect every person who wants to improve something
in linux to test on everything in the world... can you?
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: Stephan Mueller @ 2016-06-21 13:20 UTC (permalink / raw)
To: Austin S. Hemmelgarn
Cc: Theodore Ts'o, David Jaša, Andi Kleen, sandyinchina,
Jason Cooper, John Denker, H. Peter Anvin, Joe Perches,
Pavel Machek, George Spelvin, linux-crypto, linux-kernel
In-Reply-To: <6b8c8f6a-862a-3e7c-e950-75cd93cdc1f7@gmail.com>
Am Dienstag, 21. Juni 2016, 09:05:55 schrieb Austin S. Hemmelgarn:
Hi Austin,
> On 2016-06-20 14:32, Stephan Mueller wrote:
> > Am Montag, 20. Juni 2016, 13:07:32 schrieb Austin S. Hemmelgarn:
> >
> > Hi Austin,
> >
> >> On 2016-06-18 12:31, Stephan Mueller wrote:
> >>> Am Samstag, 18. Juni 2016, 10:44:08 schrieb Theodore Ts'o:
> >>>
> >>> Hi Theodore,
> >>>
> >>>> At the end of the day, with these devices you really badly need a
> >>>> hardware RNG. We can't generate randomness out of thin air. The only
> >>>> thing you really can do requires user space help, which is to generate
> >>>> keys lazily, or as late as possible, so you can gather as much entropy
> >>>> as you can --- and to feed in measurements from the WiFi (RSSI
> >>>> measurements, MAC addresses seen, etc.) This won't help much if you
> >>>> have an FBI van parked outside your house trying to carry out a
> >>>> TEMPEST attack, but hopefully it provides some protection against a
> >>>> remote attacker who isn't try to carry out an on-premises attack.
> >>>
> >>> All my measurements on such small systems like MIPS or smaller/older
> >>> ARMs
> >>> do not seem to support that statement :-)
> >>
> >> Was this on real hardware, or in a virtual machine/emulator? Because if
> >> it's not on real hardware, you're harvesting entropy from the host
> >> system, not the emulated one. While I haven't done this with MIPS or
> >> ARM systems, I've taken similar measurements on SPARC64, x86_64, and
> >> PPC64 systems comparing real hardware and emulated hardware, and the
> >> emulated hardware _always_ has higher entropy, even when running the
> >> emulator on an identical CPU to the one being emulated and using KVM
> >> acceleration and passing through all the devices possible.
> >>
> >> Even if you were testing on real hardware, I'm still rather dubious, as
> >> every single test I've ever done on any hardware (SPARC, PPC, x86, ARM,
> >> and even PA-RISC) indicates that you can't harvest entropy as
> >> effectively from a smaller CPU compared to a large one, and this effect
> >> is significantly more pronounced on RISC systems.
> >
> > It was on real hardware. As part of my Jitter RNG project, I tested all
> > major CPUs from small to big -- see Appendix F [1]. For MIPS/ARM, see the
> > trailing part of the big table.
> >
> > [1] http://www.chronox.de/jent/doc/CPU-Jitter-NPTRNG.pdf
>
> Specific things I notice about this:
> 1. QEMU systems are reporting higher values than almost anything else
> with the same ISA. This makes sense, but you don't appear to have
> accounted for the fact that you can't trust almost any of the entropy in
> a VM unless you have absolute trust in the host system, because the host
> system can do whatever the hell it wants to you, including manipulating
> timings directly (with a little patience and some time spent working on
> it, you could probably get those number to show whatever you want just
> by manipulating scheduling parameters on the host OS for the VM software).
I am not sure where you see QEMU systems listed there.
> 2. Quite a few systems have a rather distressingly low lower bound and
> still get accepted by your algorithm (a number of the S/390 systems, and
> a handful of the AMD processors in particular).
I am aware of that, but please read the entire documentation where the lower
and upper boundary comes from and how the Jitter RNG really operates. There
you will see that the lower boundary is just that: it will not be lower, but
the common case is the upper boundary.
Furthermore, the use case of the Jitter RNG is to support the DRBG seeding
with a very high reseed interval.
> 3. Your statement at the bottom of the table that 'all test systems at
> least un-optimized have a lower bound of 1 bit' is refuted by your own
> data, I count at least 2 data points where this is not the case. One of
> them is mentioned at the bottom as an outlier, and you have data to back
> this up listed in the table, but the other (MIPS 4Kec v4.8) is the only
> system of that specific type that you tested, and thus can't be claimed
> as an outlier.
You are right, I have added more and more test results to the table without
updating the statement below. I will fix that.
But note, that there is a list below that statement providing explanations
already. So, it is just that one statement that needs updating.
> 4. You state the S/390 systems gave different results when run
> un-optimized, but don't provide any data regarding this.
The pointer to appendix F.46 was supposed to cover that issue.
> 5. You discount the Pentium Celeron Mobile CPU as old and therefore not
> worth worrying about. Linux still runs on 80486 and other 'ancient'
> systems, and there are people using it on such systems. You need to
> account for this usage.
I do not account for that in the documentation. In real life though, I
certainly do -- see how the Jitter RNG is used in the kernel.
> 6. You have a significant lack of data regarding embedded systems, which
> is one of the two biggest segments of Linux's market share. You list no
> results for any pre-ARMv6 systems (Linux still runs on and is regularly
> used on ARMv4 CPU's, and it's worth also pointing out that the values on
> the ARMv6 systems are themselves below average), any MIPS systems other
> than 24k and 4k (which is not a good representation of modern embedded
> usage), any SPARC CPU's other than UltraSPARC (ideally you should have
> results on at least a couple of LEON systems as well), no tight-embedded
> PPC chips (PPC 440 processors are very widely used, as are the 7xx and
> 970 families, and Freescale's e series), and only one set of results for
> a tight-embedded x86 CPU (the Via Nano, you should ideally also have
> results on things like an Intel Quark). Overall, your test system
> selection is not entirely representative of actual Linux usage (yeah,
> ther'es a lot of x86 servers out there running Linux, there's at least
> as many embedded systems running it too though, even without including
> Android).
Perfectly valid argument. But I programmed that RNG as a hobby -- I do not
have the funds to buy all devices there are.
And http://www.chronox.de/jent.html asks for help -- if you have those
devices, please help and simply execute one application and return the data to
me.
> 7. The RISC CPU's that you actually tested have more consistency within
> a particular type than the CISC CPU's. Many of them do have higher
> values than the CISC CPU's, but a majority of the ones I see listed
> which have such high values are either old systems not designed for low
> latency, or relatively big SMP systems (which will have higher entropy
> because of larger numbers of IRQ's, as well as other factors).
Ok, run the tests on the systems you like and return the results to me.
Ciao
Stephan
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: Tomas Mraz @ 2016-06-21 13:19 UTC (permalink / raw)
To: Austin S. Hemmelgarn, Stephan Mueller
Cc: Theodore Ts'o, David Jaša, Andi Kleen, sandyinchina,
Jason Cooper, John Denker, H. Peter Anvin, Joe Perches,
Pavel Machek, George Spelvin, linux-crypto, linux-kernel
In-Reply-To: <6b8c8f6a-862a-3e7c-e950-75cd93cdc1f7@gmail.com>
On Út, 2016-06-21 at 09:05 -0400, Austin S. Hemmelgarn wrote:
> On 2016-06-20 14:32, Stephan Mueller wrote:
> >
> > [1] http://www.chronox.de/jent/doc/CPU-Jitter-NPTRNG.pdf
> Specific things I notice about this:
> 1. QEMU systems are reporting higher values than almost anything
> else
> with the same ISA. This makes sense, but you don't appear to have
> accounted for the fact that you can't trust almost any of the entropy
> in
> a VM unless you have absolute trust in the host system, because the
> host
> system can do whatever the hell it wants to you, including
> manipulating
> timings directly (with a little patience and some time spent working
> on
> it, you could probably get those number to show whatever you want
> just
> by manipulating scheduling parameters on the host OS for the VM
> software).
You have to trust the host for anything, not just for the entropy in
timings. This is completely invalid argument unless you can present a
method that one guest can manipulate timings in other guest in such a
way that _removes_ the inherent entropy from the host.
--
Tomas Mraz
No matter how far down the wrong road you've gone, turn back.
Turkish proverb
(You'll never know whether the road is wrong though.)
^ permalink raw reply
* Re: [PATCH] crypto : async implementation for sha1-mb
From: Herbert Xu @ 2016-06-21 13:07 UTC (permalink / raw)
To: Megha Dey
Cc: tim.c.chen, davem, linux-crypto, linux-kernel, fenghua.yu,
Megha Dey
In-Reply-To: <1466454346-12107-1-git-send-email-megha.dey@intel.com>
On Mon, Jun 20, 2016 at 01:25:46PM -0700, Megha Dey wrote:
>
> - desc->tfm = child;
> - desc->flags = CRYPTO_TFM_REQ_MAY_SLEEP; /* check this again */
> + ahash_request_set_tfm(desc, child);
> + ahash_request_set_callback(desc, CRYPTO_TFM_REQ_MAY_SLEEP, NULL, NULL);
The callback function needs to be set, or this will crash when
it returns -EINPROGRESS.
> if (err) {
> req->base.complete = rctx->complete;
This is not calling the completion function conditinoally, rather
it is trying to restore the original completion function when we
are done.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: Austin S. Hemmelgarn @ 2016-06-21 13:05 UTC (permalink / raw)
To: Stephan Mueller
Cc: Theodore Ts'o, David Jaša, Andi Kleen, sandyinchina,
Jason Cooper, John Denker, H. Peter Anvin, Joe Perches,
Pavel Machek, George Spelvin, linux-crypto, linux-kernel
In-Reply-To: <8999970.pstTbGZv5G@positron.chronox.de>
On 2016-06-20 14:32, Stephan Mueller wrote:
> Am Montag, 20. Juni 2016, 13:07:32 schrieb Austin S. Hemmelgarn:
>
> Hi Austin,
>
>> On 2016-06-18 12:31, Stephan Mueller wrote:
>>> Am Samstag, 18. Juni 2016, 10:44:08 schrieb Theodore Ts'o:
>>>
>>> Hi Theodore,
>>>
>>>> At the end of the day, with these devices you really badly need a
>>>> hardware RNG. We can't generate randomness out of thin air. The only
>>>> thing you really can do requires user space help, which is to generate
>>>> keys lazily, or as late as possible, so you can gather as much entropy
>>>> as you can --- and to feed in measurements from the WiFi (RSSI
>>>> measurements, MAC addresses seen, etc.) This won't help much if you
>>>> have an FBI van parked outside your house trying to carry out a
>>>> TEMPEST attack, but hopefully it provides some protection against a
>>>> remote attacker who isn't try to carry out an on-premises attack.
>>>
>>> All my measurements on such small systems like MIPS or smaller/older ARMs
>>> do not seem to support that statement :-)
>>
>> Was this on real hardware, or in a virtual machine/emulator? Because if
>> it's not on real hardware, you're harvesting entropy from the host
>> system, not the emulated one. While I haven't done this with MIPS or
>> ARM systems, I've taken similar measurements on SPARC64, x86_64, and
>> PPC64 systems comparing real hardware and emulated hardware, and the
>> emulated hardware _always_ has higher entropy, even when running the
>> emulator on an identical CPU to the one being emulated and using KVM
>> acceleration and passing through all the devices possible.
>>
>> Even if you were testing on real hardware, I'm still rather dubious, as
>> every single test I've ever done on any hardware (SPARC, PPC, x86, ARM,
>> and even PA-RISC) indicates that you can't harvest entropy as
>> effectively from a smaller CPU compared to a large one, and this effect
>> is significantly more pronounced on RISC systems.
>
> It was on real hardware. As part of my Jitter RNG project, I tested all major
> CPUs from small to big -- see Appendix F [1]. For MIPS/ARM, see the trailing
> part of the big table.
>
> [1] http://www.chronox.de/jent/doc/CPU-Jitter-NPTRNG.pdf
Specific things I notice about this:
1. QEMU systems are reporting higher values than almost anything else
with the same ISA. This makes sense, but you don't appear to have
accounted for the fact that you can't trust almost any of the entropy in
a VM unless you have absolute trust in the host system, because the host
system can do whatever the hell it wants to you, including manipulating
timings directly (with a little patience and some time spent working on
it, you could probably get those number to show whatever you want just
by manipulating scheduling parameters on the host OS for the VM software).
2. Quite a few systems have a rather distressingly low lower bound and
still get accepted by your algorithm (a number of the S/390 systems, and
a handful of the AMD processors in particular).
3. Your statement at the bottom of the table that 'all test systems at
least un-optimized have a lower bound of 1 bit' is refuted by your own
data, I count at least 2 data points where this is not the case. One of
them is mentioned at the bottom as an outlier, and you have data to back
this up listed in the table, but the other (MIPS 4Kec v4.8) is the only
system of that specific type that you tested, and thus can't be claimed
as an outlier.
4. You state the S/390 systems gave different results when run
un-optimized, but don't provide any data regarding this.
5. You discount the Pentium Celeron Mobile CPU as old and therefore not
worth worrying about. Linux still runs on 80486 and other 'ancient'
systems, and there are people using it on such systems. You need to
account for this usage.
6. You have a significant lack of data regarding embedded systems, which
is one of the two biggest segments of Linux's market share. You list no
results for any pre-ARMv6 systems (Linux still runs on and is regularly
used on ARMv4 CPU's, and it's worth also pointing out that the values on
the ARMv6 systems are themselves below average), any MIPS systems other
than 24k and 4k (which is not a good representation of modern embedded
usage), any SPARC CPU's other than UltraSPARC (ideally you should have
results on at least a couple of LEON systems as well), no tight-embedded
PPC chips (PPC 440 processors are very widely used, as are the 7xx and
970 families, and Freescale's e series), and only one set of results for
a tight-embedded x86 CPU (the Via Nano, you should ideally also have
results on things like an Intel Quark). Overall, your test system
selection is not entirely representative of actual Linux usage (yeah,
ther'es a lot of x86 servers out there running Linux, there's at least
as many embedded systems running it too though, even without including
Android).
7. The RISC CPU's that you actually tested have more consistency within
a particular type than the CISC CPU's. Many of them do have higher
values than the CISC CPU's, but a majority of the ones I see listed
which have such high values are either old systems not designed for low
latency, or relatively big SMP systems (which will have higher entropy
because of larger numbers of IRQ's, as well as other factors).
^ permalink raw reply
* Re: [PATCH v3 09/10] crypto: marvell: Add support for chaining crypto requests in TDMA mode
From: Boris Brezillon @ 2016-06-21 12:37 UTC (permalink / raw)
To: Romain Perier
Cc: Thomas Petazzoni, Russell King, Arnaud Ebalard, linux-crypto,
Gregory Clement, David S. Miller, linux-arm-kernel
In-Reply-To: <1466496520-28806-10-git-send-email-romain.perier@free-electrons.com>
On Tue, 21 Jun 2016 10:08:39 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> The Cryptographic Engines and Security Accelerators (CESA) supports the
> Multi-Packet Chain Mode. With this mode enabled, multiple tdma requests
> can be chained and processed by the hardware without software
> intervention. This mode was already activated, however the crypto
> requests were not chained together. By doing so, we reduce significantly
> the number of IRQs. Instead of being interrupted at the end of each
> crypto request, we are interrupted at the end of the last cryptographic
> request processed by the engine.
>
> This commits re-factorizes the code, changes the code architecture and
> adds the required data structures to chain cryptographic requests
> together before sending them to an engine (stopped or possibly already
> running).
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
One nit below ;).
> ---
>
> Changes in v3:
>
> - Cosmetic changes: Extra blank lines and coding style issues
> on prototypes.
>
> Changes in v2:
>
> - Reworded the commit message
> - Fixed cosmetic changes: coding styles issues, missing blank lines
> - Reworked mv_cesa_rearm_engine: lock handling is simpler
> - Removed the call to the complete operation in mv_cesa_std_process,
> in case of errors (not required)
> - Squashed the removal of the '.prepare' fields (cipher.c, hash.c)
> into another commit (see PATCH 08/10).
> - In mv_cesa_tdma_process only treat the status argument for the last
> request, use 'normal' status for the other ones.
> - Added a comment for explaining how the errors are notified to the
> cesa core.
>
> drivers/crypto/marvell/cesa.c | 115 +++++++++++++++++++++++++++++++---------
> drivers/crypto/marvell/cesa.h | 39 +++++++++++++-
> drivers/crypto/marvell/cipher.c | 2 +-
> drivers/crypto/marvell/hash.c | 6 +++
> drivers/crypto/marvell/tdma.c | 86 ++++++++++++++++++++++++++++++
> 5 files changed, 221 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index c0497ac..bb91156 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -40,14 +40,33 @@ MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if over
>
> struct mv_cesa_dev *cesa_dev;
>
> -static void mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine)
> +struct crypto_async_request *
> +mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine,
> + struct crypto_async_request **backlog)
> {
> - struct crypto_async_request *req, *backlog;
> - struct mv_cesa_ctx *ctx;
> + struct crypto_async_request *req;
>
> - backlog = crypto_get_backlog(&engine->queue);
> + *backlog = crypto_get_backlog(&engine->queue);
> req = crypto_dequeue_request(&engine->queue);
> - engine->req = req;
> +
> + if (!req)
> + return NULL;
> +
> + return req;
> +}
> +
> +static void mv_cesa_rearm_engine(struct mv_cesa_engine *engine)
> +{
> + struct crypto_async_request *req = NULL, *backlog = NULL;
> + struct mv_cesa_ctx *ctx;
> +
> +
> + spin_lock_bh(&engine->lock);
> + if (!engine->req) {
> + req = mv_cesa_dequeue_req_locked(engine, &backlog);
> + engine->req = req;
> + }
> + spin_unlock_bh(&engine->lock);
>
> if (!req)
> return;
> @@ -57,6 +76,46 @@ static void mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine)
>
> ctx = crypto_tfm_ctx(req->tfm);
> ctx->ops->step(req);
> +
> + return;
> +}
> +
> +static int mv_cesa_std_process(struct mv_cesa_engine *engine, u32 status)
> +{
> + struct crypto_async_request *req;
> + struct mv_cesa_ctx *ctx;
> + int res;
> +
> + req = engine->req;
> + ctx = crypto_tfm_ctx(req->tfm);
> + res = ctx->ops->process(req, status);
> +
> + if (res == 0) {
> + ctx->ops->complete(req);
> + mv_cesa_engine_enqueue_complete_request(engine, req);
> + } else if (res == -EINPROGRESS) {
> + ctx->ops->step(req);
> + }
> +
> + return res;
> +}
> +
> +static int mv_cesa_int_process(struct mv_cesa_engine *engine, u32 status)
> +{
> + if (engine->chain.first && engine->chain.last)
> + return mv_cesa_tdma_process(engine, status);
> +
> + return mv_cesa_std_process(engine, status);
> +}
> +
> +static inline void
> +mv_cesa_complete_req(struct mv_cesa_ctx *ctx, struct crypto_async_request *req,
> + int res)
> +{
> + ctx->ops->cleanup(req);
> + local_bh_disable();
> + req->complete(req, res);
> + local_bh_enable();
> }
>
> static irqreturn_t mv_cesa_int(int irq, void *priv)
> @@ -83,26 +142,31 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
> writel(~status, engine->regs + CESA_SA_FPGA_INT_STATUS);
> writel(~status, engine->regs + CESA_SA_INT_STATUS);
>
> + /* Process fetched requests */
> + res = mv_cesa_int_process(engine, status & mask);
> ret = IRQ_HANDLED;
> +
> spin_lock_bh(&engine->lock);
> req = engine->req;
> + if (res != -EINPROGRESS)
> + engine->req = NULL;
> spin_unlock_bh(&engine->lock);
> - if (req) {
> - ctx = crypto_tfm_ctx(req->tfm);
> - res = ctx->ops->process(req, status & mask);
> - if (res != -EINPROGRESS) {
> - spin_lock_bh(&engine->lock);
> - engine->req = NULL;
> - mv_cesa_dequeue_req_locked(engine);
> - spin_unlock_bh(&engine->lock);
> - ctx->ops->complete(req);
> - ctx->ops->cleanup(req);
> - local_bh_disable();
> - req->complete(req, res);
> - local_bh_enable();
> - } else {
> - ctx->ops->step(req);
> - }
> +
> + ctx = crypto_tfm_ctx(req->tfm);
> +
> + if (res && res != -EINPROGRESS)
> + mv_cesa_complete_req(ctx, req, res);
> +
> + /* Launch the next pending request */
> + mv_cesa_rearm_engine(engine);
> +
> + /* Iterate over the complete queue */
> + while (true) {
> + req = mv_cesa_engine_dequeue_complete_request(engine);
> + if (!req)
> + break;
> +
> + mv_cesa_complete_req(ctx, req, 0);
> }
> }
>
> @@ -116,16 +180,16 @@ int mv_cesa_queue_req(struct crypto_async_request *req,
> struct mv_cesa_engine *engine = creq->engine;
>
> spin_lock_bh(&engine->lock);
> + if (mv_cesa_req_get_type(creq) == CESA_DMA_REQ)
> + mv_cesa_tdma_chain(engine, creq);
> +
> ret = crypto_enqueue_request(&engine->queue, req);
> spin_unlock_bh(&engine->lock);
>
> if (ret != -EINPROGRESS)
> return ret;
>
> - spin_lock_bh(&engine->lock);
> - if (!engine->req)
> - mv_cesa_dequeue_req_locked(engine);
> - spin_unlock_bh(&engine->lock);
> + mv_cesa_rearm_engine(engine);
>
> return -EINPROGRESS;
> }
> @@ -496,6 +560,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
>
> crypto_init_queue(&engine->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
> atomic_set(&engine->load, 0);
> + INIT_LIST_HEAD(&engine->complete_queue);
> }
>
> cesa_dev = cesa;
> diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
> index 644be35..50a1fb2 100644
> --- a/drivers/crypto/marvell/cesa.h
> +++ b/drivers/crypto/marvell/cesa.h
> @@ -271,7 +271,9 @@ struct mv_cesa_op_ctx {
> /* TDMA descriptor flags */
> #define CESA_TDMA_DST_IN_SRAM BIT(31)
> #define CESA_TDMA_SRC_IN_SRAM BIT(30)
> -#define CESA_TDMA_TYPE_MSK GENMASK(29, 0)
> +#define CESA_TDMA_END_OF_REQ BIT(29)
> +#define CESA_TDMA_BREAK_CHAIN BIT(28)
> +#define CESA_TDMA_TYPE_MSK GENMASK(27, 0)
> #define CESA_TDMA_DUMMY 0
> #define CESA_TDMA_DATA 1
> #define CESA_TDMA_OP 2
> @@ -431,6 +433,9 @@ struct mv_cesa_dev {
> * SRAM
> * @queue: fifo of the pending crypto requests
> * @load: engine load counter, useful for load balancing
> + * @chain: list of the current tdma descriptors being processed
> + * by this engine.
> + * @complete_queue: fifo of the processed requests by the engine
> *
> * Structure storing CESA engine information.
> */
> @@ -448,6 +453,8 @@ struct mv_cesa_engine {
> struct gen_pool *pool;
> struct crypto_queue queue;
> atomic_t load;
> + struct mv_cesa_tdma_chain chain;
> + struct list_head complete_queue;
> };
>
> /**
> @@ -608,6 +615,29 @@ struct mv_cesa_ahash_req {
>
> extern struct mv_cesa_dev *cesa_dev;
>
> +
> +static inline void
> +mv_cesa_engine_enqueue_complete_request(struct mv_cesa_engine *engine,
> + struct crypto_async_request *req)
> +{
> + list_add_tail(&req->list, &engine->complete_queue);
> +}
> +
> +static inline struct crypto_async_request *
> +mv_cesa_engine_dequeue_complete_request(struct mv_cesa_engine *engine)
> +{
> + struct crypto_async_request *req;
> +
> + req = list_first_entry_or_null(&engine->complete_queue,
> + struct crypto_async_request,
> + list);
> + if (req)
> + list_del(&req->list);
> +
> + return req;
> +}
> +
> +
> static inline enum mv_cesa_req_type
> mv_cesa_req_get_type(struct mv_cesa_req *req)
> {
> @@ -689,6 +719,10 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
> int mv_cesa_queue_req(struct crypto_async_request *req,
> struct mv_cesa_req *creq);
>
> +struct crypto_async_request *
> +mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine,
> + struct crypto_async_request **backlog);
> +
> static inline struct mv_cesa_engine *mv_cesa_select_engine(int weight)
> {
> int i;
> @@ -794,6 +828,9 @@ static inline int mv_cesa_dma_process(struct mv_cesa_req *dreq,
> void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
> struct mv_cesa_engine *engine);
> void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq);
> +void mv_cesa_tdma_chain(struct mv_cesa_engine *engine,
> + struct mv_cesa_req *dreq);
> +int mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status);
>
>
> static inline void
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index 28894be..a9ca0dc 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -390,6 +390,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
> goto err_free_tdma;
>
> basereq->chain = chain;
> + basereq->chain.last->flags |= CESA_TDMA_END_OF_REQ;
>
> return 0;
>
> @@ -447,7 +448,6 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
> mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_OP_CRYPT_ONLY,
> CESA_SA_DESC_CFG_OP_MSK);
>
> - /* TODO: add a threshold for DMA usage */
> if (cesa_dev->caps->has_tdma)
> ret = mv_cesa_ablkcipher_dma_req_init(req, tmpl);
> else
> diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
> index b7cfc42..c7e5a46 100644
> --- a/drivers/crypto/marvell/hash.c
> +++ b/drivers/crypto/marvell/hash.c
> @@ -172,6 +172,9 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
> for (i = 0; i < digsize / 4; i++)
> writel_relaxed(creq->state[i], engine->regs + CESA_IVDIG(i));
>
> + mv_cesa_adjust_op(engine, &creq->op_tmpl);
> + memcpy_toio(engine->sram, &creq->op_tmpl, sizeof(creq->op_tmpl));
> +
> if (creq->cache_ptr)
> memcpy_toio(engine->sram + CESA_SA_DATA_SRAM_OFFSET,
> creq->cache, creq->cache_ptr);
> @@ -647,6 +650,9 @@ static int mv_cesa_ahash_dma_req_init(struct ahash_request *req)
> else
> creq->cache_ptr = 0;
>
> + basereq->chain.last->flags |= (CESA_TDMA_END_OF_REQ |
> + CESA_TDMA_BREAK_CHAIN);
> +
> return 0;
>
> err_free_tdma:
> diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
> index 9d944ad..8de8c83 100644
> --- a/drivers/crypto/marvell/tdma.c
> +++ b/drivers/crypto/marvell/tdma.c
> @@ -99,6 +99,92 @@ void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
> }
> }
>
> +void mv_cesa_tdma_chain(struct mv_cesa_engine *engine,
> + struct mv_cesa_req *dreq)
> +{
> + if (engine->chain.first == NULL && engine->chain.last == NULL) {
> + engine->chain.first = dreq->chain.first;
> + engine->chain.last = dreq->chain.last;
> + } else {
> + struct mv_cesa_tdma_desc *last;
> +
> + last = engine->chain.last;
> + last->next = dreq->chain.first;
> + engine->chain.last = dreq->chain.last;
> +
> + if (!(last->flags & CESA_TDMA_BREAK_CHAIN))
> + last->next_dma = dreq->chain.first->cur_dma;
> + }
> +}
> +
> +int mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status)
> +{
> + struct crypto_async_request *req = NULL;
> + struct mv_cesa_tdma_desc *tdma = NULL, *next = NULL;
> + dma_addr_t tdma_cur;
> + int res = 0;
> +
> + tdma_cur = readl(engine->regs + CESA_TDMA_CUR);
> +
> + for (tdma = engine->chain.first; tdma; tdma = next) {
> + spin_lock_bh(&engine->lock);
> + next = tdma->next;
> + spin_unlock_bh(&engine->lock);
> +
> + if (tdma->flags & CESA_TDMA_END_OF_REQ) {
> + struct crypto_async_request *backlog = NULL;
> + struct mv_cesa_ctx *ctx;
> + u32 current_status;
> +
> + spin_lock_bh(&engine->lock);
> + /*
> + * if req is NULL, this means we're processing the
> + * request in engine->req.
> + */
> + if (!req)
> + req = engine->req;
> + else
> + req = mv_cesa_dequeue_req_locked(engine,
> + &backlog);
> +
> + /* Re-chaining to the next request */
> + engine->chain.first = tdma->next;
> + tdma->next = NULL;
> +
> + /* If this is the last request, clear the chain */
> + if (engine->chain.first == NULL)
> + engine->chain.last = NULL;
> + spin_unlock_bh(&engine->lock);
> +
> + ctx = crypto_tfm_ctx(req->tfm);
> + current_status = (tdma->cur_dma == tdma_cur) ?
> + status : CESA_SA_INT_ACC0_IDMA_DONE;
> + res = ctx->ops->process(req, current_status);
> + ctx->ops->complete(req);
> +
> + if (res == 0)
> + mv_cesa_engine_enqueue_complete_request(engine,
> + req);
> +
> + if (backlog)
> + backlog->complete(backlog, -EINPROGRESS);
> + }
> +
> + if (res || tdma->cur_dma == tdma_cur)
> + break;
> + }
> +
> + /* Save the last request in error to engine->req, so that the core
> + * knows which request was fautly */
Please use the standard comment style for over 80 char comments:
/*
* <long message>
*/
> + if (res) {
> + spin_lock_bh(&engine->lock);
> + engine->req = req;
> + spin_unlock_bh(&engine->lock);
> + }
> +
> + return res;
> +}
> +
> static struct mv_cesa_tdma_desc *
> mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
> {
^ permalink raw reply
* Re: [PATCH v3 08/10] crypto: marvell: Add load balancing between engines
From: Boris Brezillon @ 2016-06-21 12:33 UTC (permalink / raw)
To: Romain Perier
Cc: Arnaud Ebalard, Gregory Clement, Thomas Petazzoni,
David S. Miller, Russell King, linux-crypto, linux-arm-kernel
In-Reply-To: <1466496520-28806-9-git-send-email-romain.perier@free-electrons.com>
On Tue, 21 Jun 2016 10:08:38 +0200
Romain Perier <romain.perier@free-electrons.com> wrote:
> This commits adds support for fine grained load balancing on
> multi-engine IPs. The engine is pre-selected based on its current load
> and on the weight of the crypto request that is about to be processed.
> The global crypto queue is also moved to each engine. These changes are
> required to allow chaining crypto requests at the DMA level. By using
> a crypto queue per engine, we make sure that we keep the state of the
> tdma chain synchronized with the crypto queue. We also reduce contention
> on 'cesa_dev->lock' and improve parallelism.
>
> Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
> ---
>
> Changes in v3:
>
> - Renamed mv_cesa_dequeue_req_unlocked => mv_cesa_dequeue_req_locked
>
> Changes in v2:
>
> - Reworded the commit message
> - Moved the code about SRAM I/O operations from this commit to
> a separated commit (see PATCH 07/10).
>
> drivers/crypto/marvell/cesa.c | 34 ++++++++++++------------
> drivers/crypto/marvell/cesa.h | 29 +++++++++++++++++----
> drivers/crypto/marvell/cipher.c | 57 ++++++++++++++++++-----------------------
> drivers/crypto/marvell/hash.c | 50 ++++++++++++++----------------------
> 4 files changed, 84 insertions(+), 86 deletions(-)
>
> diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
> index af96426..c0497ac 100644
> --- a/drivers/crypto/marvell/cesa.c
> +++ b/drivers/crypto/marvell/cesa.c
> @@ -40,16 +40,14 @@ MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if over
>
> struct mv_cesa_dev *cesa_dev;
>
> -static void mv_cesa_dequeue_req_unlocked(struct mv_cesa_engine *engine)
> +static void mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine)
> {
> struct crypto_async_request *req, *backlog;
> struct mv_cesa_ctx *ctx;
>
> - spin_lock_bh(&cesa_dev->lock);
> - backlog = crypto_get_backlog(&cesa_dev->queue);
> - req = crypto_dequeue_request(&cesa_dev->queue);
> + backlog = crypto_get_backlog(&engine->queue);
> + req = crypto_dequeue_request(&engine->queue);
> engine->req = req;
> - spin_unlock_bh(&cesa_dev->lock);
>
> if (!req)
> return;
> @@ -58,7 +56,6 @@ static void mv_cesa_dequeue_req_unlocked(struct mv_cesa_engine *engine)
> backlog->complete(backlog, -EINPROGRESS);
>
> ctx = crypto_tfm_ctx(req->tfm);
> - ctx->ops->prepare(req, engine);
> ctx->ops->step(req);
> }
>
> @@ -96,7 +93,7 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
> if (res != -EINPROGRESS) {
> spin_lock_bh(&engine->lock);
> engine->req = NULL;
> - mv_cesa_dequeue_req_unlocked(engine);
> + mv_cesa_dequeue_req_locked(engine);
> spin_unlock_bh(&engine->lock);
> ctx->ops->complete(req);
> ctx->ops->cleanup(req);
> @@ -116,21 +113,19 @@ int mv_cesa_queue_req(struct crypto_async_request *req,
> struct mv_cesa_req *creq)
> {
> int ret;
> - int i;
> + struct mv_cesa_engine *engine = creq->engine;
>
> - spin_lock_bh(&cesa_dev->lock);
> - ret = crypto_enqueue_request(&cesa_dev->queue, req);
> - spin_unlock_bh(&cesa_dev->lock);
> + spin_lock_bh(&engine->lock);
> + ret = crypto_enqueue_request(&engine->queue, req);
> + spin_unlock_bh(&engine->lock);
>
> if (ret != -EINPROGRESS)
> return ret;
>
> - for (i = 0; i < cesa_dev->caps->nengines; i++) {
> - spin_lock_bh(&cesa_dev->engines[i].lock);
> - if (!cesa_dev->engines[i].req)
> - mv_cesa_dequeue_req_unlocked(&cesa_dev->engines[i]);
> - spin_unlock_bh(&cesa_dev->engines[i].lock);
> - }
> + spin_lock_bh(&engine->lock);
> + if (!engine->req)
> + mv_cesa_dequeue_req_locked(engine);
> + spin_unlock_bh(&engine->lock);
>
> return -EINPROGRESS;
> }
> @@ -425,7 +420,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
> return -ENOMEM;
>
> spin_lock_init(&cesa->lock);
> - crypto_init_queue(&cesa->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
> +
> res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
> cesa->regs = devm_ioremap_resource(dev, res);
> if (IS_ERR(cesa->regs))
> @@ -498,6 +493,9 @@ static int mv_cesa_probe(struct platform_device *pdev)
> engine);
> if (ret)
> goto err_cleanup;
> +
> + crypto_init_queue(&engine->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
> + atomic_set(&engine->load, 0);
> }
>
> cesa_dev = cesa;
> diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
> index c463528..644be35 100644
> --- a/drivers/crypto/marvell/cesa.h
> +++ b/drivers/crypto/marvell/cesa.h
> @@ -400,7 +400,6 @@ struct mv_cesa_dev_dma {
> * @regs: device registers
> * @sram_size: usable SRAM size
> * @lock: device lock
> - * @queue: crypto request queue
> * @engines: array of engines
> * @dma: dma pools
> *
> @@ -412,7 +411,6 @@ struct mv_cesa_dev {
> struct device *dev;
> unsigned int sram_size;
> spinlock_t lock;
> - struct crypto_queue queue;
> struct mv_cesa_engine *engines;
> struct mv_cesa_dev_dma *dma;
> };
> @@ -431,6 +429,8 @@ struct mv_cesa_dev {
> * @int_mask: interrupt mask cache
> * @pool: memory pool pointing to the memory region reserved in
> * SRAM
> + * @queue: fifo of the pending crypto requests
> + * @load: engine load counter, useful for load balancing
> *
> * Structure storing CESA engine information.
> */
> @@ -446,11 +446,12 @@ struct mv_cesa_engine {
> size_t max_req_len;
> u32 int_mask;
> struct gen_pool *pool;
> + struct crypto_queue queue;
> + atomic_t load;
> };
>
> /**
> * struct mv_cesa_req_ops - CESA request operations
> - * @prepare: prepare a request to be executed on the specified engine
> * @process: process a request chunk result (should return 0 if the
> * operation, -EINPROGRESS if it needs more steps or an error
> * code)
> @@ -460,8 +461,6 @@ struct mv_cesa_engine {
> * when it is needed.
> */
> struct mv_cesa_req_ops {
> - void (*prepare)(struct crypto_async_request *req,
> - struct mv_cesa_engine *engine);
> int (*process)(struct crypto_async_request *req, u32 status);
> void (*step)(struct crypto_async_request *req);
> void (*cleanup)(struct crypto_async_request *req);
> @@ -690,6 +689,26 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
> int mv_cesa_queue_req(struct crypto_async_request *req,
> struct mv_cesa_req *creq);
>
> +static inline struct mv_cesa_engine *mv_cesa_select_engine(int weight)
> +{
> + int i;
> + u32 min_load = U32_MAX;
> + struct mv_cesa_engine *selected = NULL;
> +
> + for (i = 0; i < cesa_dev->caps->nengines; i++) {
> + struct mv_cesa_engine *engine = cesa_dev->engines + i;
> + u32 load = atomic_read(&engine->load);
> + if (load < min_load) {
> + min_load = load;
> + selected = engine;
> + }
> + }
> +
> + atomic_add(weight, &selected->load);
> +
> + return selected;
> +}
> +
> /*
> * Helper function that indicates whether a crypto request needs to be
> * cleaned up or not after being enqueued using mv_cesa_queue_req().
> diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
> index 79d4175..28894be 100644
> --- a/drivers/crypto/marvell/cipher.c
> +++ b/drivers/crypto/marvell/cipher.c
> @@ -214,6 +214,7 @@ mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
> struct mv_cesa_engine *engine = creq->base.engine;
> unsigned int ivsize;
>
> + atomic_sub(ablkreq->nbytes, &engine->load);
> ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
>
> if (mv_cesa_req_get_type(&creq->base) == CESA_DMA_REQ) {
> @@ -231,7 +232,6 @@ mv_cesa_ablkcipher_complete(struct crypto_async_request *req)
> static const struct mv_cesa_req_ops mv_cesa_ablkcipher_req_ops = {
> .step = mv_cesa_ablkcipher_step,
> .process = mv_cesa_ablkcipher_process,
> - .prepare = mv_cesa_ablkcipher_prepare,
> .cleanup = mv_cesa_ablkcipher_req_cleanup,
> .complete = mv_cesa_ablkcipher_complete,
> };
> @@ -456,29 +456,41 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
> return ret;
> }
>
> -static int mv_cesa_des_op(struct ablkcipher_request *req,
> - struct mv_cesa_op_ctx *tmpl)
> +static int mv_cesa_ablkcipher_queue_req(struct ablkcipher_request *req,
> + struct mv_cesa_op_ctx *tmpl)
> {
> - struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> - struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> int ret;
> -
> - mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_CRYPTM_DES,
> - CESA_SA_DESC_CFG_CRYPTM_MSK);
> -
> - memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES_KEY_SIZE);
> + struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> + struct mv_cesa_engine *engine;
>
> ret = mv_cesa_ablkcipher_req_init(req, tmpl);
> if (ret)
> return ret;
>
> + engine = mv_cesa_select_engine(req->nbytes);
> + mv_cesa_ablkcipher_prepare(&req->base, engine);
> +
> ret = mv_cesa_queue_req(&req->base, &creq->base);
> +
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ablkcipher_cleanup(req);
>
> return ret;
> }
>
> +static int mv_cesa_des_op(struct ablkcipher_request *req,
> + struct mv_cesa_op_ctx *tmpl)
> +{
> + struct mv_cesa_des_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> +
> + mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_CRYPTM_DES,
> + CESA_SA_DESC_CFG_CRYPTM_MSK);
> +
> + memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES_KEY_SIZE);
> +
> + return mv_cesa_ablkcipher_queue_req(req, tmpl);
> +}
> +
> static int mv_cesa_ecb_des_encrypt(struct ablkcipher_request *req)
> {
> struct mv_cesa_op_ctx tmpl;
> @@ -580,24 +592,14 @@ struct crypto_alg mv_cesa_cbc_des_alg = {
> static int mv_cesa_des3_op(struct ablkcipher_request *req,
> struct mv_cesa_op_ctx *tmpl)
> {
> - struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_des3_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> - int ret;
>
> mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_CRYPTM_3DES,
> CESA_SA_DESC_CFG_CRYPTM_MSK);
>
> memcpy(tmpl->ctx.blkcipher.key, ctx->key, DES3_EDE_KEY_SIZE);
>
> - ret = mv_cesa_ablkcipher_req_init(req, tmpl);
> - if (ret)
> - return ret;
> -
> - ret = mv_cesa_queue_req(&req->base, &creq->base);
> - if (mv_cesa_req_needs_cleanup(&req->base, ret))
> - mv_cesa_ablkcipher_cleanup(req);
> -
> - return ret;
> + return mv_cesa_ablkcipher_queue_req(req, tmpl);
> }
>
> static int mv_cesa_ecb_des3_ede_encrypt(struct ablkcipher_request *req)
> @@ -707,9 +709,8 @@ struct crypto_alg mv_cesa_cbc_des3_ede_alg = {
> static int mv_cesa_aes_op(struct ablkcipher_request *req,
> struct mv_cesa_op_ctx *tmpl)
> {
> - struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(req);
> struct mv_cesa_aes_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
> - int ret, i;
> + int i;
> u32 *key;
> u32 cfg;
>
> @@ -732,15 +733,7 @@ static int mv_cesa_aes_op(struct ablkcipher_request *req,
> CESA_SA_DESC_CFG_CRYPTM_MSK |
> CESA_SA_DESC_CFG_AES_LEN_MSK);
>
> - ret = mv_cesa_ablkcipher_req_init(req, tmpl);
> - if (ret)
> - return ret;
> -
> - ret = mv_cesa_queue_req(&req->base, &creq->base);
> - if (mv_cesa_req_needs_cleanup(&req->base, ret))
> - mv_cesa_ablkcipher_cleanup(req);
> -
> - return ret;
> + return mv_cesa_ablkcipher_queue_req(req, tmpl);
> }
>
> static int mv_cesa_ecb_aes_encrypt(struct ablkcipher_request *req)
> diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
> index e1f8acd..b7cfc42 100644
> --- a/drivers/crypto/marvell/hash.c
> +++ b/drivers/crypto/marvell/hash.c
> @@ -335,6 +335,8 @@ static void mv_cesa_ahash_complete(struct crypto_async_request *req)
> result[i] = cpu_to_be32(creq->state[i]);
> }
> }
> +
> + atomic_sub(ahashreq->nbytes, &engine->load);
> }
>
> static void mv_cesa_ahash_prepare(struct crypto_async_request *req,
> @@ -365,7 +367,6 @@ static void mv_cesa_ahash_req_cleanup(struct crypto_async_request *req)
> static const struct mv_cesa_req_ops mv_cesa_ahash_req_ops = {
> .step = mv_cesa_ahash_step,
> .process = mv_cesa_ahash_process,
> - .prepare = mv_cesa_ahash_prepare,
> .cleanup = mv_cesa_ahash_req_cleanup,
> .complete = mv_cesa_ahash_complete,
> };
> @@ -682,13 +683,13 @@ static int mv_cesa_ahash_req_init(struct ahash_request *req, bool *cached)
> return ret;
> }
>
> -static int mv_cesa_ahash_update(struct ahash_request *req)
> +static int mv_cesa_ahash_queue_req(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> + struct mv_cesa_engine *engine;
> bool cached = false;
> int ret;
>
> - creq->len += req->nbytes;
> ret = mv_cesa_ahash_req_init(req, &cached);
> if (ret)
> return ret;
> @@ -696,61 +697,48 @@ static int mv_cesa_ahash_update(struct ahash_request *req)
> if (cached)
> return 0;
>
> + engine = mv_cesa_select_engine(req->nbytes);
> + mv_cesa_ahash_prepare(&req->base, engine);
> +
> ret = mv_cesa_queue_req(&req->base, &creq->base);
> +
> if (mv_cesa_req_needs_cleanup(&req->base, ret))
> mv_cesa_ahash_cleanup(req);
>
> return ret;
> }
>
> +static int mv_cesa_ahash_update(struct ahash_request *req)
> +{
> + struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> +
> + creq->len += req->nbytes;
> +
> + return mv_cesa_ahash_queue_req(req);
> +}
> +
> static int mv_cesa_ahash_final(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> struct mv_cesa_op_ctx *tmpl = &creq->op_tmpl;
> - bool cached = false;
> - int ret;
>
> mv_cesa_set_mac_op_total_len(tmpl, creq->len);
> creq->last_req = true;
> req->nbytes = 0;
>
> - ret = mv_cesa_ahash_req_init(req, &cached);
> - if (ret)
> - return ret;
> -
> - if (cached)
> - return 0;
> -
> - ret = mv_cesa_queue_req(&req->base, &creq->base);
> - if (mv_cesa_req_needs_cleanup(&req->base, ret))
> - mv_cesa_ahash_cleanup(req);
> -
> - return ret;
> + return mv_cesa_ahash_queue_req(req);
> }
>
> static int mv_cesa_ahash_finup(struct ahash_request *req)
> {
> struct mv_cesa_ahash_req *creq = ahash_request_ctx(req);
> struct mv_cesa_op_ctx *tmpl = &creq->op_tmpl;
> - bool cached = false;
> - int ret;
>
> creq->len += req->nbytes;
> mv_cesa_set_mac_op_total_len(tmpl, creq->len);
> creq->last_req = true;
>
> - ret = mv_cesa_ahash_req_init(req, &cached);
> - if (ret)
> - return ret;
> -
> - if (cached)
> - return 0;
> -
> - ret = mv_cesa_queue_req(&req->base, &creq->base);
> - if (mv_cesa_req_needs_cleanup(&req->base, ret))
> - mv_cesa_ahash_cleanup(req);
> -
> - return ret;
> + return mv_cesa_ahash_queue_req(req);
> }
>
> static int mv_cesa_ahash_export(struct ahash_request *req, void *hash,
^ permalink raw reply
* Re: [PATCH v4 0/5] /dev/random - a new approach
From: David Jaša @ 2016-06-21 12:25 UTC (permalink / raw)
To: Theodore Ts'o
Cc: Stephan Mueller, Andi Kleen, sandyinchina, Jason Cooper,
John Denker, H. Peter Anvin, Joe Perches, Pavel Machek,
George Spelvin, linux-crypto, linux-kernel
In-Reply-To: <20160618144408.GA5344@thunk.org>
Hi,
On So, 2016-06-18 at 10:44 -0400, Theodore Ts'o wrote:
> On Fri, Jun 17, 2016 at 03:56:13PM +0200, David Jaša wrote:
> > I was thinking along the lines that "almost every important package
> > supports FreeBSD as well where they have to handle the condition so
> > option to switch to Rather Break Than Generate Weak Keys would be nice"
> > - but I didn't expect that systemd could be a roadblock here. :-/
>
> It wasn't just systemd; it also broke OpenWRT and Ubuntu Quantal
> systems from booting.
>
> > I was also thinking of little devices where OpenWRT or proprietary
> > Linux-based systems run that ended up with predictable keys way too
> > ofter (or as in OpenWRT's case, with cumbersome tutorials how to
> > generate keys elsewhere).
>
> OpenWRT and other embedded devices (a) generally use a single master
> oscillator to drive everything, and (b) often use RISC architectures
> such as MIPS.
>
> Which means that arguments of the form ``the Intel L1 / L2 cache
> architecture is ****soooo**** complicated that no human could possibly
> figure out how they would affect timing calculations, and besides, my
> generator passes FIPS 140-2 tests (never mind AES(NSA_KEY, CNTR++)
this
> also passes the FIPS 140-2 statistical tests)'' --- which I normally
> have trouble believing --- are even harder for me to believe.
>
> At the end of the day, with these devices you really badly need a
> hardware RNG.
and this.
It seems much easier to me to embed AES(NSA_KEY, CNTR++) logic directly
to HW RNG compared to tweaking of every microarchitecture to make
jitter/maxwell/havege return known numbers that are going to be mixed
with other entropy anyway (won't they?). So if I put the bits together
correctly, HW RNG helps getting more random numbers but itself is
insufficient to ensure that random numbers are truly random...
Cheers,
David Jaša
> We can't generate randomness out of thin air. The only
> thing you really can do requires user space help, which is to generate
> keys lazily, or as late as possible, so you can gather as much entropy
> as you can --- and to feed in measurements from the WiFi (RSSI
> measurements, MAC addresses seen, etc.) This won't help much if you
> have an FBI van parked outside your house trying to carry out a
> TEMPEST attack, but hopefully it provides some protection against a
> remote attacker who isn't try to carry out an on-premises attack.
>
> Cheers,
>
> - Ted
^ permalink raw reply
* [[PATCH v2] 11/11] hwrng: exynos - fixup IO accesors
From: Ben Dooks @ 2016-06-21 10:20 UTC (permalink / raw)
To: matthew.leach, k.kozlowski, linux-samsung-soc, linux-arm-kernel
Cc: linux-kernel, Ben Dooks, Matt Mackall, linux-crypto
In-Reply-To: <1466504432-24187-1-git-send-email-ben.dooks@codethink.co.uk>
The __raw IO functions are not endian safe, so use the readl_relaxed
and writel_relaxed versions of these.
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
---
CC: Matt Mackall <mpm@selenic.com>
CC: Krzysztof Kozlowski <k.kozlowski@samsung.com>
CC: linux-crypto@vger.kernel.org
CC: linux-arm-kernel@lists.infradead.org
CC: linux-samsung-soc@vger.kernel.org
---
drivers/char/hw_random/exynos-rng.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/char/hw_random/exynos-rng.c b/drivers/char/hw_random/exynos-rng.c
index ed44561..23d3585 100644
--- a/drivers/char/hw_random/exynos-rng.c
+++ b/drivers/char/hw_random/exynos-rng.c
@@ -45,12 +45,12 @@ struct exynos_rng {
static u32 exynos_rng_readl(struct exynos_rng *rng, u32 offset)
{
- return __raw_readl(rng->mem + offset);
+ return readl_relaxed(rng->mem + offset);
}
static void exynos_rng_writel(struct exynos_rng *rng, u32 val, u32 offset)
{
- __raw_writel(val, rng->mem + offset);
+ writel_relaxed(val, rng->mem + offset);
}
static int exynos_rng_configure(struct exynos_rng *exynos_rng)
--
2.8.1
^ permalink raw reply related
* [PATCH 5/5] crypto: ghash-ce - Fix cryptd reordering
From: Herbert Xu @ 2016-06-21 8:55 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
In-Reply-To: <20160621085321.GA21044@gondor.apana.org.au>
This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.
The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
This patch also removes the redundant use of cryptd in the async
init function as init never touches the FPU.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
arch/arm/crypto/ghash-ce-glue.c | 40 +++++++++++++++++-----------------------
1 file changed, 17 insertions(+), 23 deletions(-)
diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c
index 03a39fe..1568cb5 100644
--- a/arch/arm/crypto/ghash-ce-glue.c
+++ b/arch/arm/crypto/ghash-ce-glue.c
@@ -154,30 +154,23 @@ static int ghash_async_init(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
- if (!may_use_simd()) {
- memcpy(cryptd_req, req, sizeof(*req));
- ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
- return crypto_ahash_init(cryptd_req);
- } else {
- struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
- struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
-
- desc->tfm = child;
- desc->flags = req->base.flags;
- return crypto_shash_init(desc);
- }
+ desc->tfm = child;
+ desc->flags = req->base.flags;
+ return crypto_shash_init(desc);
}
static int ghash_async_update(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd()) {
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
- struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
-
+ if (!may_use_simd() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_update(cryptd_req);
@@ -190,12 +183,12 @@ static int ghash_async_update(struct ahash_request *req)
static int ghash_async_final(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd()) {
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
- struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
-
+ if (!may_use_simd() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_final(cryptd_req);
@@ -212,7 +205,8 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!may_use_simd()) {
+ if (!may_use_simd() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_digest(cryptd_req);
^ permalink raw reply related
* [PATCH 3/5] crypto: ablk_helper - Fix cryptd reordering
From: Herbert Xu @ 2016-06-21 8:55 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
In-Reply-To: <20160621085321.GA21044@gondor.apana.org.au>
This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.
The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
crypto/ablk_helper.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/crypto/ablk_helper.c b/crypto/ablk_helper.c
index e1fcf53..1441f07 100644
--- a/crypto/ablk_helper.c
+++ b/crypto/ablk_helper.c
@@ -71,7 +71,8 @@ int ablk_encrypt(struct ablkcipher_request *req)
struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- if (!may_use_simd()) {
+ if (!may_use_simd() ||
+ (in_atomic() && cryptd_ablkcipher_queued(ctx->cryptd_tfm))) {
struct ablkcipher_request *cryptd_req =
ablkcipher_request_ctx(req);
@@ -90,7 +91,8 @@ int ablk_decrypt(struct ablkcipher_request *req)
struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req);
struct async_helper_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- if (!may_use_simd()) {
+ if (!may_use_simd() ||
+ (in_atomic() && cryptd_ablkcipher_queued(ctx->cryptd_tfm))) {
struct ablkcipher_request *cryptd_req =
ablkcipher_request_ctx(req);
^ permalink raw reply related
* [PATCH 4/5] crypto: ghash-clmulni - Fix cryptd reordering
From: Herbert Xu @ 2016-06-21 8:55 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
In-Reply-To: <20160621085321.GA21044@gondor.apana.org.au>
This patch fixes an old bug where requests can be reordered because
some are processed by cryptd while others are processed directly
in softirq context.
The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
This patch also removes the redundant use of cryptd in the async
init function as init never touches the FPU.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
arch/x86/crypto/ghash-clmulni-intel_glue.c | 40 ++++++++++++-----------------
1 file changed, 17 insertions(+), 23 deletions(-)
diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c
index a69321a..0420bab 100644
--- a/arch/x86/crypto/ghash-clmulni-intel_glue.c
+++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c
@@ -168,30 +168,23 @@ static int ghash_async_init(struct ahash_request *req)
struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
- if (!irq_fpu_usable()) {
- memcpy(cryptd_req, req, sizeof(*req));
- ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
- return crypto_ahash_init(cryptd_req);
- } else {
- struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
- struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
-
- desc->tfm = child;
- desc->flags = req->base.flags;
- return crypto_shash_init(desc);
- }
+ desc->tfm = child;
+ desc->flags = req->base.flags;
+ return crypto_shash_init(desc);
}
static int ghash_async_update(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable()) {
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
- struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
-
+ if (!irq_fpu_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_update(cryptd_req);
@@ -204,12 +197,12 @@ static int ghash_async_update(struct ahash_request *req)
static int ghash_async_final(struct ahash_request *req)
{
struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable()) {
- struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
- struct ghash_async_ctx *ctx = crypto_ahash_ctx(tfm);
- struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
-
+ if (!irq_fpu_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_final(cryptd_req);
@@ -249,7 +242,8 @@ static int ghash_async_digest(struct ahash_request *req)
struct ahash_request *cryptd_req = ahash_request_ctx(req);
struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
- if (!irq_fpu_usable()) {
+ if (!irq_fpu_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
memcpy(cryptd_req, req, sizeof(*req));
ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
return crypto_ahash_digest(cryptd_req);
^ permalink raw reply related
* [PATCH 1/5] crypto: cryptd - Add helpers to check whether a tfm is queued
From: Herbert Xu @ 2016-06-21 8:55 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
In-Reply-To: <20160621085321.GA21044@gondor.apana.org.au>
This patch adds helpers to check whether a given tfm is currently
queued. This is meant to be used by ablk_helper and similar
entities to ensure that no reordering is introduced because of
requests queued in cryptd with respect to requests being processed
in softirq context.
The per-cpu queue length limit is also increased to 1000 in line
with network limits.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
crypto/cryptd.c | 132 +++++++++++++++++++++++++++++++++++++++++-------
include/crypto/cryptd.h | 5 +
2 files changed, 118 insertions(+), 19 deletions(-)
diff --git a/crypto/cryptd.c b/crypto/cryptd.c
index 7921251..cf8037a 100644
--- a/crypto/cryptd.c
+++ b/crypto/cryptd.c
@@ -22,6 +22,7 @@
#include <crypto/internal/aead.h>
#include <crypto/cryptd.h>
#include <crypto/crypto_wq.h>
+#include <linux/atomic.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/kernel.h>
@@ -31,7 +32,7 @@
#include <linux/sched.h>
#include <linux/slab.h>
-#define CRYPTD_MAX_CPU_QLEN 100
+#define CRYPTD_MAX_CPU_QLEN 1000
struct cryptd_cpu_queue {
struct crypto_queue queue;
@@ -58,6 +59,7 @@ struct aead_instance_ctx {
};
struct cryptd_blkcipher_ctx {
+ atomic_t refcnt;
struct crypto_blkcipher *child;
};
@@ -66,6 +68,7 @@ struct cryptd_blkcipher_request_ctx {
};
struct cryptd_hash_ctx {
+ atomic_t refcnt;
struct crypto_shash *child;
};
@@ -75,6 +78,7 @@ struct cryptd_hash_request_ctx {
};
struct cryptd_aead_ctx {
+ atomic_t refcnt;
struct crypto_aead *child;
};
@@ -118,11 +122,29 @@ static int cryptd_enqueue_request(struct cryptd_queue *queue,
{
int cpu, err;
struct cryptd_cpu_queue *cpu_queue;
+ struct crypto_tfm *tfm;
+ atomic_t *refcnt;
+ bool may_backlog;
cpu = get_cpu();
cpu_queue = this_cpu_ptr(queue->cpu_queue);
err = crypto_enqueue_request(&cpu_queue->queue, request);
+
+ refcnt = crypto_tfm_ctx(request->tfm);
+ may_backlog = request->flags & CRYPTO_TFM_REQ_MAY_BACKLOG;
+
+ if (err == -EBUSY && !may_backlog)
+ goto out_put_cpu;
+
queue_work_on(cpu, kcrypto_wq, &cpu_queue->work);
+
+ if (!atomic_read(refcnt))
+ goto out_put_cpu;
+
+ tfm = request->tfm;
+ atomic_inc(refcnt);
+
+out_put_cpu:
put_cpu();
return err;
@@ -206,7 +228,10 @@ static void cryptd_blkcipher_crypt(struct ablkcipher_request *req,
unsigned int len))
{
struct cryptd_blkcipher_request_ctx *rctx;
+ struct cryptd_blkcipher_ctx *ctx;
+ struct crypto_ablkcipher *tfm;
struct blkcipher_desc desc;
+ int refcnt;
rctx = ablkcipher_request_ctx(req);
@@ -222,9 +247,16 @@ static void cryptd_blkcipher_crypt(struct ablkcipher_request *req,
req->base.complete = rctx->complete;
out:
+ tfm = crypto_ablkcipher_reqtfm(req);
+ ctx = crypto_ablkcipher_ctx(tfm);
+ refcnt = atomic_read(&ctx->refcnt);
+
local_bh_disable();
rctx->complete(&req->base, err);
local_bh_enable();
+
+ if (err != -EINPROGRESS && refcnt && atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_ablkcipher(tfm);
}
static void cryptd_blkcipher_encrypt(struct crypto_async_request *req, int err)
@@ -456,6 +488,21 @@ static int cryptd_hash_enqueue(struct ahash_request *req,
return cryptd_enqueue_request(queue, &req->base);
}
+static void cryptd_hash_complete(struct ahash_request *req, int err)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct cryptd_hash_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_hash_request_ctx *rctx = ahash_request_ctx(req);
+ int refcnt = atomic_read(&ctx->refcnt);
+
+ local_bh_disable();
+ rctx->complete(&req->base, err);
+ local_bh_enable();
+
+ if (err != -EINPROGRESS && refcnt && atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_ahash(tfm);
+}
+
static void cryptd_hash_init(struct crypto_async_request *req_async, int err)
{
struct cryptd_hash_ctx *ctx = crypto_tfm_ctx(req_async->tfm);
@@ -475,9 +522,7 @@ static void cryptd_hash_init(struct crypto_async_request *req_async, int err)
req->base.complete = rctx->complete;
out:
- local_bh_disable();
- rctx->complete(&req->base, err);
- local_bh_enable();
+ cryptd_hash_complete(req, err);
}
static int cryptd_hash_init_enqueue(struct ahash_request *req)
@@ -500,9 +545,7 @@ static void cryptd_hash_update(struct crypto_async_request *req_async, int err)
req->base.complete = rctx->complete;
out:
- local_bh_disable();
- rctx->complete(&req->base, err);
- local_bh_enable();
+ cryptd_hash_complete(req, err);
}
static int cryptd_hash_update_enqueue(struct ahash_request *req)
@@ -523,9 +566,7 @@ static void cryptd_hash_final(struct crypto_async_request *req_async, int err)
req->base.complete = rctx->complete;
out:
- local_bh_disable();
- rctx->complete(&req->base, err);
- local_bh_enable();
+ cryptd_hash_complete(req, err);
}
static int cryptd_hash_final_enqueue(struct ahash_request *req)
@@ -546,9 +587,7 @@ static void cryptd_hash_finup(struct crypto_async_request *req_async, int err)
req->base.complete = rctx->complete;
out:
- local_bh_disable();
- rctx->complete(&req->base, err);
- local_bh_enable();
+ cryptd_hash_complete(req, err);
}
static int cryptd_hash_finup_enqueue(struct ahash_request *req)
@@ -575,9 +614,7 @@ static void cryptd_hash_digest(struct crypto_async_request *req_async, int err)
req->base.complete = rctx->complete;
out:
- local_bh_disable();
- rctx->complete(&req->base, err);
- local_bh_enable();
+ cryptd_hash_complete(req, err);
}
static int cryptd_hash_digest_enqueue(struct ahash_request *req)
@@ -688,7 +725,10 @@ static void cryptd_aead_crypt(struct aead_request *req,
int (*crypt)(struct aead_request *req))
{
struct cryptd_aead_request_ctx *rctx;
+ struct cryptd_aead_ctx *ctx;
crypto_completion_t compl;
+ struct crypto_aead *tfm;
+ int refcnt;
rctx = aead_request_ctx(req);
compl = rctx->complete;
@@ -697,10 +737,18 @@ static void cryptd_aead_crypt(struct aead_request *req,
goto out;
aead_request_set_tfm(req, child);
err = crypt( req );
+
out:
+ tfm = crypto_aead_reqtfm(req);
+ ctx = crypto_aead_ctx(tfm);
+ refcnt = atomic_read(&ctx->refcnt);
+
local_bh_disable();
compl(&req->base, err);
local_bh_enable();
+
+ if (err != -EINPROGRESS && refcnt && atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_aead(tfm);
}
static void cryptd_aead_encrypt(struct crypto_async_request *areq, int err)
@@ -883,6 +931,7 @@ struct cryptd_ablkcipher *cryptd_alloc_ablkcipher(const char *alg_name,
u32 type, u32 mask)
{
char cryptd_alg_name[CRYPTO_MAX_ALG_NAME];
+ struct cryptd_blkcipher_ctx *ctx;
struct crypto_tfm *tfm;
if (snprintf(cryptd_alg_name, CRYPTO_MAX_ALG_NAME,
@@ -899,6 +948,9 @@ struct cryptd_ablkcipher *cryptd_alloc_ablkcipher(const char *alg_name,
return ERR_PTR(-EINVAL);
}
+ ctx = crypto_tfm_ctx(tfm);
+ atomic_set(&ctx->refcnt, 1);
+
return __cryptd_ablkcipher_cast(__crypto_ablkcipher_cast(tfm));
}
EXPORT_SYMBOL_GPL(cryptd_alloc_ablkcipher);
@@ -910,9 +962,20 @@ struct crypto_blkcipher *cryptd_ablkcipher_child(struct cryptd_ablkcipher *tfm)
}
EXPORT_SYMBOL_GPL(cryptd_ablkcipher_child);
+bool cryptd_ablkcipher_queued(struct cryptd_ablkcipher *tfm)
+{
+ struct cryptd_blkcipher_ctx *ctx = crypto_ablkcipher_ctx(&tfm->base);
+
+ return atomic_read(&ctx->refcnt) - 1;
+}
+EXPORT_SYMBOL_GPL(cryptd_ablkcipher_queued);
+
void cryptd_free_ablkcipher(struct cryptd_ablkcipher *tfm)
{
- crypto_free_ablkcipher(&tfm->base);
+ struct cryptd_blkcipher_ctx *ctx = crypto_ablkcipher_ctx(&tfm->base);
+
+ if (atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_ablkcipher(&tfm->base);
}
EXPORT_SYMBOL_GPL(cryptd_free_ablkcipher);
@@ -920,6 +983,7 @@ struct cryptd_ahash *cryptd_alloc_ahash(const char *alg_name,
u32 type, u32 mask)
{
char cryptd_alg_name[CRYPTO_MAX_ALG_NAME];
+ struct cryptd_hash_ctx *ctx;
struct crypto_ahash *tfm;
if (snprintf(cryptd_alg_name, CRYPTO_MAX_ALG_NAME,
@@ -933,6 +997,9 @@ struct cryptd_ahash *cryptd_alloc_ahash(const char *alg_name,
return ERR_PTR(-EINVAL);
}
+ ctx = crypto_ahash_ctx(tfm);
+ atomic_set(&ctx->refcnt, 1);
+
return __cryptd_ahash_cast(tfm);
}
EXPORT_SYMBOL_GPL(cryptd_alloc_ahash);
@@ -952,9 +1019,20 @@ struct shash_desc *cryptd_shash_desc(struct ahash_request *req)
}
EXPORT_SYMBOL_GPL(cryptd_shash_desc);
+bool cryptd_ahash_queued(struct cryptd_ahash *tfm)
+{
+ struct cryptd_hash_ctx *ctx = crypto_ahash_ctx(&tfm->base);
+
+ return atomic_read(&ctx->refcnt) - 1;
+}
+EXPORT_SYMBOL_GPL(cryptd_ahash_queued);
+
void cryptd_free_ahash(struct cryptd_ahash *tfm)
{
- crypto_free_ahash(&tfm->base);
+ struct cryptd_hash_ctx *ctx = crypto_ahash_ctx(&tfm->base);
+
+ if (atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_ahash(&tfm->base);
}
EXPORT_SYMBOL_GPL(cryptd_free_ahash);
@@ -962,6 +1040,7 @@ struct cryptd_aead *cryptd_alloc_aead(const char *alg_name,
u32 type, u32 mask)
{
char cryptd_alg_name[CRYPTO_MAX_ALG_NAME];
+ struct cryptd_aead_ctx *ctx;
struct crypto_aead *tfm;
if (snprintf(cryptd_alg_name, CRYPTO_MAX_ALG_NAME,
@@ -974,6 +1053,10 @@ struct cryptd_aead *cryptd_alloc_aead(const char *alg_name,
crypto_free_aead(tfm);
return ERR_PTR(-EINVAL);
}
+
+ ctx = crypto_aead_ctx(tfm);
+ atomic_set(&ctx->refcnt, 1);
+
return __cryptd_aead_cast(tfm);
}
EXPORT_SYMBOL_GPL(cryptd_alloc_aead);
@@ -986,9 +1069,20 @@ struct crypto_aead *cryptd_aead_child(struct cryptd_aead *tfm)
}
EXPORT_SYMBOL_GPL(cryptd_aead_child);
+bool cryptd_aead_queued(struct cryptd_aead *tfm)
+{
+ struct cryptd_aead_ctx *ctx = crypto_aead_ctx(&tfm->base);
+
+ return atomic_read(&ctx->refcnt) - 1;
+}
+EXPORT_SYMBOL_GPL(cryptd_aead_queued);
+
void cryptd_free_aead(struct cryptd_aead *tfm)
{
- crypto_free_aead(&tfm->base);
+ struct cryptd_aead_ctx *ctx = crypto_aead_ctx(&tfm->base);
+
+ if (atomic_dec_and_test(&ctx->refcnt))
+ crypto_free_aead(&tfm->base);
}
EXPORT_SYMBOL_GPL(cryptd_free_aead);
diff --git a/include/crypto/cryptd.h b/include/crypto/cryptd.h
index 1547f54..bc792d5 100644
--- a/include/crypto/cryptd.h
+++ b/include/crypto/cryptd.h
@@ -31,6 +31,7 @@ static inline struct cryptd_ablkcipher *__cryptd_ablkcipher_cast(
struct cryptd_ablkcipher *cryptd_alloc_ablkcipher(const char *alg_name,
u32 type, u32 mask);
struct crypto_blkcipher *cryptd_ablkcipher_child(struct cryptd_ablkcipher *tfm);
+bool cryptd_ablkcipher_queued(struct cryptd_ablkcipher *tfm);
void cryptd_free_ablkcipher(struct cryptd_ablkcipher *tfm);
struct cryptd_ahash {
@@ -48,6 +49,8 @@ struct cryptd_ahash *cryptd_alloc_ahash(const char *alg_name,
u32 type, u32 mask);
struct crypto_shash *cryptd_ahash_child(struct cryptd_ahash *tfm);
struct shash_desc *cryptd_shash_desc(struct ahash_request *req);
+/* Must be called without moving CPUs. */
+bool cryptd_ahash_queued(struct cryptd_ahash *tfm);
void cryptd_free_ahash(struct cryptd_ahash *tfm);
struct cryptd_aead {
@@ -64,6 +67,8 @@ struct cryptd_aead *cryptd_alloc_aead(const char *alg_name,
u32 type, u32 mask);
struct crypto_aead *cryptd_aead_child(struct cryptd_aead *tfm);
+/* Must be called without moving CPUs. */
+bool cryptd_aead_queued(struct cryptd_aead *tfm);
void cryptd_free_aead(struct cryptd_aead *tfm);
^ permalink raw reply related
* [PATCH 2/5] crypto: aesni - Fix cryptd reordering problem on gcm
From: Herbert Xu @ 2016-06-21 8:55 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
In-Reply-To: <20160621085321.GA21044@gondor.apana.org.au>
This patch fixes an old bug where gcm requests can be reordered
because some are processed by cryptd while others are processed
directly in softirq context.
The fix is to always postpone to cryptd if there are currently
requests outstanding from the same tfm.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
---
arch/x86/crypto/aesni-intel_glue.c | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 5b7fa14..9e15572e 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -1098,9 +1098,12 @@ static int rfc4106_encrypt(struct aead_request *req)
struct cryptd_aead **ctx = crypto_aead_ctx(tfm);
struct cryptd_aead *cryptd_tfm = *ctx;
- aead_request_set_tfm(req, irq_fpu_usable() ?
- cryptd_aead_child(cryptd_tfm) :
- &cryptd_tfm->base);
+ tfm = &cryptd_tfm->base;
+ if (irq_fpu_usable() && (!in_atomic() ||
+ !cryptd_aead_queued(cryptd_tfm)))
+ tfm = cryptd_aead_child(cryptd_tfm);
+
+ aead_request_set_tfm(req, tfm);
return crypto_aead_encrypt(req);
}
@@ -1111,9 +1114,12 @@ static int rfc4106_decrypt(struct aead_request *req)
struct cryptd_aead **ctx = crypto_aead_ctx(tfm);
struct cryptd_aead *cryptd_tfm = *ctx;
- aead_request_set_tfm(req, irq_fpu_usable() ?
- cryptd_aead_child(cryptd_tfm) :
- &cryptd_tfm->base);
+ tfm = &cryptd_tfm->base;
+ if (irq_fpu_usable() && (!in_atomic() ||
+ !cryptd_aead_queued(cryptd_tfm)))
+ tfm = cryptd_aead_child(cryptd_tfm);
+
+ aead_request_set_tfm(req, tfm);
return crypto_aead_decrypt(req);
}
^ permalink raw reply related
* [PATCH 0/5] crypto: Fix IPsec reordering caused by cryptd
From: Herbert Xu @ 2016-06-21 8:53 UTC (permalink / raw)
To: Raj Ammanur, Steffen Klassert, Linux Crypto Mailing List
Hi:
I finally got around to working on this. I quickly gave up on the
notion of hijacking the queued requests as we may end up overwhelming
our caller.
So the solution is the obvious one of using cryptd as long as there
are requests queued there belonging to the same tfm. This is
totally untested so please tread carefully.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] crypto: use timespec64 for jent_get_nstime
From: Stephan Mueller @ 2016-06-21 8:39 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Herbert Xu, David S. Miller, y2038, Kees Cook, Alexander Kuleshov,
linux-crypto, linux-kernel, John Stultz
In-Reply-To: <3776557.os2uYrMn6M@wuerfel>
Am Dienstag, 21. Juni 2016, 10:32:23 schrieb Arnd Bergmann:
Hi Arnd,
> On Tuesday, June 21, 2016 8:20:10 AM CEST Stephan Mueller wrote:
> > Am Freitag, 17. Juni 2016, 17:59:41 schrieb Arnd Bergmann:
> >
> > Hi Arnd,
> >
> > > The jent_get_nstime() function uses __getnstimeofday() to get
> > > something similar to a 64-bit nanosecond counter. As we want
> > > to get rid of struct timespec to fix the y2038 overflow,
> > > this patch changes the code to use __getnstimeofday64()
> > > instead, which returns a timespec64 structure.
> > >
> > > Nothing changes about the algorithm, but it looks like it
> > > might be better to use
> > >
> > > *out = ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
> > >
> > > or even
> > >
> > > *out = ktime_get_raw_fast_ns();
> >
> > I tested ktime_get_raw_fast_ns and it works perfectly well for the use
> > case, i.e. the RNG behavior is indistinguishable from RDTSC on x86.
> >
> > Which time source is used for this function? I am wondering about
> > architectures other than X86.
>
> (adding John Stultz to Cc, he can correct me if I say something wrong)
>
> All ktime_get_* and *get*timeofday() functions use the same clocksource,
> which is configurable and picked from the available clocksources,
> using a combination of reliability, latency for reading and accuracy.
>
> Compared to the previous __getnstimeofday(), the difference is
>
> - using "monotonic" timebase instead of "real", so the zero time
> is when the system booted rather than Jan 1 1970
> - "raw" means we don't honor updates for the rate based on ntp,
> which is probably better as the ntp state might be observable
> over the net (it probably doesn't matter, but it can't hurt)
> - "fast" means that in very rare cases, the time might appear
> to go backwards (it probably can't happen here because you are not
> called in an NMI).
Thank you for the explanation.
>
> There may be other clocksources in the system that are more appropriate
> for the purpose of jent_get_nstime(), but I don't think we have a way
> of exposing them at the moment, because we have not needed it in the
> past.
>
> I assume what you want is the highest-resolution clocksource, and
> you don't really care about
Correct, all I need is a high-res clocksource.
>
> > Note, this function is used as a fallback when random_get_entropy is not
> > implemented. In addition the Jitter RNG has an online health test which
> > will catch the failure of the time stamp operation. Hence, even if this
> > function may not be suitable on one or the other arch, it should not hurt
> > though.
> Ok. For the ARM architecture, I think most (maybe all) of the modern ARMv7
> platforms have either an architected "global timer" or have their own
> replacement, while the majority of the older ARMv4/v5/v6 machines don't
> have one.
>
> However, there are some machines that implement "sched_clock" but don't
> implement get_cycles (which is the default for random_get_entropy()).
>
> It's possible that there are cases where it's better to call sched_clock()
> than ktime_get_raw_fast_ns() as a fallback, though I could not find
> specific examples so far.
Given your explanation above, considering that the RNG would scream in the
kernel log when the timer is too coarse and the fact that we have not had bug
reports on this issue, I would think that a replacement call that is equal to
the old __getnstimeofday is fine.
Do you want to update your patch or shall I hand in the patch?
Ciao
Stephan
^ permalink raw reply
* Re: [PATCH] crypto: use timespec64 for jent_get_nstime
From: Arnd Bergmann @ 2016-06-21 8:32 UTC (permalink / raw)
To: Stephan Mueller
Cc: Herbert Xu, Alexander Kuleshov, y2038, linux-kernel, John Stultz,
linux-crypto, David S. Miller, Kees Cook
In-Reply-To: <10630099.jHjGp8aIto@positron.chronox.de>
On Tuesday, June 21, 2016 8:20:10 AM CEST Stephan Mueller wrote:
> Am Freitag, 17. Juni 2016, 17:59:41 schrieb Arnd Bergmann:
>
> Hi Arnd,
>
> > The jent_get_nstime() function uses __getnstimeofday() to get
> > something similar to a 64-bit nanosecond counter. As we want
> > to get rid of struct timespec to fix the y2038 overflow,
> > this patch changes the code to use __getnstimeofday64()
> > instead, which returns a timespec64 structure.
> >
> > Nothing changes about the algorithm, but it looks like it
> > might be better to use
> >
> > *out = ts.tv_sec * NSEC_PER_SEC + ts.tv_nsec;
> >
> > or even
> >
> > *out = ktime_get_raw_fast_ns();
>
> I tested ktime_get_raw_fast_ns and it works perfectly well for the use case,
> i.e. the RNG behavior is indistinguishable from RDTSC on x86.
>
> Which time source is used for this function? I am wondering about
> architectures other than X86.
(adding John Stultz to Cc, he can correct me if I say something wrong)
All ktime_get_* and *get*timeofday() functions use the same clocksource,
which is configurable and picked from the available clocksources,
using a combination of reliability, latency for reading and accuracy.
Compared to the previous __getnstimeofday(), the difference is
- using "monotonic" timebase instead of "real", so the zero time
is when the system booted rather than Jan 1 1970
- "raw" means we don't honor updates for the rate based on ntp,
which is probably better as the ntp state might be observable
over the net (it probably doesn't matter, but it can't hurt)
- "fast" means that in very rare cases, the time might appear
to go backwards (it probably can't happen here because you are not
called in an NMI).
There may be other clocksources in the system that are more appropriate
for the purpose of jent_get_nstime(), but I don't think we have a way
of exposing them at the moment, because we have not needed it in the
past.
I assume what you want is the highest-resolution clocksource, and
you don't really care about
> Note, this function is used as a fallback when random_get_entropy is not
> implemented. In addition the Jitter RNG has an online health test which will
> catch the failure of the time stamp operation. Hence, even if this function
> may not be suitable on one or the other arch, it should not hurt though.
Ok. For the ARM architecture, I think most (maybe all) of the modern ARMv7
platforms have either an architected "global timer" or have their own
replacement, while the majority of the older ARMv4/v5/v6 machines don't
have one.
However, there are some machines that implement "sched_clock" but don't
implement get_cycles (which is the default for random_get_entropy()).
It's possible that there are cases where it's better to call sched_clock()
than ktime_get_raw_fast_ns() as a fallback, though I could not find
specific examples so far.
Arnd
_______________________________________________
Y2038 mailing list
Y2038@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/y2038
^ permalink raw reply
* [PATCH v3 02/10] crypto: marvell: Check engine is not already running when enabling a req
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
Add a BUG_ON() call when the driver tries to launch a crypto request
while the engine is still processing the previous one. This replaces
a silent system hang by a verbose kernel panic with the associated
backtrace to let the user know that something went wrong in the CESA
driver.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
---
Changes in v3:
- Fixed incorrectly aligned parameter for BUG_ON in
mv_cesa_ablkcipher_std_step
Changes in v2:
- Reworded the commit message
- Fixed cosmetic changes
drivers/crypto/marvell/cipher.c | 2 ++
drivers/crypto/marvell/hash.c | 2 ++
drivers/crypto/marvell/tdma.c | 2 ++
3 files changed, 6 insertions(+)
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index dcf1fce..8c1432e 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -106,6 +106,8 @@ static void mv_cesa_ablkcipher_std_step(struct ablkcipher_request *req)
mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD) &
+ CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index 7ca2e0f..80bddd7 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -237,6 +237,8 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
mv_cesa_set_int_mask(engine, CESA_SA_INT_ACCEL0_DONE);
writel_relaxed(CESA_SA_CFG_PARA_DIS, engine->regs + CESA_SA_CFG);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD) &
+ CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 7642798..8c86bb6 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -53,6 +53,8 @@ void mv_cesa_dma_step(struct mv_cesa_tdma_req *dreq)
engine->regs + CESA_SA_CFG);
writel_relaxed(dreq->chain.first->cur_dma,
engine->regs + CESA_TDMA_NEXT_ADDR);
+ BUG_ON(readl(engine->regs + CESA_SA_CMD) &
+ CESA_SA_CMD_EN_CESA_SA_ACCL0);
writel(CESA_SA_CMD_EN_CESA_SA_ACCL0, engine->regs + CESA_SA_CMD);
}
--
2.7.4
^ permalink raw reply related
* [PATCH v3 01/10] crypto: marvell: Add a macro constant for the size of the crypto queue
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
Adding a macro constant to be used for the size of the crypto queue,
instead of using a numeric value directly. It will be easier to
maintain in case we add more than one crypto queue of the same size.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index 056a754..fb403e1 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -31,6 +31,9 @@
#include "cesa.h"
+/* Limit of the crypto queue before reaching the backlog */
+#define CESA_CRYPTO_DEFAULT_MAX_QLEN 50
+
static int allhwsupport = !IS_ENABLED(CONFIG_CRYPTO_DEV_MV_CESA);
module_param_named(allhwsupport, allhwsupport, int, 0444);
MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if overlaps with the mv_cesa driver)");
@@ -416,7 +419,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
return -ENOMEM;
spin_lock_init(&cesa->lock);
- crypto_init_queue(&cesa->queue, 50);
+ crypto_init_queue(&cesa->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
cesa->regs = devm_ioremap_resource(dev, res);
if (IS_ERR(cesa->regs))
--
2.7.4
^ permalink raw reply related
* [PATCH v3 03/10] crypto: marvell: Fix wrong type check in dma functions
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
So far, the way that the type of a TDMA operation was checked was wrong.
We have to use the type mask in order to get the right part of the flag
containing the type of the operation.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
---
drivers/crypto/marvell/tdma.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 8c86bb6..de8c253 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -64,8 +64,9 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
for (tdma = dreq->chain.first; tdma;) {
struct mv_cesa_tdma_desc *old_tdma = tdma;
+ u32 type = tdma->flags & CESA_TDMA_TYPE_MSK;
- if (tdma->flags & CESA_TDMA_OP)
+ if (type == CESA_TDMA_OP)
dma_pool_free(cesa_dev->dma->op_pool, tdma->op,
le32_to_cpu(tdma->src));
@@ -90,7 +91,7 @@ void mv_cesa_dma_prepare(struct mv_cesa_tdma_req *dreq,
if (tdma->flags & CESA_TDMA_SRC_IN_SRAM)
tdma->src = cpu_to_le32(tdma->src + engine->sram_dma);
- if (tdma->flags & CESA_TDMA_OP)
+ if ((tdma->flags & CESA_TDMA_TYPE_MSK) == CESA_TDMA_OP)
mv_cesa_adjust_op(engine, tdma->op);
}
}
--
2.7.4
^ permalink raw reply related
* [PATCH v3 04/10] crypto: marvell: Copy IV vectors by DMA transfers for acipher requests
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Gregory Clement, Thomas Petazzoni, David S. Miller, Russell King,
linux-crypto, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
Add a TDMA descriptor at the end of the request for copying the
output IV vector via a DMA transfer. This is a good way for offloading
as much as processing as possible to the DMA and the crypto engine.
This is also required for processing multiple cipher requests
in chained mode, otherwise the content of the IV vector would be
overwritten by the last processed request.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
---
Changes in v3:
- Fixed coding style issues
Changes in v2:
- Reworded the commit message, the term 'asynchronously' was
ambigous
- Changed the value of CESA_TDMA_IV from 4 to 3
- Adding missing blank lines
- Rewrote the function mv_cesa_ablkcipher_process to something more
readable.
- Fixed a bug about how the type of a TDMA operation was tested in
mv_cesa_dma_cleanup and mv_cesa_dma_prepare, I created a separated
commit for that (see PATCH 03/10)
- Renamed variables in mv_cesa_dma_add_iv_op
- Removed the flag CESA_TDMA_DATA from mv_cesa_dma_add_iv_op (not
needed)
drivers/crypto/marvell/cesa.c | 4 ++++
drivers/crypto/marvell/cesa.h | 5 +++++
drivers/crypto/marvell/cipher.c | 31 ++++++++++++++++++++++---------
drivers/crypto/marvell/tdma.c | 29 +++++++++++++++++++++++++++++
4 files changed, 60 insertions(+), 9 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index fb403e1..93700cd 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -312,6 +312,10 @@ static int mv_cesa_dev_dma_init(struct mv_cesa_dev *cesa)
if (!dma->padding_pool)
return -ENOMEM;
+ dma->iv_pool = dmam_pool_create("cesa_iv", dev, 16, 1, 0);
+ if (!dma->iv_pool)
+ return -ENOMEM;
+
cesa->dma = dma;
return 0;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 74071e4..685a627 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -275,6 +275,7 @@ struct mv_cesa_op_ctx {
#define CESA_TDMA_DUMMY 0
#define CESA_TDMA_DATA 1
#define CESA_TDMA_OP 2
+#define CESA_TDMA_IV 3
/**
* struct mv_cesa_tdma_desc - TDMA descriptor
@@ -390,6 +391,7 @@ struct mv_cesa_dev_dma {
struct dma_pool *op_pool;
struct dma_pool *cache_pool;
struct dma_pool *padding_pool;
+ struct dma_pool *iv_pool;
};
/**
@@ -790,6 +792,9 @@ mv_cesa_tdma_desc_iter_init(struct mv_cesa_tdma_chain *chain)
memset(chain, 0, sizeof(*chain));
}
+int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
+ u32 size, u32 flags, gfp_t gfp_flags);
+
struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
const struct mv_cesa_op_ctx *op_templ,
bool skip_ctx,
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index ec23609..908be86 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -118,6 +118,7 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
struct mv_cesa_engine *engine = sreq->base.engine;
size_t len;
+ unsigned int ivsize;
len = sg_pcopy_from_buffer(req->dst, creq->dst_nents,
engine->sram + CESA_SA_DATA_SRAM_OFFSET,
@@ -127,6 +128,10 @@ static int mv_cesa_ablkcipher_std_process(struct ablkcipher_request *req,
if (sreq->offset < req->nbytes)
return -EINPROGRESS;
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
+ memcpy_fromio(req->info,
+ engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET, ivsize);
+
return 0;
}
@@ -135,21 +140,20 @@ static int mv_cesa_ablkcipher_process(struct crypto_async_request *req,
{
struct ablkcipher_request *ablkreq = ablkcipher_request_cast(req);
struct mv_cesa_ablkcipher_req *creq = ablkcipher_request_ctx(ablkreq);
- struct mv_cesa_ablkcipher_std_req *sreq = &creq->req.std;
- struct mv_cesa_engine *engine = sreq->base.engine;
+ struct mv_cesa_tdma_req *dreq;
+ unsigned int ivsize;
int ret;
- if (creq->req.base.type == CESA_DMA_REQ)
- ret = mv_cesa_dma_process(&creq->req.dma, status);
- else
- ret = mv_cesa_ablkcipher_std_process(ablkreq, status);
+ if (creq->req.base.type == CESA_STD_REQ)
+ return mv_cesa_ablkcipher_std_process(ablkreq, status);
+ ret = mv_cesa_dma_process(&creq->req.dma, status);
if (ret)
return ret;
- memcpy_fromio(ablkreq->info,
- engine->sram + CESA_SA_CRYPT_IV_SRAM_OFFSET,
- crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq)));
+ dreq = &creq->req.dma;
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(ablkreq));
+ memcpy_fromio(ablkreq->info, dreq->chain.last->data, ivsize);
return 0;
}
@@ -302,6 +306,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
struct mv_cesa_tdma_chain chain;
bool skip_ctx = false;
int ret;
+ unsigned int ivsize;
dreq->base.type = CESA_DMA_REQ;
dreq->chain.first = NULL;
@@ -360,6 +365,14 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
} while (mv_cesa_ablkcipher_req_iter_next_op(&iter));
+ /* Add output data for IV */
+ ivsize = crypto_ablkcipher_ivsize(crypto_ablkcipher_reqtfm(req));
+ ret = mv_cesa_dma_add_iv_op(&chain, CESA_SA_CRYPT_IV_SRAM_OFFSET,
+ ivsize, CESA_TDMA_SRC_IN_SRAM, flags);
+
+ if (ret)
+ goto err_free_tdma;
+
dreq->chain = chain;
return 0;
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index de8c253..01dda58 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -69,6 +69,9 @@ void mv_cesa_dma_cleanup(struct mv_cesa_tdma_req *dreq)
if (type == CESA_TDMA_OP)
dma_pool_free(cesa_dev->dma->op_pool, tdma->op,
le32_to_cpu(tdma->src));
+ else if (type == CESA_TDMA_IV)
+ dma_pool_free(cesa_dev->dma->iv_pool, tdma->data,
+ le32_to_cpu(tdma->dst));
tdma = tdma->next;
dma_pool_free(cesa_dev->dma->tdma_desc_pool, old_tdma,
@@ -121,6 +124,32 @@ mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
return new_tdma;
}
+int mv_cesa_dma_add_iv_op(struct mv_cesa_tdma_chain *chain, dma_addr_t src,
+ u32 size, u32 flags, gfp_t gfp_flags)
+{
+
+ struct mv_cesa_tdma_desc *tdma;
+ u8 *iv;
+ dma_addr_t dma_handle;
+
+ tdma = mv_cesa_dma_add_desc(chain, gfp_flags);
+ if (IS_ERR(tdma))
+ return PTR_ERR(tdma);
+
+ iv = dma_pool_alloc(cesa_dev->dma->iv_pool, flags, &dma_handle);
+ if (!iv)
+ return -ENOMEM;
+
+ tdma->byte_cnt = cpu_to_le32(size | BIT(31));
+ tdma->src = src;
+ tdma->dst = cpu_to_le32(dma_handle);
+ tdma->data = iv;
+
+ flags &= (CESA_TDMA_DST_IN_SRAM | CESA_TDMA_SRC_IN_SRAM);
+ tdma->flags = flags | CESA_TDMA_IV;
+ return 0;
+}
+
struct mv_cesa_op_ctx *mv_cesa_dma_add_op(struct mv_cesa_tdma_chain *chain,
const struct mv_cesa_op_ctx *op_templ,
bool skip_ctx,
--
2.7.4
^ permalink raw reply related
* [PATCH v3 10/10] crypto: marvell: Increase the size of the crypto queue
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Thomas Petazzoni, Russell King, linux-crypto, Gregory Clement,
David S. Miller, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
Now that crypto requests are chained together at the DMA level, we
increase the size of the crypto queue for each engine. The result is
that as the backlog list is reached later, it does not stop the crypto
stack from sending asychronous requests, so more cryptographic tasks
are processed by the engines.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Acked-by: Boris Brezillon <boris.brezillon@free-electrons.com>
---
drivers/crypto/marvell/cesa.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index bb91156..5147073 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -32,7 +32,7 @@
#include "cesa.h"
/* Limit of the crypto queue before reaching the backlog */
-#define CESA_CRYPTO_DEFAULT_MAX_QLEN 50
+#define CESA_CRYPTO_DEFAULT_MAX_QLEN 128
static int allhwsupport = !IS_ENABLED(CONFIG_CRYPTO_DEV_MV_CESA);
module_param_named(allhwsupport, allhwsupport, int, 0444);
--
2.7.4
^ permalink raw reply related
* [PATCH v3 09/10] crypto: marvell: Add support for chaining crypto requests in TDMA mode
From: Romain Perier @ 2016-06-21 8:08 UTC (permalink / raw)
To: Boris Brezillon, Arnaud Ebalard
Cc: Thomas Petazzoni, Russell King, linux-crypto, Gregory Clement,
David S. Miller, linux-arm-kernel
In-Reply-To: <1466496520-28806-1-git-send-email-romain.perier@free-electrons.com>
The Cryptographic Engines and Security Accelerators (CESA) supports the
Multi-Packet Chain Mode. With this mode enabled, multiple tdma requests
can be chained and processed by the hardware without software
intervention. This mode was already activated, however the crypto
requests were not chained together. By doing so, we reduce significantly
the number of IRQs. Instead of being interrupted at the end of each
crypto request, we are interrupted at the end of the last cryptographic
request processed by the engine.
This commits re-factorizes the code, changes the code architecture and
adds the required data structures to chain cryptographic requests
together before sending them to an engine (stopped or possibly already
running).
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
---
Changes in v3:
- Cosmetic changes: Extra blank lines and coding style issues
on prototypes.
Changes in v2:
- Reworded the commit message
- Fixed cosmetic changes: coding styles issues, missing blank lines
- Reworked mv_cesa_rearm_engine: lock handling is simpler
- Removed the call to the complete operation in mv_cesa_std_process,
in case of errors (not required)
- Squashed the removal of the '.prepare' fields (cipher.c, hash.c)
into another commit (see PATCH 08/10).
- In mv_cesa_tdma_process only treat the status argument for the last
request, use 'normal' status for the other ones.
- Added a comment for explaining how the errors are notified to the
cesa core.
drivers/crypto/marvell/cesa.c | 115 +++++++++++++++++++++++++++++++---------
drivers/crypto/marvell/cesa.h | 39 +++++++++++++-
drivers/crypto/marvell/cipher.c | 2 +-
drivers/crypto/marvell/hash.c | 6 +++
drivers/crypto/marvell/tdma.c | 86 ++++++++++++++++++++++++++++++
5 files changed, 221 insertions(+), 27 deletions(-)
diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c
index c0497ac..bb91156 100644
--- a/drivers/crypto/marvell/cesa.c
+++ b/drivers/crypto/marvell/cesa.c
@@ -40,14 +40,33 @@ MODULE_PARM_DESC(allhwsupport, "Enable support for all hardware (even it if over
struct mv_cesa_dev *cesa_dev;
-static void mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine)
+struct crypto_async_request *
+mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine,
+ struct crypto_async_request **backlog)
{
- struct crypto_async_request *req, *backlog;
- struct mv_cesa_ctx *ctx;
+ struct crypto_async_request *req;
- backlog = crypto_get_backlog(&engine->queue);
+ *backlog = crypto_get_backlog(&engine->queue);
req = crypto_dequeue_request(&engine->queue);
- engine->req = req;
+
+ if (!req)
+ return NULL;
+
+ return req;
+}
+
+static void mv_cesa_rearm_engine(struct mv_cesa_engine *engine)
+{
+ struct crypto_async_request *req = NULL, *backlog = NULL;
+ struct mv_cesa_ctx *ctx;
+
+
+ spin_lock_bh(&engine->lock);
+ if (!engine->req) {
+ req = mv_cesa_dequeue_req_locked(engine, &backlog);
+ engine->req = req;
+ }
+ spin_unlock_bh(&engine->lock);
if (!req)
return;
@@ -57,6 +76,46 @@ static void mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine)
ctx = crypto_tfm_ctx(req->tfm);
ctx->ops->step(req);
+
+ return;
+}
+
+static int mv_cesa_std_process(struct mv_cesa_engine *engine, u32 status)
+{
+ struct crypto_async_request *req;
+ struct mv_cesa_ctx *ctx;
+ int res;
+
+ req = engine->req;
+ ctx = crypto_tfm_ctx(req->tfm);
+ res = ctx->ops->process(req, status);
+
+ if (res == 0) {
+ ctx->ops->complete(req);
+ mv_cesa_engine_enqueue_complete_request(engine, req);
+ } else if (res == -EINPROGRESS) {
+ ctx->ops->step(req);
+ }
+
+ return res;
+}
+
+static int mv_cesa_int_process(struct mv_cesa_engine *engine, u32 status)
+{
+ if (engine->chain.first && engine->chain.last)
+ return mv_cesa_tdma_process(engine, status);
+
+ return mv_cesa_std_process(engine, status);
+}
+
+static inline void
+mv_cesa_complete_req(struct mv_cesa_ctx *ctx, struct crypto_async_request *req,
+ int res)
+{
+ ctx->ops->cleanup(req);
+ local_bh_disable();
+ req->complete(req, res);
+ local_bh_enable();
}
static irqreturn_t mv_cesa_int(int irq, void *priv)
@@ -83,26 +142,31 @@ static irqreturn_t mv_cesa_int(int irq, void *priv)
writel(~status, engine->regs + CESA_SA_FPGA_INT_STATUS);
writel(~status, engine->regs + CESA_SA_INT_STATUS);
+ /* Process fetched requests */
+ res = mv_cesa_int_process(engine, status & mask);
ret = IRQ_HANDLED;
+
spin_lock_bh(&engine->lock);
req = engine->req;
+ if (res != -EINPROGRESS)
+ engine->req = NULL;
spin_unlock_bh(&engine->lock);
- if (req) {
- ctx = crypto_tfm_ctx(req->tfm);
- res = ctx->ops->process(req, status & mask);
- if (res != -EINPROGRESS) {
- spin_lock_bh(&engine->lock);
- engine->req = NULL;
- mv_cesa_dequeue_req_locked(engine);
- spin_unlock_bh(&engine->lock);
- ctx->ops->complete(req);
- ctx->ops->cleanup(req);
- local_bh_disable();
- req->complete(req, res);
- local_bh_enable();
- } else {
- ctx->ops->step(req);
- }
+
+ ctx = crypto_tfm_ctx(req->tfm);
+
+ if (res && res != -EINPROGRESS)
+ mv_cesa_complete_req(ctx, req, res);
+
+ /* Launch the next pending request */
+ mv_cesa_rearm_engine(engine);
+
+ /* Iterate over the complete queue */
+ while (true) {
+ req = mv_cesa_engine_dequeue_complete_request(engine);
+ if (!req)
+ break;
+
+ mv_cesa_complete_req(ctx, req, 0);
}
}
@@ -116,16 +180,16 @@ int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_engine *engine = creq->engine;
spin_lock_bh(&engine->lock);
+ if (mv_cesa_req_get_type(creq) == CESA_DMA_REQ)
+ mv_cesa_tdma_chain(engine, creq);
+
ret = crypto_enqueue_request(&engine->queue, req);
spin_unlock_bh(&engine->lock);
if (ret != -EINPROGRESS)
return ret;
- spin_lock_bh(&engine->lock);
- if (!engine->req)
- mv_cesa_dequeue_req_locked(engine);
- spin_unlock_bh(&engine->lock);
+ mv_cesa_rearm_engine(engine);
return -EINPROGRESS;
}
@@ -496,6 +560,7 @@ static int mv_cesa_probe(struct platform_device *pdev)
crypto_init_queue(&engine->queue, CESA_CRYPTO_DEFAULT_MAX_QLEN);
atomic_set(&engine->load, 0);
+ INIT_LIST_HEAD(&engine->complete_queue);
}
cesa_dev = cesa;
diff --git a/drivers/crypto/marvell/cesa.h b/drivers/crypto/marvell/cesa.h
index 644be35..50a1fb2 100644
--- a/drivers/crypto/marvell/cesa.h
+++ b/drivers/crypto/marvell/cesa.h
@@ -271,7 +271,9 @@ struct mv_cesa_op_ctx {
/* TDMA descriptor flags */
#define CESA_TDMA_DST_IN_SRAM BIT(31)
#define CESA_TDMA_SRC_IN_SRAM BIT(30)
-#define CESA_TDMA_TYPE_MSK GENMASK(29, 0)
+#define CESA_TDMA_END_OF_REQ BIT(29)
+#define CESA_TDMA_BREAK_CHAIN BIT(28)
+#define CESA_TDMA_TYPE_MSK GENMASK(27, 0)
#define CESA_TDMA_DUMMY 0
#define CESA_TDMA_DATA 1
#define CESA_TDMA_OP 2
@@ -431,6 +433,9 @@ struct mv_cesa_dev {
* SRAM
* @queue: fifo of the pending crypto requests
* @load: engine load counter, useful for load balancing
+ * @chain: list of the current tdma descriptors being processed
+ * by this engine.
+ * @complete_queue: fifo of the processed requests by the engine
*
* Structure storing CESA engine information.
*/
@@ -448,6 +453,8 @@ struct mv_cesa_engine {
struct gen_pool *pool;
struct crypto_queue queue;
atomic_t load;
+ struct mv_cesa_tdma_chain chain;
+ struct list_head complete_queue;
};
/**
@@ -608,6 +615,29 @@ struct mv_cesa_ahash_req {
extern struct mv_cesa_dev *cesa_dev;
+
+static inline void
+mv_cesa_engine_enqueue_complete_request(struct mv_cesa_engine *engine,
+ struct crypto_async_request *req)
+{
+ list_add_tail(&req->list, &engine->complete_queue);
+}
+
+static inline struct crypto_async_request *
+mv_cesa_engine_dequeue_complete_request(struct mv_cesa_engine *engine)
+{
+ struct crypto_async_request *req;
+
+ req = list_first_entry_or_null(&engine->complete_queue,
+ struct crypto_async_request,
+ list);
+ if (req)
+ list_del(&req->list);
+
+ return req;
+}
+
+
static inline enum mv_cesa_req_type
mv_cesa_req_get_type(struct mv_cesa_req *req)
{
@@ -689,6 +719,10 @@ static inline bool mv_cesa_mac_op_is_first_frag(const struct mv_cesa_op_ctx *op)
int mv_cesa_queue_req(struct crypto_async_request *req,
struct mv_cesa_req *creq);
+struct crypto_async_request *
+mv_cesa_dequeue_req_locked(struct mv_cesa_engine *engine,
+ struct crypto_async_request **backlog);
+
static inline struct mv_cesa_engine *mv_cesa_select_engine(int weight)
{
int i;
@@ -794,6 +828,9 @@ static inline int mv_cesa_dma_process(struct mv_cesa_req *dreq,
void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
struct mv_cesa_engine *engine);
void mv_cesa_dma_cleanup(struct mv_cesa_req *dreq);
+void mv_cesa_tdma_chain(struct mv_cesa_engine *engine,
+ struct mv_cesa_req *dreq);
+int mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status);
static inline void
diff --git a/drivers/crypto/marvell/cipher.c b/drivers/crypto/marvell/cipher.c
index 28894be..a9ca0dc 100644
--- a/drivers/crypto/marvell/cipher.c
+++ b/drivers/crypto/marvell/cipher.c
@@ -390,6 +390,7 @@ static int mv_cesa_ablkcipher_dma_req_init(struct ablkcipher_request *req,
goto err_free_tdma;
basereq->chain = chain;
+ basereq->chain.last->flags |= CESA_TDMA_END_OF_REQ;
return 0;
@@ -447,7 +448,6 @@ static int mv_cesa_ablkcipher_req_init(struct ablkcipher_request *req,
mv_cesa_update_op_cfg(tmpl, CESA_SA_DESC_CFG_OP_CRYPT_ONLY,
CESA_SA_DESC_CFG_OP_MSK);
- /* TODO: add a threshold for DMA usage */
if (cesa_dev->caps->has_tdma)
ret = mv_cesa_ablkcipher_dma_req_init(req, tmpl);
else
diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c
index b7cfc42..c7e5a46 100644
--- a/drivers/crypto/marvell/hash.c
+++ b/drivers/crypto/marvell/hash.c
@@ -172,6 +172,9 @@ static void mv_cesa_ahash_std_step(struct ahash_request *req)
for (i = 0; i < digsize / 4; i++)
writel_relaxed(creq->state[i], engine->regs + CESA_IVDIG(i));
+ mv_cesa_adjust_op(engine, &creq->op_tmpl);
+ memcpy_toio(engine->sram, &creq->op_tmpl, sizeof(creq->op_tmpl));
+
if (creq->cache_ptr)
memcpy_toio(engine->sram + CESA_SA_DATA_SRAM_OFFSET,
creq->cache, creq->cache_ptr);
@@ -647,6 +650,9 @@ static int mv_cesa_ahash_dma_req_init(struct ahash_request *req)
else
creq->cache_ptr = 0;
+ basereq->chain.last->flags |= (CESA_TDMA_END_OF_REQ |
+ CESA_TDMA_BREAK_CHAIN);
+
return 0;
err_free_tdma:
diff --git a/drivers/crypto/marvell/tdma.c b/drivers/crypto/marvell/tdma.c
index 9d944ad..8de8c83 100644
--- a/drivers/crypto/marvell/tdma.c
+++ b/drivers/crypto/marvell/tdma.c
@@ -99,6 +99,92 @@ void mv_cesa_dma_prepare(struct mv_cesa_req *dreq,
}
}
+void mv_cesa_tdma_chain(struct mv_cesa_engine *engine,
+ struct mv_cesa_req *dreq)
+{
+ if (engine->chain.first == NULL && engine->chain.last == NULL) {
+ engine->chain.first = dreq->chain.first;
+ engine->chain.last = dreq->chain.last;
+ } else {
+ struct mv_cesa_tdma_desc *last;
+
+ last = engine->chain.last;
+ last->next = dreq->chain.first;
+ engine->chain.last = dreq->chain.last;
+
+ if (!(last->flags & CESA_TDMA_BREAK_CHAIN))
+ last->next_dma = dreq->chain.first->cur_dma;
+ }
+}
+
+int mv_cesa_tdma_process(struct mv_cesa_engine *engine, u32 status)
+{
+ struct crypto_async_request *req = NULL;
+ struct mv_cesa_tdma_desc *tdma = NULL, *next = NULL;
+ dma_addr_t tdma_cur;
+ int res = 0;
+
+ tdma_cur = readl(engine->regs + CESA_TDMA_CUR);
+
+ for (tdma = engine->chain.first; tdma; tdma = next) {
+ spin_lock_bh(&engine->lock);
+ next = tdma->next;
+ spin_unlock_bh(&engine->lock);
+
+ if (tdma->flags & CESA_TDMA_END_OF_REQ) {
+ struct crypto_async_request *backlog = NULL;
+ struct mv_cesa_ctx *ctx;
+ u32 current_status;
+
+ spin_lock_bh(&engine->lock);
+ /*
+ * if req is NULL, this means we're processing the
+ * request in engine->req.
+ */
+ if (!req)
+ req = engine->req;
+ else
+ req = mv_cesa_dequeue_req_locked(engine,
+ &backlog);
+
+ /* Re-chaining to the next request */
+ engine->chain.first = tdma->next;
+ tdma->next = NULL;
+
+ /* If this is the last request, clear the chain */
+ if (engine->chain.first == NULL)
+ engine->chain.last = NULL;
+ spin_unlock_bh(&engine->lock);
+
+ ctx = crypto_tfm_ctx(req->tfm);
+ current_status = (tdma->cur_dma == tdma_cur) ?
+ status : CESA_SA_INT_ACC0_IDMA_DONE;
+ res = ctx->ops->process(req, current_status);
+ ctx->ops->complete(req);
+
+ if (res == 0)
+ mv_cesa_engine_enqueue_complete_request(engine,
+ req);
+
+ if (backlog)
+ backlog->complete(backlog, -EINPROGRESS);
+ }
+
+ if (res || tdma->cur_dma == tdma_cur)
+ break;
+ }
+
+ /* Save the last request in error to engine->req, so that the core
+ * knows which request was fautly */
+ if (res) {
+ spin_lock_bh(&engine->lock);
+ engine->req = req;
+ spin_unlock_bh(&engine->lock);
+ }
+
+ return res;
+}
+
static struct mv_cesa_tdma_desc *
mv_cesa_dma_add_desc(struct mv_cesa_tdma_chain *chain, gfp_t flags)
{
--
2.7.4
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox