[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

> On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:

> My Armada 385 -device hangs when using marvell-cesa -driver.
> I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> I have also tried with the driver version of Linux 4.7-rc3.
> 
> Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.

I just tested the marvell-cesa with Linux 4.8-rc2 and Armada 385.
There is still the same hanging issue (originally described on
https://bugzilla.kernel.org/show_bug.cgi?id=120111).

Please let me ask which model of Armada board has been verified to work with the
marvell-cesa driver? I'm using a custom board based on 88F6820. Is this SoC
model fully supported by the driver? Do you know if there is some necessary
hardware initializations which must be done on u-boot? I'm using latest Marvell
patch set (2015_T1.0p16) on u-boot.

My test case have been:
# cryptsetup benchmark --cipher aes-cbc

and that always freezes the board.

Best Regards
Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Thomas Petazzoni]

Hello,

I'm adding Romain Perier in Cc, since he is actively working on the
CESA driver.

On Fri, 19 Aug 2016 14:54:50 +0300, radioconfusion at gmail.com wrote:

> > On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:  
> 
> > My Armada 385 -device hangs when using marvell-cesa -driver.
> > I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> > I have also tried with the driver version of Linux 4.7-rc3.
> > 
> > Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.  
> 
> I just tested the marvell-cesa with Linux 4.8-rc2 and Armada 385.
> There is still the same hanging issue (originally described on
> https://bugzilla.kernel.org/show_bug.cgi?id=120111).
> 
> Please let me ask which model of Armada board has been verified to work with the
> marvell-cesa driver?

The Armada 388 GP board.

> I'm using a custom board based on 88F6820. Is this SoC
> model fully supported by the driver? Do you know if there is some necessary
> hardware initializations which must be done on u-boot? I'm using latest Marvell
> patch set (2015_T1.0p16) on u-boot.
> 
> My test case have been:
> # cryptsetup benchmark --cipher aes-cbc
> 
> and that always freezes the board.

Romain, can you test the same scenario, and see if you can reproduce
the hang or not?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Romain Perier]

Hello,

Le 19/08/2016 14:12, Thomas Petazzoni a ?crit :
> Hello,
>
> I'm adding Romain Perier in Cc, since he is actively working on the
> CESA driver.
>
> On Fri, 19 Aug 2016 14:54:50 +0300, radioconfusion at gmail.com wrote:
>
>>> On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:
>>
>>> My Armada 385 -device hangs when using marvell-cesa -driver.
>>> I'm using Linux 4.4.10 and armada-385-db-ap.dts.
>>> I have also tried with the driver version of Linux 4.7-rc3.
>>>
>>> Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.
>>
>> I just tested the marvell-cesa with Linux 4.8-rc2 and Armada 385.
>> There is still the same hanging issue (originally described on
>> https://bugzilla.kernel.org/show_bug.cgi?id=120111).
>>
>> Please let me ask which model of Armada board has been verified to work with the
>> marvell-cesa driver?
>
> The Armada 388 GP board.

For now:
The Armada 388-DB-GP
The Armada 385-DB-AP
The Armada 375-DB


>
>> I'm using a custom board based on 88F6820. Is this SoC
>> model fully supported by the driver? Do you know if there is some necessary
>> hardware initializations which must be done on u-boot? I'm using latest Marvell
>> patch set (2015_T1.0p16) on u-boot.
>>
>> My test case have been:
>> # cryptsetup benchmark --cipher aes-cbc
>>
>> and that always freezes the board.
>
> Romain, can you test the same scenario, and see if you can reproduce
> the hang or not?

Sure,


I have did exactly the same tests with Linux 4.8-rc2 (using a rootfs 
buildroot-2016.02 over nfs) on A388-db-gp and A385-db-ap, both work just 
fine. I launched the benchmark several times.

I am running uboot, Marvell version: 2015_T1.0p5


See yourself: http://paste.ubuntu.com/23070118


Romain
-- 
Romain Perier, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Russell King - ARM Linux]

On Fri, Aug 19, 2016 at 02:12:51PM +0200, Thomas Petazzoni wrote:
> Hello,
> 
> I'm adding Romain Perier in Cc, since he is actively working on the
> CESA driver.

4.7 here on Clearfog (Armada 388):

# cryptsetup benchmark --cipher aes-cbc
# Tests are approximate using memory only (no storage IO).
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   256b    67.1 MiB/s    68.3 MiB/s

/proc/interrupts:
           CPU0       CPU1
 42:       2144          0     GIC-0  51 Level     f1090000.crypto

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Russell King - ARM Linux]

On Fri, Aug 19, 2016 at 02:08:46PM +0100, Russell King - ARM Linux wrote:
> On Fri, Aug 19, 2016 at 02:12:51PM +0200, Thomas Petazzoni wrote:
> > Hello,
> > 
> > I'm adding Romain Perier in Cc, since he is actively working on the
> > CESA driver.
> 
> 4.7 here on Clearfog (Armada 388):
> 
> # cryptsetup benchmark --cipher aes-cbc
> # Tests are approximate using memory only (no storage IO).
> #  Algorithm | Key |  Encryption |  Decryption
>      aes-cbc   256b    67.1 MiB/s    68.3 MiB/s
> 
> /proc/interrupts:
>            CPU0       CPU1
>  42:       2144          0     GIC-0  51 Level     f1090000.crypto

With the coherency fix commit reverted (as mentioned in the bug report)
I get:

# cryptsetup benchmark --cipher aes-cbc
# Tests are approximate using memory only (no storage IO).
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   256b    65.8 MiB/s    67.1 MiB/s

so not that different from before.  Run more than 10 times without issue.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Romain Perier]

I was, wondering, do you boot with a separated DTB or a legacy boot 
(concatenated to the Image) ?


Could you test with uImage.armada-385-db-ap [1], that's a legacy Image.
Also, I have uploaded a minimalistic rootfs [2], if you want.

1. http://www.free-electrons.com/~romain/pub/cesa/uImage.armada-385-db-ap
2. http://www.free-electrons.com/~romain/pub/cesa/rootfs.cpio.xz

Romain

-- 
Romain Perier, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Romain Perier]

Le 19/08/2016 16:09, Romain Perier a ?crit :
> I was, wondering, do you boot with a separated DTB or a legacy boot
> (concatenated to the Image) ?
>
>
> Could you test with uImage.armada-385-db-ap [1], that's a legacy Image.
> Also, I have uploaded a minimalistic rootfs [2], if you want.
>
> 1. http://www.free-electrons.com/~romain/pub/cesa/uImage.armada-385-db-ap
> 2. http://www.free-electrons.com/~romain/pub/cesa/rootfs.cpio.xz

I have also uploaded the zImage and a separated DTB (the cesa directory 
can be read from your browser)

Romain
-- 
Romain Perier, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Russell King - ARM Linux]

On Fri, Aug 19, 2016 at 04:09:05PM +0200, Romain Perier wrote:
> I was, wondering, do you boot with a separated DTB or a legacy boot
> (concatenated to the Image) ?
> 
> 
> Could you test with uImage.armada-385-db-ap [1], that's a legacy Image.
> Also, I have uploaded a minimalistic rootfs [2], if you want.
> 
> 1. http://www.free-electrons.com/~romain/pub/cesa/uImage.armada-385-db-ap
> 2. http://www.free-electrons.com/~romain/pub/cesa/rootfs.cpio.xz

I assume you mean Thomas rather than myself, as I have only Clearfog
platforms, which are all Armada 388.

I don't use uImage anymore, except with versions of uboot that are
unable to support zImage booting.  Where zImage booting is supported,
I'm always using a separated DTB.  Doing otherwise is insane IMHO.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Romain Perier]

Hello,

Le 19/08/2016 16:21, Russell King - ARM Linux a ?crit :
> On Fri, Aug 19, 2016 at 04:09:05PM +0200, Romain Perier wrote:
>> I was, wondering, do you boot with a separated DTB or a legacy boot
>> (concatenated to the Image) ?
>>
>>
>> Could you test with uImage.armada-385-db-ap [1], that's a legacy Image.
>> Also, I have uploaded a minimalistic rootfs [2], if you want.
>>
>> 1. http://www.free-electrons.com/~romain/pub/cesa/uImage.armada-385-db-ap
>> 2. http://www.free-electrons.com/~romain/pub/cesa/rootfs.cpio.xz
>
> I assume you mean Thomas rather than myself, as I have only Clearfog
> platforms, which are all Armada 388.

Yep, sorry. I meant, the person who has issues with cesa, i.e 
radioconfusion at gmail.com :)

>
> I don't use uImage anymore, except with versions of uboot that are
> unable to support zImage booting.  Where zImage booting is supported,
> I'm always using a separated DTB.  Doing otherwise is insane IMHO.
>

That's just, we have several ways to boot our Marvell devices, including 
legacy or non legacy boots, uImage and zImage. Anyway,
that's not the point of my previous email.

Romain
-- 
Romain Perier, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

On Wed, Aug 24, 2016 at 10:26:47PM +0100, Russell King - ARM Linux wrote:
> > The problem caused by my mistake, I was using the following kernel parameter:
> > mem=1048448k

> Try mem=1046M - reducing it by 2MB.  If that works, try mem=1047M
> instead.  Yes, that's more than you want to reserve, but please
> check how the system behaves with these.

I assume that you mean mem=1022M.

Quick testing results:
mem=256M works
mem=384M not
mem=524160k not (512M-128k)
mem=524228k works (512M-60k)
mem=512M works
mem=524348k not (512M+60k)
mem=524416k not (512M+128k)
mem=768M works
mem=896M not
mem=960M not
mem=1022M not
mem=1023M not
mem=1048000k not
mem=1048448k not (1024M-128k)
mem=1048514k not
mem=1048516k works (1024M-60k)
mem=1048518k works
mem=1048520k works
mem=1024M works

So I can reserve 0-60 kiB or 256 MiB but nothing between.
The device boots up with all listed values but cesa works with few
values only.

Best Regards
Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Thomas Petazzoni]

Hello,

On Wed, 24 Aug 2016 13:52:33 +0300, radioconfusion at gmail.com wrote:

> I assume that you mean mem=1022M.
> 
> Quick testing results:
> mem=256M works
> mem=384M not
> mem=524160k not (512M-128k)
> mem=524228k works (512M-60k)
> mem=512M works
> mem=524348k not (512M+60k)
> mem=524416k not (512M+128k)
> mem=768M works
> mem=896M not
> mem=960M not
> mem=1022M not
> mem=1023M not
> mem=1048000k not
> mem=1048448k not (1024M-128k)
> mem=1048514k not
> mem=1048516k works (1024M-60k)
> mem=1048518k works
> mem=1048520k works
> mem=1024M works

The Mbus windows need to have a power of two size, and there are
windows going from the device to the CPU for devices doing DMA. If the
memory size is not a power of two, I'm not sure we're rounding up the
window size to the next power of two. That might be the issue here.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

On Fri, 19 Aug 2016 16:09, Roman Perier wrote:
> Could you test with uImage.armada-385-db-ap [1], that's a legacy Image.
> Also, I have uploaded a minimalistic rootfs [2], if you want.
>
> 1. http://www.free-electrons.com/~romain/pub/cesa/uImage.armada-385-db-ap
> 2. http://www.free-electrons.com/~romain/pub/cesa/rootfs.cpio.xz

Huge thanks to Roman, Thomas and Russell.
The cesa works on your image and rootfs. Now it works on my system too.

The problem caused by my mistake, I was using the following kernel parameter:
mem=1048448k
My device has 1 GiB of RAM and I tried to reserve 128 kiB of memory for ram
oops. I guess this parameter caused overlapping or inaccessible area with cesa's
addresses. Am I right?
I tried to add memmap=0x20000$0x3ffe0000 without success.

Is this normal behaviour?
If yes, what is the right way to reserve RAM without killing the cesa? Should I
reserve memory at physical start point?

Best Regards,
Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Russell King - ARM Linux]

On Wed, Aug 24, 2016 at 12:15:07PM +0300, radioconfusion at gmail.com wrote:
> The problem caused by my mistake, I was using the following kernel parameter:
> mem=1048448k

Try mem=1046M - reducing it by 2MB.  If that works, try mem=1047M
instead.  Yes, that's more than you want to reserve, but please
check how the system behaves with these.

> My device has 1 GiB of RAM and I tried to reserve 128 kiB of memory for ram
> oops. I guess this parameter caused overlapping or inaccessible area with cesa's
> addresses. Am I right?
> I tried to add memmap=0x20000$0x3ffe0000 without success.

memmap= has no meaning on ARM.  It's an x86 thing, the ARM arch has
never implemented it.  I've seen this come up a few times, and I don't
know why people keep trying to use it (always with an "it doesn't seem
to work" result.)  It's even documented as being an ACPI / x86 thing
in Documentation/kernel-parameters.txt...

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

On Thu, 7 Jul 2016 15:43:14 +0200, thomas.petazzoni at free-electrons.com wrote:

> > > > My Armada 385 -device hangs when using marvell-cesa -driver.
> > > > I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> > > > I have also tried with the driver version of Linux 4.7-rc3.
> > > > 
> > > > Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.  
> > 
> > > Can you try with
> > > http://git.infradead.org/linux-mvebu.git/commitdiff/c5379ba8fccd99d5f99632c789f0393d84a57805
> > > applied ?  
> > 
> > Thanks Thomas.
> > Your patch seems to fix a random hang during crypto operations.
>
> OK, great.
>

(Some lines snipped...)

> Could you test with the latest 4.7-rc kernel + the patch I provided?

Sorry about that silence. Finally I had some time for testing this.

I took Linux 4.7-rc7 (92d21ac74a9e3c09b0b01c764e530657e4c85c49) and just put "fix HW I/O coherency related deadlocks" to top of it.
Unfortunately, the result was exact same than 4.4.10.

First, without the driver:

# cryptsetup benchmark --cipher aes-cbc

# Tests are approximate using memory only (no storage IO).
#  Algorithm | Key |  Encryption |  Decryption
      aes-cbc   256b    21.9 MiB/s    23.6 MiB/s

Loading the driver:

# modprobe marvell_cesa
[  103.213901] marvell-cesa f1090000.crypto: CESA device successfully registered

And after that:
# cryptsetup benchmark --cipher aes-cbc

was never done.

Thanks,

Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

On Thu, 7 Jul 2016 15:43:14 +0200, thomas.petazzoni at free-electrons.com wrote:

> > > > My Armada 385 -device hangs when using marvell-cesa -driver.
> > > > I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> > > > I have also tried with the driver version of Linux 4.7-rc3.
> > > > 
> > > > Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.  
> > 
> > > Can you try with
> > > http://git.infradead.org/linux-mvebu.git/commitdiff/c5379ba8fccd99d5f99632c789f0393d84a57805
> > > applied ?  
> > 
> > Thanks Thomas.
> > Your patch seems to fix a random hang during crypto operations.
>
> OK, great.
>
> > Unfortunately it seems to hit the crypto performance too.
> > The command "cryptsetup benchmark --cipher aes-cbc"
> > gives me around 35 MiB/s instead of 50 MiB/s.
> > 
> > The patch did not fix the hanging which appears instantly after enabling tdma
> > in the driver and running some crypto operation.
>
> I don't understand. Above you're saying that it fixes the problem, and
> now you're saying that it doesn't.

I'm sorry that ambiguity.
As I wrote to bugzilla (link above), there might be two different problems.

1. Without patching (Linux 4.4.10 -version):
"Cryptsetup benchmark never completes and the device locks up."
This happens always and instantly.

I found a workaround for this first problem. If I disable tdma on driver,
I can run successfully some crypto tasks (with quite poor performance),
including cryptsetup benchmark. Then I found the second problem.

2. After disabling tdma (patch provided on bugzilla):
"The device still hangs after a couple of crypto tasks, e.g. when writing a few
hundred MiB of data to btrfs which is on top of luks."

Your c5379ba8 -commit fixed the second problem for me.
Still I have the first problem when tdma is enabled.

> Could you test with the latest 4.7-rc kernel + the patch I provided?

Yes, I will.

Thanks,

Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

On Wed, 6 Jul 2016 09:06:55 +0200, thomas.petazzoni at free-electronics.com wrote:

> Hello,

> On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:

> > My Armada 385 -device hangs when using marvell-cesa -driver.
> > I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> > I have also tried with the driver version of Linux 4.7-rc3.
> > 
> > Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.

> Can you try with
> http://git.infradead.org/linux-mvebu.git/commitdiff/c5379ba8fccd99d5f99632c789f0393d84a57805
> applied ?

Thanks Thomas.
Your patch seems to fix a random hang during crypto operations.

Unfortunately it seems to hit the crypto performance too.
The command "cryptsetup benchmark --cipher aes-cbc"
gives me around 35 MiB/s instead of 50 MiB/s.

The patch did not fix the hanging which appears instantly after enabling tdma
in the driver and running some crypto operation.

Thanks,

Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Thomas Petazzoni]

Hello,

On Thu, 07 Jul 2016 12:20:58 +0300, radioconfusion at gmail.com wrote:

> > On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:  
> 
> > > My Armada 385 -device hangs when using marvell-cesa -driver.
> > > I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> > > I have also tried with the driver version of Linux 4.7-rc3.
> > > 
> > > Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.  
> 
> > Can you try with
> > http://git.infradead.org/linux-mvebu.git/commitdiff/c5379ba8fccd99d5f99632c789f0393d84a57805
> > applied ?  
> 
> Thanks Thomas.
> Your patch seems to fix a random hang during crypto operations.

OK, great.

> Unfortunately it seems to hit the crypto performance too.
> The command "cryptsetup benchmark --cipher aes-cbc"
> gives me around 35 MiB/s instead of 50 MiB/s.
> 
> The patch did not fix the hanging which appears instantly after enabling tdma
> in the driver and running some crypto operation.

I don't understand. Above you're saying that it fixes the problem, and
now you're saying that it doesn't.

Could you test with the latest 4.7-rc kernel + the patch I provided?

(i.e, don't use 4.4 + your own backports, we have no idea which patches
you have backported, and if the backport was done properly)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[radioconfusion at gmail.com]

Hello,

My Armada 385 -device hangs when using marvell-cesa -driver.
I'm using Linux 4.4.10 and armada-385-db-ap.dts.
I have also tried with the driver version of Linux 4.7-rc3.

Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.

Best Regards,
Jussi

[crypto] [marvell-cesa] Driver hangs on Armada 385 device

[Thomas Petazzoni]

Hello,

On Tue, 05 Jul 2016 11:46:30 +0300, radioconfusion at gmail.com wrote:

> My Armada 385 -device hangs when using marvell-cesa -driver.
> I'm using Linux 4.4.10 and armada-385-db-ap.dts.
> I have also tried with the driver version of Linux 4.7-rc3.
> 
> Please see https://bugzilla.kernel.org/show_bug.cgi?id=120111 for details.

Can you try with
http://git.infradead.org/linux-mvebu.git/commitdiff/c5379ba8fccd99d5f99632c789f0393d84a57805
applied ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com