[PATCH 0/2] Crypto: Remove x86 dependency on QAT drivers

[PATCH 0/2] Crypto: Remove x86 dependency on QAT drivers

[Yoan Picchi]

The QAT acceleration card can be very helpfull for some tasks like dealing
with IPSEC but it is currently restricted to be used only on x86 machine.
Looking at the code we didn't see any reasons why those drivers might not
work on other architectures. We've successfully built all of them on x86,
arm64, arm32, mips64, powerpc64, riscv64 and sparc64.

We also have tested the driver with an Intel Corporation C62x Chipset
QuickAssist Technology (rev 04) PCIe card on an arm64 server. After the numa
patch, it works with the AF_ALG crypto userland interface, allowing us to
encrypt some data with cbc for instance. We've also successfully created some
VF, bound them to DPDK, and used the card this way, thus showing some real
life usecases of x86 do work on arm64 too.

Andre Przywara (1):
  crypto: qat: replace get_current_node() with numa_node_id()

Yoan Picchi (1):
  Removes the x86 dependency on the QAT drivers

 drivers/crypto/qat/Kconfig                     | 14 +++++++-------
 drivers/crypto/qat/qat_common/adf_common_drv.h |  5 -----
 drivers/crypto/qat/qat_common/qat_algs.c       |  4 ++--
 drivers/crypto/qat/qat_common/qat_asym_algs.c  |  4 ++--
 4 files changed, 11 insertions(+), 16 deletions(-)

-- 
2.25.1

[PATCH 1/2] crypto: qat: replace get_current_node() with numa_node_id()

[Yoan Picchi]

From: Andre Przywara <andre.przywara@arm.com>

Currently the QAT driver code uses a self-defined wrapper function
called get_current_node() when it wants to learn the current NUMA node.
This implementation references the topology_physical_package_id[] array,
which more or less coincidentally contains the NUMA node id, at least
on x86.

Because this is not universal, and Linux offers a direct function to
learn the NUMA node ID, replace that function with a call to
numa_node_id(), which would work everywhere.

This fixes the QAT driver operation on arm64 machines.

Reported-by: Yoan Picchi <Yoan.Picchi@arm.com>
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Yoan Picchi <yoan.picchi@arm.com>
---
 drivers/crypto/qat/qat_common/adf_common_drv.h | 5 -----
 drivers/crypto/qat/qat_common/qat_algs.c       | 4 ++--
 drivers/crypto/qat/qat_common/qat_asym_algs.c  | 4 ++--
 3 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/drivers/crypto/qat/qat_common/adf_common_drv.h b/drivers/crypto/qat/qat_common/adf_common_drv.h
index e8c9b77c0d66..b582107db67b 100644
--- a/drivers/crypto/qat/qat_common/adf_common_drv.h
+++ b/drivers/crypto/qat/qat_common/adf_common_drv.h
@@ -49,11 +49,6 @@ struct service_hndl {
 	struct list_head list;
 };
 
-static inline int get_current_node(void)
-{
-	return topology_physical_package_id(raw_smp_processor_id());
-}
-
 int adf_service_register(struct service_hndl *service);
 int adf_service_unregister(struct service_hndl *service);
 
diff --git a/drivers/crypto/qat/qat_common/qat_algs.c b/drivers/crypto/qat/qat_common/qat_algs.c
index f998ed58457c..c0ffaebcc8b8 100644
--- a/drivers/crypto/qat/qat_common/qat_algs.c
+++ b/drivers/crypto/qat/qat_common/qat_algs.c
@@ -618,7 +618,7 @@ static int qat_alg_aead_newkey(struct crypto_aead *tfm, const u8 *key,
 {
 	struct qat_alg_aead_ctx *ctx = crypto_aead_ctx(tfm);
 	struct qat_crypto_instance *inst = NULL;
-	int node = get_current_node();
+	int node = numa_node_id();
 	struct device *dev;
 	int ret;
 
@@ -1042,7 +1042,7 @@ static int qat_alg_skcipher_newkey(struct qat_alg_skcipher_ctx *ctx,
 {
 	struct qat_crypto_instance *inst = NULL;
 	struct device *dev;
-	int node = get_current_node();
+	int node = numa_node_id();
 	int ret;
 
 	inst = qat_crypto_get_instance_node(node);
diff --git a/drivers/crypto/qat/qat_common/qat_asym_algs.c b/drivers/crypto/qat/qat_common/qat_asym_algs.c
index b0b78445418b..3701eac10bce 100644
--- a/drivers/crypto/qat/qat_common/qat_asym_algs.c
+++ b/drivers/crypto/qat/qat_common/qat_asym_algs.c
@@ -480,7 +480,7 @@ static int qat_dh_init_tfm(struct crypto_kpp *tfm)
 {
 	struct qat_dh_ctx *ctx = kpp_tfm_ctx(tfm);
 	struct qat_crypto_instance *inst =
-			qat_crypto_get_instance_node(get_current_node());
+			qat_crypto_get_instance_node(numa_node_id());
 
 	if (!inst)
 		return -EINVAL;
@@ -1218,7 +1218,7 @@ static int qat_rsa_init_tfm(struct crypto_akcipher *tfm)
 {
 	struct qat_rsa_ctx *ctx = akcipher_tfm_ctx(tfm);
 	struct qat_crypto_instance *inst =
-			qat_crypto_get_instance_node(get_current_node());
+			qat_crypto_get_instance_node(numa_node_id());
 
 	if (!inst)
 		return -EINVAL;
-- 
2.25.1

[PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Yoan Picchi]

This dependency looks outdated. After the previous patch, we have been able
to use this driver to encrypt some data and to create working VF on arm64.
We have not tested it yet on any big endian machine, hence the new dependency

Signed-off-by: Yoan Picchi <yoan.picchi@arm.com>
---
 drivers/crypto/qat/Kconfig | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/crypto/qat/Kconfig b/drivers/crypto/qat/Kconfig
index 4b90c0f22b03..afc28059274e 100644
--- a/drivers/crypto/qat/Kconfig
+++ b/drivers/crypto/qat/Kconfig
@@ -17,7 +17,7 @@ config CRYPTO_DEV_QAT
 
 config CRYPTO_DEV_QAT_DH895xCC
 	tristate "Support for Intel(R) DH895xCC"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) DH895xcc with Intel(R) QuickAssist Technology
@@ -28,7 +28,7 @@ config CRYPTO_DEV_QAT_DH895xCC
 
 config CRYPTO_DEV_QAT_C3XXX
 	tristate "Support for Intel(R) C3XXX"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C3xxx with Intel(R) QuickAssist Technology
@@ -39,7 +39,7 @@ config CRYPTO_DEV_QAT_C3XXX
 
 config CRYPTO_DEV_QAT_C62X
 	tristate "Support for Intel(R) C62X"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C62x with Intel(R) QuickAssist Technology
@@ -50,7 +50,7 @@ config CRYPTO_DEV_QAT_C62X
 
 config CRYPTO_DEV_QAT_4XXX
 	tristate "Support for Intel(R) QAT_4XXX"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) QuickAssist Technology QAT_4xxx
@@ -61,7 +61,7 @@ config CRYPTO_DEV_QAT_4XXX
 
 config CRYPTO_DEV_QAT_DH895xCCVF
 	tristate "Support for Intel(R) DH895xCC Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 
@@ -74,7 +74,7 @@ config CRYPTO_DEV_QAT_DH895xCCVF
 
 config CRYPTO_DEV_QAT_C3XXXVF
 	tristate "Support for Intel(R) C3XXX Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
@@ -86,7 +86,7 @@ config CRYPTO_DEV_QAT_C3XXXVF
 
 config CRYPTO_DEV_QAT_C62XVF
 	tristate "Support for Intel(R) C62X Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && !CPU_BIG_ENDIAN
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
-- 
2.25.1

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Rob Herring]

On Tue, Jun 07, 2022 at 04:58:40PM +0000, Yoan Picchi wrote:
> This dependency looks outdated. After the previous patch, we have been able
> to use this driver to encrypt some data and to create working VF on arm64.
> We have not tested it yet on any big endian machine, hence the new dependency

For the subject, use prefixes matching the subsystem (like you did on 
patch 1).

The only testing obligation you have is compiling for BE. If kconfig was
supposed to capture what endianness drivers have been tested or not
tested with, then lots of drivers are missing the dependency. Kconfig
depends/select entries should generally be either to prevent compile
failures (you checked PPC, RiscV, etc.?) or to hide drivers *really*
specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it 
is known not to work and not easily fixed.

Also, with the dependency, no one can test the driver without modifying 
the kernel and if it does work as-is, then one has to upstream a change 
and then wait for it to show up in distro kernels. You could mitigate 
the first part with COMPILE_TEST.

Rob

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Andre Przywara]

On Thu, 9 Jun 2022 15:36:52 -0600
Rob Herring <robh@kernel.org> wrote:

Hi,

> On Tue, Jun 07, 2022 at 04:58:40PM +0000, Yoan Picchi wrote:
> > This dependency looks outdated. After the previous patch, we have been able
> > to use this driver to encrypt some data and to create working VF on arm64.
> > We have not tested it yet on any big endian machine, hence the new dependency  
> 
> For the subject, use prefixes matching the subsystem (like you did on 
> patch 1).
> 
> The only testing obligation you have is compiling for BE.

So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
$ file qat_c62xvf.ko
qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
not stripped
qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
not stripped
qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped

All built without errors or warnings, for every of the enabled drivers.

> If kconfig was
> supposed to capture what endianness drivers have been tested or not
> tested with, then lots of drivers are missing the dependency. Kconfig
> depends/select entries should generally be either to prevent compile
> failures (you checked PPC, RiscV, etc.?) or to hide drivers *really*
> specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it 
> is known not to work and not easily fixed.

Fair enough, I leave that decision to Giovanni. I have plans to test this
with BE, but getting a BE setup on a server is not trivial, both for
userland and actual booting, so this will take some time. We just didn't
want to block this on some BE concerns.

> Also, with the dependency, no one can test the driver without modifying 
> the kernel and if it does work as-is, then one has to upstream a change 
> and then wait for it to show up in distro kernels. You could mitigate 
> the first part with COMPILE_TEST.

Yeah, that's a good point, we were already bitten by this, the initial
testing was done on a stable distro kernel (v5.4), and it worked fine
already there.

Cheers,
Andre

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Giovanni Cabiddu]

On Fri, Jun 10, 2022 at 11:48:40AM +0100, Andre Przywara wrote:
> On Thu, 9 Jun 2022 15:36:52 -0600
> Rob Herring <robh@kernel.org> wrote:
> 
> Hi,
> 
> > On Tue, Jun 07, 2022 at 04:58:40PM +0000, Yoan Picchi wrote:
> > > This dependency looks outdated. After the previous patch, we have been able
> > > to use this driver to encrypt some data and to create working VF on arm64.
> > > We have not tested it yet on any big endian machine, hence the new dependency  
> > 
> > For the subject, use prefixes matching the subsystem (like you did on 
> > patch 1).
Just to add on this, patches to the qat driver should have the following
headline:
    crypto: qat -
not
    crypto: qat:

> > 
> > The only testing obligation you have is compiling for BE.
> 
> So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
> $ file qat_c62xvf.ko
> qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
> BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
> not stripped
> qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
> version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
> not stripped
> qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
> version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
> not stripped
> qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
> BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped
> 
> All built without errors or warnings, for every of the enabled drivers.
> 
> > If kconfig was
> > supposed to capture what endianness drivers have been tested or not
> > tested with, then lots of drivers are missing the dependency. Kconfig
> > depends/select entries should generally be either to prevent compile
> > failures (you checked PPC, RiscV, etc.?) or to hide drivers *really*
> > specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it 
> > is known not to work and not easily fixed.
> 
> Fair enough, I leave that decision to Giovanni. I have plans to test this
> with BE, but getting a BE setup on a server is not trivial, both for
> userland and actual booting, so this will take some time. We just didn't
> want to block this on some BE concerns.
Just inspecting the code I can see we are not handling BE in the logic
that builds FW descriptors.
My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed
and tested.

> > Also, with the dependency, no one can test the driver without modifying 
> > the kernel and if it does work as-is, then one has to upstream a change 
> > and then wait for it to show up in distro kernels. You could mitigate 
> > the first part with COMPILE_TEST.
> 
> Yeah, that's a good point, we were already bitten by this, the initial
> testing was done on a stable distro kernel (v5.4), and it worked fine
> already there.
> 
> Cheers,
> Andre

-- 
Giovanni

Re: [PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Rob Herring]

On Mon, Jun 13, 2022 at 4:41 AM Giovanni Cabiddu
<giovanni.cabiddu@intel.com> wrote:
>
> On Fri, Jun 10, 2022 at 11:48:40AM +0100, Andre Przywara wrote:
> > On Thu, 9 Jun 2022 15:36:52 -0600
> > Rob Herring <robh@kernel.org> wrote:
> >
> > Hi,
> >
> > > On Tue, Jun 07, 2022 at 04:58:40PM +0000, Yoan Picchi wrote:
> > > > This dependency looks outdated. After the previous patch, we have been able
> > > > to use this driver to encrypt some data and to create working VF on arm64.
> > > > We have not tested it yet on any big endian machine, hence the new dependency
> > >
> > > For the subject, use prefixes matching the subsystem (like you did on
> > > patch 1).
> Just to add on this, patches to the qat driver should have the following
> headline:
>     crypto: qat -
> not
>     crypto: qat:
>
> > >
> > > The only testing obligation you have is compiling for BE.
> >
> > So I just compiled for arm64 BE, powerpc BE & LE, and riscv again:
> > $ file qat_c62xvf.ko
> > qat_c62xvf.ko: ELF 64-bit MSB relocatable, ARM aarch64, version 1 (SYSV),
> > BuildID[sha1]=630cc0ee5586c7aeb6e0ab5567ce2f2f7cc46adf, with debug_info,
> > not stripped
> > qat_c62xvf.ko: ELF 64-bit MSB relocatable, 64-bit PowerPC or cisco 7500,
> > version 1 (SYSV), BuildID[sha1]=4090ba181cf95f27108bf3ecde0776f12ef2b636,
> > not stripped
> > qat_c62xvf.ko: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500,
> > version 1 (SYSV), BuildID[sha1]=2cb0fd09d5bc36c8918fcd061c9f3dac1546cf0d,
> > not stripped
> > qat_c62xvf.ko: ELF 64-bit LSB relocatable, UCB RISC-V, version 1 (SYSV),
> > BuildID[sha1]=bfaa53df7e9aad79d3ab4c05e75ca9169227f6b8, not stripped
> >
> > All built without errors or warnings, for every of the enabled drivers.
> >
> > > If kconfig was
> > > supposed to capture what endianness drivers have been tested or not
> > > tested with, then lots of drivers are missing the dependency. Kconfig
> > > depends/select entries should generally be either to prevent compile
> > > failures (you checked PPC, RiscV, etc.?) or to hide drivers *really*
> > > specific to a platform. IMO, we should only have !CPU_BIG_ENDIAN if it
> > > is known not to work and not easily fixed.
> >
> > Fair enough, I leave that decision to Giovanni. I have plans to test this
> > with BE, but getting a BE setup on a server is not trivial, both for
> > userland and actual booting, so this will take some time. We just didn't
> > want to block this on some BE concerns.
> Just inspecting the code I can see we are not handling BE in the logic
> that builds FW descriptors.
> My preference would be to keep !CPU_BIG_ENDIAN until the driver is fixed
> and tested.

Okay, then at least add '|| COMPILE_TEST' so that allyesconfig builds build it.

Rob

[PATCH 2/2] Removes the x86 dependency on the QAT drivers

[Yoan Picchi]

This dependency looks outdated. After the previous patch, we have been able
to use this driver to encrypt some data and to create working VF on arm64.
We have not tested it yet on any big endian machine, hence the new dependency

Signed-off-by: Yoan Picchi <yoan.picchi@arm.com>
---
 drivers/crypto/qat/Kconfig | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/crypto/qat/Kconfig b/drivers/crypto/qat/Kconfig
index 4b90c0f22b03..1220cc86f910 100644
--- a/drivers/crypto/qat/Kconfig
+++ b/drivers/crypto/qat/Kconfig
@@ -17,7 +17,7 @@ config CRYPTO_DEV_QAT
 
 config CRYPTO_DEV_QAT_DH895xCC
 	tristate "Support for Intel(R) DH895xCC"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) DH895xcc with Intel(R) QuickAssist Technology
@@ -28,7 +28,7 @@ config CRYPTO_DEV_QAT_DH895xCC
 
 config CRYPTO_DEV_QAT_C3XXX
 	tristate "Support for Intel(R) C3XXX"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C3xxx with Intel(R) QuickAssist Technology
@@ -39,7 +39,7 @@ config CRYPTO_DEV_QAT_C3XXX
 
 config CRYPTO_DEV_QAT_C62X
 	tristate "Support for Intel(R) C62X"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) C62x with Intel(R) QuickAssist Technology
@@ -50,7 +50,7 @@ config CRYPTO_DEV_QAT_C62X
 
 config CRYPTO_DEV_QAT_4XXX
 	tristate "Support for Intel(R) QAT_4XXX"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select CRYPTO_DEV_QAT
 	help
 	  Support for Intel(R) QuickAssist Technology QAT_4xxx
@@ -61,7 +61,7 @@ config CRYPTO_DEV_QAT_4XXX
 
 config CRYPTO_DEV_QAT_DH895xCCVF
 	tristate "Support for Intel(R) DH895xCC Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 
@@ -74,7 +74,7 @@ config CRYPTO_DEV_QAT_DH895xCCVF
 
 config CRYPTO_DEV_QAT_C3XXXVF
 	tristate "Support for Intel(R) C3XXX Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
@@ -86,7 +86,7 @@ config CRYPTO_DEV_QAT_C3XXXVF
 
 config CRYPTO_DEV_QAT_C62XVF
 	tristate "Support for Intel(R) C62X Virtual Function"
-	depends on X86 && PCI
+	depends on PCI && (!CPU_BIG_ENDIAN || COMPILE_TEST)
 	select PCI_IOV
 	select CRYPTO_DEV_QAT
 	help
-- 
2.25.1