* Re: [PATCH v2 10/14] x86/extable: Adjust extable handling to be shadow stack compatible
From: Andrew Cooper @ 2020-05-29 19:43 UTC (permalink / raw)
To: Jan Beulich; +Cc: Xen-devel, Wei Liu, Roger Pau Monné
In-Reply-To: <b36b5868-0b7c-2b45-a994-0ff5ea170433@suse.com>
On 28/05/2020 17:15, Jan Beulich wrote:
> On 27.05.2020 21:18, Andrew Cooper wrote:
>> @@ -400,6 +400,18 @@ unsigned long get_stack_trace_bottom(unsigned long sp)
>> }
>> }
>>
>> +static unsigned long get_shstk_bottom(unsigned long sp)
>> +{
>> + switch ( get_stack_page(sp) )
>> + {
>> +#ifdef CONFIG_XEN_SHSTK
>> + case 0: return ROUNDUP(sp, IST_SHSTK_SIZE) - sizeof(unsigned long);
>> + case 5: return ROUNDUP(sp, PAGE_SIZE) - sizeof(unsigned long);
> PRIMARY_SHSTK_SLOT please and, if introduced as suggested for the
> earlier patch, IST_SHSTK_SLOT in the earlier line.
>
>> @@ -763,6 +775,56 @@ static void do_reserved_trap(struct cpu_user_regs *regs)
>> trapnr, vec_name(trapnr), regs->error_code);
>> }
>>
>> +static void extable_shstk_fixup(struct cpu_user_regs *regs, unsigned long fixup)
>> +{
>> + unsigned long ssp, *ptr, *base;
>> +
>> + asm ( "rdsspq %0" : "=r" (ssp) : "0" (1) );
>> + if ( ssp == 1 )
>> + return;
>> +
>> + ptr = _p(ssp);
>> + base = _p(get_shstk_bottom(ssp));
>> +
>> + for ( ; ptr < base; ++ptr )
>> + {
>> + /*
>> + * Search for %rip. The shstk currently looks like this:
>> + *
>> + * ... [Likely pointed to by SSP]
>> + * %cs [== regs->cs]
>> + * %rip [== regs->rip]
>> + * SSP [Likely points to 3 slots higher, above %cs]
>> + * ... [call tree to this function, likely 2/3 slots]
>> + *
>> + * and we want to overwrite %rip with fixup. There are two
>> + * complications:
>> + * 1) We cant depend on SSP values, because they won't differ by 3
>> + * slots if the exception is taken on an IST stack.
>> + * 2) There are synthetic (unrealistic but not impossible) scenarios
>> + * where %rip can end up in the call tree to this function, so we
>> + * can't check against regs->rip alone.
>> + *
>> + * Check for both reg->rip and regs->cs matching.
> Nit: regs->rip
>
>> + */
>> +
>> + if ( ptr[0] == regs->rip && ptr[1] == regs->cs )
>> + {
>> + asm ( "wrssq %[fix], %[stk]"
>> + : [stk] "=m" (*ptr)
> Could this be ptr[0], to match the if()?
>
> Considering how important it is that we don't fix up the wrong stack
> location here, I continue to wonder if we wouldn't better also
> include the SSP value on the stack in the checking, at the very
> least by way of an ASSERT() or BUG_ON().
Well no, for the reason discussed in point 1.
Its not technically an issue right now, but there is no possible way to
BUILD_BUG_ON() someone turning an exception into IST, or stopping the
use of the extable infrastructure on a #DB.
Such a check would lie in wait and either provide an unexpected tangent
to someone debugging a complicated issue (I do use #DB for a fair bit),
or become a security vulnerability.
> Since, with how the code is
> currently written, this would require a somewhat odd looking ptr[-1]
> I also wonder whether "while ( ++ptr < base )" as the loop header
> wouldn't be better. The first entry on the stack can't be the RIP
> we look for anyway, can it?
Yes it can.
>> --- a/xen/arch/x86/x86_64/entry.S
>> +++ b/xen/arch/x86/x86_64/entry.S
>> @@ -708,7 +708,16 @@ exception_with_ints_disabled:
>> call search_pre_exception_table
>> testq %rax,%rax # no fixup code for faulting EIP?
>> jz 1b
>> - movq %rax,UREGS_rip(%rsp)
>> + movq %rax,UREGS_rip(%rsp) # fixup regular stack
>> +
>> +#ifdef CONFIG_XEN_SHSTK
>> + mov $1, %edi
>> + rdsspq %rdi
>> + cmp $1, %edi
>> + je .L_exn_shstk_done
>> + wrssq %rax, (%rdi) # fixup shadow stack
> According to the comment in extable_shstk_fixup(), isn't the value
> to be replaced at 8(%rdi)?
Hmm - I think you're right. I thought I had this covered by unit tests.
~Andrew
^ permalink raw reply
* Re: [PATCH net-next] net: phy: mscc: fix PHYs using the vsc8574_probe
From: David Miller @ 2020-05-29 19:45 UTC (permalink / raw)
To: antoine.tenart
Cc: andrew, f.fainelli, hkallweit1, michael, netdev, linux-kernel
In-Reply-To: <20200529094909.1254629-1-antoine.tenart@bootlin.com>
From: Antoine Tenart <antoine.tenart@bootlin.com>
Date: Fri, 29 May 2020 11:49:09 +0200
> PHYs using the vsc8574_probe fail to be initialized and their
> config_init return -EIO leading to errors like:
> "could not attach PHY: -5".
>
> This is because when the conversion of the MSCC PHY driver to use the
> shared PHY package helpers was done, the base address retrieval and the
> base PHY read and write helpers in the driver were modified. In
> particular, the base address retrieval logic was moved from the
> config_init to the probe. But the vsc8574_probe was forgotten. This
> patch fixes it.
>
> Fixes: deb04e9c0ff2 ("net: phy: mscc: use phy_package_shared")
> Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH 2/2] perf build: Allow explicitely disabling the NO_SYSCALL_TABLE variable
From: Arnaldo Carvalho de Melo @ 2020-05-29 19:45 UTC (permalink / raw)
To: Jiri Olsa
Cc: Jiri Olsa, Namhyung Kim, Ingo Molnar, Thomas Gleixner,
Clark Williams, linux-kernel, linux-perf-users,
Arnaldo Carvalho de Melo, Adrian Hunter
In-Reply-To: <20200529180717.GF506785@krava>
Em Fri, May 29, 2020 at 08:07:17PM +0200, Jiri Olsa escreveu:
> On Fri, May 29, 2020 at 12:55:52PM -0300, Arnaldo Carvalho de Melo wrote:
> > From: Arnaldo Carvalho de Melo <acme@redhat.com>
> >
> > This is useful to see if, on x86, the legacy libaudit still works, as it
> > is used in architectures that don't have the SYSCALL_TABLE logic and we
> > want to have it tested in 'make -C tools/perf/ build-test'.
> >
> > E.g.:
> >
> > Without having audit-libs-devel installed:
> >
> > $ make NO_SYSCALL_TABLE=1 O=/tmp/build/perf -C tools/perf install-bin
> > make: Entering directory '/home/acme/git/perf/tools/perf'
> > BUILD: Doing 'make -j12' parallel build
> > <SNIP>
> > Auto-detecting system features:
> > <SNIP>
> > ... libaudit: [ OFF ]
> > ... libbfd: [ on ]
> > ... libcap: [ on ]
> > <SNIP>
> > Makefile.config:664: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev
> > <SNIP>
> >
> > After installing it:
> >
> > $ rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf
> > $ time make NO_SYSCALL_TABLE=1 O=/tmp/build/perf -C tools/perf install-bin ; perf test python
>
> heya,
> seems ok, perhaps also put it in comment to Makefile.perf
> among other NO_* stuff and to tests/make
Added this and your Acked-by (from the "seems ok") :-) Ok?
- Arnaldo
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 30e41dcd4095..e3a34af38130 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -118,6 +118,9 @@ include ../scripts/utilities.mak
#
# Define LIBBPF_DYNAMIC to enable libbpf dynamic linking.
#
+# Define NO_SYSCALL_TABLE=1 to disable the use of syscall id to/from name tables
+# generated from the kernel .tbl or unistd.h files and use, if available, libaudit
+# for doing the conversions to/from strings/id.
# As per kernel Makefile, avoid funny character set dependencies
unexport LC_ALL
^ permalink raw reply related
* Re: [PATCH] usb: dwc2: Fix shutdown callback in platform
From: Frank Mori Hess @ 2020-05-29 19:45 UTC (permalink / raw)
To: Minas Harutyunyan
Cc: Doug Anderson, Alan Stern, John Youn, Felipe Balbi,
Greg Kroah-Hartman, linux-usb@vger.kernel.org, # 4.0+
In-Reply-To: <52f936c5-7f3f-5da0-33b8-3e6d4a4008d1@synopsys.com>
On Fri, May 29, 2020 at 3:33 PM Minas Harutyunyan
<Minas.Harutyunyan@synopsys.com> wrote:
> So, can we conclude on this solution?
>
> dwc2_disable_global_interrupts(hsotg);
> synchronize_irq(hsotg->irq)
That solution is fine with me. Disabling the dwc2 interrupt sources
in initialization before the handlers are registered is also worth
doing, but it doesn't have to be in the same patch.
--
Frank
^ permalink raw reply
* Re: 5.5 kernel: using nouveau or something else just long enough to turn off Quadro RTX 4000 Mobile for hybrid graphics?
From: Marc MERLIN @ 2020-05-29 19:46 UTC (permalink / raw)
To: Ilia Mirkin; +Cc: nouveau
In-Reply-To: <CAKb7Uvhw2EYo1RR-=NGgLO3CU9QTRWchcAw1injffybZbJ-zOA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Fri, May 29, 2020 at 02:53:51PM -0400, Ilia Mirkin wrote:
> > moving to new window moves the mouse, but not windows get displayed.
>
> Do you see anything in dmesg after this is set up? I'd expect some
> errors about timeouts or something else.
Nothing other than what I pasted.
> Which kernel are you using? There have been some turing-specific fixes recently.
5.5.11. I can put 5.6 if needed.
> Also note that TLP has a problem where it forces the audio
> sub-function to always-on which prevents the GPU from suspending.
Ah, thanks for that.
I have
#RUNTIME_PM_DRIVER_BLACKLIST="amdgpu mei_me nouveau nvidia pcieport radeon"
sauron:~$ lspci |grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation TU104GLM [Quadro RTX 4000 Mobile / Max-Q] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
So you're saying that I need to blacklist 01:00.1 and without that it hangs
when suspending the powered off nvidia chip, which is what I'm experiencing
as a hang when I unplug power?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
^ permalink raw reply
* Re: [PATCH] acpi: tpm: Do not build TCPA table for TPM 2
From: Marc-André Lureau @ 2020-05-29 19:45 UTC (permalink / raw)
To: Stefan Berger; +Cc: Auger Eric, Stefan Berger, QEMU, Michael S. Tsirkin
In-Reply-To: <20200529192840.3921141-1-stefanb@linux.vnet.ibm.com>
[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]
On Fri, May 29, 2020 at 9:28 PM Stefan Berger <stefanb@linux.vnet.ibm.com>
wrote:
> From: Stefan Berger <stefanb@linux.ibm.com>
>
> Do not build a TCPA table for TPM 2 anymore but create the log area when
> building the TPM2 table. The TCPA table is only needed for TPM 1.2.
>
> Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
---
> hw/i386/acpi-build.c | 12 ++++++++----
> 1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
> index 2e15f6848e..b5669d6c65 100644
> --- a/hw/i386/acpi-build.c
> +++ b/hw/i386/acpi-build.c
> @@ -2317,6 +2317,10 @@ build_tpm2(GArray *table_data, BIOSLinker *linker,
> GArray *tcpalog)
> tpm2_ptr->log_area_minimum_length =
> cpu_to_le32(TPM_LOG_AREA_MINIMUM_SIZE);
>
> + acpi_data_push(tcpalog,
> le32_to_cpu(tpm2_ptr->log_area_minimum_length));
> + bios_linker_loader_alloc(linker, ACPI_BUILD_TPMLOG_FILE, tcpalog, 1,
> + false);
> +
> /* log area start address to be filled by Guest linker */
> bios_linker_loader_add_pointer(linker, ACPI_BUILD_TABLE_FILE,
> log_addr_offset, log_addr_size,
> @@ -2848,10 +2852,10 @@ void acpi_build(AcpiBuildTables *tables,
> MachineState *machine)
> build_hpet(tables_blob, tables->linker);
> }
> if (misc.tpm_version != TPM_VERSION_UNSPEC) {
> - acpi_add_table(table_offsets, tables_blob);
> - build_tpm_tcpa(tables_blob, tables->linker, tables->tcpalog);
> -
> - if (misc.tpm_version == TPM_VERSION_2_0) {
> + if (misc.tpm_version == TPM_VERSION_1_2) {
> + acpi_add_table(table_offsets, tables_blob);
> + build_tpm_tcpa(tables_blob, tables->linker, tables->tcpalog);
> + } else { /* TPM_VERSION_2_0 */
> acpi_add_table(table_offsets, tables_blob);
> build_tpm2(tables_blob, tables->linker, tables->tcpalog);
> }
> --
> 2.24.1
>
>
--
Marc-André Lureau
[-- Attachment #2: Type: text/html, Size: 3071 bytes --]
^ permalink raw reply
* Re: [PATCH] usb: dwc2: Fix shutdown callback in platform
From: Doug Anderson @ 2020-05-29 19:46 UTC (permalink / raw)
To: Frank Mori Hess
Cc: Minas Harutyunyan, Alan Stern, John Youn, Felipe Balbi,
Greg Kroah-Hartman, linux-usb@vger.kernel.org, # 4.0+
In-Reply-To: <CAJz5OpcP860ANLLZELB7GNSsGAY2SvataZVAMyQj_HG0udkOmg@mail.gmail.com>
Hi,
On Fri, May 29, 2020 at 12:45 PM Frank Mori Hess <fmh6jj@gmail.com> wrote:
>
> On Fri, May 29, 2020 at 3:33 PM Minas Harutyunyan
> <Minas.Harutyunyan@synopsys.com> wrote:
> > So, can we conclude on this solution?
> >
> > dwc2_disable_global_interrupts(hsotg);
> > synchronize_irq(hsotg->irq)
>
> That solution is fine with me. Disabling the dwc2 interrupt sources
> in initialization before the handlers are registered is also worth
> doing, but it doesn't have to be in the same patch.
Sounds fine to me.
-Doug
^ permalink raw reply
* [Bug 207959] Don't warn about the universal zero initializer for a structure with the 'designated_init' attribute.
From: bugzilla-daemon @ 2020-05-29 19:47 UTC (permalink / raw)
To: linux-sparse
In-Reply-To: <bug-207959-200559@https.bugzilla.kernel.org/>
https://bugzilla.kernel.org/show_bug.cgi?id=207959
--- Comment #8 from Linus Torvalds (torvalds@linux-foundation.org) ---
I'm happy to let the kernel not warn about {0} used for a pointer until I find
some egregious horror-case, and then I know that I can add
-Wuniversal-initializer to warn about it.
--
You are receiving this mail because:
You are watching the assignee of the bug.
^ permalink raw reply
* [pull request][net-next 00/11] mlx5 kTLS RX offload support 2020-05-29
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Saeed Mahameed
Hi Dave/Jakub
This series adds kTLS rx offloads support to mlx5
For more information please see tag log below.
Please pull and let me know if there is any problem.
Thanks,
Saeed.
---
The following changes since commit 394f9ebf92c899b42207d4e71465869656981ba1:
Merge branch 'hns3-next' (2020-05-28 16:39:04 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2020-05-29
for you to fetch changes up to 9956363818e903ac8ef1e2d6168e8d0cc19f155b:
net/mlx5e: kTLS, Improve rx handler function call (2020-05-29 12:39:03 -0700)
----------------------------------------------------------------
mlx5-updates-2020-05-29
mlx5 kTLS rx offload:
1) Improve hardware layouts and structure for kTLS support
2) Generalize ICOSQ (Internal Channel Operations Send Queue)
Due to the asynchronous nature of adding new kTLS flows and handling
HW asynchronous kTLS resync requests, the XSK ICOSQ was extended to
support generic async operations, such as kTLS add flow and resync, in
addition to the existing XSK usages.
3) kTLS hardware flow steering and classification:
The driver already has the means to classify TCP ipv4/6 flows to send them
to the corresponding RSS HW engine, as reflected in patches 3 through 5,
the series will add a steering layer that will hook to the driver's TCP
classifiers and will match on well known kTLS connection, in case of a
match traffic will be redirected to the kTLS decryption engine, otherwise
traffic will continue flowing normally to the TCP RSS engine.
3) kTLS add flow RX HW offload support
New offload contexts post their static/progress params WQEs
(Work Queue Element) to communicate the newly added kTLS contexts
over the per-channel async ICOSQ.
The Channel/RQ is selected according to the socket's rxq index.
A new TLS-RX workqueue is used to allow asynchronous addition of
steering rules, out of the NAPI context.
It will be also used in a downstream patch in the resync procedure.
Feature is OFF by default. Can be turned on by:
$ ethtool -K <if> tls-hw-rx-offload on
4) Added mlx5 kTLS sw stats and new counters are documented in
Documentation/networking/tls-offload.rst
rx_tls_ctx - number of TLS RX HW offload contexts added to device for
decryption.
rx_tls_ooo - number of RX packets which were part of a TLS stream
but did not arrive in the expected order and triggered the resync
procedure.
rx_tls_del - number of TLS RX HW offload contexts deleted from device
(connection has finished).
rx_tls_err - number of RX packets which were part of a TLS stream
but were not decrypted due to unexpected error in the state machine.
5) Implement the RX resync procedure.
The HW offload of TLS decryption in RX side might get out-of-sync
due to out-of-order reception of packets.
This requires SW intervention to update the HW context and get it
back in-sync.
Performance:
CPU: Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz, 24 cores, HT off
NIC: ConnectX-6 Dx 100GbE dual port
Goodput (app-layer throughput) comparison:
+---------------+-------+-------+---------+
| # connections | 1 | 4 | 8 |
+---------------+-------+-------+---------+
| SW (Gbps) | 7.26 | 24.70 | 50.30 |
+---------------+-------+-------+---------+
| HW (Gbps) | 18.50 | 64.30 | 92.90 |
+---------------+-------+-------+---------+
| Speedup | 2.55x | 2.56x | 1.85x * |
+---------------+-------+-------+---------+
* After linerate is reached, diff is observed in CPU util
----------------------------------------------------------------
Boris Pismenny (1):
net/mlx5e: Receive flow steering framework for accelerated TCP flows
Saeed Mahameed (2):
net/mlx5e: API to manipulate TTC rules destinations
net/mlx5e: kTLS, Improve rx handler function call
Tariq Toukan (8):
net/mlx5: kTLS, Improve TLS params layout structures
net/mlx5e: Turn XSK ICOSQ into a general asynchronous one
net/mlx5e: Accel, Expose flow steering API for rules add/del
net/mlx5e: kTLS, Improve TLS feature modularity
net/mlx5e: kTLS, Use kernel API to extract private offload context
net/mlx5e: kTLS, Add kTLS RX HW offload support
net/mlx5e: kTLS, Add kTLS RX stats
net/mlx5e: kTLS, Add kTLS RX resync support
Documentation/networking/tls-offload.rst | 8 +
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 3 +-
.../net/ethernet/mellanox/mlx5/core/accel/tls.h | 19 +-
drivers/net/ethernet/mellanox/mlx5/core/en.h | 22 +-
drivers/net/ethernet/mellanox/mlx5/core/en/fs.h | 26 +-
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 14 +-
.../net/ethernet/mellanox/mlx5/core/en/xsk/setup.c | 46 +-
.../net/ethernet/mellanox/mlx5/core/en/xsk/tx.c | 12 +-
.../mellanox/mlx5/core/en_accel/en_accel.h | 20 +
.../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c | 389 +++++++++++++
.../ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h | 27 +
.../ethernet/mellanox/mlx5/core/en_accel/ktls.c | 123 ++--
.../ethernet/mellanox/mlx5/core/en_accel/ktls.h | 114 +---
.../ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c | 640 +++++++++++++++++++++
.../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c | 206 ++++---
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 119 ++++
.../mellanox/mlx5/core/en_accel/ktls_txrx.h | 44 ++
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 87 +++
.../net/ethernet/mellanox/mlx5/core/en_accel/tls.c | 20 +-
.../net/ethernet/mellanox/mlx5/core/en_accel/tls.h | 8 +-
.../mellanox/mlx5/core/en_accel/tls_rxtx.c | 34 +-
.../mellanox/mlx5/core/en_accel/tls_rxtx.h | 39 +-
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 34 +-
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 84 ++-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 33 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 42 +-
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c | 24 +
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h | 15 +
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 1 -
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 12 +-
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 3 +-
include/linux/mlx5/device.h | 9 +
include/linux/mlx5/mlx5_ifc.h | 5 +-
include/linux/mlx5/qp.h | 2 +-
35 files changed, 1867 insertions(+), 421 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
^ permalink raw reply
* [net-next 01/11] net/mlx5: kTLS, Improve TLS params layout structures
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Add explicit WQE segment structures for the TLS static and progress
params.
According to the HW spec, TISN is not part of the progress params context,
take it out of it.
Rename the control segment tisn field as it could hold either a TIS
or a TIR number.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h | 2 +-
.../ethernet/mellanox/mlx5/core/en_accel/ktls.h | 2 +-
.../ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c | 14 +++++++++-----
.../mellanox/mlx5/core/en_accel/tls_rxtx.c | 2 +-
include/linux/mlx5/device.h | 9 +++++++++
include/linux/mlx5/mlx5_ifc.h | 5 +----
include/linux/mlx5/qp.h | 2 +-
7 files changed, 23 insertions(+), 13 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index bfd3e1161bc66..31cac239563de 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -182,7 +182,7 @@ mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
static inline bool mlx5e_transport_inline_tx_wqe(struct mlx5_wqe_ctrl_seg *cseg)
{
- return cseg && !!cseg->tisn;
+ return cseg && !!cseg->tis_tir_num;
}
static inline u8
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index c6180892cfcba..806ed185dd4cf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -19,7 +19,7 @@
#define MLX5E_KTLS_PROGRESS_WQE_SZ \
(offsetof(struct mlx5e_tx_wqe, tls_progress_params_ctx) + \
- MLX5_ST_SZ_BYTES(tls_progress_params))
+ sizeof(struct mlx5_wqe_tls_progress_params_seg))
#define MLX5E_KTLS_PROGRESS_WQEBBS \
(DIV_ROUND_UP(MLX5E_KTLS_PROGRESS_WQE_SZ, MLX5_SEND_WQE_BB))
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index 3cd78d9503c12..ad7300f198157 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -64,7 +64,7 @@ build_static_params(struct mlx5e_umr_wqe *wqe, u16 pc, u32 sqn,
cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
STATIC_PARAMS_DS_CNT);
cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
- cseg->tisn = cpu_to_be32(priv_tx->tisn << 8);
+ cseg->tis_tir_num = cpu_to_be32(priv_tx->tisn << 8);
ucseg->flags = MLX5_UMR_INLINE;
ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
@@ -75,10 +75,14 @@ build_static_params(struct mlx5e_umr_wqe *wqe, u16 pc, u32 sqn,
static void
fill_progress_params_ctx(void *ctx, struct mlx5e_ktls_offload_context_tx *priv_tx)
{
- MLX5_SET(tls_progress_params, ctx, tisn, priv_tx->tisn);
- MLX5_SET(tls_progress_params, ctx, record_tracker_state,
+ struct mlx5_wqe_tls_progress_params_seg *params;
+
+ params = ctx;
+
+ params->tis_tir_num = cpu_to_be32(priv_tx->tisn);
+ MLX5_SET(tls_progress_params, params->ctx, record_tracker_state,
MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
- MLX5_SET(tls_progress_params, ctx, auth_state,
+ MLX5_SET(tls_progress_params, params->ctx, auth_state,
MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD);
}
@@ -284,7 +288,7 @@ tx_post_resync_dump(struct mlx5e_txqsq *sq, skb_frag_t *frag, u32 tisn, bool fir
cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_DUMP);
cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
- cseg->tisn = cpu_to_be32(tisn << 8);
+ cseg->tis_tir_num = cpu_to_be32(tisn << 8);
cseg->fm_ce_se = first ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
fsz = skb_frag_size(frag);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
index 05454a843b289..72d26fbc8d5bf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
@@ -305,7 +305,7 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
struct mlx5e_accel_tx_tls_state *state)
{
- cseg->tisn = cpu_to_be32(state->tls_tisn << 8);
+ cseg->tis_tir_num = cpu_to_be32(state->tls_tisn << 8);
}
static int tls_update_resync_sn(struct net_device *netdev,
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 1bc27aca648bb..57db125e58021 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -458,6 +458,15 @@ enum {
MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS = 0x2,
};
+struct mlx5_wqe_tls_static_params_seg {
+ u8 ctx[MLX5_ST_SZ_BYTES(tls_static_params)];
+};
+
+struct mlx5_wqe_tls_progress_params_seg {
+ __be32 tis_tir_num;
+ u8 ctx[MLX5_ST_SZ_BYTES(tls_progress_params)];
+};
+
enum {
MLX5_SET_PORT_RESET_QKEY = 0,
MLX5_SET_PORT_GUID0 = 16,
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index fb243848132d5..e9c3c09f974f3 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -10631,16 +10631,13 @@ struct mlx5_ifc_tls_static_params_bits {
};
struct mlx5_ifc_tls_progress_params_bits {
- u8 reserved_at_0[0x8];
- u8 tisn[0x18];
-
u8 next_record_tcp_sn[0x20];
u8 hw_resync_tcp_sn[0x20];
u8 record_tracker_state[0x2];
u8 auth_state[0x2];
- u8 reserved_at_64[0x4];
+ u8 reserved_at_44[0x4];
u8 hw_offset_record_number[0x18];
};
diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index f23eb18526fe7..a635b38e48559 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -208,7 +208,7 @@ struct mlx5_wqe_ctrl_seg {
__be32 general_id;
__be32 imm;
__be32 umr_mkey;
- __be32 tisn;
+ __be32 tis_tir_num;
};
};
--
2.26.2
^ permalink raw reply related
* [net-next 02/11] net/mlx5e: Turn XSK ICOSQ into a general asynchronous one
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba
Cc: netdev, Tariq Toukan, Maxim Mikityanskiy, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
There is an upcoming demand (in downstream patches) for
an ICOSQ to be populated out of the NAPI context, asynchronously.
There is already an existing one serving XSK-related use case.
In this patch, promote this ICOSQ to serve as general async ICOSQ,
to be used for XSK and non-XSK flows.
As part of this, the reg_umr bit of the SQ context is now set
(if capable), as the general async ICOSQ should support possible
posts of UMR WQEs.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Reviewed-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 8 ++--
.../mellanox/mlx5/core/en/xsk/setup.c | 46 ++-----------------
.../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 12 ++---
.../net/ethernet/mellanox/mlx5/core/en_main.c | 24 +++++++++-
.../net/ethernet/mellanox/mlx5/core/en_txrx.c | 12 ++---
5 files changed, 42 insertions(+), 60 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index c5f8a3925b168..6937aaf24235b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -651,9 +651,11 @@ struct mlx5e_channel {
/* AF_XDP zero-copy */
struct mlx5e_rq xskrq;
struct mlx5e_xdpsq xsksq;
- struct mlx5e_icosq xskicosq;
- /* xskicosq can be accessed from any CPU - the spinlock protects it. */
- spinlock_t xskicosq_lock;
+
+ /* Async ICOSQ */
+ struct mlx5e_icosq async_icosq;
+ /* async_icosq can be accessed from any CPU - the spinlock protects it. */
+ spinlock_t async_icosq_lock;
/* data path - accessed per napi poll */
struct irq_desc *irq_desc;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
index c28cbae423310..8532e4302c9b6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
@@ -34,31 +34,15 @@ bool mlx5e_validate_xsk_param(struct mlx5e_params *params,
}
}
-static void mlx5e_build_xskicosq_param(struct mlx5e_priv *priv,
- u8 log_wq_size,
- struct mlx5e_sq_param *param)
-{
- void *sqc = param->sqc;
- void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
-
- mlx5e_build_sq_param_common(priv, param);
-
- MLX5_SET(wq, wq, log_wq_sz, log_wq_size);
-}
-
static void mlx5e_build_xsk_cparam(struct mlx5e_priv *priv,
struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
struct mlx5e_channel_param *cparam)
{
- const u8 xskicosq_size = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
-
mlx5e_build_rq_param(priv, params, xsk, &cparam->rq);
mlx5e_build_xdpsq_param(priv, params, &cparam->xdp_sq);
- mlx5e_build_xskicosq_param(priv, xskicosq_size, &cparam->icosq);
mlx5e_build_rx_cq_param(priv, params, xsk, &cparam->rx_cq);
mlx5e_build_tx_cq_param(priv, params, &cparam->tx_cq);
- mlx5e_build_ico_cq_param(priv, xskicosq_size, &cparam->icosq_cq);
}
int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
@@ -66,7 +50,6 @@ int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
struct mlx5e_channel *c)
{
struct mlx5e_channel_param *cparam;
- struct dim_cq_moder icocq_moder = {};
int err;
if (!mlx5e_validate_xsk_param(params, xsk, priv->mdev))
@@ -100,31 +83,12 @@ int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
if (unlikely(err))
goto err_close_tx_cq;
- err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->xskicosq.cq);
- if (unlikely(err))
- goto err_close_sq;
-
- /* Create a dedicated SQ for posting NOPs whenever we need an IRQ to be
- * triggered and NAPI to be called on the correct CPU.
- */
- err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->xskicosq);
- if (unlikely(err))
- goto err_close_icocq;
-
kvfree(cparam);
- spin_lock_init(&c->xskicosq_lock);
-
set_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
return 0;
-err_close_icocq:
- mlx5e_close_cq(&c->xskicosq.cq);
-
-err_close_sq:
- mlx5e_close_xdpsq(&c->xsksq);
-
err_close_tx_cq:
mlx5e_close_cq(&c->xsksq.cq);
@@ -148,28 +112,24 @@ void mlx5e_close_xsk(struct mlx5e_channel *c)
mlx5e_close_rq(&c->xskrq);
mlx5e_close_cq(&c->xskrq.cq);
- mlx5e_close_icosq(&c->xskicosq);
- mlx5e_close_cq(&c->xskicosq.cq);
mlx5e_close_xdpsq(&c->xsksq);
mlx5e_close_cq(&c->xsksq.cq);
}
void mlx5e_activate_xsk(struct mlx5e_channel *c)
{
- mlx5e_activate_icosq(&c->xskicosq);
set_bit(MLX5E_RQ_STATE_ENABLED, &c->xskrq.state);
/* TX queue is created active. */
- spin_lock(&c->xskicosq_lock);
- mlx5e_trigger_irq(&c->xskicosq);
- spin_unlock(&c->xskicosq_lock);
+ spin_lock(&c->async_icosq_lock);
+ mlx5e_trigger_irq(&c->async_icosq);
+ spin_unlock(&c->async_icosq_lock);
}
void mlx5e_deactivate_xsk(struct mlx5e_channel *c)
{
mlx5e_deactivate_rq(&c->xskrq);
/* TX queue is disabled on close. */
- mlx5e_deactivate_icosq(&c->xskicosq);
}
static int mlx5e_redirect_xsk_rqt(struct mlx5e_priv *priv, u16 ix, u32 rqn)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
index 83dce9cdb8c2f..e0b3c61af93ea 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
@@ -26,19 +26,19 @@ int mlx5e_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags)
return -ENXIO;
if (!napi_if_scheduled_mark_missed(&c->napi)) {
- /* To avoid WQE overrun, don't post a NOP if XSKICOSQ is not
+ /* To avoid WQE overrun, don't post a NOP if async_icosq is not
* active and not polled by NAPI. Return 0, because the upcoming
* activate will trigger the IRQ for us.
*/
- if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &c->xskicosq.state)))
+ if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &c->async_icosq.state)))
return 0;
- if (test_and_set_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->xskicosq.state))
+ if (test_and_set_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state))
return 0;
- spin_lock(&c->xskicosq_lock);
- mlx5e_trigger_irq(&c->xskicosq);
- spin_unlock(&c->xskicosq_lock);
+ spin_lock(&c->async_icosq_lock);
+ mlx5e_trigger_irq(&c->async_icosq);
+ spin_unlock(&c->async_icosq_lock);
}
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 2c36a04181b84..db18f602cdb2f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1817,10 +1817,14 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
struct dim_cq_moder icocq_moder = {0, 0};
int err;
- err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
+ err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->async_icosq.cq);
if (err)
return err;
+ err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
+ if (err)
+ goto err_close_async_icosq_cq;
+
err = mlx5e_open_tx_cqs(c, params, cparam);
if (err)
goto err_close_icosq_cq;
@@ -1841,10 +1845,16 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
napi_enable(&c->napi);
- err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->icosq);
+ spin_lock_init(&c->async_icosq_lock);
+
+ err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->async_icosq);
if (err)
goto err_disable_napi;
+ err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->icosq);
+ if (err)
+ goto err_close_async_icosq;
+
err = mlx5e_open_sqs(c, params, cparam);
if (err)
goto err_close_icosq;
@@ -1879,6 +1889,9 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
err_close_icosq:
mlx5e_close_icosq(&c->icosq);
+err_close_async_icosq:
+ mlx5e_close_icosq(&c->async_icosq);
+
err_disable_napi:
napi_disable(&c->napi);
@@ -1897,6 +1910,9 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
err_close_icosq_cq:
mlx5e_close_cq(&c->icosq.cq);
+err_close_async_icosq_cq:
+ mlx5e_close_cq(&c->async_icosq.cq);
+
return err;
}
@@ -1908,6 +1924,7 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
mlx5e_close_xdpsq(&c->rq_xdpsq);
mlx5e_close_sqs(c);
mlx5e_close_icosq(&c->icosq);
+ mlx5e_close_icosq(&c->async_icosq);
napi_disable(&c->napi);
if (c->xdp)
mlx5e_close_cq(&c->rq_xdpsq.cq);
@@ -1915,6 +1932,7 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
mlx5e_close_cq(&c->xdpsq.cq);
mlx5e_close_tx_cqs(c);
mlx5e_close_cq(&c->icosq.cq);
+ mlx5e_close_cq(&c->async_icosq.cq);
}
static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)
@@ -1995,6 +2013,7 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
for (tc = 0; tc < c->num_tc; tc++)
mlx5e_activate_txqsq(&c->sq[tc]);
mlx5e_activate_icosq(&c->icosq);
+ mlx5e_activate_icosq(&c->async_icosq);
mlx5e_activate_rq(&c->rq);
if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
@@ -2009,6 +2028,7 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
mlx5e_deactivate_xsk(c);
mlx5e_deactivate_rq(&c->rq);
+ mlx5e_deactivate_icosq(&c->async_icosq);
mlx5e_deactivate_icosq(&c->icosq);
for (tc = 0; tc < c->num_tc; tc++)
mlx5e_deactivate_txqsq(&c->sq[tc]);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index 8480278f2ee20..ad99f270efb5e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -149,17 +149,17 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
}
mlx5e_poll_ico_cq(&c->icosq.cq);
+ if (mlx5e_poll_ico_cq(&c->async_icosq.cq))
+ /* Don't clear the flag if nothing was polled to prevent
+ * queueing more WQEs and overflowing XSKICOSQ.
+ */
+ clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->async_icosq.state);
busy |= INDIRECT_CALL_2(rq->post_wqes,
mlx5e_post_rx_mpwqes,
mlx5e_post_rx_wqes,
rq);
if (xsk_open) {
- if (mlx5e_poll_ico_cq(&c->xskicosq.cq))
- /* Don't clear the flag if nothing was polled to prevent
- * queueing more WQEs and overflowing XSKICOSQ.
- */
- clear_bit(MLX5E_SQ_STATE_PENDING_XSK_TX, &c->xskicosq.state);
busy |= mlx5e_poll_xdpsq_cq(&xsksq->cq);
busy_xsk |= mlx5e_napi_xsk_post(xsksq, xskrq);
}
@@ -189,11 +189,11 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
mlx5e_cq_arm(&rq->cq);
mlx5e_cq_arm(&c->icosq.cq);
+ mlx5e_cq_arm(&c->async_icosq.cq);
mlx5e_cq_arm(&c->xdpsq.cq);
if (xsk_open) {
mlx5e_handle_rx_dim(xskrq);
- mlx5e_cq_arm(&c->xskicosq.cq);
mlx5e_cq_arm(&xsksq->cq);
mlx5e_cq_arm(&xskrq->cq);
}
--
2.26.2
^ permalink raw reply related
* Re: [PATCH v2 11/14] x86/alt: Adjust _alternative_instructions() to not create shadow stacks
From: Andrew Cooper @ 2020-05-29 19:46 UTC (permalink / raw)
To: Jan Beulich; +Cc: Xen-devel, Wei Liu, Roger Pau Monné
In-Reply-To: <1e6d1503-40a8-55b9-9bd3-750cf301220d@suse.com>
On 29/05/2020 13:23, Jan Beulich wrote:
> On 27.05.2020 21:18, Andrew Cooper wrote:
>> @@ -398,6 +399,19 @@ static void __init _alternative_instructions(bool force)
>> panic("Timed out waiting for alternatives self-NMI to hit\n");
>>
>> set_nmi_callback(saved_nmi_callback);
>> +
>> + /*
>> + * When Xen is using shadow stacks, the alternatives clearing CR0.WP and
>> + * writing into the mappings set dirty bits, turning the mappings into
>> + * shadow stack mappings.
>> + *
>> + * While we can execute from them, this would also permit them to be the
>> + * target of WRSS instructions, so reset the dirty after patching.
>> + */
>> + if ( cpu_has_xen_shstk )
>> + modify_xen_mappings(XEN_VIRT_START + MB(2),
>> + (unsigned long)&__2M_text_end,
>> + PAGE_HYPERVISOR_RX);
> Am I misremembering, or did you post a patch before that should
> be part of this series, as being a prereq to this change,
> making modify_xen_mappings() also respect _PAGE_DIRTY as
> requested by the caller?
No. Its the hunk you deleted from lower in this patch.
> Additionally I notice this
>
> if ( desc->Attribute & (efi_bs_revision < EFI_REVISION(2, 5)
> ? EFI_MEMORY_WP : EFI_MEMORY_RO) )
> prot &= ~_PAGE_RW;
>
> in efi_init_memory(). Afaict we will need to clear _PAGE_DIRTY
> there as well, with prot starting out as PAGE_HYPERVISOR_RWX.
Ok. I'll pull that out into a separate patch.
~Andrew
^ permalink raw reply
* [net-next 03/11] net/mlx5e: API to manipulate TTC rules destinations
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Saeed Mahameed, Tariq Toukan
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
Store the default destinations of the on-load generated TTC
(Traffic Type Classifier) rules in the ttc rules table.
Introduce TTC API functions to manipulate/restore and get the TTC rule
destination and use these API functions in arfs implementation.
This will allow a better decoupling between TTC implementation and its
users.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/en/fs.h | 16 +++-
.../net/ethernet/mellanox/mlx5/core/en_arfs.c | 34 +++-----
.../net/ethernet/mellanox/mlx5/core/en_fs.c | 84 ++++++++++++++-----
3 files changed, 86 insertions(+), 48 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
index 0416f77121096..c633579474c34 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
@@ -105,11 +105,16 @@ enum mlx5e_tunnel_types {
bool mlx5e_tunnel_inner_ft_supported(struct mlx5_core_dev *mdev);
+struct mlx5e_ttc_rule {
+ struct mlx5_flow_handle *rule;
+ struct mlx5_flow_destination default_dest;
+};
+
/* L3/L4 traffic type classifier */
struct mlx5e_ttc_table {
- struct mlx5e_flow_table ft;
- struct mlx5_flow_handle *rules[MLX5E_NUM_TT];
- struct mlx5_flow_handle *tunnel_rules[MLX5E_NUM_TUNNEL_TT];
+ struct mlx5e_flow_table ft;
+ struct mlx5e_ttc_rule rules[MLX5E_NUM_TT];
+ struct mlx5_flow_handle *tunnel_rules[MLX5E_NUM_TUNNEL_TT];
};
/* NIC prio FTS */
@@ -248,6 +253,11 @@ void mlx5e_destroy_inner_ttc_table(struct mlx5e_priv *priv,
struct mlx5e_ttc_table *ttc);
void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft);
+int mlx5e_ttc_fwd_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type,
+ struct mlx5_flow_destination *new_dest);
+struct mlx5_flow_destination
+mlx5e_ttc_get_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type);
+int mlx5e_ttc_fwd_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type);
void mlx5e_enable_cvlan_filter(struct mlx5e_priv *priv);
void mlx5e_disable_cvlan_filter(struct mlx5e_priv *priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
index 014639ea06e34..8f1564159ea4d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -90,23 +90,15 @@ static enum mlx5e_traffic_types arfs_get_tt(enum arfs_type type)
static int arfs_disable(struct mlx5e_priv *priv)
{
- struct mlx5_flow_destination dest = {};
- struct mlx5e_tir *tir = priv->indir_tir;
- int err = 0;
- int tt;
- int i;
+ int err, i;
- dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
for (i = 0; i < ARFS_NUM_TYPES; i++) {
- dest.tir_num = tir[i].tirn;
- tt = arfs_get_tt(i);
- /* Modify ttc rules destination to bypass the aRFS tables*/
- err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
- &dest, NULL);
+ /* Modify ttc rules destination back to their default */
+ err = mlx5e_ttc_fwd_default_dest(priv, arfs_get_tt(i));
if (err) {
netdev_err(priv->netdev,
- "%s: modify ttc destination failed\n",
- __func__);
+ "%s: modify ttc[%d] default destination failed, err(%d)\n",
+ __func__, arfs_get_tt(i), err);
return err;
}
}
@@ -125,21 +117,17 @@ int mlx5e_arfs_disable(struct mlx5e_priv *priv)
int mlx5e_arfs_enable(struct mlx5e_priv *priv)
{
struct mlx5_flow_destination dest = {};
- int err = 0;
- int tt;
- int i;
+ int err, i;
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
for (i = 0; i < ARFS_NUM_TYPES; i++) {
dest.ft = priv->fs.arfs.arfs_tables[i].ft.t;
- tt = arfs_get_tt(i);
/* Modify ttc rules destination to point on the aRFS FTs */
- err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
- &dest, NULL);
+ err = mlx5e_ttc_fwd_dest(priv, arfs_get_tt(i), &dest);
if (err) {
netdev_err(priv->netdev,
- "%s: modify ttc destination failed err=%d\n",
- __func__, err);
+ "%s: modify ttc[%d] dest to arfs, failed err(%d)\n",
+ __func__, arfs_get_tt(i), err);
arfs_disable(priv);
return err;
}
@@ -186,8 +174,10 @@ static int arfs_add_default_rule(struct mlx5e_priv *priv,
return -EINVAL;
}
+ /* FIXME: Must use mlx5e_ttc_get_default_dest(),
+ * but can't since TTC default is not setup yet !
+ */
dest.tir_num = tir[tt].tirn;
-
arfs_t->default_rule = mlx5_add_flow_rules(arfs_t->ft.t, NULL,
&flow_act,
&dest, 1);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index 73d3dc07331f1..6c4bdc7dd6e61 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -672,9 +672,9 @@ static void mlx5e_cleanup_ttc_rules(struct mlx5e_ttc_table *ttc)
int i;
for (i = 0; i < MLX5E_NUM_TT; i++) {
- if (!IS_ERR_OR_NULL(ttc->rules[i])) {
- mlx5_del_flow_rules(ttc->rules[i]);
- ttc->rules[i] = NULL;
+ if (!IS_ERR_OR_NULL(ttc->rules[i].rule)) {
+ mlx5_del_flow_rules(ttc->rules[i].rule);
+ ttc->rules[i].rule = NULL;
}
}
@@ -857,7 +857,8 @@ static int mlx5e_generate_ttc_table_rules(struct mlx5e_priv *priv,
struct mlx5e_ttc_table *ttc)
{
struct mlx5_flow_destination dest = {};
- struct mlx5_flow_handle **rules;
+ struct mlx5_flow_handle **trules;
+ struct mlx5e_ttc_rule *rules;
struct mlx5_flow_table *ft;
int tt;
int err;
@@ -867,39 +868,47 @@ static int mlx5e_generate_ttc_table_rules(struct mlx5e_priv *priv,
dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
for (tt = 0; tt < MLX5E_NUM_TT; tt++) {
+ struct mlx5e_ttc_rule *rule = &rules[tt];
+
if (tt == MLX5E_TT_ANY)
dest.tir_num = params->any_tt_tirn;
else
dest.tir_num = params->indir_tirn[tt];
- rules[tt] = mlx5e_generate_ttc_rule(priv, ft, &dest,
- ttc_rules[tt].etype,
- ttc_rules[tt].proto);
- if (IS_ERR(rules[tt]))
+
+ rule->rule = mlx5e_generate_ttc_rule(priv, ft, &dest,
+ ttc_rules[tt].etype,
+ ttc_rules[tt].proto);
+ if (IS_ERR(rule->rule)) {
+ err = PTR_ERR(rule->rule);
+ rule->rule = NULL;
goto del_rules;
+ }
+ rule->default_dest = dest;
}
if (!params->inner_ttc || !mlx5e_tunnel_inner_ft_supported(priv->mdev))
return 0;
- rules = ttc->tunnel_rules;
+ trules = ttc->tunnel_rules;
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
dest.ft = params->inner_ttc->ft.t;
for (tt = 0; tt < MLX5E_NUM_TUNNEL_TT; tt++) {
if (!mlx5e_tunnel_proto_supported(priv->mdev,
ttc_tunnel_rules[tt].proto))
continue;
- rules[tt] = mlx5e_generate_ttc_rule(priv, ft, &dest,
- ttc_tunnel_rules[tt].etype,
- ttc_tunnel_rules[tt].proto);
- if (IS_ERR(rules[tt]))
+ trules[tt] = mlx5e_generate_ttc_rule(priv, ft, &dest,
+ ttc_tunnel_rules[tt].etype,
+ ttc_tunnel_rules[tt].proto);
+ if (IS_ERR(trules[tt])) {
+ err = PTR_ERR(trules[tt]);
+ trules[tt] = NULL;
goto del_rules;
+ }
}
return 0;
del_rules:
- err = PTR_ERR(rules[tt]);
- rules[tt] = NULL;
mlx5e_cleanup_ttc_rules(ttc);
return err;
}
@@ -1015,33 +1024,38 @@ static int mlx5e_generate_inner_ttc_table_rules(struct mlx5e_priv *priv,
struct mlx5e_ttc_table *ttc)
{
struct mlx5_flow_destination dest = {};
- struct mlx5_flow_handle **rules;
+ struct mlx5e_ttc_rule *rules;
struct mlx5_flow_table *ft;
int err;
int tt;
ft = ttc->ft.t;
rules = ttc->rules;
-
dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+
for (tt = 0; tt < MLX5E_NUM_TT; tt++) {
+ struct mlx5e_ttc_rule *rule = &rules[tt];
+
if (tt == MLX5E_TT_ANY)
dest.tir_num = params->any_tt_tirn;
else
dest.tir_num = params->indir_tirn[tt];
- rules[tt] = mlx5e_generate_inner_ttc_rule(priv, ft, &dest,
- ttc_rules[tt].etype,
- ttc_rules[tt].proto);
- if (IS_ERR(rules[tt]))
+ rule->rule = mlx5e_generate_inner_ttc_rule(priv, ft, &dest,
+ ttc_rules[tt].etype,
+ ttc_rules[tt].proto);
+ if (IS_ERR(rule->rule)) {
+ err = PTR_ERR(rule->rule);
+ rule->rule = NULL;
goto del_rules;
+ }
+ rule->default_dest = dest;
}
return 0;
del_rules:
- err = PTR_ERR(rules[tt]);
- rules[tt] = NULL;
+
mlx5e_cleanup_ttc_rules(ttc);
return err;
}
@@ -1210,6 +1224,30 @@ int mlx5e_create_ttc_table(struct mlx5e_priv *priv, struct ttc_params *params,
return err;
}
+int mlx5e_ttc_fwd_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type,
+ struct mlx5_flow_destination *new_dest)
+{
+ return mlx5_modify_rule_destination(priv->fs.ttc.rules[type].rule, new_dest, NULL);
+}
+
+struct mlx5_flow_destination
+mlx5e_ttc_get_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type)
+{
+ struct mlx5_flow_destination *dest = &priv->fs.ttc.rules[type].default_dest;
+
+ WARN_ONCE(dest->type != MLX5_FLOW_DESTINATION_TYPE_TIR,
+ "TTC[%d] default dest is not setup yet", type);
+
+ return *dest;
+}
+
+int mlx5e_ttc_fwd_default_dest(struct mlx5e_priv *priv, enum mlx5e_traffic_types type)
+{
+ struct mlx5_flow_destination dest = mlx5e_ttc_get_default_dest(priv, type);
+
+ return mlx5_modify_rule_destination(priv->fs.ttc.rules[type].rule, &dest, NULL);
+}
+
static void mlx5e_del_l2_flow_rule(struct mlx5e_priv *priv,
struct mlx5e_l2_rule *ai)
{
--
2.26.2
^ permalink raw reply related
* [net-next 04/11] net/mlx5e: Receive flow steering framework for accelerated TCP flows
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba
Cc: netdev, Boris Pismenny, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Boris Pismenny <borisp@mellanox.com>
The framework allows creating flow tables to steer incoming traffic of
TCP sockets to the acceleration TIRs.
This is used in downstream patches for TLS, and will be used in the
future for other offloads.
Signed-off-by: Boris Pismenny <borisp@mellanox.com>
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
.../net/ethernet/mellanox/mlx5/core/en/fs.h | 10 +
.../mellanox/mlx5/core/en_accel/fs_tcp.c | 275 ++++++++++++++++++
.../mellanox/mlx5/core/en_accel/fs_tcp.h | 18 ++
.../net/ethernet/mellanox/mlx5/core/fs_core.c | 4 +-
5 files changed, 306 insertions(+), 3 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index b61e47bc16e84..8ffa1325a18f9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -74,7 +74,7 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
en_accel/ipsec_stats.o
mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
- en_accel/ktls.o en_accel/ktls_tx.o
+ en_accel/ktls.o en_accel/ktls_tx.o en_accel/fs_tcp.o
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
index c633579474c34..385cbff1caf1d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/fs.h
@@ -123,6 +123,9 @@ enum {
MLX5E_L2_FT_LEVEL,
MLX5E_TTC_FT_LEVEL,
MLX5E_INNER_TTC_FT_LEVEL,
+#ifdef CONFIG_MLX5_EN_TLS
+ MLX5E_ACCEL_FS_TCP_FT_LEVEL,
+#endif
#ifdef CONFIG_MLX5_EN_ARFS
MLX5E_ARFS_FT_LEVEL
#endif
@@ -216,6 +219,10 @@ static inline int mlx5e_arfs_enable(struct mlx5e_priv *priv) { return -EOPNOTSUP
static inline int mlx5e_arfs_disable(struct mlx5e_priv *priv) { return -EOPNOTSUPP; }
#endif
+#ifdef CONFIG_MLX5_EN_TLS
+struct mlx5e_accel_fs_tcp;
+#endif
+
struct mlx5e_flow_steering {
struct mlx5_flow_namespace *ns;
#ifdef CONFIG_MLX5_EN_RXNFC
@@ -229,6 +236,9 @@ struct mlx5e_flow_steering {
#ifdef CONFIG_MLX5_EN_ARFS
struct mlx5e_arfs_tables arfs;
#endif
+#ifdef CONFIG_MLX5_EN_TLS
+ struct mlx5e_accel_fs_tcp *accel_tcp;
+#endif
};
struct ttc_params {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
new file mode 100644
index 0000000000000..882ea296e7eec
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
@@ -0,0 +1,275 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#include <linux/netdevice.h>
+#include "en_accel/fs_tcp.h"
+#include "fs_core.h"
+
+enum accel_fs_tcp_type {
+ ACCEL_FS_IPV4_TCP,
+ ACCEL_FS_IPV6_TCP,
+ ACCEL_FS_TCP_NUM_TYPES,
+};
+
+struct mlx5e_accel_fs_tcp {
+ struct mlx5e_flow_table tables[ACCEL_FS_TCP_NUM_TYPES];
+ struct mlx5_flow_handle *default_rules[ACCEL_FS_TCP_NUM_TYPES];
+};
+
+static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
+{
+ switch (i) {
+ case ACCEL_FS_IPV4_TCP:
+ return MLX5E_TT_IPV4_TCP;
+ default: /* ACCEL_FS_IPV6_TCP */
+ return MLX5E_TT_IPV6_TCP;
+ }
+}
+
+static int accel_fs_tcp_add_default_rule(struct mlx5e_priv *priv,
+ enum accel_fs_tcp_type type)
+{
+ struct mlx5e_flow_table *accel_fs_t;
+ struct mlx5_flow_destination dest;
+ struct mlx5e_accel_fs_tcp *fs_tcp;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
+ struct mlx5_flow_handle *rule;
+ int err = 0;
+
+ fs_tcp = priv->fs.accel_tcp;
+ accel_fs_t = &fs_tcp->tables[type];
+
+ dest = mlx5e_ttc_get_default_dest(priv, fs_accel2tt(type));
+ rule = mlx5_add_flow_rules(accel_fs_t->t, NULL, &flow_act, &dest, 1);
+ if (IS_ERR(rule)) {
+ err = PTR_ERR(rule);
+ netdev_err(priv->netdev,
+ "%s: add default rule failed, accel_fs type=%d, err %d\n",
+ __func__, type, err);
+ return err;
+ }
+
+ fs_tcp->default_rules[type] = rule;
+ return 0;
+}
+
+#define MLX5E_ACCEL_FS_TCP_NUM_GROUPS (2)
+#define MLX5E_ACCEL_FS_TCP_GROUP1_SIZE (BIT(16) - 1)
+#define MLX5E_ACCEL_FS_TCP_GROUP2_SIZE (BIT(0))
+#define MLX5E_ACCEL_FS_TCP_TABLE_SIZE (MLX5E_ACCEL_FS_TCP_GROUP1_SIZE +\
+ MLX5E_ACCEL_FS_TCP_GROUP2_SIZE)
+static int accel_fs_tcp_create_groups(struct mlx5e_flow_table *ft,
+ enum accel_fs_tcp_type type)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ void *outer_headers_c;
+ int ix = 0;
+ u32 *in;
+ int err;
+ u8 *mc;
+
+ ft->g = kcalloc(MLX5E_ACCEL_FS_TCP_NUM_GROUPS, sizeof(*ft->g), GFP_KERNEL);
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in || !ft->g) {
+ kvfree(ft->g);
+ kvfree(in);
+ return -ENOMEM;
+ }
+
+ mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+ outer_headers_c = MLX5_ADDR_OF(fte_match_param, mc, outer_headers);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, ethertype);
+ switch (type) {
+ case ACCEL_FS_IPV4_TCP:
+ case ACCEL_FS_IPV6_TCP:
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_dport);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_sport);
+ break;
+ default:
+ err = -EINVAL;
+ goto out;
+ }
+
+ switch (type) {
+ case ACCEL_FS_IPV4_TCP:
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c,
+ src_ipv4_src_ipv6.ipv4_layout.ipv4);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c,
+ dst_ipv4_dst_ipv6.ipv4_layout.ipv4);
+ break;
+ case ACCEL_FS_IPV6_TCP:
+ memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
+ src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+ memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
+ dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+ break;
+ default:
+ err = -EINVAL;
+ goto out;
+ }
+
+ MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
+ MLX5_SET_CFG(in, start_flow_index, ix);
+ ix += MLX5E_ACCEL_FS_TCP_GROUP1_SIZE;
+ MLX5_SET_CFG(in, end_flow_index, ix - 1);
+ ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
+ if (IS_ERR(ft->g[ft->num_groups]))
+ goto err;
+ ft->num_groups++;
+
+ /* Default Flow Group */
+ memset(in, 0, inlen);
+ MLX5_SET_CFG(in, start_flow_index, ix);
+ ix += MLX5E_ACCEL_FS_TCP_GROUP2_SIZE;
+ MLX5_SET_CFG(in, end_flow_index, ix - 1);
+ ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
+ if (IS_ERR(ft->g[ft->num_groups]))
+ goto err;
+ ft->num_groups++;
+
+ kvfree(in);
+ return 0;
+
+err:
+ err = PTR_ERR(ft->g[ft->num_groups]);
+ ft->g[ft->num_groups] = NULL;
+out:
+ kvfree(in);
+
+ return err;
+}
+
+static int accel_fs_tcp_create_table(struct mlx5e_priv *priv, enum accel_fs_tcp_type type)
+{
+ struct mlx5e_flow_table *ft = &priv->fs.accel_tcp->tables[type];
+ struct mlx5_flow_table_attr ft_attr = {};
+ int err;
+
+ ft->num_groups = 0;
+
+ ft_attr.max_fte = MLX5E_ACCEL_FS_TCP_TABLE_SIZE;
+ ft_attr.level = MLX5E_ACCEL_FS_TCP_FT_LEVEL;
+ ft_attr.prio = MLX5E_NIC_PRIO;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
+ if (IS_ERR(ft->t)) {
+ err = PTR_ERR(ft->t);
+ ft->t = NULL;
+ return err;
+ }
+
+ netdev_dbg(priv->netdev, "Created fs accel table id %u level %u\n",
+ ft->t->id, ft->t->level);
+
+ err = accel_fs_tcp_create_groups(ft, type);
+ if (err)
+ goto err;
+
+ err = accel_fs_tcp_add_default_rule(priv, type);
+ if (err)
+ goto err;
+
+ return 0;
+err:
+ mlx5e_destroy_flow_table(ft);
+ return err;
+}
+
+static int accel_fs_tcp_disable(struct mlx5e_priv *priv)
+{
+ int err, i;
+
+ for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++) {
+ /* Modify ttc rules destination to point back to the indir TIRs */
+ err = mlx5e_ttc_fwd_default_dest(priv, fs_accel2tt(i));
+ if (err) {
+ netdev_err(priv->netdev,
+ "%s: modify ttc[%d] default destination failed, err(%d)\n",
+ __func__, fs_accel2tt(i), err);
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int accel_fs_tcp_enable(struct mlx5e_priv *priv)
+{
+ struct mlx5_flow_destination dest = {};
+ int err, i;
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
+ for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++) {
+ dest.ft = priv->fs.accel_tcp->tables[i].t;
+
+ /* Modify ttc rules destination to point on the accel_fs FTs */
+ err = mlx5e_ttc_fwd_dest(priv, fs_accel2tt(i), &dest);
+ if (err) {
+ netdev_err(priv->netdev,
+ "%s: modify ttc[%d] destination to accel failed, err(%d)\n",
+ __func__, fs_accel2tt(i), err);
+ return err;
+ }
+ }
+ return 0;
+}
+
+static void accel_fs_tcp_destroy_table(struct mlx5e_priv *priv, int i)
+{
+ struct mlx5e_accel_fs_tcp *fs_tcp;
+
+ fs_tcp = priv->fs.accel_tcp;
+ if (IS_ERR_OR_NULL(fs_tcp->tables[i].t))
+ return;
+
+ mlx5_del_flow_rules(fs_tcp->default_rules[i]);
+ mlx5e_destroy_flow_table(&fs_tcp->tables[i]);
+ fs_tcp->tables[i].t = NULL;
+}
+
+void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv)
+{
+ int i;
+
+ if (!priv->fs.accel_tcp)
+ return;
+
+ accel_fs_tcp_disable(priv);
+
+ for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++)
+ accel_fs_tcp_destroy_table(priv, i);
+
+ kfree(priv->fs.accel_tcp);
+ priv->fs.accel_tcp = NULL;
+}
+
+int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv)
+{
+ int i, err;
+
+ priv->fs.accel_tcp = kzalloc(sizeof(*priv->fs.accel_tcp), GFP_KERNEL);
+ if (!priv->fs.accel_tcp)
+ return -ENOMEM;
+
+ for (i = 0; i < ACCEL_FS_TCP_NUM_TYPES; i++) {
+ err = accel_fs_tcp_create_table(priv, i);
+ if (err)
+ goto err_destroy_tables;
+ }
+
+ err = accel_fs_tcp_enable(priv);
+ if (err)
+ goto err_destroy_tables;
+
+ return 0;
+
+err_destroy_tables:
+ while (--i >= 0)
+ accel_fs_tcp_destroy_table(priv, i);
+
+ kfree(priv->fs.accel_tcp);
+ priv->fs.accel_tcp = NULL;
+ return err;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
new file mode 100644
index 0000000000000..0df53473550af
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#ifndef __MLX5E_ACCEL_FS_TCP_H__
+#define __MLX5E_ACCEL_FS_TCP_H__
+
+#include "en.h"
+
+#ifdef CONFIG_MLX5_EN_TLS
+int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv);
+void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv);
+#else
+static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
+static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
+#endif
+
+#endif /* __MLX5E_ACCEL_FS_TCP_H__ */
+
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index 4bebbecb078d2..c3c111fb1c817 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -106,8 +106,8 @@
#define ETHTOOL_PRIO_NUM_LEVELS 1
#define ETHTOOL_NUM_PRIOS 11
#define ETHTOOL_MIN_LEVEL (KERNEL_MIN_LEVEL + ETHTOOL_NUM_PRIOS)
-/* Vlan, mac, ttc, inner ttc, aRFS */
-#define KERNEL_NIC_PRIO_NUM_LEVELS 5
+/* Vlan, mac, ttc, inner ttc, {aRFS/accel} */
+#define KERNEL_NIC_PRIO_NUM_LEVELS 6
#define KERNEL_NIC_NUM_PRIOS 1
/* One more level for tc */
#define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
--
2.26.2
^ permalink raw reply related
* [net-next 05/11] net/mlx5e: Accel, Expose flow steering API for rules add/del
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Given a socket, the function extracts the TCP/IP{4,6} ntuple
and adds rule to steering.
Another function gets the rule and deletes it.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../mellanox/mlx5/core/en_accel/fs_tcp.c | 114 ++++++++++++++++++
.../mellanox/mlx5/core/en_accel/fs_tcp.h | 9 ++
2 files changed, 123 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
index 882ea296e7eec..c5cd96ef8a7e3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.c
@@ -26,6 +26,120 @@ static enum mlx5e_traffic_types fs_accel2tt(enum accel_fs_tcp_type i)
}
}
+static void accel_fs_tcp_set_ipv4_flow(struct mlx5_flow_spec *spec, struct sock *sk)
+{
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ethertype);
+ MLX5_SET(fte_match_param, spec->match_value, outer_headers.ethertype, ETH_P_IP);
+ memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+ outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4),
+ &inet_sk(sk)->inet_daddr, 4);
+ memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+ outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4),
+ &inet_sk(sk)->inet_rcv_saddr, 4);
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+ outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4);
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+ outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4);
+}
+
+static void accel_fs_tcp_set_ipv6_flow(struct mlx5_flow_spec *spec, struct sock *sk)
+{
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ethertype);
+ MLX5_SET(fte_match_param, spec->match_value, outer_headers.ethertype, ETH_P_IPV6);
+ memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+ outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ &sk->sk_v6_daddr, 16);
+ memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value,
+ outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ &inet6_sk(sk)->saddr, 16);
+ memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
+ outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+ memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
+ outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+}
+
+void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule)
+{
+ mlx5_del_flow_rules(rule);
+}
+
+struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+ struct sock *sk, u32 tirn,
+ uint32_t flow_tag)
+{
+ struct mlx5_flow_destination dest = {};
+ struct mlx5e_flow_table *ft = NULL;
+ struct mlx5e_accel_fs_tcp *fs_tcp;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
+ struct mlx5_flow_handle *flow;
+ struct mlx5_flow_spec *spec;
+
+ spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
+ if (!spec)
+ return ERR_PTR(-ENOMEM);
+
+ fs_tcp = priv->fs.accel_tcp;
+
+ spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ switch (sk->sk_family) {
+ case AF_INET:
+ accel_fs_tcp_set_ipv4_flow(spec, sk);
+ ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
+ mlx5e_dbg(HW, priv, "%s flow is %pI4:%d -> %pI4:%d\n", __func__,
+ &inet_sk(sk)->inet_rcv_saddr,
+ inet_sk(sk)->inet_sport,
+ &inet_sk(sk)->inet_daddr,
+ inet_sk(sk)->inet_dport);
+ break;
+#if IS_ENABLED(CONFIG_IPV6)
+ case AF_INET6:
+ if (!sk->sk_ipv6only &&
+ ipv6_addr_type(&sk->sk_v6_daddr) == IPV6_ADDR_MAPPED) {
+ accel_fs_tcp_set_ipv4_flow(spec, sk);
+ ft = &fs_tcp->tables[ACCEL_FS_IPV4_TCP];
+ } else {
+ accel_fs_tcp_set_ipv6_flow(spec, sk);
+ ft = &fs_tcp->tables[ACCEL_FS_IPV6_TCP];
+ }
+ break;
+#endif
+ default:
+ break;
+ }
+
+ if (!ft) {
+ flow = ERR_PTR(-EINVAL);
+ goto out;
+ }
+
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+ outer_headers.tcp_dport);
+ MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria,
+ outer_headers.tcp_sport);
+ MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_dport,
+ ntohs(inet_sk(sk)->inet_sport));
+ MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_sport,
+ ntohs(inet_sk(sk)->inet_dport));
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ dest.tir_num = tirn;
+ if (flow_tag != MLX5_FS_DEFAULT_FLOW_TAG) {
+ spec->flow_context.flow_tag = flow_tag;
+ spec->flow_context.flags = FLOW_CONTEXT_HAS_TAG;
+ }
+
+ flow = mlx5_add_flow_rules(ft->t, spec, &flow_act, &dest, 1);
+ if (IS_ERR(flow))
+ netdev_err(priv->netdev, "mlx5_add_flow_rules() failed, flow is %ld\n",
+ PTR_ERR(flow));
+
+out:
+ kvfree(spec);
+ return flow;
+}
+
static int accel_fs_tcp_add_default_rule(struct mlx5e_priv *priv,
enum accel_fs_tcp_type type)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
index 0df53473550af..4341806a7b689 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/fs_tcp.h
@@ -9,9 +9,18 @@
#ifdef CONFIG_MLX5_EN_TLS
int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv);
void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv);
+struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+ struct sock *sk, u32 tirn,
+ uint32_t flow_tag);
+void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule);
#else
static inline int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) { return 0; }
static inline void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) {}
+static inline struct mlx5_flow_handle *mlx5e_accel_fs_add_sk(struct mlx5e_priv *priv,
+ struct sock *sk, u32 tirn,
+ uint32_t flow_tag)
+{ return NULL; }
+static inline void mlx5e_accel_fs_del_sk(struct mlx5_flow_handle *rule) {}
#endif
#endif /* __MLX5E_ACCEL_FS_TCP_H__ */
--
2.26.2
^ permalink raw reply related
* [net-next 07/11] net/mlx5e: kTLS, Use kernel API to extract private offload context
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba
Cc: netdev, Tariq Toukan, Maxim Mikityanskiy, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Modify the implementation of the private kTLS TX HW offload context
getter and setter, so it uses the kernel API functions, instead of
a local shadow structure.
A single BUILD_BUG_ON check is sufficient, remove the duplicate.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Reviewed-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 28 ++++++-------------
1 file changed, 9 insertions(+), 19 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index 349e29214b928..5a980f93c3263 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -56,37 +56,26 @@ struct mlx5e_ktls_offload_context_tx {
bool ctx_post_pending;
};
-struct mlx5e_ktls_offload_context_tx_shadow {
- struct tls_offload_context_tx tx_ctx;
- struct mlx5e_ktls_offload_context_tx *priv_tx;
-};
-
static void
mlx5e_set_ktls_tx_priv_ctx(struct tls_context *tls_ctx,
struct mlx5e_ktls_offload_context_tx *priv_tx)
{
- struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
- struct mlx5e_ktls_offload_context_tx_shadow *shadow;
+ struct mlx5e_ktls_offload_context_tx **ctx =
+ __tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_TX);
- BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
+ BUILD_BUG_ON(sizeof(struct mlx5e_ktls_offload_context_tx *) >
+ TLS_OFFLOAD_CONTEXT_SIZE_TX);
- shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
-
- shadow->priv_tx = priv_tx;
- priv_tx->tx_ctx = tx_ctx;
+ *ctx = priv_tx;
}
static struct mlx5e_ktls_offload_context_tx *
mlx5e_get_ktls_tx_priv_ctx(struct tls_context *tls_ctx)
{
- struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
- struct mlx5e_ktls_offload_context_tx_shadow *shadow;
-
- BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
-
- shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
+ struct mlx5e_ktls_offload_context_tx **ctx =
+ __tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_TX);
- return shadow->priv_tx;
+ return *ctx;
}
int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
@@ -113,6 +102,7 @@ int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
priv_tx->expected_seq = start_offload_tcp_sn;
priv_tx->crypto_info =
*(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
+ priv_tx->tx_ctx = tls_offload_ctx_tx(tls_ctx);
mlx5e_set_ktls_tx_priv_ctx(tls_ctx, priv_tx);
--
2.26.2
^ permalink raw reply related
* [net-next 06/11] net/mlx5e: kTLS, Improve TLS feature modularity
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Better separate the code into c/h files, so that kTLS internals
are exposed to the corresponding non-accel flow as follows:
- Necessary datapath functions are exposed via ktls_txrx.h.
- Necessary caps and configuration functions are exposed via ktls.h,
which became very small.
In addition, kTLS internal code sharing is done via ktls_utils.h,
which is not exposed to any non-accel file.
Add explicit WQE structures for the TLS static and progress
params, breaking the union of the static with UMR, and the progress
with PSV.
Generalize the API as a preparation for TLS RX offload support.
Move kTLS TX-specific code to the proper file.
Remove the inline tag for function in C files, let the compiler decide.
Use kzalloc/kfree for the priv_tx context.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 3 +-
drivers/net/ethernet/mellanox/mlx5/core/en.h | 14 +-
.../mellanox/mlx5/core/en_accel/ktls.c | 62 +----
.../mellanox/mlx5/core/en_accel/ktls.h | 107 ---------
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 214 +++++++++++-------
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 110 +++++++++
.../mellanox/mlx5/core/en_accel/ktls_txrx.h | 38 ++++
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 69 ++++++
.../mellanox/mlx5/core/en_accel/tls.c | 14 --
.../mellanox/mlx5/core/en_accel/tls.h | 7 -
.../mellanox/mlx5/core/en_accel/tls_rxtx.c | 14 ++
.../mellanox/mlx5/core/en_accel/tls_rxtx.h | 13 +-
.../net/ethernet/mellanox/mlx5/core/en_tx.c | 1 -
13 files changed, 378 insertions(+), 288 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 8ffa1325a18f9..70ad24fff2e23 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -74,7 +74,8 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
en_accel/ipsec_stats.o
mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
- en_accel/ktls.o en_accel/ktls_tx.o en_accel/fs_tcp.o
+ en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
+ en_accel/ktls_tx.o
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 6937aaf24235b..55b89d3900f21 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -191,13 +191,8 @@ static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
struct mlx5e_tx_wqe {
struct mlx5_wqe_ctrl_seg ctrl;
- union {
- struct {
- struct mlx5_wqe_eth_seg eth;
- struct mlx5_wqe_data_seg data[0];
- };
- u8 tls_progress_params_ctx[0];
- };
+ struct mlx5_wqe_eth_seg eth;
+ struct mlx5_wqe_data_seg data[0];
};
struct mlx5e_rx_wqe_ll {
@@ -213,10 +208,7 @@ struct mlx5e_umr_wqe {
struct mlx5_wqe_ctrl_seg ctrl;
struct mlx5_wqe_umr_ctrl_seg uctrl;
struct mlx5_mkey_seg mkc;
- union {
- struct mlx5_mtt inline_mtts[0];
- u8 tls_static_params_ctx[0];
- };
+ struct mlx5_mtt inline_mtts[0];
};
extern const char mlx5e_self_tests[][ETH_GSTRING_LEN];
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
index 452fcf59c36bf..8970ea68d005c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
@@ -3,31 +3,7 @@
#include "en.h"
#include "en_accel/ktls.h"
-
-u16 mlx5e_ktls_get_stop_room(struct mlx5e_txqsq *sq)
-{
- u16 num_dumps, stop_room = 0;
-
- num_dumps = mlx5e_ktls_dumps_num_wqes(sq, MAX_SKB_FRAGS, TLS_MAX_PAYLOAD_SIZE);
-
- stop_room += mlx5e_stop_room_for_wqe(MLX5E_KTLS_STATIC_WQEBBS);
- stop_room += mlx5e_stop_room_for_wqe(MLX5E_KTLS_PROGRESS_WQEBBS);
- stop_room += num_dumps * mlx5e_stop_room_for_wqe(MLX5E_KTLS_DUMP_WQEBBS);
-
- return stop_room;
-}
-
-static int mlx5e_ktls_create_tis(struct mlx5_core_dev *mdev, u32 *tisn)
-{
- u32 in[MLX5_ST_SZ_DW(create_tis_in)] = {};
- void *tisc;
-
- tisc = MLX5_ADDR_OF(create_tis_in, in, ctx);
-
- MLX5_SET(tisc, tisc, tls_en, 1);
-
- return mlx5e_create_tis(mdev, in, tisn);
-}
+#include "en_accel/ktls_utils.h"
static int mlx5e_ktls_add(struct net_device *netdev, struct sock *sk,
enum tls_offload_ctx_dir direction,
@@ -35,8 +11,6 @@ static int mlx5e_ktls_add(struct net_device *netdev, struct sock *sk,
u32 start_offload_tcp_sn)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- struct mlx5e_ktls_offload_context_tx *tx_priv;
- struct tls_context *tls_ctx = tls_get_ctx(sk);
struct mlx5_core_dev *mdev = priv->mdev;
int err;
@@ -46,31 +20,8 @@ static int mlx5e_ktls_add(struct net_device *netdev, struct sock *sk,
if (WARN_ON(!mlx5e_ktls_type_check(mdev, crypto_info)))
return -EOPNOTSUPP;
- tx_priv = kvzalloc(sizeof(*tx_priv), GFP_KERNEL);
- if (!tx_priv)
- return -ENOMEM;
-
- tx_priv->expected_seq = start_offload_tcp_sn;
- tx_priv->crypto_info = *(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
- mlx5e_set_ktls_tx_priv_ctx(tls_ctx, tx_priv);
-
- /* tc and underlay_qpn values are not in use for tls tis */
- err = mlx5e_ktls_create_tis(mdev, &tx_priv->tisn);
- if (err)
- goto create_tis_fail;
-
- err = mlx5_ktls_create_key(mdev, crypto_info, &tx_priv->key_id);
- if (err)
- goto encryption_key_create_fail;
+ err = mlx5e_ktls_add_tx(netdev, sk, crypto_info, start_offload_tcp_sn);
- mlx5e_ktls_tx_offload_set_pending(tx_priv);
-
- return 0;
-
-encryption_key_create_fail:
- mlx5e_destroy_tis(priv->mdev, tx_priv->tisn);
-create_tis_fail:
- kvfree(tx_priv);
return err;
}
@@ -78,13 +29,10 @@ static void mlx5e_ktls_del(struct net_device *netdev,
struct tls_context *tls_ctx,
enum tls_offload_ctx_dir direction)
{
- struct mlx5e_priv *priv = netdev_priv(netdev);
- struct mlx5e_ktls_offload_context_tx *tx_priv =
- mlx5e_get_ktls_tx_priv_ctx(tls_ctx);
+ if (direction != TLS_OFFLOAD_CTX_DIR_TX)
+ return;
- mlx5e_destroy_tis(priv->mdev, tx_priv->tisn);
- mlx5_ktls_destroy_key(priv->mdev, tx_priv->key_id);
- kvfree(tx_priv);
+ mlx5e_ktls_del_tx(netdev, tls_ctx);
}
static const struct tlsdev_ops mlx5e_ktls_ops = {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 806ed185dd4cf..69d736954977c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -7,122 +7,15 @@
#include "en.h"
#ifdef CONFIG_MLX5_EN_TLS
-#include <net/tls.h>
-#include "accel/tls.h"
-#include "en_accel/tls_rxtx.h"
-
-#define MLX5E_KTLS_STATIC_UMR_WQE_SZ \
- (offsetof(struct mlx5e_umr_wqe, tls_static_params_ctx) + \
- MLX5_ST_SZ_BYTES(tls_static_params))
-#define MLX5E_KTLS_STATIC_WQEBBS \
- (DIV_ROUND_UP(MLX5E_KTLS_STATIC_UMR_WQE_SZ, MLX5_SEND_WQE_BB))
-
-#define MLX5E_KTLS_PROGRESS_WQE_SZ \
- (offsetof(struct mlx5e_tx_wqe, tls_progress_params_ctx) + \
- sizeof(struct mlx5_wqe_tls_progress_params_seg))
-#define MLX5E_KTLS_PROGRESS_WQEBBS \
- (DIV_ROUND_UP(MLX5E_KTLS_PROGRESS_WQE_SZ, MLX5_SEND_WQE_BB))
-
-struct mlx5e_dump_wqe {
- struct mlx5_wqe_ctrl_seg ctrl;
- struct mlx5_wqe_data_seg data;
-};
-
-#define MLX5E_TLS_FETCH_UMR_WQE(sq, pi) \
- ((struct mlx5e_umr_wqe *)mlx5e_fetch_wqe(&(sq)->wq, pi, MLX5E_KTLS_STATIC_UMR_WQE_SZ))
-#define MLX5E_TLS_FETCH_PROGRESS_WQE(sq, pi) \
- ((struct mlx5e_tx_wqe *)mlx5e_fetch_wqe(&(sq)->wq, pi, MLX5E_KTLS_PROGRESS_WQE_SZ))
-#define MLX5E_TLS_FETCH_DUMP_WQE(sq, pi) \
- ((struct mlx5e_dump_wqe *)mlx5e_fetch_wqe(&(sq)->wq, pi, \
- sizeof(struct mlx5e_dump_wqe)))
-
-#define MLX5E_KTLS_DUMP_WQEBBS \
- (DIV_ROUND_UP(sizeof(struct mlx5e_dump_wqe), MLX5_SEND_WQE_BB))
-
-enum {
- MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD = 0,
- MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_OFFLOAD = 1,
- MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_AUTHENTICATION = 2,
-};
-
-enum {
- MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START = 0,
- MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_TRACKING = 1,
- MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_SEARCHING = 2,
-};
-
-struct mlx5e_ktls_offload_context_tx {
- struct tls_offload_context_tx *tx_ctx;
- struct tls12_crypto_info_aes_gcm_128 crypto_info;
- u32 expected_seq;
- u32 tisn;
- u32 key_id;
- bool ctx_post_pending;
-};
-
-struct mlx5e_ktls_offload_context_tx_shadow {
- struct tls_offload_context_tx tx_ctx;
- struct mlx5e_ktls_offload_context_tx *priv_tx;
-};
-
-static inline void
-mlx5e_set_ktls_tx_priv_ctx(struct tls_context *tls_ctx,
- struct mlx5e_ktls_offload_context_tx *priv_tx)
-{
- struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
- struct mlx5e_ktls_offload_context_tx_shadow *shadow;
-
- BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
-
- shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
-
- shadow->priv_tx = priv_tx;
- priv_tx->tx_ctx = tx_ctx;
-}
-
-static inline struct mlx5e_ktls_offload_context_tx *
-mlx5e_get_ktls_tx_priv_ctx(struct tls_context *tls_ctx)
-{
- struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
- struct mlx5e_ktls_offload_context_tx_shadow *shadow;
-
- BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
-
- shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
-
- return shadow->priv_tx;
-}
void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv);
-void mlx5e_ktls_tx_offload_set_pending(struct mlx5e_ktls_offload_context_tx *priv_tx);
-bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
- struct sk_buff *skb, int datalen,
- struct mlx5e_accel_tx_tls_state *state);
-void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
- struct mlx5e_tx_wqe_info *wi,
- u32 *dma_fifo_cc);
-u16 mlx5e_ktls_get_stop_room(struct mlx5e_txqsq *sq);
-
-static inline u8
-mlx5e_ktls_dumps_num_wqes(struct mlx5e_txqsq *sq, unsigned int nfrags,
- unsigned int sync_len)
-{
- /* Given the MTU and sync_len, calculates an upper bound for the
- * number of DUMP WQEs needed for the TX resync of a record.
- */
- return nfrags + DIV_ROUND_UP(sync_len, sq->hw_mtu);
-}
#else
static inline void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)
{
}
-static inline void
-mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
- struct mlx5e_tx_wqe_info *wi,
- u32 *dma_fifo_cc) {}
#endif
#endif /* __MLX5E_TLS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index ad7300f198157..349e29214b928 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -1,109 +1,149 @@
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
// Copyright (c) 2019 Mellanox Technologies.
-#include <linux/tls.h>
-#include "en.h"
-#include "en/txrx.h"
-#include "en_accel/ktls.h"
+#include "en_accel/ktls_txrx.h"
+#include "en_accel/ktls_utils.h"
-enum {
- MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+struct mlx5e_dump_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_wqe_data_seg data;
};
-enum {
- MLX5E_ENCRYPTION_STANDARD_TLS = 0x1,
-};
+#define MLX5E_KTLS_DUMP_WQEBBS \
+ (DIV_ROUND_UP(sizeof(struct mlx5e_dump_wqe), MLX5_SEND_WQE_BB))
-#define EXTRACT_INFO_FIELDS do { \
- salt = info->salt; \
- rec_seq = info->rec_seq; \
- salt_sz = sizeof(info->salt); \
- rec_seq_sz = sizeof(info->rec_seq); \
-} while (0)
+static u8
+mlx5e_ktls_dumps_num_wqes(struct mlx5e_txqsq *sq, unsigned int nfrags,
+ unsigned int sync_len)
+{
+ /* Given the MTU and sync_len, calculates an upper bound for the
+ * number of DUMP WQEs needed for the TX resync of a record.
+ */
+ return nfrags + DIV_ROUND_UP(sync_len, sq->hw_mtu);
+}
-static void
-fill_static_params_ctx(void *ctx, struct mlx5e_ktls_offload_context_tx *priv_tx)
+u16 mlx5e_ktls_get_stop_room(struct mlx5e_txqsq *sq)
{
- struct tls12_crypto_info_aes_gcm_128 *info = &priv_tx->crypto_info;
- char *initial_rn, *gcm_iv;
- u16 salt_sz, rec_seq_sz;
- char *salt, *rec_seq;
- u8 tls_version;
+ u16 num_dumps, stop_room = 0;
- EXTRACT_INFO_FIELDS;
+ num_dumps = mlx5e_ktls_dumps_num_wqes(sq, MAX_SKB_FRAGS, TLS_MAX_PAYLOAD_SIZE);
- gcm_iv = MLX5_ADDR_OF(tls_static_params, ctx, gcm_iv);
- initial_rn = MLX5_ADDR_OF(tls_static_params, ctx, initial_record_number);
+ stop_room += mlx5e_stop_room_for_wqe(MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS);
+ stop_room += mlx5e_stop_room_for_wqe(MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS);
+ stop_room += num_dumps * mlx5e_stop_room_for_wqe(MLX5E_KTLS_DUMP_WQEBBS);
- memcpy(gcm_iv, salt, salt_sz);
- memcpy(initial_rn, rec_seq, rec_seq_sz);
+ return stop_room;
+}
+
+static int mlx5e_ktls_create_tis(struct mlx5_core_dev *mdev, u32 *tisn)
+{
+ u32 in[MLX5_ST_SZ_DW(create_tis_in)] = {};
+ void *tisc;
- tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ tisc = MLX5_ADDR_OF(create_tis_in, in, ctx);
- MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
- MLX5_SET(tls_static_params, ctx, const_1, 1);
- MLX5_SET(tls_static_params, ctx, const_2, 2);
- MLX5_SET(tls_static_params, ctx, encryption_standard,
- MLX5E_ENCRYPTION_STANDARD_TLS);
- MLX5_SET(tls_static_params, ctx, dek_index, priv_tx->key_id);
+ MLX5_SET(tisc, tisc, tls_en, 1);
+
+ return mlx5e_create_tis(mdev, in, tisn);
}
+struct mlx5e_ktls_offload_context_tx {
+ struct tls_offload_context_tx *tx_ctx;
+ struct tls12_crypto_info_aes_gcm_128 crypto_info;
+ u32 expected_seq;
+ u32 tisn;
+ u32 key_id;
+ bool ctx_post_pending;
+};
+
+struct mlx5e_ktls_offload_context_tx_shadow {
+ struct tls_offload_context_tx tx_ctx;
+ struct mlx5e_ktls_offload_context_tx *priv_tx;
+};
+
static void
-build_static_params(struct mlx5e_umr_wqe *wqe, u16 pc, u32 sqn,
- struct mlx5e_ktls_offload_context_tx *priv_tx,
- bool fence)
+mlx5e_set_ktls_tx_priv_ctx(struct tls_context *tls_ctx,
+ struct mlx5e_ktls_offload_context_tx *priv_tx)
{
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
- struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
-
-#define STATIC_PARAMS_DS_CNT \
- DIV_ROUND_UP(MLX5E_KTLS_STATIC_UMR_WQE_SZ, MLX5_SEND_WQE_DS)
+ struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
+ struct mlx5e_ktls_offload_context_tx_shadow *shadow;
- cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR |
- (MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS << 24));
- cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
- STATIC_PARAMS_DS_CNT);
- cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
- cseg->tis_tir_num = cpu_to_be32(priv_tx->tisn << 8);
+ BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
- ucseg->flags = MLX5_UMR_INLINE;
- ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
+ shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
- fill_static_params_ctx(wqe->tls_static_params_ctx, priv_tx);
+ shadow->priv_tx = priv_tx;
+ priv_tx->tx_ctx = tx_ctx;
}
-static void
-fill_progress_params_ctx(void *ctx, struct mlx5e_ktls_offload_context_tx *priv_tx)
+static struct mlx5e_ktls_offload_context_tx *
+mlx5e_get_ktls_tx_priv_ctx(struct tls_context *tls_ctx)
{
- struct mlx5_wqe_tls_progress_params_seg *params;
+ struct tls_offload_context_tx *tx_ctx = tls_offload_ctx_tx(tls_ctx);
+ struct mlx5e_ktls_offload_context_tx_shadow *shadow;
+
+ BUILD_BUG_ON(sizeof(*shadow) > TLS_OFFLOAD_CONTEXT_SIZE_TX);
- params = ctx;
+ shadow = (struct mlx5e_ktls_offload_context_tx_shadow *)tx_ctx;
- params->tis_tir_num = cpu_to_be32(priv_tx->tisn);
- MLX5_SET(tls_progress_params, params->ctx, record_tracker_state,
- MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
- MLX5_SET(tls_progress_params, params->ctx, auth_state,
- MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD);
+ return shadow->priv_tx;
}
-static void
-build_progress_params(struct mlx5e_tx_wqe *wqe, u16 pc, u32 sqn,
- struct mlx5e_ktls_offload_context_tx *priv_tx,
- bool fence)
+int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
+ struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn)
{
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5e_ktls_offload_context_tx *priv_tx;
+ struct tls_context *tls_ctx;
+ struct mlx5_core_dev *mdev;
+ struct mlx5e_priv *priv;
+ int err;
-#define PROGRESS_PARAMS_DS_CNT \
- DIV_ROUND_UP(MLX5E_KTLS_PROGRESS_WQE_SZ, MLX5_SEND_WQE_DS)
+ tls_ctx = tls_get_ctx(sk);
+ priv = netdev_priv(netdev);
+ mdev = priv->mdev;
- cseg->opmod_idx_opcode =
- cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV |
- (MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS << 24));
- cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
- PROGRESS_PARAMS_DS_CNT);
- cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
+ priv_tx = kzalloc(sizeof(*priv_tx), GFP_KERNEL);
+ if (!priv_tx)
+ return -ENOMEM;
+
+ err = mlx5_ktls_create_key(mdev, crypto_info, &priv_tx->key_id);
+ if (err)
+ goto err_create_key;
+
+ priv_tx->expected_seq = start_offload_tcp_sn;
+ priv_tx->crypto_info =
+ *(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
+
+ mlx5e_set_ktls_tx_priv_ctx(tls_ctx, priv_tx);
+
+ err = mlx5e_ktls_create_tis(mdev, &priv_tx->tisn);
+ if (err)
+ goto err_create_tis;
+
+ priv_tx->ctx_post_pending = true;
+
+ return 0;
+
+err_create_tis:
+ mlx5_ktls_destroy_key(mdev, priv_tx->key_id);
+err_create_key:
+ kfree(priv_tx);
+ return err;
+}
+
+void mlx5e_ktls_del_tx(struct net_device *netdev, struct tls_context *tls_ctx)
+{
+ struct mlx5e_ktls_offload_context_tx *priv_tx;
+ struct mlx5_core_dev *mdev;
+ struct mlx5e_priv *priv;
+
+ priv_tx = mlx5e_get_ktls_tx_priv_ctx(tls_ctx);
+ priv = netdev_priv(netdev);
+ mdev = priv->mdev;
- fill_progress_params_ctx(wqe->tls_progress_params_ctx, priv_tx);
+ mlx5e_destroy_tis(mdev, priv_tx->tisn);
+ mlx5_ktls_destroy_key(mdev, priv_tx->key_id);
+ kfree(priv_tx);
}
static void tx_fill_wi(struct mlx5e_txqsq *sq,
@@ -119,11 +159,6 @@ static void tx_fill_wi(struct mlx5e_txqsq *sq,
};
}
-void mlx5e_ktls_tx_offload_set_pending(struct mlx5e_ktls_offload_context_tx *priv_tx)
-{
- priv_tx->ctx_post_pending = true;
-}
-
static bool
mlx5e_ktls_tx_offload_test_and_clear_pending(struct mlx5e_ktls_offload_context_tx *priv_tx)
{
@@ -139,12 +174,15 @@ post_static_params(struct mlx5e_txqsq *sq,
struct mlx5e_ktls_offload_context_tx *priv_tx,
bool fence)
{
- u16 pi, num_wqebbs = MLX5E_KTLS_STATIC_WQEBBS;
- struct mlx5e_umr_wqe *umr_wqe;
+ struct mlx5e_set_tls_static_params_wqe *wqe;
+ u16 pi, num_wqebbs;
+ num_wqebbs = MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS;
pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
- umr_wqe = MLX5E_TLS_FETCH_UMR_WQE(sq, pi);
- build_static_params(umr_wqe, sq->pc, sq->sqn, priv_tx, fence);
+ wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
+ mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_tx->crypto_info,
+ priv_tx->tisn, priv_tx->key_id, fence,
+ TLS_OFFLOAD_CTX_DIR_TX);
tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
sq->pc += num_wqebbs;
}
@@ -154,12 +192,14 @@ post_progress_params(struct mlx5e_txqsq *sq,
struct mlx5e_ktls_offload_context_tx *priv_tx,
bool fence)
{
- u16 pi, num_wqebbs = MLX5E_KTLS_PROGRESS_WQEBBS;
- struct mlx5e_tx_wqe *wqe;
+ struct mlx5e_set_tls_progress_params_wqe *wqe;
+ u16 pi, num_wqebbs;
+ num_wqebbs = MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS;
pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
- wqe = MLX5E_TLS_FETCH_PROGRESS_WQE(sq, pi);
- build_progress_params(wqe, sq->pc, sq->sqn, priv_tx, fence);
+ wqe = MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi);
+ mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_tx->tisn, fence,
+ TLS_OFFLOAD_CTX_DIR_TX);
tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
sq->pc += num_wqebbs;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
new file mode 100644
index 0000000000000..edf404eaa2752
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#include "en_accel/ktls_txrx.h"
+#include "en_accel/ktls_utils.h"
+
+enum {
+ MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+};
+
+enum {
+ MLX5E_ENCRYPTION_STANDARD_TLS = 0x1,
+};
+
+#define EXTRACT_INFO_FIELDS do { \
+ salt = info->salt; \
+ rec_seq = info->rec_seq; \
+ salt_sz = sizeof(info->salt); \
+ rec_seq_sz = sizeof(info->rec_seq); \
+} while (0)
+
+static void
+fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
+ struct tls12_crypto_info_aes_gcm_128 *info,
+ u32 key_id)
+{
+ char *initial_rn, *gcm_iv;
+ u16 salt_sz, rec_seq_sz;
+ char *salt, *rec_seq;
+ u8 tls_version;
+ u8 *ctx;
+
+ ctx = params->ctx;
+
+ EXTRACT_INFO_FIELDS;
+
+ gcm_iv = MLX5_ADDR_OF(tls_static_params, ctx, gcm_iv);
+ initial_rn = MLX5_ADDR_OF(tls_static_params, ctx, initial_record_number);
+
+ memcpy(gcm_iv, salt, salt_sz);
+ memcpy(initial_rn, rec_seq, rec_seq_sz);
+
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+
+ MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
+ MLX5_SET(tls_static_params, ctx, const_1, 1);
+ MLX5_SET(tls_static_params, ctx, const_2, 2);
+ MLX5_SET(tls_static_params, ctx, encryption_standard,
+ MLX5E_ENCRYPTION_STANDARD_TLS);
+ MLX5_SET(tls_static_params, ctx, dek_index, key_id);
+}
+
+void
+mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
+ u16 pc, u32 sqn,
+ struct tls12_crypto_info_aes_gcm_128 *info,
+ u32 tis_tir_num, u32 key_id,
+ bool fence, enum tls_offload_ctx_dir direction)
+{
+ struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+
+#define STATIC_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
+
+ cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR |
+ (MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS << 24));
+ cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
+ STATIC_PARAMS_DS_CNT);
+ cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
+ cseg->tis_tir_num = cpu_to_be32(tis_tir_num << 8);
+
+ ucseg->flags = MLX5_UMR_INLINE;
+ ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
+
+ fill_static_params(&wqe->params, info, key_id);
+}
+
+static void
+fill_progress_params(struct mlx5_wqe_tls_progress_params_seg *params, u32 tis_tir_num)
+{
+ u8 *ctx = params->ctx;
+
+ params->tis_tir_num = cpu_to_be32(tis_tir_num);
+
+ MLX5_SET(tls_progress_params, ctx, record_tracker_state,
+ MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
+ MLX5_SET(tls_progress_params, ctx, auth_state,
+ MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD);
+}
+
+void
+mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
+ u16 pc, u32 sqn,
+ u32 tis_tir_num, bool fence,
+ enum tls_offload_ctx_dir direction)
+{
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+
+#define PROGRESS_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
+
+ cseg->opmod_idx_opcode =
+ cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV |
+ (MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS << 24));
+ cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
+ PROGRESS_PARAMS_DS_CNT);
+ cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
+
+ fill_progress_params(&wqe->params, tis_tir_num);
+}
+
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
new file mode 100644
index 0000000000000..a325f34831176
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#ifndef __MLX5E_KTLS_TXRX_H__
+#define __MLX5E_KTLS_TXRX_H__
+
+#ifdef CONFIG_MLX5_EN_TLS
+
+#include <net/tls.h>
+#include "en.h"
+#include "en/txrx.h"
+
+struct mlx5e_accel_tx_tls_state {
+ u32 tls_tisn;
+};
+
+u16 mlx5e_ktls_get_stop_room(struct mlx5e_txqsq *sq);
+u8 mlx5e_ktls_dumps_num_wqebbs(struct mlx5e_txqsq *sq, unsigned int nfrags,
+ unsigned int sync_len);
+
+bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
+ struct sk_buff *skb, int datalen,
+ struct mlx5e_accel_tx_tls_state *state);
+
+void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
+ struct mlx5e_tx_wqe_info *wi,
+ u32 *dma_fifo_cc);
+#else
+static inline void
+mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
+ struct mlx5e_tx_wqe_info *wi,
+ u32 *dma_fifo_cc)
+{
+}
+
+#endif /* CONFIG_MLX5_EN_TLS */
+
+#endif /* __MLX5E_TLS_TXRX_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
new file mode 100644
index 0000000000000..d1d747cb2dcb6
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc. All rights reserved. */
+
+#ifndef __MLX5E_KTLS_UTILS_H__
+#define __MLX5E_KTLS_UTILS_H__
+
+#include <net/tls.h>
+#include "en.h"
+#include "accel/tls.h"
+
+enum {
+ MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD = 0,
+ MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_OFFLOAD = 1,
+ MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_AUTHENTICATION = 2,
+};
+
+enum {
+ MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START = 0,
+ MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_TRACKING = 1,
+ MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_SEARCHING = 2,
+};
+
+int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
+ struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);
+void mlx5e_ktls_del_tx(struct net_device *netdev, struct tls_context *tls_ctx);
+
+struct mlx5e_set_tls_static_params_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_wqe_umr_ctrl_seg uctrl;
+ struct mlx5_mkey_seg mkc;
+ struct mlx5_wqe_tls_static_params_seg params;
+};
+
+struct mlx5e_set_tls_progress_params_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_wqe_tls_progress_params_seg params;
+};
+
+#define MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS \
+ (DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_static_params_wqe), MLX5_SEND_WQE_BB))
+
+#define MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS \
+ (DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
+
+#define MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi) \
+ ((struct mlx5e_set_tls_static_params_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_static_params_wqe)))
+
+#define MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi) \
+ ((struct mlx5e_set_tls_progress_params_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_progress_params_wqe)))
+
+#define MLX5E_TLS_FETCH_DUMP_WQE(sq, pi) \
+ ((struct mlx5e_dump_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe)))
+
+void
+mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
+ u16 pc, u32 sqn,
+ struct tls12_crypto_info_aes_gcm_128 *info,
+ u32 tis_tir_num, u32 key_id,
+ bool fence, enum tls_offload_ctx_dir direction);
+void
+mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
+ u16 pc, u32 sqn,
+ u32 tis_tir_num, bool fence,
+ enum tls_offload_ctx_dir direction);
+
+#endif /* __MLX5E_TLS_UTILS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
index c27e9a609d519..fba561ffe1d4e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
@@ -240,17 +240,3 @@ void mlx5e_tls_cleanup(struct mlx5e_priv *priv)
kfree(tls);
priv->tls = NULL;
}
-
-u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
-{
- struct mlx5_core_dev *mdev = sq->channel->mdev;
-
- if (!mlx5_accel_is_tls_device(mdev))
- return 0;
-
- if (MLX5_CAP_GEN(mdev, tls_tx))
- return mlx5e_ktls_get_stop_room(sq);
-
- /* Resync SKB. */
- return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
-}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
index 9219bdb2786e2..9015f3f7792d2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
@@ -94,8 +94,6 @@ int mlx5e_tls_get_count(struct mlx5e_priv *priv);
int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data);
int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data);
-u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq);
-
#else
static inline void mlx5e_tls_build_netdev(struct mlx5e_priv *priv)
@@ -110,11 +108,6 @@ static inline int mlx5e_tls_get_count(struct mlx5e_priv *priv) { return 0; }
static inline int mlx5e_tls_get_strings(struct mlx5e_priv *priv, uint8_t *data) { return 0; }
static inline int mlx5e_tls_get_stats(struct mlx5e_priv *priv, u64 *data) { return 0; }
-static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
-{
- return 0;
-}
-
#endif
#endif /* __MLX5E_TLS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
index 72d26fbc8d5bf..4d796fea906d1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
@@ -387,3 +387,17 @@ void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
remove_metadata_hdr(skb);
*cqe_bcnt -= MLX5E_METADATA_ETHER_LEN;
}
+
+u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
+{
+ struct mlx5_core_dev *mdev = sq->channel->mdev;
+
+ if (!mlx5_accel_is_tls_device(mdev))
+ return 0;
+
+ if (MLX5_CAP_GEN(mdev, tls_tx))
+ return mlx5e_ktls_get_stop_room(sq);
+
+ /* Resync SKB. */
+ return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
index a50d0394df0ae..2a7b98531539f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
@@ -34,15 +34,15 @@
#ifndef __MLX5E_TLS_RXTX_H__
#define __MLX5E_TLS_RXTX_H__
+#include "en_accel/ktls_txrx.h"
+
#ifdef CONFIG_MLX5_EN_TLS
#include <linux/skbuff.h>
#include "en.h"
#include "en/txrx.h"
-struct mlx5e_accel_tx_tls_state {
- u32 tls_tisn;
-};
+u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq);
bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
struct sk_buff *skb, struct mlx5e_accel_tx_tls_state *state);
@@ -52,6 +52,13 @@ void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *c
void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
u32 *cqe_bcnt);
+#else
+
+static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
+{
+ return 0;
+}
+
#endif /* CONFIG_MLX5_EN_TLS */
#endif /* __MLX5E_TLS_RXTX_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 6d406063aca42..da596de3abba2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -38,7 +38,6 @@
#include "en/txrx.h"
#include "ipoib/ipoib.h"
#include "en_accel/en_accel.h"
-#include "en_accel/ktls.h"
#include "lib/clock.h"
static void mlx5e_dma_unmap_wqe_err(struct mlx5e_txqsq *sq, u8 num_dma)
--
2.26.2
^ permalink raw reply related
* [net-next 08/11] net/mlx5e: kTLS, Add kTLS RX HW offload support
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Implement driver support for the kTLS RX HW offload feature.
Resync support is added in a downstream patch.
New offload contexts post their static/progress params WQEs
over the per-channel async ICOSQ, protected under a spin-lock.
The Channel/RQ is selected according to the socket's rxq index.
Feature is OFF by default. Can be turned on by:
$ ethtool -K <if> tls-hw-rx-offload on
A new TLS-RX workqueue is used to allow asynchronous addition of
steering rules, out of the NAPI context.
It will be also used in a downstream patch in the resync procedure.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
.../ethernet/mellanox/mlx5/core/accel/tls.h | 19 +-
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 11 +
.../mellanox/mlx5/core/en_accel/en_accel.h | 20 ++
.../mellanox/mlx5/core/en_accel/ktls.c | 69 +++-
.../mellanox/mlx5/core/en_accel/ktls.h | 19 +-
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 317 ++++++++++++++++++
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 2 +-
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 20 +-
.../mellanox/mlx5/core/en_accel/ktls_txrx.h | 4 +
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 6 +
.../mellanox/mlx5/core/en_accel/tls.c | 6 +
.../mellanox/mlx5/core/en_accel/tls.h | 1 +
.../mellanox/mlx5/core/en_accel/tls_rxtx.c | 20 +-
.../mellanox/mlx5/core/en_accel/tls_rxtx.h | 4 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 9 +
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 33 +-
drivers/net/ethernet/mellanox/mlx5/core/fw.c | 3 +-
18 files changed, 534 insertions(+), 31 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 70ad24fff2e23..1e7c7f10db6e2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -75,7 +75,7 @@ mlx5_core-$(CONFIG_MLX5_EN_IPSEC) += en_accel/ipsec.o en_accel/ipsec_rxtx.o \
mlx5_core-$(CONFIG_MLX5_EN_TLS) += en_accel/tls.o en_accel/tls_rxtx.o en_accel/tls_stats.o \
en_accel/fs_tcp.o en_accel/ktls.o en_accel/ktls_txrx.o \
- en_accel/ktls_tx.o
+ en_accel/ktls_tx.o en_accel/ktls_rx.o
mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o \
steering/dr_matcher.o steering/dr_rule.o \
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/accel/tls.h b/drivers/net/ethernet/mellanox/mlx5/core/accel/tls.h
index e09bc3858d574..9f9247ddcb88d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/accel/tls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/accel/tls.h
@@ -43,9 +43,20 @@ int mlx5_ktls_create_key(struct mlx5_core_dev *mdev,
u32 *p_key_id);
void mlx5_ktls_destroy_key(struct mlx5_core_dev *mdev, u32 key_id);
+static inline bool mlx5_accel_is_ktls_tx(struct mlx5_core_dev *mdev)
+{
+ return MLX5_CAP_GEN(mdev, tls_tx);
+}
+
+static inline bool mlx5_accel_is_ktls_rx(struct mlx5_core_dev *mdev)
+{
+ return MLX5_CAP_GEN(mdev, tls_rx);
+}
+
static inline bool mlx5_accel_is_ktls_device(struct mlx5_core_dev *mdev)
{
- if (!MLX5_CAP_GEN(mdev, tls_tx))
+ if (!mlx5_accel_is_ktls_tx(mdev) &&
+ !mlx5_accel_is_ktls_rx(mdev))
return false;
if (!MLX5_CAP_GEN(mdev, log_max_dek))
@@ -67,6 +78,12 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
return false;
}
#else
+static inline bool mlx5_accel_is_ktls_tx(struct mlx5_core_dev *mdev)
+{ return false; }
+
+static inline bool mlx5_accel_is_ktls_rx(struct mlx5_core_dev *mdev)
+{ return false; }
+
static inline int
mlx5_ktls_create_key(struct mlx5_core_dev *mdev,
struct tls_crypto_info *crypto_info,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index 31cac239563de..ad1525aa5670b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -11,6 +11,10 @@
enum mlx5e_icosq_wqe_type {
MLX5E_ICOSQ_WQE_NOP,
MLX5E_ICOSQ_WQE_UMR_RX,
+#ifdef CONFIG_MLX5_EN_TLS
+ MLX5E_ICOSQ_WQE_UMR_TLS,
+ MLX5E_ICOSQ_WQE_SET_PSV_TLS,
+#endif
};
static inline bool
@@ -114,9 +118,16 @@ struct mlx5e_icosq_wqe_info {
struct {
struct mlx5e_rq *rq;
} umr;
+ struct {
+#ifdef CONFIG_MLX5_EN_TLS
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+#endif
+ } accel;
};
};
+void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq);
+
static inline u16 mlx5e_icosq_get_next_pi(struct mlx5e_icosq *sq, u16 size)
{
struct mlx5_wq_cyc *wq = &sq->wq;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
index fac145dcf2ceb..7b6abea850d44 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/en_accel.h
@@ -37,6 +37,7 @@
#include <linux/skbuff.h>
#include <linux/netdevice.h>
#include "en_accel/ipsec_rxtx.h"
+#include "en_accel/tls.h"
#include "en_accel/tls_rxtx.h"
#include "en.h"
#include "en/txrx.h"
@@ -147,4 +148,23 @@ static inline bool mlx5e_accel_tx_finish(struct mlx5e_priv *priv,
return true;
}
+static inline int mlx5e_accel_sk_get_rxq(struct sock *sk)
+{
+ int rxq = sk_rx_queue_get(sk);
+
+ if (unlikely(rxq == -1))
+ rxq = 0;
+
+ return rxq;
+}
+
+static inline int mlx5e_accel_init_rx(struct mlx5e_priv *priv)
+{
+ return mlx5e_ktls_init_rx(priv);
+}
+
+static inline void mlx5e_accel_cleanup_rx(struct mlx5e_priv *priv)
+{
+ mlx5e_ktls_cleanup_rx(priv);
+}
#endif /* __MLX5E_EN_ACCEL_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
index 8970ea68d005c..313bdbaaf9d86 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
@@ -4,6 +4,7 @@
#include "en.h"
#include "en_accel/ktls.h"
#include "en_accel/ktls_utils.h"
+#include "en_accel/fs_tcp.h"
static int mlx5e_ktls_add(struct net_device *netdev, struct sock *sk,
enum tls_offload_ctx_dir direction,
@@ -14,13 +15,13 @@ static int mlx5e_ktls_add(struct net_device *netdev, struct sock *sk,
struct mlx5_core_dev *mdev = priv->mdev;
int err;
- if (WARN_ON(direction != TLS_OFFLOAD_CTX_DIR_TX))
- return -EINVAL;
-
if (WARN_ON(!mlx5e_ktls_type_check(mdev, crypto_info)))
return -EOPNOTSUPP;
- err = mlx5e_ktls_add_tx(netdev, sk, crypto_info, start_offload_tcp_sn);
+ if (direction == TLS_OFFLOAD_CTX_DIR_TX)
+ err = mlx5e_ktls_add_tx(netdev, sk, crypto_info, start_offload_tcp_sn);
+ else
+ err = mlx5e_ktls_add_rx(netdev, sk, crypto_info, start_offload_tcp_sn);
return err;
}
@@ -29,26 +30,74 @@ static void mlx5e_ktls_del(struct net_device *netdev,
struct tls_context *tls_ctx,
enum tls_offload_ctx_dir direction)
{
- if (direction != TLS_OFFLOAD_CTX_DIR_TX)
- return;
+ if (direction == TLS_OFFLOAD_CTX_DIR_TX)
+ mlx5e_ktls_del_tx(netdev, tls_ctx);
+ else
+ mlx5e_ktls_del_rx(netdev, tls_ctx);
+}
+
+static int mlx5e_ktls_resync(struct net_device *netdev,
+ struct sock *sk, u32 seq, u8 *rcd_sn,
+ enum tls_offload_ctx_dir direction)
+{
+ if (unlikely(direction != TLS_OFFLOAD_CTX_DIR_RX))
+ return -EINVAL;
- mlx5e_ktls_del_tx(netdev, tls_ctx);
+ return mlx5e_ktls_rx_resync(netdev, sk, seq, rcd_sn);
}
static const struct tlsdev_ops mlx5e_ktls_ops = {
.tls_dev_add = mlx5e_ktls_add,
.tls_dev_del = mlx5e_ktls_del,
+ .tls_dev_resync = mlx5e_ktls_resync,
};
void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)
{
struct net_device *netdev = priv->netdev;
+ struct mlx5_core_dev *mdev = priv->mdev;
- if (!mlx5_accel_is_ktls_device(priv->mdev))
+ if (!mlx5_accel_is_ktls_device(mdev))
return;
- netdev->hw_features |= NETIF_F_HW_TLS_TX;
- netdev->features |= NETIF_F_HW_TLS_TX;
+ if (mlx5_accel_is_ktls_tx(mdev)) {
+ netdev->hw_features |= NETIF_F_HW_TLS_TX;
+ netdev->features |= NETIF_F_HW_TLS_TX;
+ }
+
+ if (mlx5_accel_is_ktls_rx(mdev))
+ netdev->hw_features |= NETIF_F_HW_TLS_RX;
netdev->tlsdev_ops = &mlx5e_ktls_ops;
}
+
+int mlx5e_ktls_set_feature_rx(struct net_device *netdev, bool enable)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ int err = 0;
+
+ mutex_lock(&priv->state_lock);
+ if (enable)
+ err = mlx5e_accel_fs_tcp_create(priv);
+ else
+ mlx5e_accel_fs_tcp_destroy(priv);
+ mutex_unlock(&priv->state_lock);
+
+ return err;
+}
+
+int mlx5e_ktls_init_rx(struct mlx5e_priv *priv)
+{
+ int err = 0;
+
+ if (priv->netdev->features & NETIF_F_HW_TLS_RX)
+ err = mlx5e_accel_fs_tcp_create(priv);
+
+ return err;
+}
+
+void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv)
+{
+ if (priv->netdev->features & NETIF_F_HW_TLS_RX)
+ mlx5e_accel_fs_tcp_destroy(priv);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 69d736954977c..baa58b62e8df0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -9,13 +9,30 @@
#ifdef CONFIG_MLX5_EN_TLS
void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv);
-
+int mlx5e_ktls_init_rx(struct mlx5e_priv *priv);
+void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv);
+int mlx5e_ktls_set_feature_rx(struct net_device *netdev, bool enable);
#else
static inline void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)
{
}
+static inline int mlx5e_ktls_init_rx(struct mlx5e_priv *priv)
+{
+ return 0;
+}
+
+static inline void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv)
+{
+}
+
+static inline int mlx5e_ktls_set_feature_rx(struct net_device *netdev, bool enable)
+{
+ netdev_warn(netdev, "kTLS is not supported\n");
+ return -EOPNOTSUPP;
+}
+
#endif
#endif /* __MLX5E_TLS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
new file mode 100644
index 0000000000000..83779cbc380a7
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+// Copyright (c) 2019 Mellanox Technologies.
+
+#include "en_accel/en_accel.h"
+#include "en_accel/ktls_txrx.h"
+#include "en_accel/ktls_utils.h"
+#include "en_accel/fs_tcp.h"
+
+struct accel_rule {
+ struct work_struct work;
+ struct mlx5e_priv *priv;
+ struct mlx5_flow_handle *rule;
+ struct sock *sk;
+};
+
+enum {
+ MLX5E_PRIV_RX_FLAG_DELETING,
+ MLX5E_NUM_PRIV_RX_FLAGS,
+};
+
+struct mlx5e_ktls_offload_context_rx {
+ struct tls12_crypto_info_aes_gcm_128 crypto_info;
+ struct accel_rule rule;
+ struct tls_offload_context_rx *rx_ctx;
+ struct completion add_ctx;
+ u32 tirn;
+ u32 key_id;
+ u32 rxq;
+ DECLARE_BITMAP(flags, MLX5E_NUM_PRIV_RX_FLAGS);
+};
+
+static int mlx5e_ktls_create_tir(struct mlx5_core_dev *mdev, u32 *tirn, u32 rqtn)
+{
+ int err, inlen;
+ void *tirc;
+ u32 *in;
+
+ inlen = MLX5_ST_SZ_BYTES(create_tir_in);
+ in = kvzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return -ENOMEM;
+
+ tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
+
+ MLX5_SET(tirc, tirc, transport_domain, mdev->mlx5e_res.td.tdn);
+ MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT);
+ MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
+ MLX5_SET(tirc, tirc, indirect_table, rqtn);
+ MLX5_SET(tirc, tirc, tls_en, 1);
+ MLX5_SET(tirc, tirc, self_lb_block,
+ MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST |
+ MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST);
+
+ err = mlx5_core_create_tir(mdev, in, tirn);
+
+ kvfree(in);
+ return err;
+}
+
+static void accel_rule_handle_work(struct work_struct *work)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct accel_rule *accel_rule;
+ struct mlx5_flow_handle *rule;
+
+ accel_rule = container_of(work, struct accel_rule, work);
+ priv_rx = container_of(accel_rule, struct mlx5e_ktls_offload_context_rx, rule);
+ if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags)))
+ goto out;
+
+ rule = mlx5e_accel_fs_add_sk(accel_rule->priv, accel_rule->sk,
+ priv_rx->tirn, MLX5_FS_DEFAULT_FLOW_TAG);
+ if (!IS_ERR_OR_NULL(rule))
+ accel_rule->rule = rule;
+out:
+ complete(&priv_rx->add_ctx);
+}
+
+static void accel_rule_init(struct accel_rule *rule, struct mlx5e_priv *priv,
+ struct sock *sk)
+{
+ INIT_WORK(&rule->work, accel_rule_handle_work);
+ rule->priv = priv;
+ rule->sk = sk;
+}
+
+static void icosq_fill_wi(struct mlx5e_icosq *sq,
+ u16 pi, u8 wqe_type, u8 num_wqebbs,
+ struct mlx5e_ktls_offload_context_rx *priv_rx)
+{
+ struct mlx5e_icosq_wqe_info *wi = &sq->db.wqe_info[pi];
+
+ *wi = (struct mlx5e_icosq_wqe_info) {
+ .wqe_type = wqe_type,
+ .num_wqebbs = num_wqebbs,
+ .accel.priv_rx = priv_rx,
+ };
+}
+
+static struct mlx5_wqe_ctrl_seg *
+post_static_params(struct mlx5e_icosq *sq,
+ struct mlx5e_ktls_offload_context_rx *priv_rx)
+{
+ struct mlx5e_set_tls_static_params_wqe *wqe;
+ u16 pi, num_wqebbs, room;
+
+ num_wqebbs = MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS;
+ room = mlx5e_stop_room_for_wqe(num_wqebbs);
+ if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, room)))
+ return ERR_PTR(-ENOSPC);
+
+ pi = mlx5e_icosq_get_next_pi(sq, num_wqebbs);
+ wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
+ mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_rx->crypto_info,
+ priv_rx->tirn, priv_rx->key_id, false,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_UMR_TLS, num_wqebbs, priv_rx);
+ sq->pc += num_wqebbs;
+
+ return &wqe->ctrl;
+}
+
+static struct mlx5_wqe_ctrl_seg *
+post_progress_params(struct mlx5e_icosq *sq,
+ struct mlx5e_ktls_offload_context_rx *priv_rx,
+ u32 next_record_tcp_sn)
+{
+ struct mlx5e_set_tls_progress_params_wqe *wqe;
+ u16 pi, num_wqebbs, room;
+
+ num_wqebbs = MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS;
+ room = mlx5e_stop_room_for_wqe(num_wqebbs);
+ if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, room)))
+ return ERR_PTR(-ENOSPC);
+
+ pi = mlx5e_icosq_get_next_pi(sq, num_wqebbs);
+ wqe = MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi);
+ mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_rx->tirn, false,
+ next_record_tcp_sn,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_SET_PSV_TLS, num_wqebbs, priv_rx);
+ sq->pc += num_wqebbs;
+
+ return &wqe->ctrl;
+}
+
+static int post_rx_param_wqes(struct mlx5e_channel *c,
+ struct mlx5e_ktls_offload_context_rx *priv_rx,
+ u32 next_record_tcp_sn)
+{
+ struct mlx5_wqe_ctrl_seg *cseg;
+ struct mlx5e_icosq *sq;
+ int err;
+
+ err = 0;
+ sq = &c->async_icosq;
+ spin_lock(&c->async_icosq_lock);
+
+ cseg = post_static_params(sq, priv_rx);
+ if (IS_ERR(cseg)) {
+ err = PTR_ERR(cseg);
+ complete(&priv_rx->add_ctx);
+ goto unlock;
+ }
+ cseg = post_progress_params(sq, priv_rx, next_record_tcp_sn);
+ if (IS_ERR(cseg)) {
+ err = PTR_ERR(cseg);
+ complete(&priv_rx->add_ctx);
+ goto unlock;
+ }
+
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, cseg);
+unlock:
+ spin_unlock(&c->async_icosq_lock);
+
+ return err;
+}
+
+static void
+mlx5e_set_ktls_rx_priv_ctx(struct tls_context *tls_ctx,
+ struct mlx5e_ktls_offload_context_rx *priv_rx)
+{
+ struct mlx5e_ktls_offload_context_rx **ctx =
+ __tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_RX);
+
+ BUILD_BUG_ON(sizeof(struct mlx5e_ktls_offload_context_rx *) >
+ TLS_OFFLOAD_CONTEXT_SIZE_RX);
+
+ *ctx = priv_rx;
+}
+
+static struct mlx5e_ktls_offload_context_rx *
+mlx5e_get_ktls_rx_priv_ctx(struct tls_context *tls_ctx)
+{
+ struct mlx5e_ktls_offload_context_rx **ctx =
+ __tls_driver_ctx(tls_ctx, TLS_OFFLOAD_CTX_DIR_RX);
+
+ return *ctx;
+}
+
+/* Re-sync */
+int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
+ u32 seq, u8 *rcd_sn)
+{
+ return -EOPNOTSUPP;
+}
+
+/* End of resync section */
+
+void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
+{
+ u8 tls_offload = get_cqe_tls_offload(cqe);
+
+ if (likely(tls_offload == CQE_TLS_OFFLOAD_NOT_DECRYPTED))
+ return;
+
+ switch (tls_offload) {
+ case CQE_TLS_OFFLOAD_DECRYPTED:
+ skb->decrypted = 1;
+ break;
+ case CQE_TLS_OFFLOAD_RESYNC:
+ break;
+ default: /* CQE_TLS_OFFLOAD_ERROR: */
+ break;
+ }
+}
+
+void mlx5e_ktls_handle_ctx_completion(struct mlx5e_icosq_wqe_info *wi)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx = wi->accel.priv_rx;
+ struct accel_rule *rule = &priv_rx->rule;
+
+ if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags))) {
+ complete(&priv_rx->add_ctx);
+ return;
+ }
+ queue_work(rule->priv->tls->rx_wq, &rule->work);
+}
+
+int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
+ struct tls_crypto_info *crypto_info,
+ u32 start_offload_tcp_sn)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct tls_context *tls_ctx;
+ struct mlx5_core_dev *mdev;
+ struct mlx5e_priv *priv;
+ int rxq, err;
+ u32 rqtn;
+
+ tls_ctx = tls_get_ctx(sk);
+ priv = netdev_priv(netdev);
+ mdev = priv->mdev;
+ priv_rx = kzalloc(sizeof(*priv_rx), GFP_KERNEL);
+ if (unlikely(!priv_rx))
+ return -ENOMEM;
+
+ err = mlx5_ktls_create_key(mdev, crypto_info, &priv_rx->key_id);
+ if (err)
+ goto err_create_key;
+
+ priv_rx->crypto_info =
+ *(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
+ priv_rx->rx_ctx = tls_offload_ctx_rx(tls_ctx);
+ priv_rx->rxq = mlx5e_accel_sk_get_rxq(sk);
+
+ mlx5e_set_ktls_rx_priv_ctx(tls_ctx, priv_rx);
+
+ rxq = priv_rx->rxq;
+ rqtn = priv->direct_tir[rxq].rqt.rqtn;
+
+ err = mlx5e_ktls_create_tir(mdev, &priv_rx->tirn, rqtn);
+ if (err)
+ goto err_create_tir;
+
+ init_completion(&priv_rx->add_ctx);
+ accel_rule_init(&priv_rx->rule, priv, sk);
+ err = post_rx_param_wqes(priv->channels.c[rxq], priv_rx, start_offload_tcp_sn);
+ if (err)
+ goto err_post_wqes;
+
+ return 0;
+
+err_post_wqes:
+ mlx5_core_destroy_tir(mdev, priv_rx->tirn);
+err_create_tir:
+ mlx5_ktls_destroy_key(mdev, priv_rx->key_id);
+err_create_key:
+ kfree(priv_rx);
+ return err;
+}
+
+void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5_core_dev *mdev;
+ struct mlx5e_priv *priv;
+
+ priv = netdev_priv(netdev);
+ mdev = priv->mdev;
+
+ priv_rx = mlx5e_get_ktls_rx_priv_ctx(tls_ctx);
+ set_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags);
+ if (!cancel_work_sync(&priv_rx->rule.work))
+ /* completion is needed, as the priv_rx in the add flow
+ * is maintained on the wqe info (wi), not on the socket.
+ */
+ wait_for_completion(&priv_rx->add_ctx);
+
+ if (priv_rx->rule.rule)
+ mlx5e_accel_fs_del_sk(priv_rx->rule.rule);
+
+ mlx5_core_destroy_tir(mdev, priv_rx->tirn);
+ mlx5_ktls_destroy_key(mdev, priv_rx->key_id);
+ kfree(priv_rx);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index 5a980f93c3263..9c34ffa55b328 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -188,7 +188,7 @@ post_progress_params(struct mlx5e_txqsq *sq,
num_wqebbs = MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS;
pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
wqe = MLX5E_TLS_FETCH_SET_PROGRESS_PARAMS_WQE(sq, pi);
- mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_tx->tisn, fence,
+ mlx5e_ktls_build_progress_params(wqe, sq->pc, sq->sqn, priv_tx->tisn, fence, 0,
TLS_OFFLOAD_CTX_DIR_TX);
tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
sq->pc += num_wqebbs;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index edf404eaa2752..c1f1ad32ca4c2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -59,11 +59,13 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
{
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ u8 opmod = direction == TLS_OFFLOAD_CTX_DIR_TX ?
+ MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS :
+ MLX5_OPC_MOD_TLS_TIR_STATIC_PARAMS;
#define STATIC_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
- cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR |
- (MLX5_OPC_MOD_TLS_TIS_STATIC_PARAMS << 24));
+ cseg->opmod_idx_opcode = cpu_to_be32((pc << 8) | MLX5_OPCODE_UMR | (opmod << 24));
cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
STATIC_PARAMS_DS_CNT);
cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
@@ -76,12 +78,15 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
}
static void
-fill_progress_params(struct mlx5_wqe_tls_progress_params_seg *params, u32 tis_tir_num)
+fill_progress_params(struct mlx5_wqe_tls_progress_params_seg *params, u32 tis_tir_num,
+ u32 next_record_tcp_sn)
{
u8 *ctx = params->ctx;
params->tis_tir_num = cpu_to_be32(tis_tir_num);
+ MLX5_SET(tls_progress_params, ctx, next_record_tcp_sn,
+ next_record_tcp_sn);
MLX5_SET(tls_progress_params, ctx, record_tracker_state,
MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_START);
MLX5_SET(tls_progress_params, ctx, auth_state,
@@ -92,19 +97,22 @@ void
mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
u16 pc, u32 sqn,
u32 tis_tir_num, bool fence,
+ u32 next_record_tcp_sn,
enum tls_offload_ctx_dir direction)
{
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ u8 opmod = direction == TLS_OFFLOAD_CTX_DIR_TX ?
+ MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS :
+ MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS;
#define PROGRESS_PARAMS_DS_CNT DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS)
cseg->opmod_idx_opcode =
- cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV |
- (MLX5_OPC_MOD_TLS_TIS_PROGRESS_PARAMS << 24));
+ cpu_to_be32((pc << 8) | MLX5_OPCODE_SET_PSV | (opmod << 24));
cseg->qpn_ds = cpu_to_be32((sqn << MLX5_WQE_CTRL_QPN_SHIFT) |
PROGRESS_PARAMS_DS_CNT);
cseg->fm_ce_se = fence ? MLX5_FENCE_MODE_INITIATOR_SMALL : 0;
- fill_progress_params(&wqe->params, tis_tir_num);
+ fill_progress_params(&wqe->params, tis_tir_num, next_record_tcp_sn);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
index a325f34831176..7d7b63ae5c350 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
@@ -21,6 +21,10 @@ u8 mlx5e_ktls_dumps_num_wqebbs(struct mlx5e_txqsq *sq, unsigned int nfrags,
bool mlx5e_ktls_handle_tx_skb(struct tls_context *tls_ctx, struct mlx5e_txqsq *sq,
struct sk_buff *skb, int datalen,
struct mlx5e_accel_tx_tls_state *state);
+void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
+
+void mlx5e_ktls_handle_ctx_completion(struct mlx5e_icosq_wqe_info *wi);
void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
struct mlx5e_tx_wqe_info *wi,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
index d1d747cb2dcb6..af8117d4bf884 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
@@ -23,6 +23,11 @@ enum {
int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,
struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);
void mlx5e_ktls_del_tx(struct net_device *netdev, struct tls_context *tls_ctx);
+int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
+ struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn);
+void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx);
+int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
+ u32 seq, u8 *rcd_sn);
struct mlx5e_set_tls_static_params_wqe {
struct mlx5_wqe_ctrl_seg ctrl;
@@ -64,6 +69,7 @@ void
mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
u16 pc, u32 sqn,
u32 tis_tir_num, bool fence,
+ u32 next_record_tcp_sn,
enum tls_offload_ctx_dir direction);
#endif /* __MLX5E_TLS_UTILS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
index fba561ffe1d4e..dd9bbf3830564 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.c
@@ -197,6 +197,7 @@ void mlx5e_tls_build_netdev(struct mlx5e_priv *priv)
return;
}
+ /* FPGA */
if (!mlx5_accel_is_tls_device(priv->mdev))
return;
@@ -226,6 +227,10 @@ int mlx5e_tls_init(struct mlx5e_priv *priv)
if (!tls)
return -ENOMEM;
+ tls->rx_wq = create_singlethread_workqueue("mlx5e_tls_rx");
+ if (!tls->rx_wq)
+ return -ENOMEM;
+
priv->tls = tls;
return 0;
}
@@ -237,6 +242,7 @@ void mlx5e_tls_cleanup(struct mlx5e_priv *priv)
if (!tls)
return;
+ destroy_workqueue(tls->rx_wq);
kfree(tls);
priv->tls = NULL;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
index 9015f3f7792d2..ca0c2ebb41a1c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls.h
@@ -53,6 +53,7 @@ struct mlx5e_tls_sw_stats {
struct mlx5e_tls {
struct mlx5e_tls_sw_stats sw_stats;
+ struct workqueue_struct *rx_wq;
};
struct mlx5e_tls_offload_context_tx {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
index 4d796fea906d1..182841322ce42 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
@@ -278,9 +278,10 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
if (WARN_ON_ONCE(tls_ctx->netdev != netdev))
goto err_out;
- if (MLX5_CAP_GEN(sq->channel->mdev, tls_tx))
+ if (mlx5_accel_is_ktls_tx(sq->channel->mdev))
return mlx5e_ktls_handle_tx_skb(tls_ctx, sq, skb, datalen, state);
+ /* FPGA */
skb_seq = ntohl(tcp_hdr(skb)->seq);
context = mlx5e_get_tls_tx_context(tls_ctx);
expected_seq = context->expected_seq;
@@ -354,12 +355,16 @@ static int tls_update_resync_sn(struct net_device *netdev,
return 0;
}
-void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
- u32 *cqe_bcnt)
+void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
{
struct mlx5e_tls_metadata *mdata;
struct mlx5e_priv *priv;
+ if (likely(mlx5_accel_is_ktls_rx(rq->mdev)))
+ return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
+
+ /* FPGA */
if (!is_metadata_hdr_valid(skb))
return;
@@ -370,13 +375,13 @@ void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
skb->decrypted = 1;
break;
case SYNDROM_RESYNC_REQUEST:
- tls_update_resync_sn(netdev, skb, mdata);
- priv = netdev_priv(netdev);
+ tls_update_resync_sn(rq->netdev, skb, mdata);
+ priv = netdev_priv(rq->netdev);
atomic64_inc(&priv->tls->sw_stats.rx_tls_resync_request);
break;
case SYNDROM_AUTH_FAILED:
/* Authentication failure will be observed and verified by kTLS */
- priv = netdev_priv(netdev);
+ priv = netdev_priv(rq->netdev);
atomic64_inc(&priv->tls->sw_stats.rx_tls_auth_fail);
break;
default:
@@ -395,9 +400,10 @@ u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
if (!mlx5_accel_is_tls_device(mdev))
return 0;
- if (MLX5_CAP_GEN(mdev, tls_tx))
+ if (mlx5_accel_is_ktls_device(mdev))
return mlx5e_ktls_get_stop_room(sq);
+ /* FPGA */
/* Resync SKB. */
return mlx5e_stop_room_for_wqe(MLX5_SEND_WQE_MAX_WQEBBS);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
index 2a7b98531539f..8bb7906740425 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
@@ -49,8 +49,8 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
struct mlx5e_accel_tx_tls_state *state);
-void mlx5e_tls_handle_rx_skb(struct net_device *netdev, struct sk_buff *skb,
- u32 *cqe_bcnt);
+void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
#else
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index db18f602cdb2f..1b173d0aad1f4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1441,6 +1441,7 @@ void mlx5e_close_icosq(struct mlx5e_icosq *sq)
struct mlx5e_channel *c = sq->channel;
mlx5e_destroy_sq(c->mdev, sq->sqn);
+ mlx5e_free_icosq_descs(sq);
mlx5e_free_icosq(sq);
}
@@ -3851,6 +3852,7 @@ int mlx5e_set_features(struct net_device *netdev, netdev_features_t features)
#ifdef CONFIG_MLX5_EN_ARFS
err |= MLX5E_HANDLE_FEATURE(NETIF_F_NTUPLE, set_feature_arfs);
#endif
+ err |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_TLS_RX, mlx5e_ktls_set_feature_rx);
if (err) {
netdev->features = oper_features;
@@ -5133,8 +5135,14 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
if (err)
goto err_destroy_flow_steering;
+ err = mlx5e_accel_init_rx(priv);
+ if (err)
+ goto err_tc_nic_cleanup;
+
return 0;
+err_tc_nic_cleanup:
+ mlx5e_tc_nic_cleanup(priv);
err_destroy_flow_steering:
mlx5e_destroy_flow_steering(priv);
err_destroy_xsk_tirs:
@@ -5158,6 +5166,7 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
static void mlx5e_cleanup_nic_rx(struct mlx5e_priv *priv)
{
+ mlx5e_accel_cleanup_rx(priv);
mlx5e_tc_nic_cleanup(priv);
mlx5e_destroy_flow_steering(priv);
mlx5e_destroy_direct_tirs(priv, priv->xsk_tir);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 6b3c82da199ce..d5cce529190d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -578,6 +578,30 @@ bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq)
return !!err;
}
+void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
+{
+ u16 sqcc;
+
+ sqcc = sq->cc;
+
+ while (sqcc != sq->pc) {
+ struct mlx5e_icosq_wqe_info *wi;
+ u16 ci;
+
+ ci = mlx5_wq_cyc_ctr2ix(&sq->wq, sqcc);
+ wi = &sq->db.wqe_info[ci];
+ sqcc += wi->num_wqebbs;
+#ifdef CONFIG_MLX5_EN_TLS
+ switch (wi->wqe_type) {
+ case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
+ mlx5e_ktls_handle_ctx_completion(wi);
+ break;
+ }
+#endif
+ }
+ sq->cc = sqcc;
+}
+
int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
{
struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
@@ -633,6 +657,13 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
break;
case MLX5E_ICOSQ_WQE_NOP:
break;
+#ifdef CONFIG_MLX5_EN_TLS
+ case MLX5E_ICOSQ_WQE_UMR_TLS:
+ break;
+ case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
+ mlx5e_ktls_handle_ctx_completion(wi);
+ break;
+#endif
default:
netdev_WARN_ONCE(cq->channel->netdev,
"Bad WQE type in ICOSQ WQE info: 0x%x\n",
@@ -983,7 +1014,7 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
skb->mac_len = ETH_HLEN;
#ifdef CONFIG_MLX5_EN_TLS
- mlx5e_tls_handle_rx_skb(netdev, skb, &cqe_bcnt);
+ mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
#endif
if (lro_num_seg > 1) {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
index a5fbe73435081..c3095863372cd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
@@ -35,6 +35,7 @@
#include <linux/module.h>
#include "mlx5_core.h"
#include "../../mlxfw/mlxfw.h"
+#include "accel/tls.h"
enum {
MCQS_IDENTIFIER_BOOT_IMG = 0x1,
@@ -236,7 +237,7 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)
return err;
}
- if (MLX5_CAP_GEN(dev, tls_tx)) {
+ if (mlx5_accel_is_ktls_tx(dev) || mlx5_accel_is_ktls_rx(dev)) {
err = mlx5_core_get_caps(dev, MLX5_CAP_TLS);
if (err)
return err;
--
2.26.2
^ permalink raw reply related
* [net-next 09/11] net/mlx5e: kTLS, Add kTLS RX stats
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Add global and per-channel ethtool SW stats for the device
offload.
Document the new counters in tls-offload.rst.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
Documentation/networking/tls-offload.rst | 8 +++++++
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 10 ++++++++
.../ethernet/mellanox/mlx5/core/en_stats.c | 24 +++++++++++++++++++
.../ethernet/mellanox/mlx5/core/en_stats.h | 15 ++++++++++++
4 files changed, 57 insertions(+)
diff --git a/Documentation/networking/tls-offload.rst b/Documentation/networking/tls-offload.rst
index f914e81fd3a64..44c4b19647746 100644
--- a/Documentation/networking/tls-offload.rst
+++ b/Documentation/networking/tls-offload.rst
@@ -428,6 +428,14 @@ by the driver:
which were part of a TLS stream.
* ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets
which were successfully decrypted.
+ * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for
+ decryption.
+ * ``rx_tls_ooo`` - number of RX packets which were part of a TLS stream
+ but did not arrive in the expected order and triggered the resync procedure.
+ * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
+ (connection has finished).
+ * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
+ but were not decrypted due to unexpected error in the state machine.
* ``tx_tls_encrypted_packets`` - number of TX packets passed to the device
for encryption of their TLS payload.
* ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index 83779cbc380a7..13e666403a5df 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -6,6 +6,9 @@
#include "en_accel/ktls_utils.h"
#include "en_accel/fs_tcp.h"
+#define KTLS_STATS_INC(priv, rxq, fld) \
+ ((priv)->channels.c[rxq]->rq.stats->fld++)
+
struct accel_rule {
struct work_struct work;
struct mlx5e_priv *priv;
@@ -218,10 +221,14 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
switch (tls_offload) {
case CQE_TLS_OFFLOAD_DECRYPTED:
skb->decrypted = 1;
+ rq->stats->tls_decrypted_packets++;
+ rq->stats->tls_decrypted_bytes += *cqe_bcnt;
break;
case CQE_TLS_OFFLOAD_RESYNC:
+ rq->stats->tls_ooo++;
break;
default: /* CQE_TLS_OFFLOAD_ERROR: */
+ rq->stats->tls_err++;
break;
}
}
@@ -280,6 +287,8 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
if (err)
goto err_post_wqes;
+ KTLS_STATS_INC(priv, rxq, tls_ctx);
+
return 0;
err_post_wqes:
@@ -308,6 +317,7 @@ void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx)
*/
wait_for_completion(&priv_rx->add_ctx);
+ KTLS_STATS_INC(priv, priv_rx->rxq, tls_del);
if (priv_rx->rule.rule)
mlx5e_accel_fs_del_sk(priv_rx->rule.rule);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
index f009fe09e99b3..13d64e6142069 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
@@ -163,6 +163,14 @@ static const struct counter_desc sw_stats_desc[] = {
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_congst_umr) },
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_arfs_err) },
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_recover) },
+#ifdef CONFIG_MLX5_EN_TLS
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_decrypted_packets) },
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_decrypted_bytes) },
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_ctx) },
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_del) },
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_ooo) },
+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_tls_err) },
+#endif
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_events) },
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_arm) },
@@ -275,6 +283,14 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
s->rx_congst_umr += rq_stats->congst_umr;
s->rx_arfs_err += rq_stats->arfs_err;
s->rx_recover += rq_stats->recover;
+#ifdef CONFIG_MLX5_EN_TLS
+ s->rx_tls_decrypted_packets += rq_stats->tls_decrypted_packets;
+ s->rx_tls_decrypted_bytes += rq_stats->tls_decrypted_bytes;
+ s->rx_tls_ctx += rq_stats->tls_ctx;
+ s->rx_tls_del += rq_stats->tls_del;
+ s->rx_tls_ooo += rq_stats->tls_ooo;
+ s->rx_tls_err += rq_stats->tls_err;
+#endif
s->ch_events += ch_stats->events;
s->ch_poll += ch_stats->poll;
s->ch_arm += ch_stats->arm;
@@ -1475,6 +1491,14 @@ static const struct counter_desc rq_stats_desc[] = {
{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, congst_umr) },
{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, arfs_err) },
{ MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, recover) },
+#ifdef CONFIG_MLX5_EN_TLS
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_decrypted_packets) },
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_decrypted_bytes) },
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_ctx) },
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_del) },
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_ooo) },
+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, tls_err) },
+#endif
};
static const struct counter_desc sq_stats_desc[] = {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
index 2b83ba9907140..b8d4488c9b328 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
@@ -186,6 +186,13 @@ struct mlx5e_sw_stats {
u64 tx_tls_skip_no_sync_data;
u64 tx_tls_drop_no_sync_data;
u64 tx_tls_drop_bypass_req;
+
+ u64 rx_tls_decrypted_packets;
+ u64 rx_tls_decrypted_bytes;
+ u64 rx_tls_ctx;
+ u64 rx_tls_del;
+ u64 rx_tls_ooo;
+ u64 rx_tls_err;
#endif
u64 rx_xsk_packets;
@@ -305,6 +312,14 @@ struct mlx5e_rq_stats {
u64 congst_umr;
u64 arfs_err;
u64 recover;
+#ifdef CONFIG_MLX5_EN_TLS
+ u64 tls_decrypted_packets;
+ u64 tls_decrypted_bytes;
+ u64 tls_ctx;
+ u64 tls_del;
+ u64 tls_ooo;
+ u64 tls_err;
+#endif
};
struct mlx5e_sq_stats {
--
2.26.2
^ permalink raw reply related
* [net-next 11/11] net/mlx5e: kTLS, Improve rx handler function call
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
Prior to this patch mlx5e tls rx handler was called unconditionally on
all rx packets and the decision whether a packet is a valid tls packet
is done inside that function. A function call can be expensive especially
for regular rx packet rate. To avoid this, check the tls validity before
jumping into the tls rx handler.
While at it, split between kTLS device offload rx handler and FPGA tls rx
handler using a similar method.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 7 +----
.../mellanox/mlx5/core/en_accel/tls_rxtx.c | 12 +++------
.../mellanox/mlx5/core/en_accel/tls_rxtx.h | 26 +++++++++++++++++--
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 5 ++--
4 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index 703ce78d54043..b0e7eb92d7174 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -481,12 +481,7 @@ int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
{
- u8 tls_offload = get_cqe_tls_offload(cqe);
-
- if (likely(tls_offload == CQE_TLS_OFFLOAD_NOT_DECRYPTED))
- return;
-
- switch (tls_offload) {
+ switch (get_cqe_tls_offload(cqe)) {
case CQE_TLS_OFFLOAD_DECRYPTED:
skb->decrypted = 1;
rq->stats->tls_decrypted_packets++;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
index 182841322ce42..b0c31d49ff8db 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.c
@@ -355,19 +355,13 @@ static int tls_update_resync_sn(struct net_device *netdev,
return 0;
}
-void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
- struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
+/* FPGA tls rx handler */
+void mlx5e_tls_handle_rx_skb_metadata(struct mlx5e_rq *rq, struct sk_buff *skb,
+ u32 *cqe_bcnt)
{
struct mlx5e_tls_metadata *mdata;
struct mlx5e_priv *priv;
- if (likely(mlx5_accel_is_ktls_rx(rq->mdev)))
- return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
-
- /* FPGA */
- if (!is_metadata_hdr_valid(skb))
- return;
-
/* Use the metadata */
mdata = (struct mlx5e_tls_metadata *)(skb->data + ETH_HLEN);
switch (mdata->content.recv.syndrome) {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
index 8bb7906740425..08cb0f91fcd09 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/tls_rxtx.h
@@ -34,6 +34,7 @@
#ifndef __MLX5E_TLS_RXTX_H__
#define __MLX5E_TLS_RXTX_H__
+#include "accel/accel.h"
#include "en_accel/ktls_txrx.h"
#ifdef CONFIG_MLX5_EN_TLS
@@ -49,11 +50,32 @@ bool mlx5e_tls_handle_tx_skb(struct net_device *netdev, struct mlx5e_txqsq *sq,
void mlx5e_tls_handle_tx_wqe(struct mlx5e_txqsq *sq, struct mlx5_wqe_ctrl_seg *cseg,
struct mlx5e_accel_tx_tls_state *state);
-void mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
- struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
+void mlx5e_tls_handle_rx_skb_metadata(struct mlx5e_rq *rq, struct sk_buff *skb,
+ u32 *cqe_bcnt);
+
+static inline bool mlx5e_accel_is_tls(struct mlx5_cqe64 *cqe, struct sk_buff *skb)
+{
+ return get_cqe_tls_offload(cqe) || is_metadata_hdr_valid(skb);
+}
+
+static inline void
+mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt)
+{
+ if (likely(mlx5e_accel_is_tls(cqe, skb)))
+ return mlx5e_ktls_handle_rx_skb(rq, skb, cqe, cqe_bcnt);
+
+ /* FPGA */
+ return mlx5e_tls_handle_rx_skb_metadata(rq, skb, cqe_bcnt);
+}
#else
+static inline bool
+mlx5e_accel_is_tls(struct mlx5_cqe64 *cqe, struct sk_buff *skb) { return false; }
+static inline void
+mlx5e_tls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
+ struct mlx5_cqe64 *cqe, u32 *cqe_bcnt) {}
static inline u16 mlx5e_tls_get_stop_room(struct mlx5e_txqsq *sq)
{
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 19bcd49224526..35ff190daaa1e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -1019,9 +1019,8 @@ static inline void mlx5e_build_rx_skb(struct mlx5_cqe64 *cqe,
skb->mac_len = ETH_HLEN;
-#ifdef CONFIG_MLX5_EN_TLS
- mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
-#endif
+ if (unlikely(mlx5e_accel_is_tls(cqe, skb)))
+ mlx5e_tls_handle_rx_skb(rq, skb, cqe, &cqe_bcnt);
if (lro_num_seg > 1) {
mlx5e_lro_update_hdr(skb, cqe, cqe_bcnt);
--
2.26.2
^ permalink raw reply related
* [net-next 10/11] net/mlx5e: kTLS, Add kTLS RX resync support
From: Saeed Mahameed @ 2020-05-29 19:46 UTC (permalink / raw)
To: David S. Miller, kuba; +Cc: netdev, Tariq Toukan, Saeed Mahameed
In-Reply-To: <20200529194641.243989-1-saeedm@mellanox.com>
From: Tariq Toukan <tariqt@mellanox.com>
Implement the RX resync procedure.
The HW offload of TLS decryption in RX side might get out-of-sync
due to out-of-order reception of packets.
This requires SW intervention to update the HW context and get it
back in-sync.
Performance:
CPU: Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz, 24 cores, HT off
NIC: ConnectX-6 Dx 100GbE dual port
Goodput (app-layer throughput) comparison:
+---------------+-------+-------+---------+
| # connections | 1 | 4 | 8 |
+---------------+-------+-------+---------+
| SW (Gbps) | 7.26 | 24.70 | 50.30 |
+---------------+-------+-------+---------+
| HW (Gbps) | 18.50 | 64.30 | 92.90 |
+---------------+-------+-------+---------+
| Speedup | 2.55x | 2.56x | 1.85x * |
+---------------+-------+-------+---------+
* After linerate is reached, diff is observed in CPU util.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/en/txrx.h | 1 +
.../mellanox/mlx5/core/en_accel/ktls_rx.c | 322 +++++++++++++++++-
.../mellanox/mlx5/core/en_accel/ktls_tx.c | 2 +-
.../mellanox/mlx5/core/en_accel/ktls_txrx.c | 7 +-
.../mellanox/mlx5/core/en_accel/ktls_txrx.h | 2 +
.../mellanox/mlx5/core/en_accel/ktls_utils.h | 14 +-
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 6 +
7 files changed, 347 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index ad1525aa5670b..934c18263de9a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -14,6 +14,7 @@ enum mlx5e_icosq_wqe_type {
#ifdef CONFIG_MLX5_EN_TLS
MLX5E_ICOSQ_WQE_UMR_TLS,
MLX5E_ICOSQ_WQE_SET_PSV_TLS,
+ MLX5E_ICOSQ_WQE_GET_PSV_TLS,
#endif
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index 13e666403a5df..703ce78d54043 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -1,7 +1,9 @@
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
// Copyright (c) 2019 Mellanox Technologies.
+#include <net/inet6_hashtables.h>
#include "en_accel/en_accel.h"
+#include "en_accel/tls.h"
#include "en_accel/ktls_txrx.h"
#include "en_accel/ktls_utils.h"
#include "en_accel/fs_tcp.h"
@@ -16,6 +18,26 @@ struct accel_rule {
struct sock *sk;
};
+#define PROGRESS_PARAMS_WRITE_UNIT 64
+#define PROGRESS_PARAMS_PADDED_SIZE \
+ (ALIGN(sizeof(struct mlx5_wqe_tls_progress_params_seg), \
+ PROGRESS_PARAMS_WRITE_UNIT))
+
+struct mlx5e_ktls_rx_resync_ctx {
+ struct work_struct work;
+ struct mlx5e_priv *priv;
+ dma_addr_t dma_addr;
+ refcount_t refcnt;
+ u32 hw_seq;
+ u32 sw_seq;
+ __be64 sw_rcd_sn_be;
+
+ union {
+ struct mlx5_wqe_tls_progress_params_seg progress;
+ u8 pad[PROGRESS_PARAMS_PADDED_SIZE];
+ } ____cacheline_aligned_in_smp;
+};
+
enum {
MLX5E_PRIV_RX_FLAG_DELETING,
MLX5E_NUM_PRIV_RX_FLAGS,
@@ -30,6 +52,9 @@ struct mlx5e_ktls_offload_context_rx {
u32 key_id;
u32 rxq;
DECLARE_BITMAP(flags, MLX5E_NUM_PRIV_RX_FLAGS);
+
+ /* resync */
+ struct mlx5e_ktls_rx_resync_ctx resync;
};
static int mlx5e_ktls_create_tir(struct mlx5_core_dev *mdev, u32 *tirn, u32 rqtn)
@@ -115,7 +140,8 @@ post_static_params(struct mlx5e_icosq *sq,
pi = mlx5e_icosq_get_next_pi(sq, num_wqebbs);
wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_rx->crypto_info,
- priv_rx->tirn, priv_rx->key_id, false,
+ priv_rx->tirn, priv_rx->key_id,
+ priv_rx->resync.hw_seq, false,
TLS_OFFLOAD_CTX_DIR_RX);
icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_UMR_TLS, num_wqebbs, priv_rx);
sq->pc += num_wqebbs;
@@ -202,10 +228,252 @@ mlx5e_get_ktls_rx_priv_ctx(struct tls_context *tls_ctx)
}
/* Re-sync */
+static struct mlx5_wqe_ctrl_seg *
+resync_post_get_progress_params(struct mlx5e_icosq *sq,
+ struct mlx5e_ktls_offload_context_rx *priv_rx)
+{
+ struct mlx5e_ktls_rx_resync_ctx *resync = &priv_rx->resync;
+ struct mlx5e_get_tls_progress_params_wqe *wqe;
+ struct mlx5_wqe_ctrl_seg *cseg;
+ struct mlx5_seg_get_psv *psv;
+ u16 pi;
+
+ dma_sync_single_for_device(resync->priv->mdev->device,
+ resync->dma_addr,
+ PROGRESS_PARAMS_PADDED_SIZE,
+ DMA_FROM_DEVICE);
+ BUILD_BUG_ON(MLX5E_KTLS_GET_PROGRESS_WQEBBS != 1);
+ if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)))
+ return ERR_PTR(-ENOSPC);
+
+ pi = mlx5e_icosq_get_next_pi(sq, 1);
+ wqe = MLX5E_TLS_FETCH_GET_PROGRESS_PARAMS_WQE(sq, pi);
+
+#define GET_PSV_DS_CNT (DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_DS))
+
+ cseg = &wqe->ctrl;
+ cseg->opmod_idx_opcode =
+ cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_GET_PSV |
+ (MLX5_OPC_MOD_TLS_TIR_PROGRESS_PARAMS << 24));
+ cseg->qpn_ds =
+ cpu_to_be32((sq->sqn << MLX5_WQE_CTRL_QPN_SHIFT) | GET_PSV_DS_CNT);
+
+ psv = &wqe->psv;
+ psv->num_psv = 1 << 4;
+ psv->l_key = sq->channel->mkey_be;
+ psv->psv_index[0] = cpu_to_be32(priv_rx->tirn);
+ psv->va = cpu_to_be64(priv_rx->resync.dma_addr);
+
+ icosq_fill_wi(sq, pi, MLX5E_ICOSQ_WQE_GET_PSV_TLS, 1, priv_rx);
+ sq->pc++;
+
+ return cseg;
+}
+
+/* Function is called with elevated refcount.
+ * It decreases it only if no WQE is posted.
+ */
+static void resync_handle_work(struct work_struct *work)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync;
+ struct mlx5_wqe_ctrl_seg *cseg;
+ struct mlx5e_channel *c;
+ struct mlx5e_icosq *sq;
+ struct mlx5_wq_cyc *wq;
+
+ resync = container_of(work, struct mlx5e_ktls_rx_resync_ctx, work);
+ priv_rx = container_of(resync, struct mlx5e_ktls_offload_context_rx, resync);
+
+ if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags))) {
+ refcount_dec(&resync->refcnt);
+ return;
+ }
+
+ c = resync->priv->channels.c[priv_rx->rxq];
+ sq = &c->async_icosq;
+ wq = &sq->wq;
+
+ spin_lock(&c->async_icosq_lock);
+
+ cseg = resync_post_get_progress_params(sq, priv_rx);
+ if (IS_ERR(cseg)) {
+ refcount_dec(&resync->refcnt);
+ goto unlock;
+ }
+ mlx5e_notify_hw(wq, sq->pc, sq->uar_map, cseg);
+unlock:
+ spin_unlock(&c->async_icosq_lock);
+}
+
+static void resync_init(struct mlx5e_ktls_rx_resync_ctx *resync, struct mlx5e_priv *priv)
+{
+ INIT_WORK(&resync->work, resync_handle_work);
+ resync->priv = priv;
+ refcount_set(&resync->refcnt, 1);
+}
+
+/* Function can be called with the refcount being either elevated or not.
+ * It does not affect the refcount.
+ */
+static int resync_handle_seq_match(struct mlx5e_ktls_offload_context_rx *priv_rx,
+ struct mlx5e_channel *c)
+{
+ struct tls12_crypto_info_aes_gcm_128 *info = &priv_rx->crypto_info;
+ u8 tracker_state, auth_state, *ctx;
+ struct mlx5_wqe_ctrl_seg *cseg;
+ struct mlx5e_icosq *sq;
+ int err;
+
+ ctx = priv_rx->resync.progress.ctx;
+ tracker_state = MLX5_GET(tls_progress_params, ctx, record_tracker_state);
+ auth_state = MLX5_GET(tls_progress_params, ctx, auth_state);
+
+ if (tracker_state != MLX5E_TLS_PROGRESS_PARAMS_RECORD_TRACKER_STATE_TRACKING ||
+ auth_state != MLX5E_TLS_PROGRESS_PARAMS_AUTH_STATE_NO_OFFLOAD)
+ return 0;
+
+ memcpy(info->rec_seq, &priv_rx->resync.sw_rcd_sn_be, sizeof(info->rec_seq));
+ err = 0;
+
+ sq = &c->async_icosq;
+ spin_lock(&c->async_icosq_lock);
+
+ cseg = post_static_params(sq, priv_rx);
+ if (IS_ERR(cseg)) {
+ err = PTR_ERR(cseg);
+ goto unlock;
+ }
+ /* Do not increment priv_rx refcnt, CQE handling is empty */
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, cseg);
+unlock:
+ spin_unlock(&c->async_icosq_lock);
+
+ return err;
+}
+
+/* Function is called with elevated refcount, it decreases it. */
+void mlx5e_ktls_handle_get_psv_completion(struct mlx5e_icosq_wqe_info *wi,
+ struct mlx5e_icosq *sq)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx = wi->accel.priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync = &priv_rx->resync;
+
+ if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags)))
+ goto out;
+
+ dma_sync_single_for_cpu(resync->priv->mdev->device,
+ resync->dma_addr,
+ PROGRESS_PARAMS_PADDED_SIZE,
+ DMA_FROM_DEVICE);
+
+ resync->hw_seq = MLX5_GET(tls_progress_params, resync->progress.ctx, hw_resync_tcp_sn);
+ if (resync->hw_seq == resync->sw_seq) {
+ struct mlx5e_channel *c =
+ container_of(sq, struct mlx5e_channel, async_icosq);
+
+ resync_handle_seq_match(priv_rx, c);
+ }
+out:
+ refcount_dec(&resync->refcnt);
+}
+
+/* Runs in NAPI.
+ * Function elevates the refcount, unless no work is queued.
+ */
+static bool resync_queue_get_psv(struct sock *sk)
+{
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync;
+
+ priv_rx = mlx5e_get_ktls_rx_priv_ctx(tls_get_ctx(sk));
+ if (unlikely(!priv_rx))
+ return false;
+
+ if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags)))
+ return false;
+
+ resync = &priv_rx->resync;
+ refcount_inc(&resync->refcnt);
+ if (unlikely(!queue_work(resync->priv->tls->rx_wq, &resync->work))) {
+ refcount_dec(&resync->refcnt);
+ return false;
+ }
+
+ return true;
+}
+
+/* Runs in NAPI */
+static void resync_update_sn(struct net_device *netdev, struct sk_buff *skb)
+{
+ struct ethhdr *eth = (struct ethhdr *)(skb->data);
+ struct sock *sk = NULL;
+ int network_depth = 0;
+ struct iphdr *iph;
+ struct tcphdr *th;
+
+ __vlan_get_protocol(skb, eth->h_proto, &network_depth);
+ iph = (struct iphdr *)(skb->data + network_depth);
+
+ if (iph->version == 4) {
+ th = (void *)iph + sizeof(struct iphdr);
+
+ sk = inet_lookup_established(dev_net(netdev), &tcp_hashinfo,
+ iph->saddr, th->source, iph->daddr,
+ th->dest, netdev->ifindex);
+#if IS_ENABLED(CONFIG_IPV6)
+ } else {
+ struct ipv6hdr *ipv6h = (struct ipv6hdr *)iph;
+
+ th = (void *)ipv6h + sizeof(struct ipv6hdr);
+
+ sk = __inet6_lookup_established(dev_net(netdev), &tcp_hashinfo,
+ &ipv6h->saddr, th->source,
+ &ipv6h->daddr, ntohs(th->dest),
+ netdev->ifindex, 0);
+#endif
+ }
+
+ if (unlikely(!sk || sk->sk_state == TCP_TIME_WAIT))
+ return;
+
+ if (unlikely(!resync_queue_get_psv(sk)))
+ return;
+
+ skb->sk = sk;
+ skb->destructor = sock_edemux;
+
+ tls_offload_rx_force_resync_request(sk);
+}
+
int mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk,
u32 seq, u8 *rcd_sn)
{
- return -EOPNOTSUPP;
+ struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync;
+
+ priv_rx = mlx5e_get_ktls_rx_priv_ctx(tls_get_ctx(sk));
+ if (unlikely(!priv_rx))
+ return 0;
+
+ resync = &priv_rx->resync;
+ resync->sw_rcd_sn_be = *(__be64 *)rcd_sn;
+ resync->sw_seq = seq - (TLS_HEADER_SIZE - 1);
+
+ if (resync->hw_seq == resync->sw_seq) {
+ struct mlx5e_priv *priv;
+ struct mlx5e_channel *c;
+
+ priv = netdev_priv(netdev);
+ c = priv->channels.c[priv_rx->rxq];
+
+ if (!resync_handle_seq_match(priv_rx, c))
+ return 0;
+ }
+
+ tls_offload_rx_force_resync_request(sk);
+
+ return 0;
}
/* End of resync section */
@@ -225,6 +493,7 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
rq->stats->tls_decrypted_bytes += *cqe_bcnt;
break;
case CQE_TLS_OFFLOAD_RESYNC:
+ resync_update_sn(rq->netdev, skb);
rq->stats->tls_ooo++;
break;
default: /* CQE_TLS_OFFLOAD_ERROR: */
@@ -250,15 +519,18 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
u32 start_offload_tcp_sn)
{
struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync;
struct tls_context *tls_ctx;
struct mlx5_core_dev *mdev;
struct mlx5e_priv *priv;
+ struct device *pdev;
int rxq, err;
u32 rqtn;
tls_ctx = tls_get_ctx(sk);
priv = netdev_priv(netdev);
mdev = priv->mdev;
+ pdev = mdev->device;
priv_rx = kzalloc(sizeof(*priv_rx), GFP_KERNEL);
if (unlikely(!priv_rx))
return -ENOMEM;
@@ -282,7 +554,14 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
goto err_create_tir;
init_completion(&priv_rx->add_ctx);
+ resync = &priv_rx->resync;
+ resync->dma_addr = dma_map_single(pdev, &resync->progress,
+ PROGRESS_PARAMS_PADDED_SIZE, DMA_FROM_DEVICE);
+ if (unlikely(dma_mapping_error(pdev, resync->dma_addr)))
+ goto err_map_resync;
+
accel_rule_init(&priv_rx->rule, priv, sk);
+ resync_init(resync, priv);
err = post_rx_param_wqes(priv->channels.c[rxq], priv_rx, start_offload_tcp_sn);
if (err)
goto err_post_wqes;
@@ -292,6 +571,9 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
return 0;
err_post_wqes:
+ dma_unmap_single(pdev, resync->dma_addr,
+ PROGRESS_PARAMS_PADDED_SIZE, DMA_FROM_DEVICE);
+err_map_resync:
mlx5_core_destroy_tir(mdev, priv_rx->tirn);
err_create_tir:
mlx5_ktls_destroy_key(mdev, priv_rx->key_id);
@@ -300,27 +582,63 @@ int mlx5e_ktls_add_rx(struct net_device *netdev, struct sock *sk,
return err;
}
+/* Elevated refcount on the resync object means there are
+ * outstanding operations (uncompleted GET_PSV WQEs) that
+ * will read the resync / priv_rx objects once completed.
+ * Wait for them to avoid use-after-free.
+ */
+static void wait_for_resync(struct net_device *netdev,
+ struct mlx5e_ktls_rx_resync_ctx *resync)
+{
+#define MLX5E_KTLS_RX_RESYNC_TIMEOUT 20000 /* msecs */
+ unsigned long exp_time = jiffies + msecs_to_jiffies(MLX5E_KTLS_RX_RESYNC_TIMEOUT);
+ unsigned int refcnt;
+
+ do {
+ refcnt = refcount_read(&resync->refcnt);
+ if (refcnt == 1)
+ return;
+
+ msleep(20);
+ } while (time_before(jiffies, exp_time));
+
+ netdev_warn(netdev,
+ "Failed waiting for kTLS RX resync refcnt to be released (%u).\n",
+ refcnt);
+}
+
void mlx5e_ktls_del_rx(struct net_device *netdev, struct tls_context *tls_ctx)
{
struct mlx5e_ktls_offload_context_rx *priv_rx;
+ struct mlx5e_ktls_rx_resync_ctx *resync;
struct mlx5_core_dev *mdev;
struct mlx5e_priv *priv;
+ struct device *pdev;
priv = netdev_priv(netdev);
mdev = priv->mdev;
+ pdev = mdev->device;
priv_rx = mlx5e_get_ktls_rx_priv_ctx(tls_ctx);
set_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags);
+ mlx5e_set_ktls_rx_priv_ctx(tls_ctx, NULL);
+ napi_synchronize(&priv->channels.c[priv_rx->rxq]->napi);
if (!cancel_work_sync(&priv_rx->rule.work))
/* completion is needed, as the priv_rx in the add flow
* is maintained on the wqe info (wi), not on the socket.
*/
wait_for_completion(&priv_rx->add_ctx);
+ resync = &priv_rx->resync;
+ if (cancel_work_sync(&resync->work))
+ refcount_dec(&resync->refcnt);
+ wait_for_resync(netdev, resync);
KTLS_STATS_INC(priv, priv_rx->rxq, tls_del);
if (priv_rx->rule.rule)
mlx5e_accel_fs_del_sk(priv_rx->rule.rule);
+ dma_unmap_single(pdev, resync->dma_addr, PROGRESS_PARAMS_PADDED_SIZE,
+ DMA_FROM_DEVICE);
mlx5_core_destroy_tir(mdev, priv_rx->tirn);
mlx5_ktls_destroy_key(mdev, priv_rx->key_id);
kfree(priv_rx);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
index 9c34ffa55b328..0e6698d1b4ca9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c
@@ -171,7 +171,7 @@ post_static_params(struct mlx5e_txqsq *sq,
pi = mlx5e_txqsq_get_next_pi(sq, num_wqebbs);
wqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);
mlx5e_ktls_build_static_params(wqe, sq->pc, sq->sqn, &priv_tx->crypto_info,
- priv_tx->tisn, priv_tx->key_id, fence,
+ priv_tx->tisn, priv_tx->key_id, 0, fence,
TLS_OFFLOAD_CTX_DIR_TX);
tx_fill_wi(sq, pi, num_wqebbs, 0, NULL);
sq->pc += num_wqebbs;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index c1f1ad32ca4c2..ac29aeb8af492 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -22,7 +22,7 @@ enum {
static void
fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
struct tls12_crypto_info_aes_gcm_128 *info,
- u32 key_id)
+ u32 key_id, u32 resync_tcp_sn)
{
char *initial_rn, *gcm_iv;
u16 salt_sz, rec_seq_sz;
@@ -47,6 +47,7 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
MLX5_SET(tls_static_params, ctx, const_2, 2);
MLX5_SET(tls_static_params, ctx, encryption_standard,
MLX5E_ENCRYPTION_STANDARD_TLS);
+ MLX5_SET(tls_static_params, ctx, resync_tcp_sn, resync_tcp_sn);
MLX5_SET(tls_static_params, ctx, dek_index, key_id);
}
@@ -54,7 +55,7 @@ void
mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
u16 pc, u32 sqn,
struct tls12_crypto_info_aes_gcm_128 *info,
- u32 tis_tir_num, u32 key_id,
+ u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
bool fence, enum tls_offload_ctx_dir direction)
{
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
@@ -74,7 +75,7 @@ mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
ucseg->flags = MLX5_UMR_INLINE;
ucseg->bsf_octowords = cpu_to_be16(MLX5_ST_SZ_BYTES(tls_static_params) / 16);
- fill_static_params(&wqe->params, info, key_id);
+ fill_static_params(&wqe->params, info, key_id, resync_tcp_sn);
}
static void
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
index 7d7b63ae5c350..66e5063a22ce4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.h
@@ -25,6 +25,8 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
struct mlx5_cqe64 *cqe, u32 *cqe_bcnt);
void mlx5e_ktls_handle_ctx_completion(struct mlx5e_icosq_wqe_info *wi);
+void mlx5e_ktls_handle_get_psv_completion(struct mlx5e_icosq_wqe_info *wi,
+ struct mlx5e_icosq *sq);
void mlx5e_ktls_tx_handle_resync_dump_comp(struct mlx5e_txqsq *sq,
struct mlx5e_tx_wqe_info *wi,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
index af8117d4bf884..3dbc45e29828b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h
@@ -41,12 +41,20 @@ struct mlx5e_set_tls_progress_params_wqe {
struct mlx5_wqe_tls_progress_params_seg params;
};
+struct mlx5e_get_tls_progress_params_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_seg_get_psv psv;
+};
+
#define MLX5E_TLS_SET_STATIC_PARAMS_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_static_params_wqe), MLX5_SEND_WQE_BB))
#define MLX5E_TLS_SET_PROGRESS_PARAMS_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_set_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
+#define MLX5E_KTLS_GET_PROGRESS_WQEBBS \
+ (DIV_ROUND_UP(sizeof(struct mlx5e_get_tls_progress_params_wqe), MLX5_SEND_WQE_BB))
+
#define MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi) \
((struct mlx5e_set_tls_static_params_wqe *)\
mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_static_params_wqe)))
@@ -55,6 +63,10 @@ struct mlx5e_set_tls_progress_params_wqe {
((struct mlx5e_set_tls_progress_params_wqe *)\
mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_set_tls_progress_params_wqe)))
+#define MLX5E_TLS_FETCH_GET_PROGRESS_PARAMS_WQE(sq, pi) \
+ ((struct mlx5e_get_tls_progress_params_wqe *)\
+ mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_get_tls_progress_params_wqe)))
+
#define MLX5E_TLS_FETCH_DUMP_WQE(sq, pi) \
((struct mlx5e_dump_wqe *)\
mlx5e_fetch_wqe(&(sq)->wq, pi, sizeof(struct mlx5e_dump_wqe)))
@@ -63,7 +75,7 @@ void
mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,
u16 pc, u32 sqn,
struct tls12_crypto_info_aes_gcm_128 *info,
- u32 tis_tir_num, u32 key_id,
+ u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,
bool fence, enum tls_offload_ctx_dir direction);
void
mlx5e_ktls_build_progress_params(struct mlx5e_set_tls_progress_params_wqe *wqe,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index d5cce529190d5..19bcd49224526 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -596,6 +596,9 @@ void mlx5e_free_icosq_descs(struct mlx5e_icosq *sq)
case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
mlx5e_ktls_handle_ctx_completion(wi);
break;
+ case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
+ mlx5e_ktls_handle_get_psv_completion(wi, sq);
+ break;
}
#endif
}
@@ -663,6 +666,9 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
case MLX5E_ICOSQ_WQE_SET_PSV_TLS:
mlx5e_ktls_handle_ctx_completion(wi);
break;
+ case MLX5E_ICOSQ_WQE_GET_PSV_TLS:
+ mlx5e_ktls_handle_get_psv_completion(wi, sq);
+ break;
#endif
default:
netdev_WARN_ONCE(cq->channel->netdev,
--
2.26.2
^ permalink raw reply related
* Re: [PATCH 3/4] dt-bindings: timer: renesas,cmt: Document r8a7742 CMT support
From: Geert Uytterhoeven @ 2020-05-29 19:48 UTC (permalink / raw)
To: Rob Herring
Cc: Lad Prabhakar, Magnus Damm, Linux-Renesas,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
Linux Kernel Mailing List, Prabhakar
In-Reply-To: <20200529191402.GA2771130@bogus>
Hi Rob,
On Fri, May 29, 2020 at 9:14 PM Rob Herring <robh@kernel.org> wrote:
> On Fri, May 29, 2020 at 02:53:02PM +0200, Geert Uytterhoeven wrote:
> > On Wed, May 27, 2020 at 11:19 PM Lad Prabhakar
> > <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> > > Document SoC specific compatible strings for r8a7742. No driver change
> > > is needed as the fallback strings will activate the right code.
> > >
> > > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > > Reviewed-by: Marian-Cristian Rotariu <marian-cristian.rotariu.rb@bp.renesas.com>
> >
> > Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
> >
> > Please note this DT binding is under yamlization, cfr.
> > "[PATCH v2] dt-bindings: timer: renesas: cmt: Convert to json-schema"
> > (20200505155127.4836-1-geert+renesas@glider.be).
>
> Do I need to pick that one up? Doesn't look like it's been applied
> AFAICT.
Would be nice. Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply
* Re: [PATCH net-next] net: phy: marvell: unlock after phy_select_page() failure
From: David Miller @ 2020-05-29 19:48 UTC (permalink / raw)
To: dan.carpenter
Cc: andrew, f.fainelli, hkallweit1, linux, kuba, netdev,
kernel-janitors
In-Reply-To: <20200529100207.GB1304852@mwanda>
From: Dan Carpenter <dan.carpenter@oracle.com>
Date: Fri, 29 May 2020 13:02:07 +0300
> We need to call phy_restore_page() even if phy_select_page() fails.
> Otherwise we are holding the phy_lock_mdio_bus() lock. This requirement
> is documented at the start of the phy_select_page() function.
>
> Fixes: a618e86da91d ("net : phy: marvell: Speedup TDR data retrieval by only changing page once")
> Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH net-next] net: phy: marvell: unlock after phy_select_page() failure
From: David Miller @ 2020-05-29 19:48 UTC (permalink / raw)
To: dan.carpenter
Cc: andrew, f.fainelli, hkallweit1, linux, kuba, netdev,
kernel-janitors
In-Reply-To: <20200529100207.GB1304852@mwanda>
From: Dan Carpenter <dan.carpenter@oracle.com>
Date: Fri, 29 May 2020 13:02:07 +0300
> We need to call phy_restore_page() even if phy_select_page() fails.
> Otherwise we are holding the phy_lock_mdio_bus() lock. This requirement
> is documented at the start of the phy_select_page() function.
>
> Fixes: a618e86da91d ("net : phy: marvell: Speedup TDR data retrieval by only changing page once")
> Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Applied, thanks.
^ permalink raw reply
* Re: request to add pfr-manager repo to CI
From: Andrew Geissler @ 2020-05-29 19:51 UTC (permalink / raw)
To: Puli, Apparao; +Cc: vikram.bodireddy, OpenBMC Maillist
In-Reply-To: <63ac8cfd-86b9-43fe-91a0-60f60b4469ee@linux.intel.com>
> On May 29, 2020, at 10:33 AM, Puli, Apparao <apparao.puli@linux.intel.com> wrote:
>
> Hi Andrew,
>
> Can you please add "pfr-manager" repo to CI builds?
Hey Apparao, sure, done.
>
> https://github.com/openbmc/pfr-manager
>
> https://gerrit.openbmc-project.xyz/#/q/project:openbmc/pfr-manager
>
>
> Thanks,
>
> -Appu
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.