From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andre Tomt Subject: Re: kTLS in combination with mlx4 is very unstable Date: Mon, 23 Apr 2018 13:44:13 +0200 Message-ID: <91d415bd-e919-dc87-d9ea-d8c30a7e90f0@tomt.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: davejwatson@fb.com, Tariq Toukan To: netdev Return-path: Received: from mail1.ugh.no ([178.79.162.34]:56880 "EHLO mail1.ugh.no" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754710AbeDWLoP (ORCPT ); Mon, 23 Apr 2018 07:44:15 -0400 In-Reply-To: Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 22. april 2018 23:21, Andre Tomt wrote: > Hello! > > kTLS looks fun, so I decided to play with it. It is quite spiffy - > however with mlx4 I get kernel crashes I'm not seeing when testing on > ixgbe. > > For testing I'm using a git build of the "stream reflector" cubemap[1] > configured with kTLS and 8 worker threads running on 4 physical cores, > loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV. > > The kernel seems to get increasingly unstable as I load it up with > client connections. At about 9Gbps and 700 connections, it is okay at > least for a while - it might run fine for say 45 minutes. Once it gets > to 20 - 30Gbps, the kernel will usually start spewing OOPSes within > minutes and the traffic drops. > > Some bad interaction between mlx4 and kTLS? > > Hardware is a quad core Xeon-D 1520 using a dual port Mellanox > ConnectX-3 VPI with a single 40Gbps ethernet link configured. Mellanox > mlx4 driver settings are kernel.org upstream defaults. Interface is > configured with FQ qdisc and sockets are using BBR congestion control. > > Tested on kernel 4.14.34, 4.15.17, and 4.16.2 - 4.16.3. > > [1] https://git.sesse.net/?p=cubemap > > First OOPS (from 4.16.3) It also blows up with a similar trace on 4.17-rc2.