From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1031C10F11 for ; Sun, 14 Apr 2019 02:06:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AD2422148E for ; Sun, 14 Apr 2019 02:06:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727198AbfDNCG0 (ORCPT ); Sat, 13 Apr 2019 22:06:26 -0400 Received: from mail1.ugh.no ([178.79.162.34]:43902 "EHLO mail1.ugh.no" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727160AbfDNCG0 (ORCPT ); Sat, 13 Apr 2019 22:06:26 -0400 X-Greylist: delayed 593 seconds by postgrey-1.27 at vger.kernel.org; Sat, 13 Apr 2019 22:06:24 EDT Received: from localhost (localhost [127.0.0.1]) by mail1.ugh.no (Postfix) with ESMTP id F18C924AF7C; Sun, 14 Apr 2019 03:56:30 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at catastrophix.ugh.no Received: from mail1.ugh.no ([127.0.0.1]) by localhost (catastrophix.ugh.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7BXEJ-dURA99; Sun, 14 Apr 2019 03:56:29 +0200 (CEST) Received: from [10.255.99.11] (unknown [185.176.245.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: andre@tomt.net) by mail.ugh.no (Postfix) with ESMTPSA id BCD3824AF6C; Sun, 14 Apr 2019 03:56:29 +0200 (CEST) Subject: Re: kTLS broken somewhere between 4.18 and 5.0 To: "Steinar H. Gunderson" , netdev@vger.kernel.org, John Fastabend , Daniel Borkmann References: <20190413153435.73ecztpnbivrckvj@sesse.net> From: Andre Tomt Message-ID: <4cdfca7f-bee0-56f5-e512-8ad2e4e6dfcf@tomt.net> Date: Sun, 14 Apr 2019 03:56:28 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <20190413153435.73ecztpnbivrckvj@sesse.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 13.04.2019 17:34, Steinar H. Gunderson wrote: > Hi, > > I've been using kTLS for a while, with my video reflector Cubemap > (https://git.sesse.net/?p=cubemap). After I upgraded my server from > 4.18.11 to 5.0.6, seemingly I've started seeing corruption. The data sent > with send() (HTTP headers, HLS playlists) appears to be fine, but sendfile() > (actual video data, from a file on tmpfs) is not; after ~20 kB of data > (19626 in one test here), the data appears to be randomly corrupted. Diffing > non-TLS (good) and TLS (bad) video data: > > 00004c70: fa 70 c5 71 b5 f5 b7 ac 74 b0 ca 80 02 4c 06 3f .p.q....t....L.? > 00004c80: 5c 5b 0c b3 e0 a0 c3 21 93 d3 6e 65 36 70 0a 27 \[.....!..ne6p.' > 00004c90: 84 67 16 2c 95 c0 55 e1 04 76 52 10 50 5d 00 26 .g.,..U..vR.P].& > > -00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 18 c0 06 20 02 54 ...p~...^~... .T > +00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 0a 60 9f 1f 97 f2 ...p~...^~.`.... > > -00004cb0: 1e 4c c1 71 7d 0b 91 28 23 98 09 ae c4 95 ae 7f .L.q}..(#....... > +00004cb0: 6e 17 50 03 67 fa 2f 83 b0 88 eb fc 54 f2 0b 00 n.P.g./.....T... > > -00004cc0: a2 92 20 b8 f2 b6 72 2a e8 7e d7 27 99 65 56 70 .. ...r*.~.'.eVp > +00004cc0: 6c 9e a1 02 b4 30 11 25 d7 58 b0 0c c0 6c e1 bd l....0.%.X...l.. > > It never appears to get back into sync after that. Interestingly, it is > _consistently_ wrong; if I download the same fragment multiple times, it > breaks at the same place and gives the same garbage (but different fragments > give different divergence points). Tested with both wget and Chrome. > Does anyone know what could be wrong? > > (It is, unfortunately, not easy for me to reboot this server at will, so a > bisect could be hard.) > > Please Cc me on any replies, I'm not subscribed to netdev. > > /* Steinar */ > Reproduced and bisected, the problem showed up in v4.20-rc1. Unfortunately the commit seems to have some significant dependencies so I was unable to verify by reverting it on 4.20. Adding John and Daniel. d3b18ad31f93d0b6bae105c679018a1ba7daa9ca is the first bad commit commit d3b18ad31f93d0b6bae105c679018a1ba7daa9ca Author: John Fastabend Date: Sat Oct 13 02:46:01 2018 +0200 tls: add bpf support to sk_msg handling This work adds BPF sk_msg verdict program support to kTLS allowing BPF and kTLS to be combined together. Previously kTLS and sk_msg verdict programs were mutually exclusive in the ULP layer which created challenges for the orchestrator when trying to apply TCP based policy, for example. To resolve this, leveraging the work from previous patches that consolidates the use of sk_msg, we can finally enable BPF sk_msg verdict programs so they continue to run after the kTLS socket is created. No change in behavior when kTLS is not used in combination with BPF, the kselftest suite for kTLS also runs successfully. Joint work with Daniel. Signed-off-by: John Fastabend Signed-off-by: Daniel Borkmann Signed-off-by: Alexei Starovoitov :040000 040000 107a1f08fa7d54610292047bbd360a6bf9fff78a 1ef18a0495f094c6d771a371f6c05f849daff512 M include :040000 040000 56e6ebd0c6dc0a5aa8d371332cae6bff6cdcc1ff 44ddc12947dce93b449e468ed6862d475c33f32b M net $ git bisect log git bisect start '--' 'net' 'include/net' # good: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19 git bisect good 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d # bad: [8fe28cb58bcb235034b64cbbb7550a8a43fd88be] Linux 4.20 git bisect bad 8fe28cb58bcb235034b64cbbb7550a8a43fd88be # bad: [aa563d7bca6e882ec2bdae24603c8f016401a144] iov_iter: Separate type from direction and use accessor functions git bisect bad aa563d7bca6e882ec2bdae24603c8f016401a144 # good: [85dd3da43dd59b9220d9cba4f933a3dc0ea6faa5] cfg80211: combine wdev/netdev unregister code git bisect good 85dd3da43dd59b9220d9cba4f933a3dc0ea6faa5 # good: [9000a457a0c84883874a844ef94adf26f633f3b4] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next git bisect good 9000a457a0c84883874a844ef94adf26f633f3b4 # good: [1a3aea2534f4f3083f29b2b047aa83a9d6c777a4] net: bridge: fix a possible memory leak in __vlan_add git bisect good 1a3aea2534f4f3083f29b2b047aa83a9d6c777a4 # bad: [cb167893f41e21e6bd283d78e53489289dc0592d] net: Plumb support for filtering ipv4 and ipv6 multicast route dumps git bisect bad cb167893f41e21e6bd283d78e53489289dc0592d # bad: [5ef0ae84f02a4dbe0e09f89c6481ac13649cb19b] bpf: Fix IPv6 dport byte-order in bpf_sk_lookup git bisect bad 5ef0ae84f02a4dbe0e09f89c6481ac13649cb19b # good: [604326b41a6fb9b4a78b6179335decee0365cd8c] bpf, sockmap: convert to generic sk_msg interface git bisect good 604326b41a6fb9b4a78b6179335decee0365cd8c # good: [924ad65ed01ee0eec5d2a3280c01c394343d6df7] tls: replace poll implementation with read hook git bisect good 924ad65ed01ee0eec5d2a3280c01c394343d6df7 # bad: [8a615c6b0352a9ec56151b6c95d68e0a2eef5cf0] bpf: Allow sk_lookup with IPv6 module git bisect bad 8a615c6b0352a9ec56151b6c95d68e0a2eef5cf0 # bad: [d3b18ad31f93d0b6bae105c679018a1ba7daa9ca] tls: add bpf support to sk_msg handling git bisect bad d3b18ad31f93d0b6bae105c679018a1ba7daa9ca # first bad commit: [d3b18ad31f93d0b6bae105c679018a1ba7daa9ca] tls: add bpf support to sk_msg handling