From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24B17C001DE for ; Wed, 26 Jul 2023 15:21:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C72C161283; Wed, 26 Jul 2023 15:21:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org C72C161283 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1690384904; bh=LkN1Mo+KeOTOJ71XmuOcOIuPcyikLGJw60Bq1QiTgvc=; h=Date:From:To:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=AN8V8MowqPfjxE8N5w0GQbWFm5ISOVXeJ0RG1kSzSXvcuTW3wXs1ypQgJs3INe5gU fHYMYpV4rilkuuYB2EEi7yY/88xpe6OCtuLe3ZSuIJU7BlGKYY3mmO2PpHwUVLQ2es FfC48uLkpoluxDq22Z4XnWwYiU/5t2mI9qUCsV0h5ehLS0GDa6NmhicIp+AGYoloDA AaQrwJkEDQdI8H4M/XM9k5p3l6Pf+gRdi20+e1ZN2V5XJdtZIbESVOV1xVi9IPtUsW CzOTp8ixKoxMD5J1vSvLX2hetGzgNGWq9BSD267TwNb9088QaUdWKtwS3fjTEaW0iz 5Agu2pmIiHXtA== X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P9jTSAtAZMrY; Wed, 26 Jul 2023 15:21:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id B3AEF6126A; Wed, 26 Jul 2023 15:21:43 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B3AEF6126A Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id AFF1C1BF5A2 for ; Tue, 25 Jul 2023 23:50:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 93F246105B for ; Tue, 25 Jul 2023 23:50:59 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 93F246105B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4aXnkeGVfnZV for ; Tue, 25 Jul 2023 23:50:58 +0000 (UTC) Received: from mail-oo1-xc2d.google.com (mail-oo1-xc2d.google.com [IPv6:2607:f8b0:4864:20::c2d]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6061B60B7C for ; Tue, 25 Jul 2023 23:50:58 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 6061B60B7C Received: by mail-oo1-xc2d.google.com with SMTP id 006d021491bc7-563531a3ad2so3748168eaf.3 for ; Tue, 25 Jul 2023 16:50:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=cGUgXHyEHM5RssvL8yDlvuaaVQfdF0c17N3qQe2jOR4EXh2Z6+0ZCnl4EU8+KJkvXg Rs/359HYWwWNQ4b39qja++AcpmUOJyKYrA+45YPPERDNk+205q63wKnlghMjtf7zPIhD 8d6pl2TZHGi2+pdyiOiznqIrL2HBGXveTno9aVoWSf12E39Jrg2Ik1wuYSrQQdIHnrj9 uYCowaetFz7WE2Ez9OxanN25+HPSJvIy6mlUxSjkvX3I0Ibjv6w4cmWbvUXl34XicvYD czPF3qIB8V/4JcHOk2KTdTOb1v8GtVBwG5tR8+xFJzNF+zq8d86Oq1pPuPkwYY/9XpCL vGkw== X-Gm-Message-State: ABy/qLY4nPEXf4ja8WuQFy687lU0w2n6y3mmyaChZB0H/6asWc6bBINZ Y85Q3BmMsM2l8kWu0UPqtGc= X-Google-Smtp-Source: APBJJlHalXYmOhaEU6n05SNM4i/kOD7ydZDReFjh8w9A4vRIkL/8696/tstPkBJZxOJSkhZD6kYQEg== X-Received: by 2002:a05:6808:1588:b0:3a4:232c:5d7e with SMTP id t8-20020a056808158800b003a4232c5d7emr476635oiw.5.1690329056959; Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: from debian.me ([103.131.18.64]) by smtp.gmail.com with ESMTPSA id rj14-20020a17090b3e8e00b00267fe43f518sm110915pjb.23.2023.07.25.16.50.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: by debian.me (Postfix, from userid 1000) id AD3F981944A1; Wed, 26 Jul 2023 06:50:53 +0700 (WIB) Date: Wed, 26 Jul 2023 06:50:52 +0700 From: Bagas Sanjaya To: Andrzej Kacprowski , Krystian Pradzynski , Stanislaw Gruszka , Jacek Lawrynowicz , Oded Gabbay , Jesse Brandeburg , Tony Nguyen , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , "Linux regression tracking (Thorsten Leemhuis)" , hq.dev+kernel@msdfc.xyz, Linus Torvalds Message-ID: References: MIME-Version: 1.0 In-Reply-To: X-Mailman-Approved-At: Wed, 26 Jul 2023 15:21:41 +0000 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=NcnLZAsl/ELgMGRZHqKtVn9uEWSI7mXT9tyTc3vBN6zCSfgBl5nNIoHcUbeHj/1bb1 7TVWKeBXM4LVNKY3H4EEIOjYge9df3vlFwHI5CNoqM7xcvxhsu41x9CyjR/s9+wixj5Q OiUwuFf/2P+3gbuIAhnbc0y6qkVIo/7TPUHbRhhemeQF4w15VIgeziXSgkZuBvGvtxky 75O3kuuW/8j7jeeC/gZJihv/Rj6o7OCKwuC6zsTWPp+SuTkt+qSvgZPDcM7pNgSYv5Dl ukVrDRTuD6SWugwpqvBKVt9kQVcW7K0gpLRcQzBha0H4iC5ImojFMbgRPMdTRzsIYpid 42jA== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=NcnLZAsl Subject: Re: [Intel-wired-lan] Fwd: Unexplainable packet drop starting at v6.4 X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Linux Networking , Linux Intel Ethernet Drivers , Linux Kernel Mailing List , Linux DRI Development , Linux Regressions Content-Type: multipart/mixed; boundary="===============0149142977627090154==" Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" --===============0149142977627090154== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OUAsi7azhbgmAo1h" Content-Disposition: inline --OUAsi7azhbgmAo1h Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 18, 2023 at 07:51:24AM +0700, Bagas Sanjaya wrote: > Hi, >=20 > I notice a regression report on Bugzilla [1]. Quoting from it: >=20 > > Hi, > >=20 > > After I updated to 6.4 through Archlinux kernel update, suddenly I noti= ced random packet losses on my routers like nodes. I have these networking = relevant config on my nodes > >=20 > > 1. Using archlinux > > 2. Network config through systemd-networkd > > 3. Using bird2 for BGP routing, but not relevant to this bug. > > 4. Using nftables for traffic control, but seems not relevant to this b= ug.=20 > > 5. Not using fail2ban like dymanic filtering tools, at least at L3/L4 l= evel > >=20 > > After I ruled out systemd-networkd, nftables related issues. I tracked = down issues to kernel. > >=20 > > Here's the tcpdump I'm seeing on one side of my node "" > >=20 > > ``` > > sudo tcpdump -i fios_wan port 38851 > > tcpdump: verbose output suppressed, use -v[v]... for full protocol deco= de > > listening on fios_wan, link-type EN10MB (Ethernet), snapshot length 262= 144 bytes > > 10:33:06.073236 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:11.406607 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:16.739969 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:21.859856 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:27.193176 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 5 packets captured > > 5 packets received by filter > > 0 packets dropped by kernel > > ``` > >=20 > > But on the other side "[REDACTED_PUBLIC_IPv4_1]", tcpdump is replying p= ackets in this wireguard stream. So packet is lost somewhere in the link. > >=20 > > From the otherside, I can do "mtr" to "[BOS1_NODE]"'s public IP and fou= nd the moment the link got lost is right at "[BOS1_NODE]", that means "[BOS= 1_NODE]"'s networking stack completely drop the inbound packets from specif= ic ip addresses. > >=20 > > Some more digging > >=20 > > 1. This situation began after booting in different delays. Sometimes ca= n trigger after 30 seconds after booting, and sometimes will be after 18 ho= urs or more. > > 2. It can envolve into worse case that when I do "ip neigh show", the i= pv4 ARP table and ipv6 neighbor discovery start to appear as "invalid", mea= ning the internet is completely loss. > > 3. When this happened to wan facing interface, it seems OK with lan fac= ing interfaces. WAN interface was using Intel X710-T4L using i40e and lan s= ide was using virtio > > 4. I tried to bisect in between 6.3 and 6.4, and the first bad commit i= t reports was "a3efabee5878b8d7b1863debb78cb7129d07a346". But this is not r= elevant to networking at all, maybe it's the wrong commit to look at. At th= e meantime, because I haven't found a reproducible way of 100% trigger the = issue, it may be the case during bisect some "good" commits are actually ba= d.=20 > > 5. I also tried to look at "dmesg", nothing interesting pop up. But I'l= l make it available upon request. > >=20 > > This is my first bug reports. Sorry for any confusion it may lead to an= d thanks for reading. >=20 > See Bugzilla for the full thread. >=20 > Thorsten: The reporter had a bad bisect (some bad commits were marked as = good > instead), hence SoB chain for culprit (unrelated) ipvu commit is in To: > list. I also asked the reporter (also in To:) to provide dmesg and request > rerunning bisection, but he doesn't currently have a reliable reproducer. > Is it the best I can do? >=20 > Anyway, I'm adding this regression to be tracked in regzbot: >=20 > #regzbot introduced: a3efabee5878b8 https://bugzilla.kernel.org/show_bug.= cgi?id=3D217678 > #regzbot title: packet drop on Intel X710-T4L due to ipvu boot fix >=20 This time, the bisection points out to v6.4 networking pull, so: #regzbot introduced: 6e98b09da931a0 (also Cc: Linus.) Thanks. --=20 An old man doll... just what I always wanted! - Clara --OUAsi7azhbgmAo1h Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZMBf0wAKCRD2uYlJVVFO o+/YAP0Z6eCcYl71Y1kT2UYGDBIwMXXiM7+aR40lhmu0mcdmbAEA9m/ui3/uZX51 DmktMr6iQDC9/1h00DKNiilDimu++go= =+BBU -----END PGP SIGNATURE----- --OUAsi7azhbgmAo1h-- --===============0149142977627090154== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan@osuosl.org https://lists.osuosl.org/mailman/listinfo/intel-wired-lan --===============0149142977627090154==-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5669426B68 for ; Tue, 25 Jul 2023 23:50:58 +0000 (UTC) Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-56597d949b1so3757172eaf.1 for ; Tue, 25 Jul 2023 16:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=NcnLZAsl/ELgMGRZHqKtVn9uEWSI7mXT9tyTc3vBN6zCSfgBl5nNIoHcUbeHj/1bb1 7TVWKeBXM4LVNKY3H4EEIOjYge9df3vlFwHI5CNoqM7xcvxhsu41x9CyjR/s9+wixj5Q OiUwuFf/2P+3gbuIAhnbc0y6qkVIo/7TPUHbRhhemeQF4w15VIgeziXSgkZuBvGvtxky 75O3kuuW/8j7jeeC/gZJihv/Rj6o7OCKwuC6zsTWPp+SuTkt+qSvgZPDcM7pNgSYv5Dl ukVrDRTuD6SWugwpqvBKVt9kQVcW7K0gpLRcQzBha0H4iC5ImojFMbgRPMdTRzsIYpid 42jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=Foolbt8PZ2rmGa+hMri+jH7XvAunNWHBGuQHk5AmtmKiV9L2PQoWI8lV5e0z8NYV+5 VOI2a7LDNyFOXtwg5riI2XZ5iq2UQ/regVf2GG7g4G+qT/5F62ynIw7nQpFoAaLlN1ib eddqoHjkXOCjim+GakhOZ6U8IS33RasBwzqoczRHK+Gf2KOcGSadWjEmd2zk6e6gTYKs LUInGlPpGKgj9igSBeJAPowi0qxaduyB95RuINN6OUL6LBamp1gEUyHfLUh//Rg2Lduy Kk6iN2X9e5JwPLzSdTc55VwUefqZrOm/U5aD7xZBsoYBmB6ObyyMRalDWOKNp3M/f9D4 cVBg== X-Gm-Message-State: ABy/qLb9NZLwji+m+m2O8PLPvsJJXJ5kb+f5tpybIE9fejXwJ6JgJbRJ A+zyCz77vPdF+9p71HnjvQE= X-Google-Smtp-Source: APBJJlHalXYmOhaEU6n05SNM4i/kOD7ydZDReFjh8w9A4vRIkL/8696/tstPkBJZxOJSkhZD6kYQEg== X-Received: by 2002:a05:6808:1588:b0:3a4:232c:5d7e with SMTP id t8-20020a056808158800b003a4232c5d7emr476635oiw.5.1690329056959; Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: from debian.me ([103.131.18.64]) by smtp.gmail.com with ESMTPSA id rj14-20020a17090b3e8e00b00267fe43f518sm110915pjb.23.2023.07.25.16.50.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: by debian.me (Postfix, from userid 1000) id AD3F981944A1; Wed, 26 Jul 2023 06:50:53 +0700 (WIB) Date: Wed, 26 Jul 2023 06:50:52 +0700 From: Bagas Sanjaya To: Andrzej Kacprowski , Krystian Pradzynski , Stanislaw Gruszka , Jacek Lawrynowicz , Oded Gabbay , Jesse Brandeburg , Tony Nguyen , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , "Linux regression tracking (Thorsten Leemhuis)" , hq.dev+kernel@msdfc.xyz, Linus Torvalds Cc: Linux Kernel Mailing List , Linux Regressions , Linux DRI Development , Linux Networking , Linux Intel Ethernet Drivers Subject: Re: Fwd: Unexplainable packet drop starting at v6.4 Message-ID: References: Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OUAsi7azhbgmAo1h" Content-Disposition: inline In-Reply-To: --OUAsi7azhbgmAo1h Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 18, 2023 at 07:51:24AM +0700, Bagas Sanjaya wrote: > Hi, >=20 > I notice a regression report on Bugzilla [1]. Quoting from it: >=20 > > Hi, > >=20 > > After I updated to 6.4 through Archlinux kernel update, suddenly I noti= ced random packet losses on my routers like nodes. I have these networking = relevant config on my nodes > >=20 > > 1. Using archlinux > > 2. Network config through systemd-networkd > > 3. Using bird2 for BGP routing, but not relevant to this bug. > > 4. Using nftables for traffic control, but seems not relevant to this b= ug.=20 > > 5. Not using fail2ban like dymanic filtering tools, at least at L3/L4 l= evel > >=20 > > After I ruled out systemd-networkd, nftables related issues. I tracked = down issues to kernel. > >=20 > > Here's the tcpdump I'm seeing on one side of my node "" > >=20 > > ``` > > sudo tcpdump -i fios_wan port 38851 > > tcpdump: verbose output suppressed, use -v[v]... for full protocol deco= de > > listening on fios_wan, link-type EN10MB (Ethernet), snapshot length 262= 144 bytes > > 10:33:06.073236 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:11.406607 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:16.739969 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:21.859856 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:27.193176 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 5 packets captured > > 5 packets received by filter > > 0 packets dropped by kernel > > ``` > >=20 > > But on the other side "[REDACTED_PUBLIC_IPv4_1]", tcpdump is replying p= ackets in this wireguard stream. So packet is lost somewhere in the link. > >=20 > > From the otherside, I can do "mtr" to "[BOS1_NODE]"'s public IP and fou= nd the moment the link got lost is right at "[BOS1_NODE]", that means "[BOS= 1_NODE]"'s networking stack completely drop the inbound packets from specif= ic ip addresses. > >=20 > > Some more digging > >=20 > > 1. This situation began after booting in different delays. Sometimes ca= n trigger after 30 seconds after booting, and sometimes will be after 18 ho= urs or more. > > 2. It can envolve into worse case that when I do "ip neigh show", the i= pv4 ARP table and ipv6 neighbor discovery start to appear as "invalid", mea= ning the internet is completely loss. > > 3. When this happened to wan facing interface, it seems OK with lan fac= ing interfaces. WAN interface was using Intel X710-T4L using i40e and lan s= ide was using virtio > > 4. I tried to bisect in between 6.3 and 6.4, and the first bad commit i= t reports was "a3efabee5878b8d7b1863debb78cb7129d07a346". But this is not r= elevant to networking at all, maybe it's the wrong commit to look at. At th= e meantime, because I haven't found a reproducible way of 100% trigger the = issue, it may be the case during bisect some "good" commits are actually ba= d.=20 > > 5. I also tried to look at "dmesg", nothing interesting pop up. But I'l= l make it available upon request. > >=20 > > This is my first bug reports. Sorry for any confusion it may lead to an= d thanks for reading. >=20 > See Bugzilla for the full thread. >=20 > Thorsten: The reporter had a bad bisect (some bad commits were marked as = good > instead), hence SoB chain for culprit (unrelated) ipvu commit is in To: > list. I also asked the reporter (also in To:) to provide dmesg and request > rerunning bisection, but he doesn't currently have a reliable reproducer. > Is it the best I can do? >=20 > Anyway, I'm adding this regression to be tracked in regzbot: >=20 > #regzbot introduced: a3efabee5878b8 https://bugzilla.kernel.org/show_bug.= cgi?id=3D217678 > #regzbot title: packet drop on Intel X710-T4L due to ipvu boot fix >=20 This time, the bisection points out to v6.4 networking pull, so: #regzbot introduced: 6e98b09da931a0 (also Cc: Linus.) Thanks. --=20 An old man doll... just what I always wanted! - Clara --OUAsi7azhbgmAo1h Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZMBf0wAKCRD2uYlJVVFO o+/YAP0Z6eCcYl71Y1kT2UYGDBIwMXXiM7+aR40lhmu0mcdmbAEA9m/ui3/uZX51 DmktMr6iQDC9/1h00DKNiilDimu++go= =+BBU -----END PGP SIGNATURE----- --OUAsi7azhbgmAo1h-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1A57EB64DD for ; Tue, 25 Jul 2023 23:51:00 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9E18C10E1B9; Tue, 25 Jul 2023 23:50:59 +0000 (UTC) Received: from mail-oi1-x22e.google.com (mail-oi1-x22e.google.com [IPv6:2607:f8b0:4864:20::22e]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1969010E1B9 for ; Tue, 25 Jul 2023 23:50:58 +0000 (UTC) Received: by mail-oi1-x22e.google.com with SMTP id 5614622812f47-3a3fbfb616dso3654387b6e.3 for ; Tue, 25 Jul 2023 16:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=NcnLZAsl/ELgMGRZHqKtVn9uEWSI7mXT9tyTc3vBN6zCSfgBl5nNIoHcUbeHj/1bb1 7TVWKeBXM4LVNKY3H4EEIOjYge9df3vlFwHI5CNoqM7xcvxhsu41x9CyjR/s9+wixj5Q OiUwuFf/2P+3gbuIAhnbc0y6qkVIo/7TPUHbRhhemeQF4w15VIgeziXSgkZuBvGvtxky 75O3kuuW/8j7jeeC/gZJihv/Rj6o7OCKwuC6zsTWPp+SuTkt+qSvgZPDcM7pNgSYv5Dl ukVrDRTuD6SWugwpqvBKVt9kQVcW7K0gpLRcQzBha0H4iC5ImojFMbgRPMdTRzsIYpid 42jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690329057; x=1690933857; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MntZLPg9Q2TkUipdw5URIw2GbDHNmtTpFo/G9AqaGl8=; b=AsIMNifx9gSAsyeMz62OnI3/ru287V0IHYBo6sYFGvFSdAfL108OM0KGnvz8SyKMRd gVtUUAIICMl4V0f36JZ3ibI3Fsxsv1yBBv3NBOwFMs60jvdk/Wdnj5fJEPwIOa2JtqDE Bo2raQ1HMsPE+NV8zG2g8jh0kz5xCHqbsDsODlsqwJTNjlligf6b5CTDMjf4mD1YhgxX oxJaM24SeHO8mWCcFX8K+5P/pehfl5136WwSg1n+kl4JynKK1arTEqJWrpyxcGj/V8uJ Zs2GaaUfgbBOf99ORxeRGr9EUrUe4FJbENv0nLA9X3bV9k6t2NZb/sc3JPo3546BiUbp stHg== X-Gm-Message-State: ABy/qLZkOs+6+DIuubZmfsi1wsm/ITuB+g/3Al7iEM7ItRCraSN2C1Ls P9FRDi3VdcNuAe6+9SG69Yg= X-Google-Smtp-Source: APBJJlHalXYmOhaEU6n05SNM4i/kOD7ydZDReFjh8w9A4vRIkL/8696/tstPkBJZxOJSkhZD6kYQEg== X-Received: by 2002:a05:6808:1588:b0:3a4:232c:5d7e with SMTP id t8-20020a056808158800b003a4232c5d7emr476635oiw.5.1690329056959; Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: from debian.me ([103.131.18.64]) by smtp.gmail.com with ESMTPSA id rj14-20020a17090b3e8e00b00267fe43f518sm110915pjb.23.2023.07.25.16.50.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jul 2023 16:50:56 -0700 (PDT) Received: by debian.me (Postfix, from userid 1000) id AD3F981944A1; Wed, 26 Jul 2023 06:50:53 +0700 (WIB) Date: Wed, 26 Jul 2023 06:50:52 +0700 From: Bagas Sanjaya To: Andrzej Kacprowski , Krystian Pradzynski , Stanislaw Gruszka , Jacek Lawrynowicz , Oded Gabbay , Jesse Brandeburg , Tony Nguyen , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , "Linux regression tracking (Thorsten Leemhuis)" , hq.dev+kernel@msdfc.xyz, Linus Torvalds Subject: Re: Fwd: Unexplainable packet drop starting at v6.4 Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="OUAsi7azhbgmAo1h" Content-Disposition: inline In-Reply-To: X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Linux Networking , Linux Intel Ethernet Drivers , Linux Kernel Mailing List , Linux DRI Development , Linux Regressions Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" --OUAsi7azhbgmAo1h Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 18, 2023 at 07:51:24AM +0700, Bagas Sanjaya wrote: > Hi, >=20 > I notice a regression report on Bugzilla [1]. Quoting from it: >=20 > > Hi, > >=20 > > After I updated to 6.4 through Archlinux kernel update, suddenly I noti= ced random packet losses on my routers like nodes. I have these networking = relevant config on my nodes > >=20 > > 1. Using archlinux > > 2. Network config through systemd-networkd > > 3. Using bird2 for BGP routing, but not relevant to this bug. > > 4. Using nftables for traffic control, but seems not relevant to this b= ug.=20 > > 5. Not using fail2ban like dymanic filtering tools, at least at L3/L4 l= evel > >=20 > > After I ruled out systemd-networkd, nftables related issues. I tracked = down issues to kernel. > >=20 > > Here's the tcpdump I'm seeing on one side of my node "" > >=20 > > ``` > > sudo tcpdump -i fios_wan port 38851 > > tcpdump: verbose output suppressed, use -v[v]... for full protocol deco= de > > listening on fios_wan, link-type EN10MB (Ethernet), snapshot length 262= 144 bytes > > 10:33:06.073236 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:11.406607 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:16.739969 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:21.859856 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 10:33:27.193176 IP [BOS1_NODE].38851 > [REDACTED_PUBLIC_IPv4_1].38851: = UDP, length 148 > > 5 packets captured > > 5 packets received by filter > > 0 packets dropped by kernel > > ``` > >=20 > > But on the other side "[REDACTED_PUBLIC_IPv4_1]", tcpdump is replying p= ackets in this wireguard stream. So packet is lost somewhere in the link. > >=20 > > From the otherside, I can do "mtr" to "[BOS1_NODE]"'s public IP and fou= nd the moment the link got lost is right at "[BOS1_NODE]", that means "[BOS= 1_NODE]"'s networking stack completely drop the inbound packets from specif= ic ip addresses. > >=20 > > Some more digging > >=20 > > 1. This situation began after booting in different delays. Sometimes ca= n trigger after 30 seconds after booting, and sometimes will be after 18 ho= urs or more. > > 2. It can envolve into worse case that when I do "ip neigh show", the i= pv4 ARP table and ipv6 neighbor discovery start to appear as "invalid", mea= ning the internet is completely loss. > > 3. When this happened to wan facing interface, it seems OK with lan fac= ing interfaces. WAN interface was using Intel X710-T4L using i40e and lan s= ide was using virtio > > 4. I tried to bisect in between 6.3 and 6.4, and the first bad commit i= t reports was "a3efabee5878b8d7b1863debb78cb7129d07a346". But this is not r= elevant to networking at all, maybe it's the wrong commit to look at. At th= e meantime, because I haven't found a reproducible way of 100% trigger the = issue, it may be the case during bisect some "good" commits are actually ba= d.=20 > > 5. I also tried to look at "dmesg", nothing interesting pop up. But I'l= l make it available upon request. > >=20 > > This is my first bug reports. Sorry for any confusion it may lead to an= d thanks for reading. >=20 > See Bugzilla for the full thread. >=20 > Thorsten: The reporter had a bad bisect (some bad commits were marked as = good > instead), hence SoB chain for culprit (unrelated) ipvu commit is in To: > list. I also asked the reporter (also in To:) to provide dmesg and request > rerunning bisection, but he doesn't currently have a reliable reproducer. > Is it the best I can do? >=20 > Anyway, I'm adding this regression to be tracked in regzbot: >=20 > #regzbot introduced: a3efabee5878b8 https://bugzilla.kernel.org/show_bug.= cgi?id=3D217678 > #regzbot title: packet drop on Intel X710-T4L due to ipvu boot fix >=20 This time, the bisection points out to v6.4 networking pull, so: #regzbot introduced: 6e98b09da931a0 (also Cc: Linus.) Thanks. --=20 An old man doll... just what I always wanted! - Clara --OUAsi7azhbgmAo1h Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZMBf0wAKCRD2uYlJVVFO o+/YAP0Z6eCcYl71Y1kT2UYGDBIwMXXiM7+aR40lhmu0mcdmbAEA9m/ui3/uZX51 DmktMr6iQDC9/1h00DKNiilDimu++go= =+BBU -----END PGP SIGNATURE----- --OUAsi7azhbgmAo1h--