From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andre Tomt Subject: Re: Fw: [Bug 201423] New: eth0: hw csum failure Date: Sun, 4 Nov 2018 06:43:03 +0100 Message-ID: References: <20181015081519.0bf076bc@xeon-e3> <4693819f-4a76-532f-9b24-d4328183c807@gmail.com> <91545596-f932-8834-f613-feda3edc9b84@tomt.net> <869fbb53-a0a5-95f9-2c77-c3ae3f6d181f@tomt.net> <6bf2dc00-ce6e-99a8-202e-1cca4469da67@tomt.net> <973f5f0e-fc25-7a99-70b5-3b53f7c69fca@tomt.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Stephen Hemminger , netdev , rossi.f@inwind.it, Dimitris Michailidis To: Eric Dumazet , Eric Dumazet Return-path: Received: from mail1.ugh.no ([178.79.162.34]:56494 "EHLO mail1.ugh.no" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727034AbeKDO4u (ORCPT ); Sun, 4 Nov 2018 09:56:50 -0500 In-Reply-To: <973f5f0e-fc25-7a99-70b5-3b53f7c69fca@tomt.net> Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 31.10.2018 05:08, Andre Tomt wrote: > On 30.10.2018 12:04, Andre Tomt wrote: >> On 30.10.2018 11:58, Andre Tomt wrote: >>> On 27.10.2018 23:41, Andre Tomt wrote: >>>> On 26.10.2018 13:45, Andre Tomt wrote: >>>>> On 25.10.2018 19:38, Eric Dumazet wrote: >>>>>> >>>>>> >>>>>> On 10/24/2018 12:41 PM, Andre Tomt wrote: >>>>>>> >>>>>>> It eventually showed up again with mlx4, on 4.18.16 + fix and >>>>>>> also on 4.19. I still do not have a useful packet capture. >>>>>>> >>>>>>> It is running a torrent client serving up various linux >>>>>>> distributions. >>>>>>> >>>>>> >>>>>> Have you also applied this fix ? >>>>>> >>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913 >>>>>> >>>>>> >>>>> >>>>> No. I've applied it now to 4.19 and will report back if anything >>>>> shows up. >>>> >>>> Just hit it on the simpler server; no VRF, no tunnels, no >>>> nat/conntrack. Only a basic stateless nftables ruleset and a vlan >>>> netdev (unlikely to be the one triggering this I guess; it has only >>>> v4 traffic). >>> >>> I'm currently testing 4.19 with the recomended commit added, plus >>> these to sort out some GRO issues (on a hunch, unsure if related): >>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218 >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7 >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894 >>> >>> >>> and I *think* it is behaving better now? it's not conclusive as it >>> could take a while to trip in this environment but some of the test >>> servers have not shown anything bad in almost 24h. >> >> Sorry, s/some of the/none of the > > I think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At > least for my workload. Servers are now 51-61 hours in, no splats. I also > added ntp pool traffic to one of them to make things a little more > exciting. > > Not sure what is needed for 4.18, I dont have the mental bandwidth to > test that right now. Also no idea about the similar looking mlx5 splats > reported elsewhere. As expected conntrack/nat + vlan + forwarding still splats. sch_cake, IFB and VRF was removed from this setup. Here is a conntrack splat without IFB/VRF/Cake inteference: > [34458.506346] wanib: hw csum failure > [34458.506371] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-1 #1 > [34458.506374] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 2.0 06/13/2018 > [34458.506377] Call Trace: > [34458.506381] > [34458.506388] dump_stack+0x5c/0x80 > [34458.506392] __skb_checksum_complete+0xac/0xc0 > [34458.506402] icmp_error+0x1c8/0x1f0 [nf_conntrack] > [34458.506406] ? skb_copy_bits+0x13d/0x220 > [34458.506411] nf_conntrack_in+0xd8/0x390 [nf_conntrack] > [34458.506416] ? ___pskb_trim+0x192/0x330 > [34458.506421] nf_hook_slow+0x43/0xc0 > [34458.506426] ip_rcv+0x90/0xb0 > [34458.506430] ? ip_rcv_finish_core.isra.0+0x310/0x310 > [34458.506435] __netif_receive_skb_one_core+0x42/0x50 > [34458.506438] netif_receive_skb_internal+0x24/0xb0 > [34458.506441] napi_gro_frags+0x177/0x210 > [34458.506446] mlx4_en_process_rx_cq+0x8df/0xb50 [mlx4_en] > [34458.506459] ? mlx4_eq_int+0x38f/0xcb0 [mlx4_core] > [34458.506463] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en] > [34458.506466] net_rx_action+0xe1/0x2c0 > [34458.506469] __do_softirq+0xe7/0x2d3 > [34458.506475] irq_exit+0x96/0xd0 > [34458.506478] do_IRQ+0x85/0xd0 > [34458.506483] common_interrupt+0xf/0xf > [34458.506486] > [34458.506491] RIP: 0010:cpuidle_enter_state+0xb9/0x320 > [34458.506495] Code: e8 3c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3b 02 00 00 31 ff e8 5e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3 > [34458.506497] RSP: 0018:ffff978d41943ea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb > [34458.506500] RAX: ffff8d8f6fa60fc0 RBX: 00001f56ff07af28 RCX: 000000000000001f > [34458.506501] RDX: 00001f56ff07af28 RSI: 000000003a2e90d6 RDI: 0000000000000000 > [34458.506503] RBP: ffff8d8f6fa698c0 R08: 0000000000000002 R09: 0000000000020840 > [34458.506504] R10: 0004ea58f2899595 R11: ffff8d8f6fa601e8 R12: 0000000000000001 > [34458.506505] R13: ffffffff8a0ac638 R14: 0000000000000001 R15: 0000000000000000 > [34458.506509] ? cpuidle_enter_state+0x94/0x320 > [34458.506512] do_idle+0x1e4/0x220 > [34458.506515] cpu_startup_entry+0x5f/0x70 > [34458.506519] start_secondary+0x185/0x1a0 > [34458.506521] secondary_startup_64+0xa4/0xb0 Stateless filtered non-forwarding host still looks like it has been fixed (the udp6_gro_* splats are still all gone). Also seems fine when moving the traffic over a vlan device. These fixes went into 4.19.1-rc1 (checksum_complete + unlink gro packets on overflow fixes)