WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Nelson, Shannon]

Using a slightly modified version of udpspam (see diff below - hopefully not mangled by corporate email servers), where I set the SO_NO_CHECK socket option and can specify a large buffer size, I can reliably get the following WARN trace.  I have reproduced this on both ixgbe and i40e drivers using "udpspam-no-check <target-ip> 6000".

It looks to me like this is in the Tx path before we get to the actual NIC drivers, but I may be wrong.


[ 1757.644324] ------------[ cut here ]------------
[ 1757.644333] WARNING: CPU: 22 PID: 5537 at net/core/dev.c:2423 skb_warn_bad_offload+0x104/0x111()
[ 1757.644340] ixgbe: caps=(0x0000080660314bb3, 0x0000000000000000) len=6092 data_len=6000 gso_size=1528 gso_type=1026 ip_summed=0
[ 1757.644343] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink vxlan ip6_udp_tunnel udp_tunnel rfcomm xt_CHECKSUM bnep bluetooth rfkill tun fuse ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_mangle iptable_security iptable_raw x86_pkg_temp_thermal coretemp kvm_intel kvm joydev iTCO_wdt ipmi_devintf iTCO_vendor_support irqbypass crct10dif_pclmul ixgbe igb crc32_pclmul ipmi_si ptp pps_core sb_edac lpc_ich crc32c_intel pcspkr edac_core i2c_i801 mfd_core mdio ipmi_msghandler mei_me shpchp tpm_tis ioatdma mei dca wmi tpm binfmt_misc uinput mgag200 i2c_algo_bit drm_kms_helper
[ 1757.644443]  ttm drm isci libsas firewire_ohci firewire_core i2c_core scsi_transport_sas crc_itu_t
[ 1757.644475] CPU: 22 PID: 5537 Comm: udpspam-no-chec Tainted: G        W       4.4.0-rc3+ #1
[ 1757.644480] Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.03.0003.041920141333 04/19/2014
[ 1757.644482]  0000000000000000 00000000c92b03e4 ffff88081907b410 ffffffff8138f918
[ 1757.644488]  ffff88081907b458 ffff88081907b448 ffffffff8109c036 ffff8808196e9500
[ 1757.644494]  ffff880816f60000 0000000000000402 0000000000000000 ffff88080e8eb2ac
[ 1757.644499] Call Trace:
[ 1757.644509]  [<ffffffff8138f918>] dump_stack+0x44/0x5c
[ 1757.644514]  [<ffffffff8109c036>] warn_slowpath_common+0x86/0xc0
[ 1757.644518]  [<ffffffff8109c0cc>] warn_slowpath_fmt+0x5c/0x80
[ 1757.644523]  [<ffffffff813958cc>] ? ___ratelimit+0x8c/0xf0
[ 1757.644539]  [<ffffffff8174c77c>] skb_warn_bad_offload+0x104/0x111
[ 1757.644549]  [<ffffffff81649d1f>] __skb_gso_segment+0x7f/0xd0
[ 1757.644563]  [<ffffffff8164a08f>] validate_xmit_skb.isra.104.part.105+0x11f/0x2a0
[ 1757.644572]  [<ffffffff8164a5fb>] validate_xmit_skb_list+0x3b/0x60
[ 1757.644579]  [<ffffffff8166e591>] sch_direct_xmit+0xc1/0x1f0
[ 1757.644585]  [<ffffffff8164a92b>] __dev_queue_xmit+0x21b/0x510
[ 1757.644589]  [<ffffffff8164ac30>] dev_queue_xmit+0x10/0x20
[ 1757.644593]  [<ffffffff8168b66f>] ip_finish_output2+0x23f/0x310
[ 1757.644598]  [<ffffffff8168c179>] ip_finish_output+0x139/0x1f0
[ 1757.644605]  [<ffffffff8167e9d6>] ? nf_hook_slow+0x76/0xd0
[ 1757.644610]  [<ffffffff8168d64e>] ip_output+0x6e/0xe0
[ 1757.644615]  [<ffffffff8168cd02>] ? __ip_local_out+0x42/0x100
[ 1757.644620]  [<ffffffff8168c040>] ? ip_fragment.constprop.49+0x80/0x80
[ 1757.644627]  [<ffffffff8168cdf5>] ip_local_out+0x35/0x40
[ 1757.644634]  [<ffffffff816d195d>] iptunnel_xmit+0x12d/0x150
[ 1757.644640]  [<ffffffffa036326a>] udp_tunnel_xmit_skb+0xea/0x100 [udp_tunnel]
[ 1757.644648]  [<ffffffffa038a506>] vxlan_xmit_one+0xac6/0x1280 [vxlan]
[ 1757.644659]  [<ffffffff810ef332>] ? vprintk_emit+0x2f2/0x4f0
[ 1757.644675]  [<ffffffff81193772>] ? printk+0x5d/0x74
[ 1757.644681]  [<ffffffff8109c045>] ? warn_slowpath_common+0x95/0xc0
[ 1757.644688]  [<ffffffffa038bf32>] vxlan_xmit+0x172/0xd44 [vxlan]
[ 1757.644694]  [<ffffffff816c26a3>] ? inet_gso_segment+0x163/0x360
[ 1757.644711]  [<ffffffff8164a43e>] dev_hard_start_xmit+0x22e/0x3b0
[ 1757.644721]  [<ffffffff8164ab24>] __dev_queue_xmit+0x414/0x510
[ 1757.644734]  [<ffffffff8164ac30>] dev_queue_xmit+0x10/0x20
[ 1757.644747]  [<ffffffff8168b66f>] ip_finish_output2+0x23f/0x310
[ 1757.644758]  [<ffffffff8168c179>] ip_finish_output+0x139/0x1f0
[ 1757.644763]  [<ffffffff8167e9d6>] ? nf_hook_slow+0x76/0xd0
[ 1757.644768]  [<ffffffff8168d64e>] ip_output+0x6e/0xe0
[ 1757.644775]  [<ffffffff8168cd02>] ? __ip_local_out+0x42/0x100
[ 1757.644780]  [<ffffffff8168c040>] ? ip_fragment.constprop.49+0x80/0x80
[ 1757.644785]  [<ffffffff8168cdf5>] ip_local_out+0x35/0x40
[ 1757.644793]  [<ffffffff8168e0c9>] ip_send_skb+0x19/0x40
[ 1757.644800]  [<ffffffff816b41cd>] udp_send_skb+0x16d/0x270
[ 1757.644807]  [<ffffffff816b50f8>] udp_sendmsg+0x2c8/0x9a0
[ 1757.644812]  [<ffffffff8168b1b0>] ? ip_reply_glue_bits+0x60/0x60
[ 1757.644825]  [<ffffffff816c2387>] inet_sendmsg+0x67/0xa0
[ 1757.644838]  [<ffffffff8162bf88>] sock_sendmsg+0x38/0x50
[ 1757.644852]  [<ffffffff8162c442>] SYSC_sendto+0x102/0x190
[ 1757.644860]  [<ffffffff8113842f>] ? __audit_syscall_entry+0xaf/0x100
[ 1757.644867]  [<ffffffff81003176>] ? do_audit_syscall_entry+0x66/0x70
[ 1757.644873]  [<ffffffff810037ef>] ? syscall_trace_enter_phase1+0x11f/0x140
[ 1757.644879]  [<ffffffff81096c6d>] ? syscall_slow_exit_work+0x3f/0x9f
[ 1757.644883]  [<ffffffff8162cf5e>] SyS_sendto+0xe/0x10
[ 1757.644890]  [<ffffffff8175946e>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 1757.644895] ---[ end trace ec9dfd887c59f41f ]---


Here are the udpspam.c diffs from the original that I found at
http://oss.sgi.com/archives/netdev/2001-10/txtmtEDzF63p0.txt


--- udpspam.c	2015-12-14 16:56:18.287053786 -0800
+++ udpspam-no-check.c	2015-12-14 17:02:21.979047972 -0800
@@ -42,8 +42,8 @@ typedef unsigned long dword;
 
 /* Globals */
 
-#define PAYLOAD_SIZE 4
-static char payload[PAYLOAD_SIZE];
+static int payload_size = 4;
+static char *payload;
 
 /* Socket functions */
 
@@ -85,6 +85,8 @@ static int get_udp_socket(){
 
 	option = 0;
 	setsockopt(fd,SOL_SOCKET,SO_BROADCAST,(void *)&option,sizeof(option));
+	option = 1;
+	setsockopt(fd,SOL_SOCKET,SO_NO_CHECK,(void *)&option,sizeof(option));
 
 	udp.sin_addr.s_addr = htonl(INADDR_ANY);
 	udp.sin_port = htons(0);
@@ -123,9 +125,19 @@ int main(int argc,char *argv[]){
 		exit(-1);
 
 	socket_connect_address(fd,&remote);
-	memset(payload,0,PAYLOAD_SIZE);
+
+	if (argc == 3) {
+		errno = 0;
+		payload_size = strtol(argv[2], NULL, 0);
+		if (errno) {
+			fprintf(stderr, "bad payload size '%s'\n", argv[2]);
+			exit(1);
+		}
+	}
+	payload = malloc(payload_size);
+	memset(payload,0,payload_size);
 	while (1)
-		sendto(fd,payload,PAYLOAD_SIZE,0,(struct sockaddr *)&remote,sizeof(remote));
+		sendto(fd,payload,payload_size,0,(struct sockaddr *)&remote,sizeof(remote));
 
 	exit(0);
 }


======================================================================
Mr. Shannon Nelson                        Network Division, Intel Corp.
Shannon.Nelson@intel.com                I don't speak for Intel
(503) 712-7659                    Parents can't afford to be squeamish

Re: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Jesse Brandeburg]

On Tue, 15 Dec 2015 01:35:27 +0000
"Nelson, Shannon" <shannon.nelson@intel.com> wrote:

> Using a slightly modified version of udpspam (see diff below - hopefully
> not mangled by corporate email servers), where I set the SO_NO_CHECK
> socket option and can specify a large buffer size, I can reliably get
> the following WARN trace.  I have reproduced this on both ixgbe and
> i40e drivers using "udpspam-no-check <target-ip> 6000".
> 
> It looks to me like this is in the Tx path before we get to the actual
> NIC drivers, but I may be wrong.
> 
> 
> [ 1757.644324] ------------[ cut here ]------------
> [ 1757.644333] WARNING: CPU: 22 PID: 5537 at net/core/dev.c:2423 skb_warn_bad_offload+0x104/0x111()

Was anyone able to take a look at this?  There is a pretty clear
reproducer in the original mail.

I know SO_NO_CHECK is a lightly used option, but it does exist and we
found a userspace program that tries to use it.

Re: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Hannes Frederic Sowa]

On 15.12.2015 02:35, Nelson, Shannon wrote:
> Using a slightly modified version of udpspam (see diff below - hopefully not mangled by corporate email servers), where I set the SO_NO_CHECK socket option and can specify a large buffer size, I can reliably get the following WARN trace.  I have reproduced this on both ixgbe and i40e drivers using "udpspam-no-check <target-ip> 6000".
>
> It looks to me like this is in the Tx path before we get to the actual NIC drivers, but I may be wrong.

Does UFO offloading on the tunnel interface fix this error?

ethtool -K vxlan ufo off

If yes we can't feed SO_NO_CHECK udp packets into UFO as gso/ufo 
requires CHECKSUM_PARTIAL (or add some more logic into the skb or query 
socket status from skb_gso_segment).

Thanks,
Hannes

Re: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Michal Kubecek]

On Thu, Jan 07, 2016 at 03:36:27AM +0100, Hannes Frederic Sowa wrote:
> On 15.12.2015 02:35, Nelson, Shannon wrote:
> >Using a slightly modified version of udpspam (see diff below -
> >hopefully not mangled by corporate email servers), where I set the
> >SO_NO_CHECK socket option and can specify a large buffer size, I can
> >reliably get the following WARN trace.  I have reproduced this on
> >both ixgbe and i40e drivers using "udpspam-no-check <target-ip>
> >6000".
> >
> >It looks to me like this is in the Tx path before we get to the
> >actual NIC drivers, but I may be wrong.
> 
> Does UFO offloading on the tunnel interface fix this error?
> 
> ethtool -K vxlan ufo off
> 
> If yes we can't feed SO_NO_CHECK udp packets into UFO as gso/ufo
> requires CHECKSUM_PARTIAL (or add some more logic into the skb or
> query socket status from skb_gso_segment).

I believe you are right. This sounds like the issue addressed by commit
acf8dd0a9d0b ("udp: only allow UFO for packets from SOCK_DGRAM sockets")
except for a regular SOCK_DGRAM socket with SO_NO_CHECK option. If so,
changing both instances of the check

	    (sk->sk_type == SOCK_DGRAM)

added by this commit to

	    (sk->sk_type == SOCK_DGRAM) && !sk->sk_no_check_tx

should fix the problem (and it should be done even if the issue reported
here is caused by something else).

                                                        Michal Kubecek

Re: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Michal Kubecek]

On Thu, Jan 07, 2016 at 07:46:40AM +0100, Michal Kubecek wrote:
> On Thu, Jan 07, 2016 at 03:36:27AM +0100, Hannes Frederic Sowa wrote:
> > On 15.12.2015 02:35, Nelson, Shannon wrote:
> > >Using a slightly modified version of udpspam (see diff below -
> > >hopefully not mangled by corporate email servers), where I set the
> > >SO_NO_CHECK socket option and can specify a large buffer size, I can
> > >reliably get the following WARN trace.  I have reproduced this on
> > >both ixgbe and i40e drivers using "udpspam-no-check <target-ip>
> > >6000".
> > >
> > >It looks to me like this is in the Tx path before we get to the
> > >actual NIC drivers, but I may be wrong.
> > 
> > Does UFO offloading on the tunnel interface fix this error?
> > 
> > ethtool -K vxlan ufo off
> > 
> > If yes we can't feed SO_NO_CHECK udp packets into UFO as gso/ufo
> > requires CHECKSUM_PARTIAL (or add some more logic into the skb or
> > query socket status from skb_gso_segment).
> 
> I believe you are right. This sounds like the issue addressed by commit
> acf8dd0a9d0b ("udp: only allow UFO for packets from SOCK_DGRAM sockets")
> except for a regular SOCK_DGRAM socket with SO_NO_CHECK option. If so,
> changing both instances of the check
> 
> 	    (sk->sk_type == SOCK_DGRAM)
> 
> added by this commit to
> 
> 	    (sk->sk_type == SOCK_DGRAM) && !sk->sk_no_check_tx
> 
> should fix the problem (and it should be done even if the issue reported
> here is caused by something else).

Correction: the IPv6 case will need "!udp_get_no_check6_tx(sk)" instead.

Michal Kubecek

Re: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Michal Kubecek]

On Thu, Jan 07, 2016 at 08:27:42AM +0100, Michal Kubecek wrote:
> On Thu, Jan 07, 2016 at 07:46:40AM +0100, Michal Kubecek wrote:
> > On Thu, Jan 07, 2016 at 03:36:27AM +0100, Hannes Frederic Sowa wrote:
> > > On 15.12.2015 02:35, Nelson, Shannon wrote:
> > > >Using a slightly modified version of udpspam (see diff below -
> > > >hopefully not mangled by corporate email servers), where I set the
> > > >SO_NO_CHECK socket option and can specify a large buffer size, I can
> > > >reliably get the following WARN trace.  I have reproduced this on
> > > >both ixgbe and i40e drivers using "udpspam-no-check <target-ip>
> > > >6000".
> > > >
> > > >It looks to me like this is in the Tx path before we get to the
> > > >actual NIC drivers, but I may be wrong.
> > > 
> > > Does UFO offloading on the tunnel interface fix this error?
> > > 
> > > ethtool -K vxlan ufo off
> > > 
> > > If yes we can't feed SO_NO_CHECK udp packets into UFO as gso/ufo
> > > requires CHECKSUM_PARTIAL (or add some more logic into the skb or
> > > query socket status from skb_gso_segment).
> > 
> > I believe you are right. This sounds like the issue addressed by commit
> > acf8dd0a9d0b ("udp: only allow UFO for packets from SOCK_DGRAM sockets")
> > except for a regular SOCK_DGRAM socket with SO_NO_CHECK option. If so,
> > changing both instances of the check
> > 
> > 	    (sk->sk_type == SOCK_DGRAM)
> > 
> > added by this commit to
> > 
> > 	    (sk->sk_type == SOCK_DGRAM) && !sk->sk_no_check_tx
> > 
> > should fix the problem (and it should be done even if the issue reported
> > here is caused by something else).
> 
> Correction: the IPv6 case will need "!udp_get_no_check6_tx(sk)" instead.

Shannon, if disabling UFO works around the problem, please check if this
patch helps:


From: Michal Kubecek <mkubecek@suse.cz>
Date: Fri, 8 Jan 2016 18:23:40 +0100
Subject: [PATCH] udp: disallow UFO for sockets with SO_NO_CHECK option

Commit acf8dd0a9d0b ("udp: only allow UFO for packets from SOCK_DGRAM
sockets") disallows UFO for packets sent from raw sockets. We need to do
the same also for SOCK_DGRAM sockets with SO_NO_CHECK option, even if
for a bit different reason: while such socket would override the
CHECKSUM_PARTIAL set by ip_ufo_append_data(), gso_size is still set and
bad offloading flags warning is triggered in __skb_gso_segment().

In the IPv6 case, SO_NO_CHECK option is ignored but we need to disallow
UFO for packets sent by sockets with UDP_NO_CHECK6_TX option.

Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
---
 net/ipv4/ip_output.c  | 2 +-
 net/ipv6/ip6_output.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 4233cbe47052..36ac9f3a6451 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -921,7 +921,7 @@ static int __ip_append_data(struct sock *sk,
 	if (((length > mtu) || (skb && skb_is_gso(skb))) &&
 	    (sk->sk_protocol == IPPROTO_UDP) &&
 	    (rt->dst.dev->features & NETIF_F_UFO) && !rt->dst.header_len &&
-	    (sk->sk_type == SOCK_DGRAM)) {
+	    (sk->sk_type == SOCK_DGRAM) && !sk->sk_no_check_tx) {
 		err = ip_ufo_append_data(sk, queue, getfrag, from, length,
 					 hh_len, fragheaderlen, transhdrlen,
 					 maxfraglen, flags);
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index e6a7bd15b9b7..6473889f1736 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1353,7 +1353,7 @@ emsgsize:
 	     (skb && skb_is_gso(skb))) &&
 	    (sk->sk_protocol == IPPROTO_UDP) &&
 	    (rt->dst.dev->features & NETIF_F_UFO) &&
-	    (sk->sk_type == SOCK_DGRAM)) {
+	    (sk->sk_type == SOCK_DGRAM) && !udp_get_no_check6_tx(sk)) {
 		err = ip6_ufo_append_data(sk, queue, getfrag, from, length,
 					  hh_len, fragheaderlen,
 					  transhdrlen, mtu, flags, fl6);
-- 
2.7.0

RE: WARN trace - skb_warn_bad_offload - vxlan - large udp packet - udp checksum disabled

[Nelson, Shannon]

> From: Michal Kubecek [mailto:mkubecek@suse.cz]
>
> Shannon, if disabling UFO works around the problem, please check if this
> patch helps:
> 
> 
> From: Michal Kubecek <mkubecek@suse.cz>
> Date: Fri, 8 Jan 2016 18:23:40 +0100
> Subject: [PATCH] udp: disallow UFO for sockets with SO_NO_CHECK option
> 

Hi Michal,

Yes, that worked for my test case.

Acked-by: Shannon Nelson <shannon.nelson@intel.com>

Thanks,
sln