* tainted warnings with tcp splicing in 3.7.1 @ 2013-01-09 13:01 Christian Becker 2013-01-09 14:50 ` Lukas Tribus 2013-01-09 17:01 ` Eric Dumazet 0 siblings, 2 replies; 17+ messages in thread From: Christian Becker @ 2013-01-09 13:01 UTC (permalink / raw) To: netdev@vger.kernel.org Hi, we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO. Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy. We´ve built the kernel yesterday from the 3.7.1 sources without any modifications. The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs: Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) Emulex Corporation OneConnect 10Gb NIC (Driver be2net) The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic. We´ve downgraded again to our old kernel version, but i guess you should take a look at this. There are two kinds of messages: an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------ Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- Jan 9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 Jan 9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net Jan 9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1 Jan 9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace: Jan 9 11:34:28 srv11 kernel: [ 1081.573762] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c Jan 9 11:34:28 srv11 kernel: [ 1081.596750] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a Jan 9 11:34:28 srv11 kernel: [ 1081.617853] [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe Jan 9 11:34:28 srv11 kernel: [ 1081.639111] [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc Jan 9 11:34:28 srv11 kernel: [ 1081.661541] [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194 Jan 9 11:34:28 srv11 kernel: [ 1081.681164] [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b Jan 9 11:34:28 srv11 kernel: [ 1081.703355] [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223 Jan 9 11:34:28 srv11 kernel: [ 1081.724912] [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0 Jan 9 11:34:28 srv11 kernel: [ 1081.744525] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b Jan 9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]--- Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]- Jan 9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0 Jan 9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net Jan 9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G W 3.7.1 #1 Jan 9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace: Jan 9 11:52:42 srv11 kernel: [ 2175.122727] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c Jan 9 11:52:42 srv11 kernel: [ 2175.147250] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a Jan 9 11:52:42 srv11 kernel: [ 2175.170289] [<ffffffff81295f64>] ? release_sock+0xe6/0x11c Jan 9 11:52:43 srv11 kernel: [ 2175.192639] [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de Jan 9 11:52:43 srv11 kernel: [ 2175.215020] [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946 Jan 9 11:52:43 srv11 kernel: [ 2175.237682] [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75 Jan 9 11:52:43 srv11 kernel: [ 2175.259727] [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e Jan 9 11:52:43 srv11 kernel: [ 2175.281111] [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50 Jan 9 11:52:43 srv11 kernel: [ 2175.300573] [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120 Jan 9 11:52:43 srv11 kernel: [ 2175.320073] [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6 Jan 9 11:52:43 srv11 kernel: [ 2175.341151] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b Jan 9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]--- As you can see here, the messages are appearing every couple of minutes: Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() Jan 9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1 2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker @ 2013-01-09 14:50 ` Lukas Tribus 2013-01-09 17:01 ` Eric Dumazet 1 sibling, 0 replies; 17+ messages in thread From: Lukas Tribus @ 2013-01-09 14:50 UTC (permalink / raw) To: netdev For the record: Another user reported similar warnings in a 3.5.0 kernel back in September [1] on the haproxy mailing list with the following kernel warnings: [142654.793193] ------------[ cut here ]------------ [142654.793395] WARNING: at net/ipv4/tcp.c:1301 tcp_cleanup_rbuf+0x54/0x150() [142654.793972] Hardware name: System Product Name [142654.794573] cleanup rbuf bug: copied 68D1EF11 seq 68CFF65F rcvnxt 68D3565D [142654.795165] Modules linked in: ixgbe(O) binfmt_misc 8021q fcoe garp stp llc libfcoe libfc scsi_transport_fc scsi_tgt ip6t_REJECT nf _conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec nouveau snd_hwdep igb snd_seq snd_seq_device snd_pcm eeepc_wmi asus_wmi ttm drm_kms_helper sparse_keymap snd_timer snd drm coretemp rfkill i2c_algo_bit mxm_wmi wmi video lpc_ich mfd_core crc32c_intel r8169 i2c_i801 i2c_core soundcore snd_page_alloc mii mdio pcspkr serio_raw ghash_clmulni_intel microcode uinput [last unloaded: ixgbe] [142654.798215] Pid: 18374, comm: haproxy Tainted: G W O 3.5.0 #1 [142654.798838] Call Trace: [142654.799440] [<ffffffff810422cf>] warn_slowpath_common+0x7f/0xc0 [142654.800031] [<ffffffff810423c6>] warn_slowpath_fmt+0x46/0x50 [142654.800653] [<ffffffff8147c270>] ? sock_pipe_buf_release+0x20/0x20 [142654.801237] [<ffffffff814cf294>] tcp_cleanup_rbuf+0x54/0x150 [142654.801847] [<ffffffff814d0ae1>] tcp_read_sock+0x1b1/0x200 [142654.802440] [<ffffffff81472777>] ? sock_sendpage+0x27/0x30 [142654.803037] [<ffffffff814ccd60>] ? tcp_done+0x90/0x90 [142654.803644] [<ffffffff814d0bf0>] tcp_splice_read+0xc0/0x250 [142654.804239] [<ffffffff814726b2>] sock_splice_read+0x62/0x80 [142654.804843] [<ffffffff8118c73b>] do_splice_to+0x7b/0xa0 [142654.805457] [<ffffffff8118e850>] sys_splice+0x540/0x560 [142654.806040] [<ffffffff8159aed2>] system_call_fastpath+0x16/0x1b [142654.806646] ---[ end trace 46d7fb693af33fde ]--- [1] http://permalink.gmane.org/gmane.comp.web.haproxy/9559 - ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker 2013-01-09 14:50 ` Lukas Tribus @ 2013-01-09 17:01 ` Eric Dumazet 2013-01-09 17:09 ` Eric Dumazet 1 sibling, 1 reply; 17+ messages in thread From: Eric Dumazet @ 2013-01-09 17:01 UTC (permalink / raw) To: Christian Becker; +Cc: netdev@vger.kernel.org On Wed, 2013-01-09 at 13:01 +0000, Christian Becker wrote: > Hi, > > we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO. > > Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy. > > We´ve built the kernel yesterday from the 3.7.1 sources without any modifications. > > The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs: > > Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) > Emulex Corporation OneConnect 10Gb NIC (Driver be2net) > > The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic. > > We´ve downgraded again to our old kernel version, but i guess you should take a look at this. > > There are two kinds of messages: > > an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------ > Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- > Jan 9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 > Jan 9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net > Jan 9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1 > Jan 9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace: > Jan 9 11:34:28 srv11 kernel: [ 1081.573762] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c > Jan 9 11:34:28 srv11 kernel: [ 1081.596750] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a > Jan 9 11:34:28 srv11 kernel: [ 1081.617853] [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe > Jan 9 11:34:28 srv11 kernel: [ 1081.639111] [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc > Jan 9 11:34:28 srv11 kernel: [ 1081.661541] [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194 > Jan 9 11:34:28 srv11 kernel: [ 1081.681164] [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b > Jan 9 11:34:28 srv11 kernel: [ 1081.703355] [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223 > Jan 9 11:34:28 srv11 kernel: [ 1081.724912] [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0 > Jan 9 11:34:28 srv11 kernel: [ 1081.744525] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b > Jan 9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]--- > > Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]- > Jan 9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0 > Jan 9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net > Jan 9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G W 3.7.1 #1 > Jan 9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace: > Jan 9 11:52:42 srv11 kernel: [ 2175.122727] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c > Jan 9 11:52:42 srv11 kernel: [ 2175.147250] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a > Jan 9 11:52:42 srv11 kernel: [ 2175.170289] [<ffffffff81295f64>] ? release_sock+0xe6/0x11c > Jan 9 11:52:43 srv11 kernel: [ 2175.192639] [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de > Jan 9 11:52:43 srv11 kernel: [ 2175.215020] [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946 > Jan 9 11:52:43 srv11 kernel: [ 2175.237682] [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75 > Jan 9 11:52:43 srv11 kernel: [ 2175.259727] [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e > Jan 9 11:52:43 srv11 kernel: [ 2175.281111] [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50 > Jan 9 11:52:43 srv11 kernel: [ 2175.300573] [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120 > Jan 9 11:52:43 srv11 kernel: [ 2175.320073] [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6 > Jan 9 11:52:43 srv11 kernel: [ 2175.341151] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b > Jan 9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]--- > > As you can see here, the messages are appearing every couple of minutes: > > Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > Jan 9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()-- Thanks for this report This might be a bug because of TCP collapsing A "netstat -s" would have been a very useful information. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-09 17:01 ` Eric Dumazet @ 2013-01-09 17:09 ` Eric Dumazet 2013-01-10 6:59 ` Eric Dumazet 0 siblings, 1 reply; 17+ messages in thread From: Eric Dumazet @ 2013-01-09 17:09 UTC (permalink / raw) To: Christian Becker; +Cc: netdev@vger.kernel.org, Willy Tarreau On Wed, 2013-01-09 at 09:01 -0800, Eric Dumazet wrote: > On Wed, 2013-01-09 at 13:01 +0000, Christian Becker wrote: > > Hi, > > > > we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO. > > > > Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy. > > > > We´ve built the kernel yesterday from the 3.7.1 sources without any modifications. > > > > The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs: > > > > Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) > > Emulex Corporation OneConnect 10Gb NIC (Driver be2net) > > > > The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic. > > > > We´ve downgraded again to our old kernel version, but i guess you should take a look at this. > > > > There are two kinds of messages: > > > > an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------ > > Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- > > Jan 9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 > > Jan 9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb > > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi > > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net > > Jan 9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1 > > Jan 9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace: > > Jan 9 11:34:28 srv11 kernel: [ 1081.573762] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c > > Jan 9 11:34:28 srv11 kernel: [ 1081.596750] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a > > Jan 9 11:34:28 srv11 kernel: [ 1081.617853] [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe > > Jan 9 11:34:28 srv11 kernel: [ 1081.639111] [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc > > Jan 9 11:34:28 srv11 kernel: [ 1081.661541] [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194 > > Jan 9 11:34:28 srv11 kernel: [ 1081.681164] [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b > > Jan 9 11:34:28 srv11 kernel: [ 1081.703355] [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223 > > Jan 9 11:34:28 srv11 kernel: [ 1081.724912] [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0 > > Jan 9 11:34:28 srv11 kernel: [ 1081.744525] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b > > Jan 9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]--- > > > > Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]- > > Jan 9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0 > > Jan 9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb > > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi > > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net > > Jan 9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G W 3.7.1 #1 > > Jan 9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace: > > Jan 9 11:52:42 srv11 kernel: [ 2175.122727] [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c > > Jan 9 11:52:42 srv11 kernel: [ 2175.147250] [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a > > Jan 9 11:52:42 srv11 kernel: [ 2175.170289] [<ffffffff81295f64>] ? release_sock+0xe6/0x11c > > Jan 9 11:52:43 srv11 kernel: [ 2175.192639] [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de > > Jan 9 11:52:43 srv11 kernel: [ 2175.215020] [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946 > > Jan 9 11:52:43 srv11 kernel: [ 2175.237682] [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75 > > Jan 9 11:52:43 srv11 kernel: [ 2175.259727] [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e > > Jan 9 11:52:43 srv11 kernel: [ 2175.281111] [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50 > > Jan 9 11:52:43 srv11 kernel: [ 2175.300573] [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120 > > Jan 9 11:52:43 srv11 kernel: [ 2175.320073] [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6 > > Jan 9 11:52:43 srv11 kernel: [ 2175.341151] [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b > > Jan 9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]--- > > > > As you can see here, the messages are appearing every couple of minutes: > > > > Jan 9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > > Jan 9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de() > > Jan 9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()-- > > Thanks for this report > > This might be a bug because of TCP collapsing > > A "netstat -s" would have been a very useful information. > > My feeling is that tcp_recv_skb() should eat skbs instead of only finding the right one Thats because skb_splice_bits() releases the socket lock before calling splice_to_pipe() Once socket is released, other incoming TCP frames can be processed, and the skb we are actually processing might be 'collapsed' into smaller units. Christian, if I send you patches, are you OK to test them ? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-09 17:09 ` Eric Dumazet @ 2013-01-10 6:59 ` Eric Dumazet 2013-01-10 7:21 ` Willy Tarreau ` (2 more replies) 0 siblings, 3 replies; 17+ messages in thread From: Eric Dumazet @ 2013-01-10 6:59 UTC (permalink / raw) To: Christian Becker, David Miller; +Cc: netdev@vger.kernel.org, Willy Tarreau From: Eric Dumazet <edumazet@google.com> On Wed, 2013-01-09 at 09:09 -0800, Eric Dumazet wrote: > My feeling is that tcp_recv_skb() should eat skbs instead of only > finding the right one > Thats indeed the case. > Thats because skb_splice_bits() releases the socket lock before calling > splice_to_pipe() > > Once socket is released, other incoming TCP frames can be processed, and > the skb we are actually processing might be 'collapsed' into smaller > units. > > Christian, if I send you patches, are you OK to test them ? > > Here is the patch fixing this issue. Thanks a lot for your report, this is a very very old bug. GRO being more deployed, and with TCP coalescing as well, chances to trigger this bug increased a lot. To reproduce it, I had to force MSS=400 and stress the receiver, adding extra delays in skb_splice_bits() with socket lock being not held. [PATCH] tcp: fix splice() and tcp collapsing interaction Under unusual circumstances, TCP collapse can split a big GRO TCP packet while its being used in a splice(socket->pipe) operation. skb_splice_bits() releases the socket lock before calling splice_to_pipe(). [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 To fix this problem, we must eat skbs in tcp_recv_skb(). Remove the inline keyword from tcp_recv_skb() definition since it has three call sites. Reported-by: Christian Becker <c.becker@traviangames.com> Cc: Willy Tarreau <w@1wt.eu> Signed-off-by: Eric Dumazet <edumazet@google.com> --- net/ipv4/tcp.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 1ca2536..1f901be 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait) } #endif -static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) +static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) { struct sk_buff *skb; u32 offset; - skb_queue_walk(&sk->sk_receive_queue, skb) { + while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) { offset = seq - TCP_SKB_CB(skb)->seq; if (tcp_hdr(skb)->syn) offset--; @@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) *off = offset; return skb; } + /* This looks weird, but this can happen if TCP collapsing + * splitted a fat GRO packet, while we released socket lock + * in skb_splice_bits() + */ + sk_eat_skb(sk, skb, false); } return NULL; } @@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, tcp_rcv_space_adjust(sk); /* Clean up data we have read: This will do ACK frames. */ - if (copied > 0) + if (copied > 0) { + tcp_recv_skb(sk, seq, &offset); tcp_cleanup_rbuf(sk, copied); + } return copied; } EXPORT_SYMBOL(tcp_read_sock); ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 6:59 ` Eric Dumazet @ 2013-01-10 7:21 ` Willy Tarreau 2013-01-10 15:29 ` Eric Dumazet 2013-01-10 18:27 ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus 2013-01-10 22:39 ` David Miller 2 siblings, 1 reply; 17+ messages in thread From: Willy Tarreau @ 2013-01-10 7:21 UTC (permalink / raw) To: Eric Dumazet; +Cc: Christian Becker, David Miller, netdev@vger.kernel.org On Wed, Jan 09, 2013 at 10:59:09PM -0800, Eric Dumazet wrote: > From: Eric Dumazet <edumazet@google.com> > > On Wed, 2013-01-09 at 09:09 -0800, Eric Dumazet wrote: > > > My feeling is that tcp_recv_skb() should eat skbs instead of only > > finding the right one > > > > Thats indeed the case. > > > Thats because skb_splice_bits() releases the socket lock before calling > > splice_to_pipe() > > > > Once socket is released, other incoming TCP frames can be processed, and > > the skb we are actually processing might be 'collapsed' into smaller > > units. > > > > Christian, if I send you patches, are you OK to test them ? > > > > > > Here is the patch fixing this issue. > > Thanks a lot for your report, this is a very very old bug. > > GRO being more deployed, and with TCP coalescing as well, chances to > trigger this bug increased a lot. > > To reproduce it, I had to force MSS=400 and stress the receiver, > adding extra delays in skb_splice_bits() with socket lock being not > held. FWIW, I tested your patch here and did not notice any regression compared to last week-end tests, at various MTU size combinations. Cheers, Willy ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 7:21 ` Willy Tarreau @ 2013-01-10 15:29 ` Eric Dumazet 2013-01-10 16:20 ` Eric Dumazet 2013-01-12 0:46 ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet 0 siblings, 2 replies; 17+ messages in thread From: Eric Dumazet @ 2013-01-10 15:29 UTC (permalink / raw) To: Willy Tarreau; +Cc: Christian Becker, David Miller, netdev@vger.kernel.org On Thu, 2013-01-10 at 08:21 +0100, Willy Tarreau wrote: > FWIW, I tested your patch here and did not notice any regression > compared to last week-end tests, at various MTU size combinations. > Thanks Willy Tested-by: Willy Tarreau <w@1wt.eu> I believe I need to fix net-next, commit 9ca1b22d6d228177e6f929f6818a1cd3d5e30c4a (net: splice: avoid high order page splitting) missed the loopback case : the skb->head might need several linear_to_page() calls. I'll send a patch after full testing. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 15:29 ` Eric Dumazet @ 2013-01-10 16:20 ` Eric Dumazet 2013-01-10 18:22 ` Rick Jones 2013-01-12 0:46 ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet 1 sibling, 1 reply; 17+ messages in thread From: Eric Dumazet @ 2013-01-10 16:20 UTC (permalink / raw) To: Willy Tarreau, Rick Jones Cc: Christian Becker, David Miller, netdev@vger.kernel.org On Thu, 2013-01-10 at 07:29 -0800, Eric Dumazet wrote: > On Thu, 2013-01-10 at 08:21 +0100, Willy Tarreau wrote: > > > FWIW, I tested your patch here and did not notice any regression > > compared to last week-end tests, at various MTU size combinations. > > > > Thanks Willy I also want to thanks Rick, as the latest netperf has splice() support. Thanks Rick ! ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 16:20 ` Eric Dumazet @ 2013-01-10 18:22 ` Rick Jones 2013-01-10 18:42 ` Eric Dumazet 0 siblings, 1 reply; 17+ messages in thread From: Rick Jones @ 2013-01-10 18:22 UTC (permalink / raw) To: Eric Dumazet Cc: Willy Tarreau, Christian Becker, David Miller, netdev@vger.kernel.org On 01/10/2013 08:20 AM, Eric Dumazet wrote: > I also want to thanks Rick, as the latest netperf has splice() support. > > Thanks Rick ! You are quite welcome - and thank you for helping me get it to actually work :) Those wishing to try it themselves should grab the top-of-trunk netperf bits from http://www.netperf.org/svn/netperf2/trunk . The use of splice() is gated by a test-specific -V option: raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain () port 0 AF_INET : copy avoidance : demo Remote Local Remote Elapsed Throughput Throughput Send Socket Recv Socket Send Time Units Size Size Size (sec) Final Final 1661688 4194304 16384 10.00 26103.14 10^6bits/s You should see that "copy avoidance" appearing in the test banner. It will also "take" for things like a migrated TCP_mumble test. For those cases where you don't see a throughput change, enabling CPU utilization measurement and looking at that and service demand should show a difference. happy benchmarking, rick jones ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 18:22 ` Rick Jones @ 2013-01-10 18:42 ` Eric Dumazet 2013-01-10 18:49 ` Rick Jones 0 siblings, 1 reply; 17+ messages in thread From: Eric Dumazet @ 2013-01-10 18:42 UTC (permalink / raw) To: Rick Jones Cc: Willy Tarreau, Christian Becker, David Miller, netdev@vger.kernel.org On Thu, 2013-01-10 at 10:22 -0800, Rick Jones wrote: > On 01/10/2013 08:20 AM, Eric Dumazet wrote: > > I also want to thanks Rick, as the latest netperf has splice() support. > > > > Thanks Rick ! > > You are quite welcome - and thank you for helping me get it to actually > work :) > > Those wishing to try it themselves should grab the top-of-trunk netperf > bits from http://www.netperf.org/svn/netperf2/trunk . The use of > splice() is gated by a test-specific -V option: > > raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V > OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to > localhost.localdomain () port 0 AF_INET : copy avoidance : demo > Remote Local Remote Elapsed Throughput Throughput > Send Socket Recv Socket Send Time Units > Size Size Size (sec) > Final Final > 1661688 4194304 16384 10.00 26103.14 10^6bits/s > > You should see that "copy avoidance" appearing in the test banner. It > will also "take" for things like a migrated TCP_mumble test. For those > cases where you don't see a throughput change, enabling CPU utilization > measurement and looking at that and service demand should show a difference. Thanks Rick ! Can we use zero copy for the sender as well (sendfile() or vmsplice()) ? Here are some results : Reference time (no splice()) # ./netperf -H 127.0.0.1 -t omni -- -d recv OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 () port 0 AF_INET catcher: timer popped with times_up != 0 Remote Local Remote Elapsed Throughput Throughput Send Socket Recv Socket Send Time Units Size Size Size (sec) Final Final 2097152 2097152 16384 10.00 33237.74 10^6bits/s zero copy at receiver (splice(socket ->pipe), splice(pipe -> /dev/null)) # ./netperf -H 127.0.0.1 -t omni -- -d recv -V OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 () port 0 AF_INET : copy avoidance catcher: timer popped with times_up != 0 Remote Local Remote Elapsed Throughput Throughput Send Socket Recv Socket Send Time Units Size Size Size (sec) Final Final 1325580 2097152 16384 10.00 51980.60 10^6bits/s ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 18:42 ` Eric Dumazet @ 2013-01-10 18:49 ` Rick Jones 2013-01-10 19:43 ` Willy Tarreau 0 siblings, 1 reply; 17+ messages in thread From: Rick Jones @ 2013-01-10 18:49 UTC (permalink / raw) To: Eric Dumazet Cc: Willy Tarreau, Christian Becker, David Miller, netdev@vger.kernel.org On 01/10/2013 10:42 AM, Eric Dumazet wrote: > Can we use zero copy for the sender as well (sendfile() or vmsplice()) ? Not at present. The TCP_SENDFILE test has not been "migrated" and there is no corresponding sendfile() test in the omni code, so the attempt to enable copy-avoidance won't "take." I'll take that as a feature request for netperf. rick ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 18:49 ` Rick Jones @ 2013-01-10 19:43 ` Willy Tarreau 0 siblings, 0 replies; 17+ messages in thread From: Willy Tarreau @ 2013-01-10 19:43 UTC (permalink / raw) To: Rick Jones Cc: Eric Dumazet, Christian Becker, David Miller, netdev@vger.kernel.org On Thu, Jan 10, 2013 at 10:49:17AM -0800, Rick Jones wrote: > On 01/10/2013 10:42 AM, Eric Dumazet wrote: > >Can we use zero copy for the sender as well (sendfile() or vmsplice()) ? > > Not at present. The TCP_SENDFILE test has not been "migrated" and there > is no corresponding sendfile() test in the omni code, so the attempt to > enable copy-avoidance won't "take." Note that in my httpterm server, I'm using tee+splice(). I had a problem with vmsplice() in the past, I don't remember which one. For the "inject" http client, I'm using recv(MSG_TRUNC) which is always faster than splice() and does the job well, considering that the goal is to count and destroy data very fast. By combining them both on the same machine over the loopback, I see 70 Gbps on a core-2 duo 2.66 GHz. These are obviously not real gigs, but at least it shows what the network stack can do when we remove memory copies ! Regards, Willy ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH net-next] net: splice: fix __splice_segment() 2013-01-10 15:29 ` Eric Dumazet 2013-01-10 16:20 ` Eric Dumazet @ 2013-01-12 0:46 ` Eric Dumazet 2013-01-12 0:48 ` David Miller 1 sibling, 1 reply; 17+ messages in thread From: Eric Dumazet @ 2013-01-12 0:46 UTC (permalink / raw) To: Willy Tarreau, David Miller; +Cc: netdev From: Eric Dumazet <edumazet@google.com> commit 9ca1b22d6d2 (net: splice: avoid high order page splitting) forgot that skb->head could need a copy into several page frags. This could be the case for loopback traffic mostly. Also remove now useless skb argument from linear_to_page() and __splice_segment() prototypes. Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Willy Tarreau <w@1wt.eu> --- net/core/skbuff.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 1e1b9ea..2568c44 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1652,7 +1652,7 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i) static struct page *linear_to_page(struct page *page, unsigned int *len, unsigned int *offset, - struct sk_buff *skb, struct sock *sk) + struct sock *sk) { struct page_frag *pfrag = sk_page_frag(sk); @@ -1685,14 +1685,14 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd, static bool spd_fill_page(struct splice_pipe_desc *spd, struct pipe_inode_info *pipe, struct page *page, unsigned int *len, unsigned int offset, - struct sk_buff *skb, bool linear, + bool linear, struct sock *sk) { if (unlikely(spd->nr_pages == MAX_SKB_FRAGS)) return true; if (linear) { - page = linear_to_page(page, len, &offset, skb, sk); + page = linear_to_page(page, len, &offset, sk); if (!page) return true; } @@ -1711,13 +1711,11 @@ static bool spd_fill_page(struct splice_pipe_desc *spd, static bool __splice_segment(struct page *page, unsigned int poff, unsigned int plen, unsigned int *off, - unsigned int *len, struct sk_buff *skb, + unsigned int *len, struct splice_pipe_desc *spd, bool linear, struct sock *sk, struct pipe_inode_info *pipe) { - unsigned int flen; - if (!*len) return true; @@ -1732,12 +1730,16 @@ static bool __splice_segment(struct page *page, unsigned int poff, plen -= *off; *off = 0; - flen = min(*len, plen); - - if (spd_fill_page(spd, pipe, page, &flen, poff, skb, linear, sk)) - return true; + do { + unsigned int flen = min(*len, plen); - *len -= flen; + if (spd_fill_page(spd, pipe, page, &flen, poff, + linear, sk)) + return true; + poff += flen; + plen -= flen; + *len -= flen; + } while (*len && plen); return false; } @@ -1760,7 +1762,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, if (__splice_segment(virt_to_page(skb->data), (unsigned long) skb->data & (PAGE_SIZE - 1), skb_headlen(skb), - offset, len, skb, spd, + offset, len, spd, skb_head_is_locked(skb), sk, pipe)) return true; @@ -1773,7 +1775,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe, if (__splice_segment(skb_frag_page(f), f->page_offset, skb_frag_size(f), - offset, len, skb, spd, false, sk, pipe)) + offset, len, spd, false, sk, pipe)) return true; } ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH net-next] net: splice: fix __splice_segment() 2013-01-12 0:46 ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet @ 2013-01-12 0:48 ` David Miller 0 siblings, 0 replies; 17+ messages in thread From: David Miller @ 2013-01-12 0:48 UTC (permalink / raw) To: eric.dumazet; +Cc: w, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Fri, 11 Jan 2013 16:46:37 -0800 > From: Eric Dumazet <edumazet@google.com> > > commit 9ca1b22d6d2 (net: splice: avoid high order page splitting) > forgot that skb->head could need a copy into several page frags. > > This could be the case for loopback traffic mostly. > > Also remove now useless skb argument from linear_to_page() > and __splice_segment() prototypes. > > Signed-off-by: Eric Dumazet <edumazet@google.com> > Cc: Willy Tarreau <w@1wt.eu> Applied, thanks Eric. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1 2013-01-10 6:59 ` Eric Dumazet 2013-01-10 7:21 ` Willy Tarreau @ 2013-01-10 18:27 ` Lukas Tribus 2013-01-10 18:37 ` Eric Dumazet 2013-01-10 22:39 ` David Miller 2 siblings, 1 reply; 17+ messages in thread From: Lukas Tribus @ 2013-01-10 18:27 UTC (permalink / raw) To: eric.dumazet; +Cc: netdev Hi Eric, this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well? Thanks, Lukas > [PATCH] tcp: fix splice() and tcp collapsing interaction > > Under unusual circumstances, TCP collapse can split a big GRO TCP packet > while its being used in a splice(socket->pipe) operation. > > skb_splice_bits() releases the socket lock before calling > splice_to_pipe(). > > [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- > [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 > > To fix this problem, we must eat skbs in tcp_recv_skb(). > > Remove the inline keyword from tcp_recv_skb() definition since > it has three call sites. > > Reported-by: Christian Becker <c.becker@traviangames.com> > Cc: Willy Tarreau <w@1wt.eu> > Signed-off-by: Eric Dumazet <edumazet@google.com> > --- > net/ipv4/tcp.c | 13 ++++++++++--- > 1 file changed, 10 insertions(+), 3 deletions(-) > > diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c > index 1ca2536..1f901be 100644 > --- a/net/ipv4/tcp.c > +++ b/net/ipv4/tcp.c > @@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait) > } > #endif > > -static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) > +static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) > { > struct sk_buff *skb; > u32 offset; > > - skb_queue_walk(&sk->sk_receive_queue, skb) { > + while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) { > offset = seq - TCP_SKB_CB(skb)->seq; > if (tcp_hdr(skb)->syn) > offset--; > @@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) > *off = offset; > return skb; > } > + /* This looks weird, but this can happen if TCP collapsing > + * splitted a fat GRO packet, while we released socket lock > + * in skb_splice_bits() > + */ > + sk_eat_skb(sk, skb, false); > } > return NULL; > } > @@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, > tcp_rcv_space_adjust(sk); > > /* Clean up data we have read: This will do ACK frames. */ > - if (copied > 0) > + if (copied > 0) { > + tcp_recv_skb(sk, seq, &offset); > tcp_cleanup_rbuf(sk, copied); > + } > return copied; > } > EXPORT_SYMBOL(tcp_read_sock); > ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1 2013-01-10 18:27 ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus @ 2013-01-10 18:37 ` Eric Dumazet 0 siblings, 0 replies; 17+ messages in thread From: Eric Dumazet @ 2013-01-10 18:37 UTC (permalink / raw) To: Lukas Tribus; +Cc: netdev On Thu, 2013-01-10 at 19:27 +0100, Lukas Tribus wrote: > Hi Eric, > > this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well? > Yes it can, net/ipv4 contains generic TCP code. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1 2013-01-10 6:59 ` Eric Dumazet 2013-01-10 7:21 ` Willy Tarreau 2013-01-10 18:27 ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus @ 2013-01-10 22:39 ` David Miller 2 siblings, 0 replies; 17+ messages in thread From: David Miller @ 2013-01-10 22:39 UTC (permalink / raw) To: eric.dumazet; +Cc: c.becker, netdev, w From: Eric Dumazet <eric.dumazet@gmail.com> Date: Wed, 09 Jan 2013 22:59:09 -0800 > [PATCH] tcp: fix splice() and tcp collapsing interaction > > Under unusual circumstances, TCP collapse can split a big GRO TCP packet > while its being used in a splice(socket->pipe) operation. > > skb_splice_bits() releases the socket lock before calling > splice_to_pipe(). > > [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc() > [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]- > [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13 > > To fix this problem, we must eat skbs in tcp_recv_skb(). > > Remove the inline keyword from tcp_recv_skb() definition since > it has three call sites. > > Reported-by: Christian Becker <c.becker@traviangames.com> > Cc: Willy Tarreau <w@1wt.eu> > Signed-off-by: Eric Dumazet <edumazet@google.com> Applied. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-01-12 0:48 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker 2013-01-09 14:50 ` Lukas Tribus 2013-01-09 17:01 ` Eric Dumazet 2013-01-09 17:09 ` Eric Dumazet 2013-01-10 6:59 ` Eric Dumazet 2013-01-10 7:21 ` Willy Tarreau 2013-01-10 15:29 ` Eric Dumazet 2013-01-10 16:20 ` Eric Dumazet 2013-01-10 18:22 ` Rick Jones 2013-01-10 18:42 ` Eric Dumazet 2013-01-10 18:49 ` Rick Jones 2013-01-10 19:43 ` Willy Tarreau 2013-01-12 0:46 ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet 2013-01-12 0:48 ` David Miller 2013-01-10 18:27 ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus 2013-01-10 18:37 ` Eric Dumazet 2013-01-10 22:39 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).