Re: Connection tracking and soft lockups with certain field values

[Andrey Korolyov <andrey@xdel.ru>]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Thu, May 28, 2015 at 10:41 PM, Andrey Korolyov <andrey@xdel.ru> wrote:
> Hi,
>
> I am currently playing with SYNPROXY target to optimize SYN filtering
> performance and by occasion found that TCP SYN packets containing port
> 0 can result in a soft lockup when conntrack is enabled just by
> itself, given high packet ratio (I`ve reached 450kpps so far with 60b
> packets on a /32<->/32 flood with enabled flow control at the media
> level and middle-level E3 Xeon on receiver side). Same flood with port
>> 0 going just well, producing same ceil numbers but without visible
> lockups in kernel log. I`ve tested the issue on a broad range of 3.x
> kernels and all of them are seemingly affected. Fast and dirty grep
> revealed special conditions for port 0 only for protocol-specific
> helpers, but there are none of them.
>
> Please find both same captures and traceback below.

Attached trace without GSO, at its presence can be somehow confusing
in a previous sample. The testbed using net.nf_conntrack_max =
2000000, forgot to mention that previously.

[-- Attachment #2: conntrack-port-0-gso-off.txt --]
[-- Type: text/plain, Size: 4494 bytes --]

[52671.706307] BUG: soft lockup - CPU#0 stuck for 24s! [rcuos/0:18]
[52671.706331] Modules linked in: ixgbe mdio xt_CT iptable_raw ipt_SYNPROXY nf_synproxy_core nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter ip_tables x_tables tun openvswitch nfsd auth_rpcgss oid_registry nfs_acl nfs lockd dns_resolver fscache sunrpc bridge stp llc w83627ehf hwmon_vid loop fuse dm_crypt dm_mod coretemp kvm_intel snd_pcm kvm snd_page_alloc snd_timer crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 ablk_helper snd cryptd lrw iTCO_wdt soundcore iTCO_vendor_support gf128mul glue_helper joydev evdev pcspkr video processor button i2c_i801 lpc_ich mfd_core shpchp ext4 crc16 jbd2 mbcache microcode sg sd_mod crc_t10dif hid_generic usbhid hid ahci libahci libata mpt2sas igb raid_class i2c_algo_bit dca scsi_transport_sas ptp pps_core i2c_core ehci_pci
[52671.706364]  scsi_mod ehci_hcd xhci_hcd usbcore usb_common thermal fan thermal_sys
[52671.706369] CPU: 0 PID: 18 Comm: rcuos/0 Not tainted 3.10-0.bpo.3-amd64 #1 Debian 3.10.11-1~bpo70+19
[52671.706370] Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 2.00 04/24/2014
[52671.706372] task: ffff88040d528790 ti: ffff88040d532000 task.ti: ffff88040d532000
[52671.706373] RIP: 0010:[<ffffffff812bf927>]  [<ffffffff812bf927>] sock_wfree+0x42/0x5b
[52671.706377] RSP: 0018:ffff88041fc03b18  EFLAGS: 00000202
[52671.706378] RAX: ffffffff812bf801 RBX: ffff88041fc10110 RCX: 0000000000000024
[52671.706379] RDX: ffff8802e0751610 RSI: 00000000ffffffff RDI: ffff88041fc0fe40
[52671.706380] RBP: 0000000000000300 R08: 0000000000000000 R09: 0000000000000000
[52671.706381] R10: ffff88041fc0ff44 R11: 0000000000000001 R12: ffff88041fc03a88
[52671.706382] R13: ffffffff8139705d R14: 0000000000000300 R15: ffff88041fc0fe40
[52671.706383] FS:  0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
[52671.706384] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[52671.706385] CR2: 00007fb866cc2140 CR3: 000000000160c000 CR4: 00000000001407f0
[52671.706386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[52671.706387] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[52671.706388] Stack:
[52671.706389]  ffff88040ac7b000 ffff88040ac7aa00 ffff88040ac7b000 ffffffff812f8747
[52671.706391]  ffff88041fc0fe40 ffffffff812fb0d8 ffff88041fc03b80 ffff88041fc03b78
[52671.706392]  ffffffff00000040 0000000000000286 ffffffff817dda00 ffffffff0214140a
[52671.706394] Call Trace:
[52671.706395]  <IRQ> 

[52671.706399]  [<ffffffff812f8747>] ? skb_orphan+0x12/0x27
[52671.706402]  [<ffffffff812fb0d8>] ? ip_send_unicast_reply+0x243/0x297
[52671.706406]  [<ffffffff810477bf>] ? mod_timer+0x7b/0x89
[52671.706409]  [<ffffffff813120c2>] ? tcp_v4_send_reset+0x2db/0x324
[52671.706411]  [<ffffffff81312e56>] ? tcp_v4_rcv+0x387/0x559
[52671.706413]  [<ffffffff812f6017>] ? __xfrm_policy_check2.constprop.9+0x50/0x50
[52671.706415]  [<ffffffff812f6117>] ? ip_local_deliver_finish+0x100/0x176
[52671.706418]  [<ffffffff812cd640>] ? __netif_receive_skb_core+0x447/0x4bf
[52671.706420]  [<ffffffff812cd893>] ? netif_receive_skb+0x4c/0x7d
[52671.706422]  [<ffffffff812ce013>] ? napi_gro_receive+0x35/0x76
[52671.706427]  [<ffffffffa04ab24c>] ? ixgbe_poll+0xbc9/0xe0a [ixgbe]
[52671.706429]  [<ffffffff812cddaa>] ? net_rx_action+0xa7/0x1e1
[52671.706431]  [<ffffffff8106442c>] ? account_system_time+0x113/0x12c
[52671.706433]  [<ffffffff81041683>] ? __do_softirq+0xf1/0x216
[52671.706436]  [<ffffffff8139781c>] ? call_softirq+0x1c/0x30
[52671.706436]  <EOI> 

[52671.706439]  [<ffffffff8100eade>] ? do_softirq+0x3a/0x78
[52671.706440]  [<ffffffff8104142e>] ? _local_bh_enable_ip.isra.11+0x6a/0x88
[52671.706443]  [<ffffffff810a2413>] ? rcu_nocb_kthread+0x25e/0x298
[52671.706445]  [<ffffffff810573e3>] ? abort_exclusive_wait+0x79/0x79
[52671.706447]  [<ffffffff810a21b5>] ? force_qs_rnp+0x120/0x120
[52671.706448]  [<ffffffff810a21b5>] ? force_qs_rnp+0x120/0x120
[52671.706450]  [<ffffffff81056a54>] ? kthread+0x81/0x89
[52671.706452]  [<ffffffff810126f9>] ? paravirt_read_tsc+0x5/0x8
[52671.706454]  [<ffffffff810569d3>] ? __kthread_parkme+0x5d/0x5d
[52671.706457]  [<ffffffff813963bc>] ? ret_from_fork+0x7c/0xb0
[52671.706458]  [<ffffffff810569d3>] ? __kthread_parkme+0x5d/0x5d
[52671.706459] Code: e6 ff ff 84 c0 75 1d 8d 7d ff bd 01 00 00 00 48 8d b3 04 01 00 00 e8 7d e3 ff ff 48 89 df ff 93 70 02 00 00 f0 29 ab 04 01 00 00 <40> 0f 94 c5 40 84 ed 74 0b 48 89 df 5b 5b 5d e9 94 fe ff ff 41