* [bisect] 3.18 oops in tcp_v4_send_reset()
@ 2014-12-09 16:00 dann frazier
2014-12-09 16:13 ` Daniel Borkmann
2014-12-09 16:16 ` Eric Dumazet
0 siblings, 2 replies; 9+ messages in thread
From: dann frazier @ 2014-12-09 16:00 UTC (permalink / raw)
To: netdev; +Cc: Eric Dumazet, Alexander Duyck, David S. Miller
I'm observing a very reproducible oops which I have bisected down to
commit ca777ef:
tcp: remove dst refcount false sharing for prequeue mode
I'm reproducing using the juju application, and this occurs when
tearing down a local lxc container (juju bootstrap/juju
destroy-environment local). Also worth noting that I'm on an
arm64 system. I'll follow up w/ results once I've attempted to
reproduce on x86, and if I'm able to create a simpler reproducer.
[ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018
[ 540.922254] pgd = ffffffc3ea9bb000
[ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000
[ 540.933902] Internal error: Oops: 96000006 [#1] SMP
[ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf
_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib
ahci xgene_enet
[ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65
[ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000
[ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4
[ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4
[ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145
[ 540.992897] sp : ffffffc3e6513a60
[ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600
[ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000
[ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8
[ 541.012099] x23: 0000000000000000 x22: 0000000000000000
[ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000
[ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d
[ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c
[ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a
[ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e
[ 541.043910] x11: 0000000000000000 x10: 0000000000000000
[ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db
[ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014
[ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000
[ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c
[ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac
[ 541.075722]
[ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058)
[ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000)
[ 541.089585] 3a60: e6513b20 ffffffc3 0069bc08 ffffffc0 e655e6e0 ffffffc3 e7a8c600 ffffffc3
[ 541.097720] 3a80: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
[ 541.105855] 3aa0: e7a8c690 ffffffc3 e7a8c600 ffffffc3 00000000 00000000 bccb9990 bf7ddf62
[ 541.113990] 3ac0: 00000000 00000450 00000000 00000000 00000000 00000000 00000000 00000000
[ 541.122124] 3ae0: e6513ab8 ffffffc3 00000014 00000000 00000000 02001afe 00000008 00000000
[ 541.130259] 3b00: 00000000 00000000 00628130 ffffffc0 e6513b40 ffffffc3 dc8cb000 cb88537f
[ 541.138394] 3b20: e6513b80 ffffffc3 00686ff4 ffffffc0 e7a8c600 ffffffc3 e7a8cb08 ffffffc3
[ 541.146528] 3b40: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
[ 541.154662] 3b60: 00000000 00000000 00628130 ffffffc0 e7a8c600 ffffffc3 00000000 00000000
[ 541.162797] 3b80: e6513ba0 ffffffc3 006880d0 ffffffc0 00000000 00000000 00000005 00000000
[ 541.170931] 3ba0: e6513c50 ffffffc3 006b1f14 ffffffc0 e6513d20 ffffffc3 e6513de8 ffffffc3
[ 541.179066] 3bc0: 00000000 00000000 e6513de8 ffffffc3 efc54f00 ffffffc3 00000005 00000000
[ 541.187200] 3be0: 00000119 00000000 0000003f 00000000 00ab8000 ffffffc0 e6510000 ffffffc3
[ 541.195335] 3c00: efc54f00 ffffffc3 0000003d 00000000 e6513ba0 ffffffc3 00000040 00000000
[ 541.203469] 3c20: e6513d20 ffffffc3 009be400 ffffffc0 92000007 00000000 fe45d400 ffffffc0
[ 541.211604] 3c40: eb2fc7e0 ffffffc3 ffffffff 7fffffff e6513ca0 ffffffc3 006241ac ffffffc0
[ 541.219738] 3c60: 00000005 00000000 e6513d20 ffffffc3 e6513ca0 ffffffc3 efc54f00 ffffffc3
[ 541.227873] 3c80: 00000005 00000000 ffffffff 00000000 e6513d20 ffffffc3 00000005 00000000
[ 541.236009] 3ca0: e6513d60 ffffffc3 0062421c ffffffc0 e6513de8 ffffffc3 e99e1d00 ffffffc3
[ 541.244144] 3cc0: 00000005 00000000 18006fe3 0000007f 80000000 00000000 00000015 00000000
[ 541.252278] 3ce0: e6513ec8 ffffffc3 e99e1d00 ffffffc3 0000003d 00000000 00000000 00000005
[ 541.260413] 3d00: efc54f00 ffffffc3 00000015 00000000 00000000 00000000 e6513d20 ffffffc3
[ 541.268547] 3d20: 00000000 00000000 00000000 ffffffc3 e6513dd8 ffffffc3 00000001 00000000
[ 541.276682] 3d40: 00000000 00000000 00000000 00000000 00000000 00000000 e6513de8 ffffffc3
[ 541.284817] 3d60: e6513da0 ffffffc3 002006c0 ffffffc0 e6513ec8 ffffffc3 00364ea0 ffffffc0
[ 541.292952] 3d80: e99e1d00 ffffffc3 e6513dd8 ffffffc3 00000001 00000000 00000000 00000000
[ 541.301086] 3da0: e6513e40 ffffffc3 00201294 ffffffc0 00000005 00000000 e99e1d00 ffffffc3
[ 541.309221] 3dc0: 18006fe3 0000007f e6513ec8 ffffffc3 e6513e00 ffffffc3 18006fe3 0000007f
[ 541.317355] 3de0: 00000005 00000000 e99e1d00 ffffffc3 00000000 00000000 00000000 00000000
[ 541.325490] 3e00: e6513ce8 ffffffc3 fe45d400 ffffffc0 00000000 00000000 00000000 00000000
[ 541.333624] 3e20: 00000005 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 541.341758] 3e40: e6513e80 ffffffc3 00201c38 ffffffc0 e99e1d01 ffffffc3 e99e1d00 ffffffc3
[ 541.349893] 3e60: ffffffff ffffffff b273b864 0000007f 80000000 00000000 b2737188 0000007f
[ 541.358027] 3e80: 841faef0 0000007f 0008425c ffffffc0 00000000 00000000 00000005 00000000
[ 541.366162] 3ea0: ffffffff ffffffff 00000001 00000000 18006fe3 0000007f 00000005 00000000
[ 541.374296] 3ec0: 00000000 00000000 00000000 00000000 00000024 00000000 18006fe3 0000007f
[ 541.382431] 3ee0: 00000005 00000000 841fbeb8 0000007f 841faeac 0000007f 841fc4a0 0000007f
[ 541.390565] 3f00: ffffffbb 00000000 00000000 00000000 0000003f 00000000 93ccf8ed 00e370ef
[ 541.398700] 3f20: 0000009e 00000000 00000070 00000000 93ccf8ed 000000ef 00000009 00000000
[ 541.406834] 3f40: 0000009b 00000000 00000095 00000000 00000000 00000000 b273b810 0000007f
[ 541.414968] 3f60: 000000aa 00000000 180016b0 0000007f 00000005 00000000 18006fe3 0000007f
[ 541.423103] 3f80: 00000005 00000000 00000005 00000000 18006fe3 0000007f 00000000 00000000
[ 541.431237] 3fa0: 841fc900 0000007f 0000feff 00000000 180012e0 0000007f 841faef0 0000007f
[ 541.439372] 3fc0: b273b84c 0000007f 841faee0 0000007f b273b864 0000007f 80000000 00000000
[ 541.447506] 3fe0: 00000024 00000000 0000003f 00000000 ed238e70 ffffffbe ed238ea8 ffffffbe
[ 541.455640] Call trace:
[ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4
[ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350
[ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc
[ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c
[ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4
[ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118
[ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74
[ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0
[ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8
[ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0
[ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80)
[ 541.518108] ---[ end trace 524a277a323ba5bd ]---
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier @ 2014-12-09 16:13 ` Daniel Borkmann 2014-12-09 16:16 ` Eric Dumazet 1 sibling, 0 replies; 9+ messages in thread From: Daniel Borkmann @ 2014-12-09 16:13 UTC (permalink / raw) To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On 12/09/2014 05:00 PM, dann frazier wrote: > I'm observing a very reproducible oops which I have bisected down to > commit ca777ef: > > tcp: remove dst refcount false sharing for prequeue mode > > I'm reproducing using the juju application, and this occurs when > tearing down a local lxc container (juju bootstrap/juju > destroy-environment local). Also worth noting that I'm on an > arm64 system. I'll follow up w/ results once I've attempted to > reproduce on x86, and if I'm able to create a simpler reproducer. It should be fixed in: https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3 Does your kernel include that commit? > [ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018 > [ 540.922254] pgd = ffffffc3ea9bb000 > [ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000 > [ 540.933902] Internal error: Oops: 96000006 [#1] SMP > [ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf > _conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib > ahci xgene_enet > [ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65 > [ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000 > [ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4 > [ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4 > [ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145 > [ 540.992897] sp : ffffffc3e6513a60 > [ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600 > [ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000 > [ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8 > [ 541.012099] x23: 0000000000000000 x22: 0000000000000000 > [ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000 > [ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d > [ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c > [ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a > [ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e > [ 541.043910] x11: 0000000000000000 x10: 0000000000000000 > [ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db > [ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014 > [ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000 > [ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c > [ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac > [ 541.075722] > [ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058) > [ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000) > [ 541.089585] 3a60: e6513b20 ffffffc3 0069bc08 ffffffc0 e655e6e0 ffffffc3 e7a8c600 ffffffc3 > [ 541.097720] 3a80: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3 > [ 541.105855] 3aa0: e7a8c690 ffffffc3 e7a8c600 ffffffc3 00000000 00000000 bccb9990 bf7ddf62 > [ 541.113990] 3ac0: 00000000 00000450 00000000 00000000 00000000 00000000 00000000 00000000 > [ 541.122124] 3ae0: e6513ab8 ffffffc3 00000014 00000000 00000000 02001afe 00000008 00000000 > [ 541.130259] 3b00: 00000000 00000000 00628130 ffffffc0 e6513b40 ffffffc3 dc8cb000 cb88537f > [ 541.138394] 3b20: e6513b80 ffffffc3 00686ff4 ffffffc0 e7a8c600 ffffffc3 e7a8cb08 ffffffc3 > [ 541.146528] 3b40: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3 > [ 541.154662] 3b60: 00000000 00000000 00628130 ffffffc0 e7a8c600 ffffffc3 00000000 00000000 > [ 541.162797] 3b80: e6513ba0 ffffffc3 006880d0 ffffffc0 00000000 00000000 00000005 00000000 > [ 541.170931] 3ba0: e6513c50 ffffffc3 006b1f14 ffffffc0 e6513d20 ffffffc3 e6513de8 ffffffc3 > [ 541.179066] 3bc0: 00000000 00000000 e6513de8 ffffffc3 efc54f00 ffffffc3 00000005 00000000 > [ 541.187200] 3be0: 00000119 00000000 0000003f 00000000 00ab8000 ffffffc0 e6510000 ffffffc3 > [ 541.195335] 3c00: efc54f00 ffffffc3 0000003d 00000000 e6513ba0 ffffffc3 00000040 00000000 > [ 541.203469] 3c20: e6513d20 ffffffc3 009be400 ffffffc0 92000007 00000000 fe45d400 ffffffc0 > [ 541.211604] 3c40: eb2fc7e0 ffffffc3 ffffffff 7fffffff e6513ca0 ffffffc3 006241ac ffffffc0 > [ 541.219738] 3c60: 00000005 00000000 e6513d20 ffffffc3 e6513ca0 ffffffc3 efc54f00 ffffffc3 > [ 541.227873] 3c80: 00000005 00000000 ffffffff 00000000 e6513d20 ffffffc3 00000005 00000000 > [ 541.236009] 3ca0: e6513d60 ffffffc3 0062421c ffffffc0 e6513de8 ffffffc3 e99e1d00 ffffffc3 > [ 541.244144] 3cc0: 00000005 00000000 18006fe3 0000007f 80000000 00000000 00000015 00000000 > [ 541.252278] 3ce0: e6513ec8 ffffffc3 e99e1d00 ffffffc3 0000003d 00000000 00000000 00000005 > [ 541.260413] 3d00: efc54f00 ffffffc3 00000015 00000000 00000000 00000000 e6513d20 ffffffc3 > [ 541.268547] 3d20: 00000000 00000000 00000000 ffffffc3 e6513dd8 ffffffc3 00000001 00000000 > [ 541.276682] 3d40: 00000000 00000000 00000000 00000000 00000000 00000000 e6513de8 ffffffc3 > [ 541.284817] 3d60: e6513da0 ffffffc3 002006c0 ffffffc0 e6513ec8 ffffffc3 00364ea0 ffffffc0 > [ 541.292952] 3d80: e99e1d00 ffffffc3 e6513dd8 ffffffc3 00000001 00000000 00000000 00000000 > [ 541.301086] 3da0: e6513e40 ffffffc3 00201294 ffffffc0 00000005 00000000 e99e1d00 ffffffc3 > [ 541.309221] 3dc0: 18006fe3 0000007f e6513ec8 ffffffc3 e6513e00 ffffffc3 18006fe3 0000007f > [ 541.317355] 3de0: 00000005 00000000 e99e1d00 ffffffc3 00000000 00000000 00000000 00000000 > [ 541.325490] 3e00: e6513ce8 ffffffc3 fe45d400 ffffffc0 00000000 00000000 00000000 00000000 > [ 541.333624] 3e20: 00000005 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > [ 541.341758] 3e40: e6513e80 ffffffc3 00201c38 ffffffc0 e99e1d01 ffffffc3 e99e1d00 ffffffc3 > [ 541.349893] 3e60: ffffffff ffffffff b273b864 0000007f 80000000 00000000 b2737188 0000007f > [ 541.358027] 3e80: 841faef0 0000007f 0008425c ffffffc0 00000000 00000000 00000005 00000000 > [ 541.366162] 3ea0: ffffffff ffffffff 00000001 00000000 18006fe3 0000007f 00000005 00000000 > [ 541.374296] 3ec0: 00000000 00000000 00000000 00000000 00000024 00000000 18006fe3 0000007f > [ 541.382431] 3ee0: 00000005 00000000 841fbeb8 0000007f 841faeac 0000007f 841fc4a0 0000007f > [ 541.390565] 3f00: ffffffbb 00000000 00000000 00000000 0000003f 00000000 93ccf8ed 00e370ef > [ 541.398700] 3f20: 0000009e 00000000 00000070 00000000 93ccf8ed 000000ef 00000009 00000000 > [ 541.406834] 3f40: 0000009b 00000000 00000095 00000000 00000000 00000000 b273b810 0000007f > [ 541.414968] 3f60: 000000aa 00000000 180016b0 0000007f 00000005 00000000 18006fe3 0000007f > [ 541.423103] 3f80: 00000005 00000000 00000005 00000000 18006fe3 0000007f 00000000 00000000 > [ 541.431237] 3fa0: 841fc900 0000007f 0000feff 00000000 180012e0 0000007f 841faef0 0000007f > [ 541.439372] 3fc0: b273b84c 0000007f 841faee0 0000007f b273b864 0000007f 80000000 00000000 > [ 541.447506] 3fe0: 00000024 00000000 0000003f 00000000 ed238e70 ffffffbe ed238ea8 ffffffbe > [ 541.455640] Call trace: > [ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4 > [ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350 > [ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc > [ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c > [ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4 > [ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118 > [ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74 > [ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0 > [ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8 > [ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0 > [ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80) > [ 541.518108] ---[ end trace 524a277a323ba5bd ]--- > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier 2014-12-09 16:13 ` Daniel Borkmann @ 2014-12-09 16:16 ` Eric Dumazet 2014-12-09 16:18 ` Eric Dumazet 1 sibling, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2014-12-09 16:16 UTC (permalink / raw) To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote: > I'm observing a very reproducible oops which I have bisected down to > commit ca777ef: > > tcp: remove dst refcount false sharing for prequeue mode > > I'm reproducing using the juju application, and this occurs when > tearing down a local lxc container (juju bootstrap/juju > destroy-environment local). Also worth noting that I'm on an > arm64 system. I'll follow up w/ results once I've attempted to > reproduce on x86, and if I'm able to create a simpler reproducer. > > [ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018 > [ 540.922254] pgd = ffffffc3ea9bb000 > [ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000 > [ 540.933902] Internal error: Oops: 96000006 [#1] SMP > [ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf > _conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib > ahci xgene_enet > [ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65 > [ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000 > [ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4 > [ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4 > [ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145 > [ 540.992897] sp : ffffffc3e6513a60 > [ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600 > [ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000 > [ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8 > [ 541.012099] x23: 0000000000000000 x22: 0000000000000000 > [ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000 > [ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d > [ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c > [ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a > [ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e > [ 541.043910] x11: 0000000000000000 x10: 0000000000000000 > [ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db > [ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014 > [ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000 > [ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c > [ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac > [ 541.075722] > [ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058) > [ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000) > [ 541.455640] Call trace: > [ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4 > [ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350 > [ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc > [ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c > [ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4 > [ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118 > [ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74 > [ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0 > [ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8 > [ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0 > [ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80) > [ 541.518108] ---[ end trace 524a277a323ba5bd ]--- > -- Following patch should have fixed this http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 16:16 ` Eric Dumazet @ 2014-12-09 16:18 ` Eric Dumazet 2014-12-09 16:28 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2014-12-09 16:18 UTC (permalink / raw) To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote: > On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote: > > I'm observing a very reproducible oops which I have bisected down to > > commit ca777ef: > > Following patch should have fixed this > > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3 > Oh well, fix was not complete. I'll submit a followup patch. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 16:18 ` Eric Dumazet @ 2014-12-09 16:28 ` Eric Dumazet 2014-12-09 17:10 ` Dann Frazier 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2014-12-09 16:28 UTC (permalink / raw) To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On Tue, 2014-12-09 at 08:18 -0800, Eric Dumazet wrote: > On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote: > > On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote: > > > I'm observing a very reproducible oops which I have bisected down to > > > commit ca777ef: > > > > > Following patch should have fixed this > > > > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3 > > > > Oh well, fix was not complete. I'll submit a followup patch. > Could you try following fix before I send official patch ? Thanks ! diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 33f5ff068c7958515e0f63792883a58fb5d6a341..a3f72d7fc06c07c43e1c00b67970eaee074e4593 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) arg.iov[0].iov_base = (unsigned char *)&rep; arg.iov[0].iov_len = sizeof(rep.th); + net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev); #ifdef CONFIG_TCP_MD5SIG hash_location = tcp_parse_md5sig_option(th); if (!sk && hash_location) { @@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) * Incoming packet is checked with md5 hash with finding key, * no RST generated if md5 hash doesn't match. */ - sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev), + sk1 = __inet_lookup_listener(net, &tcp_hashinfo, ip_hdr(skb)->saddr, th->source, ip_hdr(skb)->daddr, ntohs(th->source), inet_iif(skb)); @@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) if (sk) arg.bound_dev_if = sk->sk_bound_dev_if; - net = dev_net(skb_dst(skb)->dev); arg.tos = ip_hdr(skb)->tos; ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 16:28 ` Eric Dumazet @ 2014-12-09 17:10 ` Dann Frazier 2014-12-09 17:23 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Dann Frazier @ 2014-12-09 17:10 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On Tue, Dec 9, 2014 at 9:28 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2014-12-09 at 08:18 -0800, Eric Dumazet wrote: >> On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote: >> > On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote: >> > > I'm observing a very reproducible oops which I have bisected down to >> > > commit ca777ef: >> >> > >> > Following patch should have fixed this >> > >> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3 >> > >> >> Oh well, fix was not complete. I'll submit a followup patch. >> > > Could you try following fix before I send official patch ? This patch does appear to resolve the issue, thanks Eric! -dann > Thanks ! > > diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c > index 33f5ff068c7958515e0f63792883a58fb5d6a341..a3f72d7fc06c07c43e1c00b67970eaee074e4593 100644 > --- a/net/ipv4/tcp_ipv4.c > +++ b/net/ipv4/tcp_ipv4.c > @@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) > arg.iov[0].iov_base = (unsigned char *)&rep; > arg.iov[0].iov_len = sizeof(rep.th); > > + net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev); > #ifdef CONFIG_TCP_MD5SIG > hash_location = tcp_parse_md5sig_option(th); > if (!sk && hash_location) { > @@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) > * Incoming packet is checked with md5 hash with finding key, > * no RST generated if md5 hash doesn't match. > */ > - sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev), > + sk1 = __inet_lookup_listener(net, > &tcp_hashinfo, ip_hdr(skb)->saddr, > th->source, ip_hdr(skb)->daddr, > ntohs(th->source), inet_iif(skb)); > @@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) > if (sk) > arg.bound_dev_if = sk->sk_bound_dev_if; > > - net = dev_net(skb_dst(skb)->dev); > arg.tos = ip_hdr(skb)->tos; > ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt, > ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset() 2014-12-09 17:10 ` Dann Frazier @ 2014-12-09 17:23 ` Eric Dumazet 2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2014-12-09 17:23 UTC (permalink / raw) To: Dann Frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller On Tue, 2014-12-09 at 10:10 -0700, Dann Frazier wrote: > This patch does appear to resolve the issue, thanks Eric! Thanks Dann I need to cook a proper patch including ipv6 changes. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH net] tcp: fix more NULL deref after prequeue changes 2014-12-09 17:23 ` Eric Dumazet @ 2014-12-09 17:56 ` Eric Dumazet 2014-12-10 2:39 ` David Miller 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2014-12-09 17:56 UTC (permalink / raw) To: Dann Frazier, David S. Miller; +Cc: netdev From: Eric Dumazet <edumazet@google.com> When I cooked commit c3658e8d0f1 ("tcp: fix possible NULL dereference in tcp_vX_send_reset()") I missed other spots we could deref a NULL skb_dst(skb) Again, if a socket is provided, we do not need skb_dst() to get a pointer to network namespace : sock_net(sk) is good enough. Reported-by: Dann Frazier <dann.frazier@canonical.com> Bisected-by: Dann Frazier <dann.frazier@canonical.com> Tested-by: Dann Frazier <dann.frazier@canonical.com> Signed-off-by: Eric Dumazet <edumazet@google.com> Fixes: ca777eff51f7 ("tcp: remove dst refcount false sharing for prequeue mode") --- net/ipv4/tcp_ipv4.c | 4 ++-- net/ipv6/tcp_ipv6.c | 28 ++++++++++++++-------------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 147be2024290..ef7089ca86e2 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) arg.iov[0].iov_base = (unsigned char *)&rep; arg.iov[0].iov_len = sizeof(rep.th); + net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev); #ifdef CONFIG_TCP_MD5SIG hash_location = tcp_parse_md5sig_option(th); if (!sk && hash_location) { @@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) * Incoming packet is checked with md5 hash with finding key, * no RST generated if md5 hash doesn't match. */ - sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev), + sk1 = __inet_lookup_listener(net, &tcp_hashinfo, ip_hdr(skb)->saddr, th->source, ip_hdr(skb)->daddr, ntohs(th->source), inet_iif(skb)); @@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb) if (sk) arg.bound_dev_if = sk->sk_bound_dev_if; - net = dev_net(skb_dst(skb)->dev); arg.tos = ip_hdr(skb)->tos; ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index dc495ae2ead0..c277951d783b 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -787,16 +787,16 @@ static const struct tcp_request_sock_ops tcp_request_sock_ipv6_ops = { .queue_hash_add = inet6_csk_reqsk_queue_hash_add, }; -static void tcp_v6_send_response(struct sk_buff *skb, u32 seq, u32 ack, u32 win, - u32 tsval, u32 tsecr, int oif, - struct tcp_md5sig_key *key, int rst, u8 tclass, - u32 label) +static void tcp_v6_send_response(struct sock *sk, struct sk_buff *skb, u32 seq, + u32 ack, u32 win, u32 tsval, u32 tsecr, + int oif, struct tcp_md5sig_key *key, int rst, + u8 tclass, u32 label) { const struct tcphdr *th = tcp_hdr(skb); struct tcphdr *t1; struct sk_buff *buff; struct flowi6 fl6; - struct net *net = dev_net(skb_dst(skb)->dev); + struct net *net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev); struct sock *ctl_sk = net->ipv6.tcp_sk; unsigned int tot_len = sizeof(struct tcphdr); struct dst_entry *dst; @@ -946,7 +946,7 @@ static void tcp_v6_send_reset(struct sock *sk, struct sk_buff *skb) (th->doff << 2); oif = sk ? sk->sk_bound_dev_if : 0; - tcp_v6_send_response(skb, seq, ack_seq, 0, 0, 0, oif, key, 1, 0, 0); + tcp_v6_send_response(sk, skb, seq, ack_seq, 0, 0, 0, oif, key, 1, 0, 0); #ifdef CONFIG_TCP_MD5SIG release_sk1: @@ -957,13 +957,13 @@ release_sk1: #endif } -static void tcp_v6_send_ack(struct sk_buff *skb, u32 seq, u32 ack, - u32 win, u32 tsval, u32 tsecr, int oif, +static void tcp_v6_send_ack(struct sock *sk, struct sk_buff *skb, u32 seq, + u32 ack, u32 win, u32 tsval, u32 tsecr, int oif, struct tcp_md5sig_key *key, u8 tclass, u32 label) { - tcp_v6_send_response(skb, seq, ack, win, tsval, tsecr, oif, key, 0, tclass, - label); + tcp_v6_send_response(sk, skb, seq, ack, win, tsval, tsecr, oif, key, 0, + tclass, label); } static void tcp_v6_timewait_ack(struct sock *sk, struct sk_buff *skb) @@ -971,7 +971,7 @@ static void tcp_v6_timewait_ack(struct sock *sk, struct sk_buff *skb) struct inet_timewait_sock *tw = inet_twsk(sk); struct tcp_timewait_sock *tcptw = tcp_twsk(sk); - tcp_v6_send_ack(skb, tcptw->tw_snd_nxt, tcptw->tw_rcv_nxt, + tcp_v6_send_ack(sk, skb, tcptw->tw_snd_nxt, tcptw->tw_rcv_nxt, tcptw->tw_rcv_wnd >> tw->tw_rcv_wscale, tcp_time_stamp + tcptw->tw_ts_offset, tcptw->tw_ts_recent, tw->tw_bound_dev_if, tcp_twsk_md5_key(tcptw), @@ -986,10 +986,10 @@ static void tcp_v6_reqsk_send_ack(struct sock *sk, struct sk_buff *skb, /* sk->sk_state == TCP_LISTEN -> for regular TCP_SYN_RECV * sk->sk_state == TCP_SYN_RECV -> for Fast Open. */ - tcp_v6_send_ack(skb, (sk->sk_state == TCP_LISTEN) ? + tcp_v6_send_ack(sk, skb, (sk->sk_state == TCP_LISTEN) ? tcp_rsk(req)->snt_isn + 1 : tcp_sk(sk)->snd_nxt, - tcp_rsk(req)->rcv_nxt, - req->rcv_wnd, tcp_time_stamp, req->ts_recent, sk->sk_bound_dev_if, + tcp_rsk(req)->rcv_nxt, req->rcv_wnd, + tcp_time_stamp, req->ts_recent, sk->sk_bound_dev_if, tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->daddr), 0, 0); } ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH net] tcp: fix more NULL deref after prequeue changes 2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet @ 2014-12-10 2:39 ` David Miller 0 siblings, 0 replies; 9+ messages in thread From: David Miller @ 2014-12-10 2:39 UTC (permalink / raw) To: eric.dumazet; +Cc: dann.frazier, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 09 Dec 2014 09:56:08 -0800 > From: Eric Dumazet <edumazet@google.com> > > When I cooked commit c3658e8d0f1 ("tcp: fix possible NULL dereference in > tcp_vX_send_reset()") I missed other spots we could deref a NULL > skb_dst(skb) > > Again, if a socket is provided, we do not need skb_dst() to get a > pointer to network namespace : sock_net(sk) is good enough. > > Reported-by: Dann Frazier <dann.frazier@canonical.com> > Bisected-by: Dann Frazier <dann.frazier@canonical.com> > Tested-by: Dann Frazier <dann.frazier@canonical.com> > Signed-off-by: Eric Dumazet <edumazet@google.com> > Fixes: ca777eff51f7 ("tcp: remove dst refcount false sharing for prequeue mode") Applied and queued up for 3.18-stable. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-12-10 2:39 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier 2014-12-09 16:13 ` Daniel Borkmann 2014-12-09 16:16 ` Eric Dumazet 2014-12-09 16:18 ` Eric Dumazet 2014-12-09 16:28 ` Eric Dumazet 2014-12-09 17:10 ` Dann Frazier 2014-12-09 17:23 ` Eric Dumazet 2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet 2014-12-10 2:39 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox