* [bisect] 3.18 oops in tcp_v4_send_reset()
@ 2014-12-09 16:00 dann frazier
2014-12-09 16:13 ` Daniel Borkmann
2014-12-09 16:16 ` Eric Dumazet
0 siblings, 2 replies; 9+ messages in thread
From: dann frazier @ 2014-12-09 16:00 UTC (permalink / raw)
To: netdev; +Cc: Eric Dumazet, Alexander Duyck, David S. Miller
I'm observing a very reproducible oops which I have bisected down to
commit ca777ef:
tcp: remove dst refcount false sharing for prequeue mode
I'm reproducing using the juju application, and this occurs when
tearing down a local lxc container (juju bootstrap/juju
destroy-environment local). Also worth noting that I'm on an
arm64 system. I'll follow up w/ results once I've attempted to
reproduce on x86, and if I'm able to create a simpler reproducer.
[ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018
[ 540.922254] pgd = ffffffc3ea9bb000
[ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000
[ 540.933902] Internal error: Oops: 96000006 [#1] SMP
[ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf
_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib
ahci xgene_enet
[ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65
[ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000
[ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4
[ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4
[ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145
[ 540.992897] sp : ffffffc3e6513a60
[ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600
[ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000
[ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8
[ 541.012099] x23: 0000000000000000 x22: 0000000000000000
[ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000
[ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d
[ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c
[ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a
[ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e
[ 541.043910] x11: 0000000000000000 x10: 0000000000000000
[ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db
[ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014
[ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000
[ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c
[ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac
[ 541.075722]
[ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058)
[ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000)
[ 541.089585] 3a60: e6513b20 ffffffc3 0069bc08 ffffffc0 e655e6e0 ffffffc3 e7a8c600 ffffffc3
[ 541.097720] 3a80: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
[ 541.105855] 3aa0: e7a8c690 ffffffc3 e7a8c600 ffffffc3 00000000 00000000 bccb9990 bf7ddf62
[ 541.113990] 3ac0: 00000000 00000450 00000000 00000000 00000000 00000000 00000000 00000000
[ 541.122124] 3ae0: e6513ab8 ffffffc3 00000014 00000000 00000000 02001afe 00000008 00000000
[ 541.130259] 3b00: 00000000 00000000 00628130 ffffffc0 e6513b40 ffffffc3 dc8cb000 cb88537f
[ 541.138394] 3b20: e6513b80 ffffffc3 00686ff4 ffffffc0 e7a8c600 ffffffc3 e7a8cb08 ffffffc3
[ 541.146528] 3b40: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
[ 541.154662] 3b60: 00000000 00000000 00628130 ffffffc0 e7a8c600 ffffffc3 00000000 00000000
[ 541.162797] 3b80: e6513ba0 ffffffc3 006880d0 ffffffc0 00000000 00000000 00000005 00000000
[ 541.170931] 3ba0: e6513c50 ffffffc3 006b1f14 ffffffc0 e6513d20 ffffffc3 e6513de8 ffffffc3
[ 541.179066] 3bc0: 00000000 00000000 e6513de8 ffffffc3 efc54f00 ffffffc3 00000005 00000000
[ 541.187200] 3be0: 00000119 00000000 0000003f 00000000 00ab8000 ffffffc0 e6510000 ffffffc3
[ 541.195335] 3c00: efc54f00 ffffffc3 0000003d 00000000 e6513ba0 ffffffc3 00000040 00000000
[ 541.203469] 3c20: e6513d20 ffffffc3 009be400 ffffffc0 92000007 00000000 fe45d400 ffffffc0
[ 541.211604] 3c40: eb2fc7e0 ffffffc3 ffffffff 7fffffff e6513ca0 ffffffc3 006241ac ffffffc0
[ 541.219738] 3c60: 00000005 00000000 e6513d20 ffffffc3 e6513ca0 ffffffc3 efc54f00 ffffffc3
[ 541.227873] 3c80: 00000005 00000000 ffffffff 00000000 e6513d20 ffffffc3 00000005 00000000
[ 541.236009] 3ca0: e6513d60 ffffffc3 0062421c ffffffc0 e6513de8 ffffffc3 e99e1d00 ffffffc3
[ 541.244144] 3cc0: 00000005 00000000 18006fe3 0000007f 80000000 00000000 00000015 00000000
[ 541.252278] 3ce0: e6513ec8 ffffffc3 e99e1d00 ffffffc3 0000003d 00000000 00000000 00000005
[ 541.260413] 3d00: efc54f00 ffffffc3 00000015 00000000 00000000 00000000 e6513d20 ffffffc3
[ 541.268547] 3d20: 00000000 00000000 00000000 ffffffc3 e6513dd8 ffffffc3 00000001 00000000
[ 541.276682] 3d40: 00000000 00000000 00000000 00000000 00000000 00000000 e6513de8 ffffffc3
[ 541.284817] 3d60: e6513da0 ffffffc3 002006c0 ffffffc0 e6513ec8 ffffffc3 00364ea0 ffffffc0
[ 541.292952] 3d80: e99e1d00 ffffffc3 e6513dd8 ffffffc3 00000001 00000000 00000000 00000000
[ 541.301086] 3da0: e6513e40 ffffffc3 00201294 ffffffc0 00000005 00000000 e99e1d00 ffffffc3
[ 541.309221] 3dc0: 18006fe3 0000007f e6513ec8 ffffffc3 e6513e00 ffffffc3 18006fe3 0000007f
[ 541.317355] 3de0: 00000005 00000000 e99e1d00 ffffffc3 00000000 00000000 00000000 00000000
[ 541.325490] 3e00: e6513ce8 ffffffc3 fe45d400 ffffffc0 00000000 00000000 00000000 00000000
[ 541.333624] 3e20: 00000005 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 541.341758] 3e40: e6513e80 ffffffc3 00201c38 ffffffc0 e99e1d01 ffffffc3 e99e1d00 ffffffc3
[ 541.349893] 3e60: ffffffff ffffffff b273b864 0000007f 80000000 00000000 b2737188 0000007f
[ 541.358027] 3e80: 841faef0 0000007f 0008425c ffffffc0 00000000 00000000 00000005 00000000
[ 541.366162] 3ea0: ffffffff ffffffff 00000001 00000000 18006fe3 0000007f 00000005 00000000
[ 541.374296] 3ec0: 00000000 00000000 00000000 00000000 00000024 00000000 18006fe3 0000007f
[ 541.382431] 3ee0: 00000005 00000000 841fbeb8 0000007f 841faeac 0000007f 841fc4a0 0000007f
[ 541.390565] 3f00: ffffffbb 00000000 00000000 00000000 0000003f 00000000 93ccf8ed 00e370ef
[ 541.398700] 3f20: 0000009e 00000000 00000070 00000000 93ccf8ed 000000ef 00000009 00000000
[ 541.406834] 3f40: 0000009b 00000000 00000095 00000000 00000000 00000000 b273b810 0000007f
[ 541.414968] 3f60: 000000aa 00000000 180016b0 0000007f 00000005 00000000 18006fe3 0000007f
[ 541.423103] 3f80: 00000005 00000000 00000005 00000000 18006fe3 0000007f 00000000 00000000
[ 541.431237] 3fa0: 841fc900 0000007f 0000feff 00000000 180012e0 0000007f 841faef0 0000007f
[ 541.439372] 3fc0: b273b84c 0000007f 841faee0 0000007f b273b864 0000007f 80000000 00000000
[ 541.447506] 3fe0: 00000024 00000000 0000003f 00000000 ed238e70 ffffffbe ed238ea8 ffffffbe
[ 541.455640] Call trace:
[ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4
[ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350
[ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc
[ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c
[ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4
[ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118
[ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74
[ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0
[ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8
[ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0
[ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80)
[ 541.518108] ---[ end trace 524a277a323ba5bd ]---
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier
@ 2014-12-09 16:13 ` Daniel Borkmann
2014-12-09 16:16 ` Eric Dumazet
1 sibling, 0 replies; 9+ messages in thread
From: Daniel Borkmann @ 2014-12-09 16:13 UTC (permalink / raw)
To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On 12/09/2014 05:00 PM, dann frazier wrote:
> I'm observing a very reproducible oops which I have bisected down to
> commit ca777ef:
>
> tcp: remove dst refcount false sharing for prequeue mode
>
> I'm reproducing using the juju application, and this occurs when
> tearing down a local lxc container (juju bootstrap/juju
> destroy-environment local). Also worth noting that I'm on an
> arm64 system. I'll follow up w/ results once I've attempted to
> reproduce on x86, and if I'm able to create a simpler reproducer.
It should be fixed in:
https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3
Does your kernel include that commit?
> [ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018
> [ 540.922254] pgd = ffffffc3ea9bb000
> [ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000
> [ 540.933902] Internal error: Oops: 96000006 [#1] SMP
> [ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf
> _conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib
> ahci xgene_enet
> [ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65
> [ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000
> [ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4
> [ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4
> [ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145
> [ 540.992897] sp : ffffffc3e6513a60
> [ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600
> [ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000
> [ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8
> [ 541.012099] x23: 0000000000000000 x22: 0000000000000000
> [ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000
> [ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d
> [ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c
> [ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a
> [ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e
> [ 541.043910] x11: 0000000000000000 x10: 0000000000000000
> [ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db
> [ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014
> [ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000
> [ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c
> [ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac
> [ 541.075722]
> [ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058)
> [ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000)
> [ 541.089585] 3a60: e6513b20 ffffffc3 0069bc08 ffffffc0 e655e6e0 ffffffc3 e7a8c600 ffffffc3
> [ 541.097720] 3a80: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
> [ 541.105855] 3aa0: e7a8c690 ffffffc3 e7a8c600 ffffffc3 00000000 00000000 bccb9990 bf7ddf62
> [ 541.113990] 3ac0: 00000000 00000450 00000000 00000000 00000000 00000000 00000000 00000000
> [ 541.122124] 3ae0: e6513ab8 ffffffc3 00000014 00000000 00000000 02001afe 00000008 00000000
> [ 541.130259] 3b00: 00000000 00000000 00628130 ffffffc0 e6513b40 ffffffc3 dc8cb000 cb88537f
> [ 541.138394] 3b20: e6513b80 ffffffc3 00686ff4 ffffffc0 e7a8c600 ffffffc3 e7a8cb08 ffffffc3
> [ 541.146528] 3b40: 00000000 00000000 00000001 00000000 009be3c0 ffffffc0 e7a8cacc ffffffc3
> [ 541.154662] 3b60: 00000000 00000000 00628130 ffffffc0 e7a8c600 ffffffc3 00000000 00000000
> [ 541.162797] 3b80: e6513ba0 ffffffc3 006880d0 ffffffc0 00000000 00000000 00000005 00000000
> [ 541.170931] 3ba0: e6513c50 ffffffc3 006b1f14 ffffffc0 e6513d20 ffffffc3 e6513de8 ffffffc3
> [ 541.179066] 3bc0: 00000000 00000000 e6513de8 ffffffc3 efc54f00 ffffffc3 00000005 00000000
> [ 541.187200] 3be0: 00000119 00000000 0000003f 00000000 00ab8000 ffffffc0 e6510000 ffffffc3
> [ 541.195335] 3c00: efc54f00 ffffffc3 0000003d 00000000 e6513ba0 ffffffc3 00000040 00000000
> [ 541.203469] 3c20: e6513d20 ffffffc3 009be400 ffffffc0 92000007 00000000 fe45d400 ffffffc0
> [ 541.211604] 3c40: eb2fc7e0 ffffffc3 ffffffff 7fffffff e6513ca0 ffffffc3 006241ac ffffffc0
> [ 541.219738] 3c60: 00000005 00000000 e6513d20 ffffffc3 e6513ca0 ffffffc3 efc54f00 ffffffc3
> [ 541.227873] 3c80: 00000005 00000000 ffffffff 00000000 e6513d20 ffffffc3 00000005 00000000
> [ 541.236009] 3ca0: e6513d60 ffffffc3 0062421c ffffffc0 e6513de8 ffffffc3 e99e1d00 ffffffc3
> [ 541.244144] 3cc0: 00000005 00000000 18006fe3 0000007f 80000000 00000000 00000015 00000000
> [ 541.252278] 3ce0: e6513ec8 ffffffc3 e99e1d00 ffffffc3 0000003d 00000000 00000000 00000005
> [ 541.260413] 3d00: efc54f00 ffffffc3 00000015 00000000 00000000 00000000 e6513d20 ffffffc3
> [ 541.268547] 3d20: 00000000 00000000 00000000 ffffffc3 e6513dd8 ffffffc3 00000001 00000000
> [ 541.276682] 3d40: 00000000 00000000 00000000 00000000 00000000 00000000 e6513de8 ffffffc3
> [ 541.284817] 3d60: e6513da0 ffffffc3 002006c0 ffffffc0 e6513ec8 ffffffc3 00364ea0 ffffffc0
> [ 541.292952] 3d80: e99e1d00 ffffffc3 e6513dd8 ffffffc3 00000001 00000000 00000000 00000000
> [ 541.301086] 3da0: e6513e40 ffffffc3 00201294 ffffffc0 00000005 00000000 e99e1d00 ffffffc3
> [ 541.309221] 3dc0: 18006fe3 0000007f e6513ec8 ffffffc3 e6513e00 ffffffc3 18006fe3 0000007f
> [ 541.317355] 3de0: 00000005 00000000 e99e1d00 ffffffc3 00000000 00000000 00000000 00000000
> [ 541.325490] 3e00: e6513ce8 ffffffc3 fe45d400 ffffffc0 00000000 00000000 00000000 00000000
> [ 541.333624] 3e20: 00000005 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [ 541.341758] 3e40: e6513e80 ffffffc3 00201c38 ffffffc0 e99e1d01 ffffffc3 e99e1d00 ffffffc3
> [ 541.349893] 3e60: ffffffff ffffffff b273b864 0000007f 80000000 00000000 b2737188 0000007f
> [ 541.358027] 3e80: 841faef0 0000007f 0008425c ffffffc0 00000000 00000000 00000005 00000000
> [ 541.366162] 3ea0: ffffffff ffffffff 00000001 00000000 18006fe3 0000007f 00000005 00000000
> [ 541.374296] 3ec0: 00000000 00000000 00000000 00000000 00000024 00000000 18006fe3 0000007f
> [ 541.382431] 3ee0: 00000005 00000000 841fbeb8 0000007f 841faeac 0000007f 841fc4a0 0000007f
> [ 541.390565] 3f00: ffffffbb 00000000 00000000 00000000 0000003f 00000000 93ccf8ed 00e370ef
> [ 541.398700] 3f20: 0000009e 00000000 00000070 00000000 93ccf8ed 000000ef 00000009 00000000
> [ 541.406834] 3f40: 0000009b 00000000 00000095 00000000 00000000 00000000 b273b810 0000007f
> [ 541.414968] 3f60: 000000aa 00000000 180016b0 0000007f 00000005 00000000 18006fe3 0000007f
> [ 541.423103] 3f80: 00000005 00000000 00000005 00000000 18006fe3 0000007f 00000000 00000000
> [ 541.431237] 3fa0: 841fc900 0000007f 0000feff 00000000 180012e0 0000007f 841faef0 0000007f
> [ 541.439372] 3fc0: b273b84c 0000007f 841faee0 0000007f b273b864 0000007f 80000000 00000000
> [ 541.447506] 3fe0: 00000024 00000000 0000003f 00000000 ed238e70 ffffffbe ed238ea8 ffffffbe
> [ 541.455640] Call trace:
> [ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4
> [ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350
> [ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc
> [ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c
> [ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4
> [ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118
> [ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74
> [ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0
> [ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8
> [ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0
> [ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80)
> [ 541.518108] ---[ end trace 524a277a323ba5bd ]---
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier
2014-12-09 16:13 ` Daniel Borkmann
@ 2014-12-09 16:16 ` Eric Dumazet
2014-12-09 16:18 ` Eric Dumazet
1 sibling, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-12-09 16:16 UTC (permalink / raw)
To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote:
> I'm observing a very reproducible oops which I have bisected down to
> commit ca777ef:
>
> tcp: remove dst refcount false sharing for prequeue mode
>
> I'm reproducing using the juju application, and this occurs when
> tearing down a local lxc container (juju bootstrap/juju
> destroy-environment local). Also worth noting that I'm on an
> arm64 system. I'll follow up w/ results once I've attempted to
> reproduce on x86, and if I'm able to create a simpler reproducer.
>
> [ 540.914174] Unable to handle kernel NULL pointer dereference at virtual address 00000018
> [ 540.922254] pgd = ffffffc3ea9bb000
> [ 540.925646] [00000018] *pgd=00000043e7bfb003, *pud=00000043e7bfb003, *pmd=0000000000000000
> [ 540.933902] Internal error: Oops: 96000006 [#1] SMP
> [ 540.938754] Modules linked in: veth xt_CHECKSUM xt_tcpudp iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf
> _conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc ip_tables x_tables ahci_xgene libahci_platform lib
> ahci xgene_enet
> [ 540.962592] CPU: 4 PID: 2788 Comm: mongod Not tainted 3.18.0 #65
> [ 540.968566] task: ffffffc0fe45d400 ti: ffffffc3e6510000 task.ti: ffffffc3e6510000
> [ 540.976014] PC is at tcp_v4_send_reset+0x2ec/0x3e4
> [ 540.980778] LR is at tcp_v4_send_reset+0x3c8/0x3e4
> [ 540.985542] pc : [<ffffffc00069b5dc>] lr : [<ffffffc00069b6b8>] pstate: 80000145
> [ 540.992897] sp : ffffffc3e6513a60
> [ 540.996192] x29: ffffffc3e6513a60 x28: ffffffc3e7a8c600
> [ 541.001494] x27: 0000000000000000 x26: ffffffc3e6510000
> [ 541.006796] x25: 0000000000000000 x24: ffffffc3e6513ab8
> [ 541.012099] x23: 0000000000000000 x22: 0000000000000000
> [ 541.017401] x21: ffffffc3e7a8c600 x20: ffffffc000b65000
> [ 541.022703] x19: ffffffc3e655e6e0 x18: 000000000000000d
> [ 541.028005] x17: 0000007fb2735e10 x16: ffffffc00012052c
> [ 541.033306] x15: 0000007fb2728590 x14: 282039363638333a
> [ 541.038608] x13: 0000000062df7dbf x12: 206e6f697463656e
> [ 541.043910] x11: 0000000000000000 x10: 0000000000000000
> [ 541.049212] x9 : 00000000000012d1 x8 : 00000000000346db
> [ 541.054515] x7 : 0000000000000018 x6 : 0000000000000014
> [ 541.059817] x5 : ffffffc3e6513ae0 x4 : 0000000000000000
> [ 541.065118] x3 : ffffffc0fe6d70ac x2 : ffffffc3e655e71c
> [ 541.070420] x1 : ffffffc3e655e6e0 x0 : 00000000000000ac
> [ 541.075722]
> [ 541.077202] Process mongod (pid: 2788, stack limit = 0xffffffc3e6510058)
> [ 541.083868] Stack: (0xffffffc3e6513a60 to 0xffffffc3e6514000)
> [ 541.455640] Call trace:
> [ 541.458074] [<ffffffc00069b5dc>] tcp_v4_send_reset+0x2ec/0x3e4
> [ 541.463877] [<ffffffc00069bc04>] tcp_v4_do_rcv+0xfc/0x350
> [ 541.469247] [<ffffffc000686ff0>] tcp_prequeue_process+0x98/0xdc
> [ 541.475134] [<ffffffc0006880cc>] tcp_recvmsg+0x4c8/0xa0c
> [ 541.480419] [<ffffffc0006b1f10>] inet_recvmsg+0x98/0xb4
> [ 541.485618] [<ffffffc0006241a8>] sock_aio_read.part.12+0xf0/0x118
> [ 541.491679] [<ffffffc000624218>] sock_aio_read+0x48/0x74
> [ 541.496964] [<ffffffc0002006bc>] do_sync_read+0x8c/0xd0
> [ 541.502161] [<ffffffc000201290>] vfs_read+0x128/0x1a8
> [ 541.507185] [<ffffffc000201c34>] SyS_read+0x50/0xb0
> [ 541.512037] Code: 927ff884 b9408ba6 910203a5 8b000063 (f9400c80)
> [ 541.518108] ---[ end trace 524a277a323ba5bd ]---
> --
Following patch should have fixed this
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 16:16 ` Eric Dumazet
@ 2014-12-09 16:18 ` Eric Dumazet
2014-12-09 16:28 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-12-09 16:18 UTC (permalink / raw)
To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote:
> On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote:
> > I'm observing a very reproducible oops which I have bisected down to
> > commit ca777ef:
>
> Following patch should have fixed this
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3
>
Oh well, fix was not complete. I'll submit a followup patch.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 16:18 ` Eric Dumazet
@ 2014-12-09 16:28 ` Eric Dumazet
2014-12-09 17:10 ` Dann Frazier
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-12-09 16:28 UTC (permalink / raw)
To: dann frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On Tue, 2014-12-09 at 08:18 -0800, Eric Dumazet wrote:
> On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote:
> > On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote:
> > > I'm observing a very reproducible oops which I have bisected down to
> > > commit ca777ef:
>
> >
> > Following patch should have fixed this
> >
> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3
> >
>
> Oh well, fix was not complete. I'll submit a followup patch.
>
Could you try following fix before I send official patch ?
Thanks !
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 33f5ff068c7958515e0f63792883a58fb5d6a341..a3f72d7fc06c07c43e1c00b67970eaee074e4593 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
arg.iov[0].iov_base = (unsigned char *)&rep;
arg.iov[0].iov_len = sizeof(rep.th);
+ net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
#ifdef CONFIG_TCP_MD5SIG
hash_location = tcp_parse_md5sig_option(th);
if (!sk && hash_location) {
@@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
* Incoming packet is checked with md5 hash with finding key,
* no RST generated if md5 hash doesn't match.
*/
- sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev),
+ sk1 = __inet_lookup_listener(net,
&tcp_hashinfo, ip_hdr(skb)->saddr,
th->source, ip_hdr(skb)->daddr,
ntohs(th->source), inet_iif(skb));
@@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
if (sk)
arg.bound_dev_if = sk->sk_bound_dev_if;
- net = dev_net(skb_dst(skb)->dev);
arg.tos = ip_hdr(skb)->tos;
ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt,
ip_hdr(skb)->saddr, ip_hdr(skb)->daddr,
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 16:28 ` Eric Dumazet
@ 2014-12-09 17:10 ` Dann Frazier
2014-12-09 17:23 ` Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Dann Frazier @ 2014-12-09 17:10 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On Tue, Dec 9, 2014 at 9:28 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Tue, 2014-12-09 at 08:18 -0800, Eric Dumazet wrote:
>> On Tue, 2014-12-09 at 08:16 -0800, Eric Dumazet wrote:
>> > On Tue, 2014-12-09 at 09:00 -0700, dann frazier wrote:
>> > > I'm observing a very reproducible oops which I have bisected down to
>> > > commit ca777ef:
>>
>> >
>> > Following patch should have fixed this
>> >
>> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c3658e8d0f10147fc86018be7f11668246c156d3
>> >
>>
>> Oh well, fix was not complete. I'll submit a followup patch.
>>
>
> Could you try following fix before I send official patch ?
This patch does appear to resolve the issue, thanks Eric!
-dann
> Thanks !
>
> diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
> index 33f5ff068c7958515e0f63792883a58fb5d6a341..a3f72d7fc06c07c43e1c00b67970eaee074e4593 100644
> --- a/net/ipv4/tcp_ipv4.c
> +++ b/net/ipv4/tcp_ipv4.c
> @@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
> arg.iov[0].iov_base = (unsigned char *)&rep;
> arg.iov[0].iov_len = sizeof(rep.th);
>
> + net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
> #ifdef CONFIG_TCP_MD5SIG
> hash_location = tcp_parse_md5sig_option(th);
> if (!sk && hash_location) {
> @@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
> * Incoming packet is checked with md5 hash with finding key,
> * no RST generated if md5 hash doesn't match.
> */
> - sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev),
> + sk1 = __inet_lookup_listener(net,
> &tcp_hashinfo, ip_hdr(skb)->saddr,
> th->source, ip_hdr(skb)->daddr,
> ntohs(th->source), inet_iif(skb));
> @@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
> if (sk)
> arg.bound_dev_if = sk->sk_bound_dev_if;
>
> - net = dev_net(skb_dst(skb)->dev);
> arg.tos = ip_hdr(skb)->tos;
> ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt,
> ip_hdr(skb)->saddr, ip_hdr(skb)->daddr,
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [bisect] 3.18 oops in tcp_v4_send_reset()
2014-12-09 17:10 ` Dann Frazier
@ 2014-12-09 17:23 ` Eric Dumazet
2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-12-09 17:23 UTC (permalink / raw)
To: Dann Frazier; +Cc: netdev, Eric Dumazet, Alexander Duyck, David S. Miller
On Tue, 2014-12-09 at 10:10 -0700, Dann Frazier wrote:
> This patch does appear to resolve the issue, thanks Eric!
Thanks Dann
I need to cook a proper patch including ipv6 changes.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH net] tcp: fix more NULL deref after prequeue changes
2014-12-09 17:23 ` Eric Dumazet
@ 2014-12-09 17:56 ` Eric Dumazet
2014-12-10 2:39 ` David Miller
0 siblings, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2014-12-09 17:56 UTC (permalink / raw)
To: Dann Frazier, David S. Miller; +Cc: netdev
From: Eric Dumazet <edumazet@google.com>
When I cooked commit c3658e8d0f1 ("tcp: fix possible NULL dereference in
tcp_vX_send_reset()") I missed other spots we could deref a NULL
skb_dst(skb)
Again, if a socket is provided, we do not need skb_dst() to get a
pointer to network namespace : sock_net(sk) is good enough.
Reported-by: Dann Frazier <dann.frazier@canonical.com>
Bisected-by: Dann Frazier <dann.frazier@canonical.com>
Tested-by: Dann Frazier <dann.frazier@canonical.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Fixes: ca777eff51f7 ("tcp: remove dst refcount false sharing for prequeue mode")
---
net/ipv4/tcp_ipv4.c | 4 ++--
net/ipv6/tcp_ipv6.c | 28 ++++++++++++++--------------
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 147be2024290..ef7089ca86e2 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -623,6 +623,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
arg.iov[0].iov_base = (unsigned char *)&rep;
arg.iov[0].iov_len = sizeof(rep.th);
+ net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
#ifdef CONFIG_TCP_MD5SIG
hash_location = tcp_parse_md5sig_option(th);
if (!sk && hash_location) {
@@ -633,7 +634,7 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
* Incoming packet is checked with md5 hash with finding key,
* no RST generated if md5 hash doesn't match.
*/
- sk1 = __inet_lookup_listener(dev_net(skb_dst(skb)->dev),
+ sk1 = __inet_lookup_listener(net,
&tcp_hashinfo, ip_hdr(skb)->saddr,
th->source, ip_hdr(skb)->daddr,
ntohs(th->source), inet_iif(skb));
@@ -681,7 +682,6 @@ static void tcp_v4_send_reset(struct sock *sk, struct sk_buff *skb)
if (sk)
arg.bound_dev_if = sk->sk_bound_dev_if;
- net = dev_net(skb_dst(skb)->dev);
arg.tos = ip_hdr(skb)->tos;
ip_send_unicast_reply(net, skb, &TCP_SKB_CB(skb)->header.h4.opt,
ip_hdr(skb)->saddr, ip_hdr(skb)->daddr,
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index dc495ae2ead0..c277951d783b 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -787,16 +787,16 @@ static const struct tcp_request_sock_ops tcp_request_sock_ipv6_ops = {
.queue_hash_add = inet6_csk_reqsk_queue_hash_add,
};
-static void tcp_v6_send_response(struct sk_buff *skb, u32 seq, u32 ack, u32 win,
- u32 tsval, u32 tsecr, int oif,
- struct tcp_md5sig_key *key, int rst, u8 tclass,
- u32 label)
+static void tcp_v6_send_response(struct sock *sk, struct sk_buff *skb, u32 seq,
+ u32 ack, u32 win, u32 tsval, u32 tsecr,
+ int oif, struct tcp_md5sig_key *key, int rst,
+ u8 tclass, u32 label)
{
const struct tcphdr *th = tcp_hdr(skb);
struct tcphdr *t1;
struct sk_buff *buff;
struct flowi6 fl6;
- struct net *net = dev_net(skb_dst(skb)->dev);
+ struct net *net = sk ? sock_net(sk) : dev_net(skb_dst(skb)->dev);
struct sock *ctl_sk = net->ipv6.tcp_sk;
unsigned int tot_len = sizeof(struct tcphdr);
struct dst_entry *dst;
@@ -946,7 +946,7 @@ static void tcp_v6_send_reset(struct sock *sk, struct sk_buff *skb)
(th->doff << 2);
oif = sk ? sk->sk_bound_dev_if : 0;
- tcp_v6_send_response(skb, seq, ack_seq, 0, 0, 0, oif, key, 1, 0, 0);
+ tcp_v6_send_response(sk, skb, seq, ack_seq, 0, 0, 0, oif, key, 1, 0, 0);
#ifdef CONFIG_TCP_MD5SIG
release_sk1:
@@ -957,13 +957,13 @@ release_sk1:
#endif
}
-static void tcp_v6_send_ack(struct sk_buff *skb, u32 seq, u32 ack,
- u32 win, u32 tsval, u32 tsecr, int oif,
+static void tcp_v6_send_ack(struct sock *sk, struct sk_buff *skb, u32 seq,
+ u32 ack, u32 win, u32 tsval, u32 tsecr, int oif,
struct tcp_md5sig_key *key, u8 tclass,
u32 label)
{
- tcp_v6_send_response(skb, seq, ack, win, tsval, tsecr, oif, key, 0, tclass,
- label);
+ tcp_v6_send_response(sk, skb, seq, ack, win, tsval, tsecr, oif, key, 0,
+ tclass, label);
}
static void tcp_v6_timewait_ack(struct sock *sk, struct sk_buff *skb)
@@ -971,7 +971,7 @@ static void tcp_v6_timewait_ack(struct sock *sk, struct sk_buff *skb)
struct inet_timewait_sock *tw = inet_twsk(sk);
struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
- tcp_v6_send_ack(skb, tcptw->tw_snd_nxt, tcptw->tw_rcv_nxt,
+ tcp_v6_send_ack(sk, skb, tcptw->tw_snd_nxt, tcptw->tw_rcv_nxt,
tcptw->tw_rcv_wnd >> tw->tw_rcv_wscale,
tcp_time_stamp + tcptw->tw_ts_offset,
tcptw->tw_ts_recent, tw->tw_bound_dev_if, tcp_twsk_md5_key(tcptw),
@@ -986,10 +986,10 @@ static void tcp_v6_reqsk_send_ack(struct sock *sk, struct sk_buff *skb,
/* sk->sk_state == TCP_LISTEN -> for regular TCP_SYN_RECV
* sk->sk_state == TCP_SYN_RECV -> for Fast Open.
*/
- tcp_v6_send_ack(skb, (sk->sk_state == TCP_LISTEN) ?
+ tcp_v6_send_ack(sk, skb, (sk->sk_state == TCP_LISTEN) ?
tcp_rsk(req)->snt_isn + 1 : tcp_sk(sk)->snd_nxt,
- tcp_rsk(req)->rcv_nxt,
- req->rcv_wnd, tcp_time_stamp, req->ts_recent, sk->sk_bound_dev_if,
+ tcp_rsk(req)->rcv_nxt, req->rcv_wnd,
+ tcp_time_stamp, req->ts_recent, sk->sk_bound_dev_if,
tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->daddr),
0, 0);
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH net] tcp: fix more NULL deref after prequeue changes
2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet
@ 2014-12-10 2:39 ` David Miller
0 siblings, 0 replies; 9+ messages in thread
From: David Miller @ 2014-12-10 2:39 UTC (permalink / raw)
To: eric.dumazet; +Cc: dann.frazier, netdev
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 09 Dec 2014 09:56:08 -0800
> From: Eric Dumazet <edumazet@google.com>
>
> When I cooked commit c3658e8d0f1 ("tcp: fix possible NULL dereference in
> tcp_vX_send_reset()") I missed other spots we could deref a NULL
> skb_dst(skb)
>
> Again, if a socket is provided, we do not need skb_dst() to get a
> pointer to network namespace : sock_net(sk) is good enough.
>
> Reported-by: Dann Frazier <dann.frazier@canonical.com>
> Bisected-by: Dann Frazier <dann.frazier@canonical.com>
> Tested-by: Dann Frazier <dann.frazier@canonical.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Fixes: ca777eff51f7 ("tcp: remove dst refcount false sharing for prequeue mode")
Applied and queued up for 3.18-stable.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-12-10 2:39 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-12-09 16:00 [bisect] 3.18 oops in tcp_v4_send_reset() dann frazier
2014-12-09 16:13 ` Daniel Borkmann
2014-12-09 16:16 ` Eric Dumazet
2014-12-09 16:18 ` Eric Dumazet
2014-12-09 16:28 ` Eric Dumazet
2014-12-09 17:10 ` Dann Frazier
2014-12-09 17:23 ` Eric Dumazet
2014-12-09 17:56 ` [PATCH net] tcp: fix more NULL deref after prequeue changes Eric Dumazet
2014-12-10 2:39 ` David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox