Kernel access of bad area

Kernel access of bad area

[Tamtamis, Panagiotis]

[-- Attachment #1: Type: text/plain, Size: 4052 bytes --]

Hello to all,

We had the following dump at netfilter code:


Unable to handle kernel paging request for data at address 0x7fe3fb80
Faulting instruction address: 0xf15a69b4
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 OCC
Modules linked in: nf_conntrack_netlink pppoe pppox ppp_generic slhc
ppp_drv(O) msp(O) xt_nat iptable_raw xt_CT xt_mark xt_DSCP ipt_MASQUERADE
iptable_nat nf_nat_ipv4 nf_nat xt_limit xt_TCPMSS iptable_mangle
nfnetlink_queue nfnetlink_log nfnetlink httpk(O) nf_conntrack_ipv6
nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4
nf_defrag_ipv4 xt_pkttype xt_conntrack nf_conntrack iptable_filter ip_tables
x_tables i2c_dev tun drv_vxt(O) avmfritz mISDN_isac mISDN_l1 falc_e1
drv_tapi(O) tlani_fpga mISDN_core drv_ifxos(O) cma_card iomengine cma_mfc
tdmswitch bmod_scc lsb hdlc_scc scc_core icgx common_irq board_control akse
CPU: 1 PID: 17461 Comm: kworker/u4:0 Tainted: G           O 3.12.19-rt30 #1
Workqueue: DSP_MGMT dsp_rsrv [msp]
task: db91df80 ti: efb5e000 task.ti: d52ec000
NIP: f15a69b4 LR: f15a6958 CTR: c0037aa0
REGS: efb5fbf0 TRAP: 0300   Tainted: G           O  (3.12.19-rt30)
MSR: 00029000 <CE,EE,ME>  CR: 28ef2324  XER: 00000000
DEAR: 7fe3fb80, ESR: 00000000

GPR00: 00000000 efb5fca0 db91df80 c0771960 c078d988 efb5e000 0000c37c
c078d988 
GPR08: 0005b524 c0a88368 00cd5000 7fe3fb78 c0037aa0 00000000 d3cfdc00
c5782b40 
GPR16: 00000020 ef4da618 00000001 000086dd 00000000 c0772cc0 80000000
c37d7a2d 
GPR24: 00000014 f15b3e28 00000000 f15e5980 000030df efb5fcf4 7fe3fb78
c0771960 
NIP [f15a69b4] ____nf_conntrack_find+0x88/0x1a8 [nf_conntrack]
LR [f15a6958] ____nf_conntrack_find+0x2c/0x1a8 [nf_conntrack]
Call Trace:
[efb5fca0] [00000015] 0x15 (unreliable)
[efb5fcc0] [f15a6b14] __nf_conntrack_find_get+0x40/0x198 [nf_conntrack]
[efb5fce0] [f15a8b54] nf_conntrack_in+0x384/0x700 [nf_conntrack]
[efb5fd50] [f15e42ac] ipv4_conntrack_in+0x24/0x34 [nf_conntrack_ipv4]
[efb5fd60] [c047e364] nf_iterate+0x98/0xfc
[efb5fd90] [c047e43c] nf_hook_slow+0x74/0x158
[efb5fdd0] [c04857b4] ip_rcv+0x388/0x4f0
[efb5fe00] [c044ed3c] __netif_receive_skb_core+0x504/0x6d0
[efb5fe60] [c0450f64] netif_receive_skb+0x3c/0xd0
[efb5fe90] [c0453d88] napi_gro_receive+0xb4/0xec
[efb5fea0] [c0378e24] gfar_process_frame+0xac/0x188
[efb5fed0] [c037b08c] gfar_clean_rx_ring+0x194/0x46c
[efb5ff40] [c037b3a4] gfar_poll_rx_sq+0x40/0xac
[efb5ff60] [c0451310] net_rx_action+0x110/0x1c8
[efb5ff90] [c0037dd0] __do_softirq+0x10c/0x1c8
[efb5fff0] [c000e444] call_do_softirq+0x24/0x3c
[efb5dfb0] [c000494c] do_softirq+0x8c/0xb4
[efb5dfd0] [c0038790] irq_exit+0x7c/0x90
[efb5dfe0] [c0004624] __do_irq+0x4c/0x94
[efb5dff0] [c000e480] call_do_irq+0x24/0x3c
[d52edc10] [c00046f8] do_IRQ+0x8c/0xe0
[d52edc30] [c0010880] ret_from_except+0x0/0x18
--- Exception: 501 at vprintk_emit+0x1f0/0x4ac
    LR = vprintk_emit+0x254/0x4ac
[d52edd40] [c0563648] printk+0x68/0x78
[d52edd80] [f2728980] fEvent+0x218/0x230 [msp]
[d52ede00] [f277230c] dspCheckCommonMessage+0x32c/0x9c8 [msp]
[d52ede40] [f275bc84] dsp_rsrv+0x188/0x274 [msp]
[d52edea0] [c004b894] process_one_work+0x120/0x348
[d52edec0] [c004c658] worker_thread+0xf0/0x2c4
[d52edef0] [c0052dfc] kthread+0x98/0x9c
[d52edf40] [c00102fc] ret_from_kernel_thread+0x5c/0x64
--- Exception: 0 at   (null)
    LR =   (null)
Instruction dump:
80050008 813f047c 5400103a 7d47002e 7d09502e 38080001 7c09512e 816b0000 
71600001 408200f0 813d0000 7d7e5b78 <800b0008> 7f890000 409effc8 813d0004 
---[ end trace 7536e18f69aeba2e ]---


From what I have seen this part of code has not been changed a lot since
latest kernel version.
The crash seems to be at 

static inline int nf_inet_addr_cmp(const union nf_inet_addr *a1,
                                    const union nf_inet_addr *a2)
 {
         return a1->all[0] == a2->all[0] &&
                a1->all[1] == a2->all[1] &&
                a1->all[2] == a2->all[2] &&
                a1->all[3] == a2->all[3];
 }


Is it possible a race condition to exist somewhere in the code and not all
parts be protected with locks?

Thanks,
Tamis


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4874 bytes --]

RE: Kernel access of bad area

[Tamtamis, Panagiotis]

[-- Attachment #1: Type: text/plain, Size: 4982 bytes --]

The kernel version is 3.12.19-rt30

At that particular part of code the only addition was function
"nf_ct_key_equal" which just tests "nf_ct_is_confirmed(ct)"

But from the dump the crash seems to be on "nf_ct_tuple_equal" before even
" nf_ct_is_confirmed" is called.

-----Original Message-----
From: Feng Gao [mailto:gfree.wind@outlook.com] 
Sent: Thursday, September 17, 2015 12:06 PM
To: Tamtamis, Panagiotis; netfilter@vger.kernel.org
Subject: 答复: Kernel access of bad area

You should show your kernel version.
I remember there was one similar bug in netfilter codes before. 

-----邮件原件-----
发件人: netfilter-owner@vger.kernel.org [mailto:netfilter-owner@vger.kernel.
org] 代表 Tamtamis, Panagiotis
发送时间: 2015年9月17日 16:59
收件人: netfilter@vger.kernel.org
主题: Kernel access of bad area

Hello to all,

We had the following dump at netfilter code:


Unable to handle kernel paging request for data at address 0x7fe3fb80
Faulting instruction address: 0xf15a69b4
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 OCC
Modules linked in: nf_conntrack_netlink pppoe pppox ppp_generic slhc
ppp_drv(O) msp(O) xt_nat iptable_raw xt_CT xt_mark xt_DSCP ipt_MASQUERADE
iptable_nat nf_nat_ipv4 nf_nat xt_limit xt_TCPMSS iptable_mangle
nfnetlink_queue nfnetlink_log nfnetlink httpk(O) nf_conntrack_ipv6
nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4
nf_defrag_ipv4 xt_pkttype xt_conntrack nf_conntrack iptable_filter ip_tables
x_tables i2c_dev tun drv_vxt(O) avmfritz mISDN_isac mISDN_l1 falc_e1
drv_tapi(O) tlani_fpga mISDN_core drv_ifxos(O) cma_card iomengine cma_mfc
tdmswitch bmod_scc lsb hdlc_scc scc_core icgx common_irq board_control akse
CPU: 1 PID: 17461 Comm: kworker/u4:0 Tainted: G           O 3.12.19-rt30 #1
Workqueue: DSP_MGMT dsp_rsrv [msp]
task: db91df80 ti: efb5e000 task.ti: d52ec000
NIP: f15a69b4 LR: f15a6958 CTR: c0037aa0
REGS: efb5fbf0 TRAP: 0300   Tainted: G           O  (3.12.19-rt30)
MSR: 00029000 <CE,EE,ME>  CR: 28ef2324  XER: 00000000
DEAR: 7fe3fb80, ESR: 00000000

GPR00: 00000000 efb5fca0 db91df80 c0771960 c078d988 efb5e000 0000c37c
c078d988 
GPR08: 0005b524 c0a88368 00cd5000 7fe3fb78 c0037aa0 00000000 d3cfdc00
c5782b40 
GPR16: 00000020 ef4da618 00000001 000086dd 00000000 c0772cc0 80000000
c37d7a2d 
GPR24: 00000014 f15b3e28 00000000 f15e5980 000030df efb5fcf4 7fe3fb78
c0771960 
NIP [f15a69b4] ____nf_conntrack_find+0x88/0x1a8 [nf_conntrack]
LR [f15a6958] ____nf_conntrack_find+0x2c/0x1a8 [nf_conntrack]
Call Trace:
[efb5fca0] [00000015] 0x15 (unreliable)
[efb5fcc0] [f15a6b14] __nf_conntrack_find_get+0x40/0x198 [nf_conntrack]
[efb5fce0] [f15a8b54] nf_conntrack_in+0x384/0x700 [nf_conntrack]
[efb5fd50] [f15e42ac] ipv4_conntrack_in+0x24/0x34 [nf_conntrack_ipv4]
[efb5fd60] [c047e364] nf_iterate+0x98/0xfc
[efb5fd90] [c047e43c] nf_hook_slow+0x74/0x158
[efb5fdd0] [c04857b4] ip_rcv+0x388/0x4f0
[efb5fe00] [c044ed3c] __netif_receive_skb_core+0x504/0x6d0
[efb5fe60] [c0450f64] netif_receive_skb+0x3c/0xd0
[efb5fe90] [c0453d88] napi_gro_receive+0xb4/0xec
[efb5fea0] [c0378e24] gfar_process_frame+0xac/0x188
[efb5fed0] [c037b08c] gfar_clean_rx_ring+0x194/0x46c
[efb5ff40] [c037b3a4] gfar_poll_rx_sq+0x40/0xac
[efb5ff60] [c0451310] net_rx_action+0x110/0x1c8
[efb5ff90] [c0037dd0] __do_softirq+0x10c/0x1c8
[efb5fff0] [c000e444] call_do_softirq+0x24/0x3c
[efb5dfb0] [c000494c] do_softirq+0x8c/0xb4
[efb5dfd0] [c0038790] irq_exit+0x7c/0x90
[efb5dfe0] [c0004624] __do_irq+0x4c/0x94
[efb5dff0] [c000e480] call_do_irq+0x24/0x3c
[d52edc10] [c00046f8] do_IRQ+0x8c/0xe0
[d52edc30] [c0010880] ret_from_except+0x0/0x18
--- Exception: 501 at vprintk_emit+0x1f0/0x4ac
    LR = vprintk_emit+0x254/0x4ac
[d52edd40] [c0563648] printk+0x68/0x78
[d52edd80] [f2728980] fEvent+0x218/0x230 [msp]
[d52ede00] [f277230c] dspCheckCommonMessage+0x32c/0x9c8 [msp]
[d52ede40] [f275bc84] dsp_rsrv+0x188/0x274 [msp]
[d52edea0] [c004b894] process_one_work+0x120/0x348
[d52edec0] [c004c658] worker_thread+0xf0/0x2c4
[d52edef0] [c0052dfc] kthread+0x98/0x9c
[d52edf40] [c00102fc] ret_from_kernel_thread+0x5c/0x64
--- Exception: 0 at   (null)
    LR =   (null)
Instruction dump:
80050008 813f047c 5400103a 7d47002e 7d09502e 38080001 7c09512e 816b0000 
71600001 408200f0 813d0000 7d7e5b78 <800b0008> 7f890000 409effc8 813d0004 
---[ end trace 7536e18f69aeba2e ]---


From what I have seen this part of code has not been changed a lot since
latest kernel version.
The crash seems to be at 

static inline int nf_inet_addr_cmp(const union nf_inet_addr *a1,
                                    const union nf_inet_addr *a2)
 {
         return a1->all[0] == a2->all[0] &&
                a1->all[1] == a2->all[1] &&
                a1->all[2] == a2->all[2] &&
                a1->all[3] == a2->all[3];
 }


Is it possible a race condition to exist somewhere in the code and not all
parts be protected with locks?

Thanks,
Tamis


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4874 bytes --]

Re: Kernel access of bad area

[Pablo Neira Ayuso]

On Thu, Sep 17, 2015 at 08:58:55AM +0000, Tamtamis, Panagiotis wrote:
> Hello to all,
> 
> We had the following dump at netfilter code:
> 
> 
> Unable to handle kernel paging request for data at address 0x7fe3fb80
> Faulting instruction address: 0xf15a69b4
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=2 OCC
> Modules linked in: nf_conntrack_netlink pppoe pppox ppp_generic slhc
> ppp_drv(O) msp(O) xt_nat iptable_raw xt_CT xt_mark xt_DSCP ipt_MASQUERADE
> iptable_nat nf_nat_ipv4 nf_nat xt_limit xt_TCPMSS iptable_mangle
> nfnetlink_queue nfnetlink_log nfnetlink httpk(O) nf_conntrack_ipv6
> nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4
> nf_defrag_ipv4 xt_pkttype xt_conntrack nf_conntrack iptable_filter ip_tables
> x_tables i2c_dev tun drv_vxt(O) avmfritz mISDN_isac mISDN_l1 falc_e1
> drv_tapi(O) tlani_fpga mISDN_core drv_ifxos(O) cma_card iomengine cma_mfc
> tdmswitch bmod_scc lsb hdlc_scc scc_core icgx common_irq board_control akse
> CPU: 1 PID: 17461 Comm: kworker/u4:0 Tainted: G           O 3.12.19-rt30 #1
> Workqueue: DSP_MGMT dsp_rsrv [msp]
> task: db91df80 ti: efb5e000 task.ti: d52ec000
> NIP: f15a69b4 LR: f15a6958 CTR: c0037aa0
> REGS: efb5fbf0 TRAP: 0300   Tainted: G           O  (3.12.19-rt30)
> MSR: 00029000 <CE,EE,ME>  CR: 28ef2324  XER: 00000000
> DEAR: 7fe3fb80, ESR: 00000000
> 
> GPR00: 00000000 efb5fca0 db91df80 c0771960 c078d988 efb5e000 0000c37c
> c078d988 
> GPR08: 0005b524 c0a88368 00cd5000 7fe3fb78 c0037aa0 00000000 d3cfdc00
> c5782b40 
> GPR16: 00000020 ef4da618 00000001 000086dd 00000000 c0772cc0 80000000
> c37d7a2d 
> GPR24: 00000014 f15b3e28 00000000 f15e5980 000030df efb5fcf4 7fe3fb78
> c0771960 
> NIP [f15a69b4] ____nf_conntrack_find+0x88/0x1a8 [nf_conntrack]
> LR [f15a6958] ____nf_conntrack_find+0x2c/0x1a8 [nf_conntrack]
> Call Trace:
> [efb5fca0] [00000015] 0x15 (unreliable)
> [efb5fcc0] [f15a6b14] __nf_conntrack_find_get+0x40/0x198 [nf_conntrack]
> [efb5fce0] [f15a8b54] nf_conntrack_in+0x384/0x700 [nf_conntrack]
> [efb5fd50] [f15e42ac] ipv4_conntrack_in+0x24/0x34 [nf_conntrack_ipv4]
> [efb5fd60] [c047e364] nf_iterate+0x98/0xfc
> [efb5fd90] [c047e43c] nf_hook_slow+0x74/0x158
> [efb5fdd0] [c04857b4] ip_rcv+0x388/0x4f0

I've requested submission of these patches to -stable quite recently:

http://patchwork.ozlabs.org/patch/516773/
http://patchwork.ozlabs.org/patch/516772/

RE: Kernel access of bad area

[Tamtamis, Panagiotis]

Thanks a lot for your input.
I will merge those patches with our current version of kernel.
It seems a good solution candidate.

-----Original Message-----
From: Pablo Neira Ayuso [mailto:pablo@netfilter.org]
Sent: Thursday, September 17, 2015 1:52 PM
To: Tamtamis, Panagiotis
Cc: netfilter@vger.kernel.org
Subject: Re: Kernel access of bad area

On Thu, Sep 17, 2015 at 08:58:55AM +0000, Tamtamis, Panagiotis wrote:
> Hello to all,
>
> We had the following dump at netfilter code:
>
>
> Unable to handle kernel paging request for data at address 0x7fe3fb80
> Faulting instruction address: 0xf15a69b4
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=2 OCC
> Modules linked in: nf_conntrack_netlink pppoe pppox ppp_generic slhc
> ppp_drv(O) msp(O) xt_nat iptable_raw xt_CT xt_mark xt_DSCP ipt_MASQUERADE
> iptable_nat nf_nat_ipv4 nf_nat xt_limit xt_TCPMSS iptable_mangle
> nfnetlink_queue nfnetlink_log nfnetlink httpk(O) nf_conntrack_ipv6
> nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4
> nf_defrag_ipv4 xt_pkttype xt_conntrack nf_conntrack iptable_filter ip_tables
> x_tables i2c_dev tun drv_vxt(O) avmfritz mISDN_isac mISDN_l1 falc_e1
> drv_tapi(O) tlani_fpga mISDN_core drv_ifxos(O) cma_card iomengine cma_mfc
> tdmswitch bmod_scc lsb hdlc_scc scc_core icgx common_irq board_control akse
> CPU: 1 PID: 17461 Comm: kworker/u4:0 Tainted: G           O 3.12.19-rt30 #1
> Workqueue: DSP_MGMT dsp_rsrv [msp]
> task: db91df80 ti: efb5e000 task.ti: d52ec000
> NIP: f15a69b4 LR: f15a6958 CTR: c0037aa0
> REGS: efb5fbf0 TRAP: 0300   Tainted: G           O  (3.12.19-rt30)
> MSR: 00029000 <CE,EE,ME>  CR: 28ef2324  XER: 00000000
> DEAR: 7fe3fb80, ESR: 00000000
>
> GPR00: 00000000 efb5fca0 db91df80 c0771960 c078d988 efb5e000 0000c37c
> c078d988
> GPR08: 0005b524 c0a88368 00cd5000 7fe3fb78 c0037aa0 00000000 d3cfdc00
> c5782b40
> GPR16: 00000020 ef4da618 00000001 000086dd 00000000 c0772cc0 80000000
> c37d7a2d
> GPR24: 00000014 f15b3e28 00000000 f15e5980 000030df efb5fcf4 7fe3fb78
> c0771960
> NIP [f15a69b4] ____nf_conntrack_find+0x88/0x1a8 [nf_conntrack]
> LR [f15a6958] ____nf_conntrack_find+0x2c/0x1a8 [nf_conntrack]
> Call Trace:
> [efb5fca0] [00000015] 0x15 (unreliable)
> [efb5fcc0] [f15a6b14] __nf_conntrack_find_get+0x40/0x198 [nf_conntrack]
> [efb5fce0] [f15a8b54] nf_conntrack_in+0x384/0x700 [nf_conntrack]
> [efb5fd50] [f15e42ac] ipv4_conntrack_in+0x24/0x34 [nf_conntrack_ipv4]
> [efb5fd60] [c047e364] nf_iterate+0x98/0xfc
> [efb5fd90] [c047e43c] nf_hook_slow+0x74/0x158
> [efb5fdd0] [c04857b4] ip_rcv+0x388/0x4f0

I've requested submission of these patches to -stable quite recently:

http://patchwork.ozlabs.org/patch/516773/
http://patchwork.ozlabs.org/patch/516772/