* Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
@ 2014-05-06 15:16 Stephen Hemminger
2014-05-06 22:30 ` Cong Wang
0 siblings, 1 reply; 5+ messages in thread
From: Stephen Hemminger @ 2014-05-06 15:16 UTC (permalink / raw)
To: netdev
Begin forwarded message:
Date: Mon, 5 May 2014 23:48:49 -0700
From: "bugzilla-daemon@bugzilla.kernel.org" <bugzilla-daemon@bugzilla.kernel.org>
To: "stephen@networkplumber.org" <stephen@networkplumber.org>
Subject: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
https://bugzilla.kernel.org/show_bug.cgi?id=75571
Bug ID: 75571
Summary: using tc action mirred results BUG at
net/core/skbuff.c:119
Product: Networking
Version: 2.5
Kernel Version: 3.2.58
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: shemminger@linux-foundation.org
Reporter: d77190@mail.ru
Regression: No
Hello!
Got this bug while setting large shaper ruleset (about 4k hash rules) on mirred
ifb interface at boot time:
Attached archive consists of used shaper scripts.
[ 19.355181] ------------[ cut here ]------------
[ 19.356148] kernel BUG at net/core/skbuff.c:119!
[ 19.356148] invalid opcode: 0000 [#1] SMP
[ 19.356148] Modules linked in: pppoe pppox ppp_generic slhc sch_sfq sch_htb
act_mirred cls_u32 sch_ingress ipt_REJECT ext4 jbd2 ipt_MASQUERADE xt_multiport
xt_rateest xt_RATEEST xt_layer7 xt_HL xt_TPROXY nf_tproxy_core xt_set
xt_connmark ipt_DF ipt_ULOG xt_socket ip6_tables nf_defrag_ipv6 xt_mark
xt_conntrack xt_TCPMSS xt_tcpudp xt_DSCP xt_dscp iptable_raw iptable_mangle
iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter
ip_tables x_tables ip_set_hash_net ip_set nfnetlink cpufreq_ondemand
acpi_cpufreq freq_table processor coretemp mperf ipip hwmon macvlan ifb tun
ipmi_watchdog ipmi_si ipmi_msghandler pcspkr
[ 19.356148]
[ 19.356148] Pid: 2305, comm: openvpn Not tainted 3.2.58 #1 Intel Corporation
S3210SH/S3210SH
[ 19.356148] EIP: 0060:[<c05cb9e4>] EFLAGS: 00210296 CPU: 0
[ 19.356148] EIP is at skb_push+0x84/0x90
[ 19.356148] EAX: 0000008b EBX: ef7a7a8e ECX: 00000000 EDX: fffe7ddb
[ 19.356148] ESI: f1af8800 EDI: ebc1e840 EBP: f540be24 ESP: f540bdf8
[ 19.356148] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 19.356148] Process openvpn (pid: 2305, ti=f540a000 task=f568b190
task.ti=f1b42000)
[ 19.356148] Stack:
[ 19.356148] c0988ec4 f9ace566 000000ce 00000094 ef7a7a00 ef7a79c0 ef7a7a8e
ef7a7b40
[ 19.356148] f1af8800 f157ed00 ebc1e600 f540be48 f9ace566 f140c800 00000000
f2790800
[ 19.356148] 00000001 f9ace450 f1459220 ebc1e600 f540be60 c05f0cff f540bf1c
00000000
[ 19.356148] Call Trace:
[ 19.356148] [<f9ace566>] ? tcf_mirred+0x116/0x15c [act_mirred]
[ 19.356148] [<f9ace566>] tcf_mirred+0x116/0x15c [act_mirred]
[ 19.356148] [<f9ace450>] ? tcf_mirred_dump+0x110/0x110 [act_mirred]
[ 19.356148] [<c05f0cff>] tcf_action_exec+0x3f/0x80
[ 19.356148] [<f9ac70d2>] u32_classify+0x212/0x374 [cls_u32]
[ 19.356148] [<f83e59e6>] ? ipip_rcv+0x136/0x220 [ipip]
[ 19.356148] [<c05f779a>] ? nf_hook_slow+0x5a/0x110
[ 19.356148] [<c063c5a3>] ? tunnel4_rcv+0x33/0x90
[ 19.356148] [<c05fe0bf>] ? ip_local_deliver_finish+0x9f/0x230
[ 19.356148] [<c05ed981>] tc_classify_compat+0x31/0x70
[ 19.356148] [<c05ee8c4>] tc_classify+0x44/0xb0
[ 19.356148] [<f9abe0f1>] ingress_enqueue+0x21/0x80 [sch_ingress]
[ 19.356148] [<c05d2740>] __netif_receive_skb+0x2e0/0x4e0
[ 19.356148] [<c05d2a0a>] process_backlog+0xca/0x1a0
[ 19.356148] [<c05d4929>] net_rx_action+0xc9/0x1c0
[ 19.356148] [<c013e95b>] __do_softirq+0x7b/0x110
[ 19.356148] [<c013e8e0>] ? irq_enter+0x70/0x70
[ 19.356148] [<c013e8e0>] ? irq_enter+0x70/0x70
[ 19.356148] <IRQ>
[ 19.356148] [<c05d4b30>] ? netif_rx_ni+0x20/0x30
[ 19.356148] [<f836e959>] ? tun_get_user+0x1a9/0x3e0 [tun]
[ 19.356148] [<f836ec2e>] ? tun_chr_aio_write+0x6e/0xb0 [tun]
[ 19.356148] [<c01bb3bc>] ? do_sync_write+0xac/0xf0
[ 19.356148] [<c010b487>] ? init_fpu+0xd7/0x160
[ 19.356148] [<c0123b30>] ? mm_fault_error+0x130/0x130
[ 19.356148] [<c01bbeca>] ? vfs_write+0x9a/0x170
[ 19.673799] [<c01bb310>] ? do_sync_readv_writev+0xe0/0xe0
[ 19.673799] [<c01bc06d>] ? sys_write+0x3d/0x70
[ 19.673799] [<c06d291c>] ? sysenter_do_call+0x12/0x2c
[ 19.673799] Code: 5c 24 18 8b 88 a8 00 00 00 89 54 24 0c 89 4c 24 10 8b 40
50 c7 04 24 c4 8e 98 c0 89 44 24 08 8b 45 04 89 44 24 04 e8 e9 3b 10 00 <0f> 0b
eb fe 90 8d b4 26 00 00 00 00 55 89 e5 56 53 83 ec 24 8b
[ 19.673799] EIP: [<c05cb9e4>] skb_push+0x84/0x90 SS:ESP 0068:f540bdf8
[ 19.726460] ---[ end trace c5c43cec4bd0fcd8 ]---
[ 19.731801] Kernel panic - not syncing: Fatal exception in interrupt
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
2014-05-06 15:16 Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119 Stephen Hemminger
@ 2014-05-06 22:30 ` Cong Wang
2014-05-08 10:13 ` Jamal Hadi Salim
0 siblings, 1 reply; 5+ messages in thread
From: Cong Wang @ 2014-05-06 22:30 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: netdev, d77190, Jamal Hadi Salim
On Tue, May 6, 2014 at 8:16 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> Hello!
> Got this bug while setting large shaper ruleset (about 4k hash rules) on mirred
> ifb interface at boot time:
> Attached archive consists of used shaper scripts.
>
> [ 19.355181] ------------[ cut here ]------------
> [ 19.356148] kernel BUG at net/core/skbuff.c:119!
> [ 19.356148] invalid opcode: 0000 [#1] SMP
> [ 19.356148] Modules linked in: pppoe pppox ppp_generic slhc sch_sfq sch_htb
> act_mirred cls_u32 sch_ingress ipt_REJECT ext4 jbd2 ipt_MASQUERADE xt_multiport
> xt_rateest xt_RATEEST xt_layer7 xt_HL xt_TPROXY nf_tproxy_core xt_set
> xt_connmark ipt_DF ipt_ULOG xt_socket ip6_tables nf_defrag_ipv6 xt_mark
> xt_conntrack xt_TCPMSS xt_tcpudp xt_DSCP xt_dscp iptable_raw iptable_mangle
> iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter
> ip_tables x_tables ip_set_hash_net ip_set nfnetlink cpufreq_ondemand
> acpi_cpufreq freq_table processor coretemp mperf ipip hwmon macvlan ifb tun
> ipmi_watchdog ipmi_si ipmi_msghandler pcspkr
> [ 19.356148]
> [ 19.356148] Pid: 2305, comm: openvpn Not tainted 3.2.58 #1 Intel Corporation
> S3210SH/S3210SH
> [ 19.356148] EIP: 0060:[<c05cb9e4>] EFLAGS: 00210296 CPU: 0
> [ 19.356148] EIP is at skb_push+0x84/0x90
> [ 19.356148] EAX: 0000008b EBX: ef7a7a8e ECX: 00000000 EDX: fffe7ddb
> [ 19.356148] ESI: f1af8800 EDI: ebc1e840 EBP: f540be24 ESP: f540bdf8
> [ 19.356148] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> [ 19.356148] Process openvpn (pid: 2305, ti=f540a000 task=f568b190
> task.ti=f1b42000)
> [ 19.356148] Stack:
> [ 19.356148] c0988ec4 f9ace566 000000ce 00000094 ef7a7a00 ef7a79c0 ef7a7a8e
> ef7a7b40
> [ 19.356148] f1af8800 f157ed00 ebc1e600 f540be48 f9ace566 f140c800 00000000
> f2790800
> [ 19.356148] 00000001 f9ace450 f1459220 ebc1e600 f540be60 c05f0cff f540bf1c
> 00000000
> [ 19.356148] Call Trace:
> [ 19.356148] [<f9ace566>] ? tcf_mirred+0x116/0x15c [act_mirred]
> [ 19.356148] [<f9ace566>] tcf_mirred+0x116/0x15c [act_mirred]
> [ 19.356148] [<f9ace450>] ? tcf_mirred_dump+0x110/0x110 [act_mirred]
> [ 19.356148] [<c05f0cff>] tcf_action_exec+0x3f/0x80
> [ 19.356148] [<f9ac70d2>] u32_classify+0x212/0x374 [cls_u32]
> [ 19.356148] [<f83e59e6>] ? ipip_rcv+0x136/0x220 [ipip]
> [ 19.356148] [<c05f779a>] ? nf_hook_slow+0x5a/0x110
> [ 19.356148] [<c063c5a3>] ? tunnel4_rcv+0x33/0x90
> [ 19.356148] [<c05fe0bf>] ? ip_local_deliver_finish+0x9f/0x230
> [ 19.356148] [<c05ed981>] tc_classify_compat+0x31/0x70
> [ 19.356148] [<c05ee8c4>] tc_classify+0x44/0xb0
> [ 19.356148] [<f9abe0f1>] ingress_enqueue+0x21/0x80 [sch_ingress]
> [ 19.356148] [<c05d2740>] __netif_receive_skb+0x2e0/0x4e0
> [ 19.356148] [<c05d2a0a>] process_backlog+0xca/0x1a0
> [ 19.356148] [<c05d4929>] net_rx_action+0xc9/0x1c0
> [ 19.356148] [<c013e95b>] __do_softirq+0x7b/0x110
> [ 19.356148] [<c013e8e0>] ? irq_enter+0x70/0x70
> [ 19.356148] [<c013e8e0>] ? irq_enter+0x70/0x70
> [ 19.356148] <IRQ>
> [ 19.356148] [<c05d4b30>] ? netif_rx_ni+0x20/0x30
> [ 19.356148] [<f836e959>] ? tun_get_user+0x1a9/0x3e0 [tun]
> [ 19.356148] [<f836ec2e>] ? tun_chr_aio_write+0x6e/0xb0 [tun]
> [ 19.356148] [<c01bb3bc>] ? do_sync_write+0xac/0xf0
> [ 19.356148] [<c010b487>] ? init_fpu+0xd7/0x160
> [ 19.356148] [<c0123b30>] ? mm_fault_error+0x130/0x130
> [ 19.356148] [<c01bbeca>] ? vfs_write+0x9a/0x170
> [ 19.673799] [<c01bb310>] ? do_sync_readv_writev+0xe0/0xe0
> [ 19.673799] [<c01bc06d>] ? sys_write+0x3d/0x70
> [ 19.673799] [<c06d291c>] ? sysenter_do_call+0x12/0x2c
> [ 19.673799] Code: 5c 24 18 8b 88 a8 00 00 00 89 54 24 0c 89 4c 24 10 8b 40
> 50 c7 04 24 c4 8e 98 c0 89 44 24 08 8b 45 04 89 44 24 04 e8 e9 3b 10 00 <0f> 0b
> eb fe 90 8d b4 26 00 00 00 00 55 89 e5 56 53 83 ec 24 8b
> [ 19.673799] EIP: [<c05cb9e4>] skb_push+0x84/0x90 SS:ESP 0068:f540bdf8
> [ 19.726460] ---[ end trace c5c43cec4bd0fcd8 ]---
> [ 19.731801] Kernel panic - not syncing: Fatal exception in interrupt
>
Hmm, looks like we pushed wrong header size here. tcf_mirred_init()
checks if we could push hard header by checking the type of the target device,
while in tcf_mirred() we push the ->hard_header_len of the source device
where this skb is received. I am not sure at all.
Jamal, does the following patch make any sense for you?
---
diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
index 4f912c0..8199540 100644
--- a/net/sched/act_mirred.c
+++ b/net/sched/act_mirred.c
@@ -157,7 +157,7 @@ static int tcf_mirred(struct sk_buff *skb, const
struct tc_action *a,
if (!(at & AT_EGRESS)) {
if (m->tcfm_ok_push)
- skb_push(skb2, skb2->dev->hard_header_len);
+ skb_push(skb2, dev->hard_header_len);
}
/* mirror is always swallowed */
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
2014-05-06 22:30 ` Cong Wang
@ 2014-05-08 10:13 ` Jamal Hadi Salim
2014-05-09 18:31 ` Cong Wang
0 siblings, 1 reply; 5+ messages in thread
From: Jamal Hadi Salim @ 2014-05-08 10:13 UTC (permalink / raw)
To: Cong Wang, Stephen Hemminger; +Cc: netdev, d77190
Hi Cong,
Sorry for the latency - on travel mode.
On 05/06/14 18:30, Cong Wang wrote:
> On Tue, May 6, 2014 at 8:16 AM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>
> Hmm, looks like we pushed wrong header size here. tcf_mirred_init()
> checks if we could push hard header by checking the type of the target device,
> while in tcf_mirred() we push the ->hard_header_len of the source device
> where this skb is received. I am not sure at all.
>
> Jamal, does the following patch make any sense for you?
>
Dont feel like that patch would help. Scratching my head....
One way to tell is to: ifconfig down the source (which seems like tunnel
device) and install all the rules then last step is ifconfig first ifb,
then the tunnel device. And if it works with current kernel patch is not
needed. It feels like some race condition between installing
rule and using it i.e rule gets installed and when still trying to
advertise to user space, a packet arrives and starts using it.
If the above works - then the question to d77190@mail.ru is what kernel
this is and when did this last work as installed?
[My gut feeling is it may have something to do with the rcu changes
perhaps - but not sure.]
cheers,
jamal
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
2014-05-08 10:13 ` Jamal Hadi Salim
@ 2014-05-09 18:31 ` Cong Wang
2014-05-11 11:29 ` Jamal Hadi Salim
0 siblings, 1 reply; 5+ messages in thread
From: Cong Wang @ 2014-05-09 18:31 UTC (permalink / raw)
To: Jamal Hadi Salim; +Cc: Stephen Hemminger, netdev, d77190
On Thu, May 8, 2014 at 3:13 AM, Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> Hi Cong,
>
> Sorry for the latency - on travel mode.
>
>
> On 05/06/14 18:30, Cong Wang wrote:
>>
>> On Tue, May 6, 2014 at 8:16 AM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>
>
>>
>> Hmm, looks like we pushed wrong header size here. tcf_mirred_init()
>> checks if we could push hard header by checking the type of the target
>> device,
>> while in tcf_mirred() we push the ->hard_header_len of the source device
>> where this skb is received. I am not sure at all.
>>
>> Jamal, does the following patch make any sense for you?
>>
>
> Dont feel like that patch would help. Scratching my head....
> One way to tell is to: ifconfig down the source (which seems like tunnel
> device) and install all the rules then last step is ifconfig first ifb,
> then the tunnel device. And if it works with current kernel patch is not
> needed. It feels like some race condition between installing
> rule and using it i.e rule gets installed and when still trying to
> advertise to user space, a packet arrives and starts using it.
> If the above works - then the question to d77190@mail.ru is what kernel
> this is and when did this last work as installed?
It's an old kernel, 3.2.58. :-/
Please try to reproduce it on the latest kernel if possible.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119
2014-05-09 18:31 ` Cong Wang
@ 2014-05-11 11:29 ` Jamal Hadi Salim
0 siblings, 0 replies; 5+ messages in thread
From: Jamal Hadi Salim @ 2014-05-11 11:29 UTC (permalink / raw)
To: Cong Wang; +Cc: Stephen Hemminger, netdev, d77190
On 05/09/14 14:31, Cong Wang wrote:
> It's an old kernel, 3.2.58. :-/
>
The email wasnt clear.
> Please try to reproduce it on the latest kernel if possible.
Also - try what i said in ifconfig up interfaces last and
see if that resolves it (that would still point to some
lock/rcu changes that may have happened earlier than your
changes; i.e something is no longer protecting user space
from data path activity).
cheers,
jamal
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-05-11 11:30 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-05-06 15:16 Fw: [Bug 75571] New: using tc action mirred results BUG at net/core/skbuff.c:119 Stephen Hemminger
2014-05-06 22:30 ` Cong Wang
2014-05-08 10:13 ` Jamal Hadi Salim
2014-05-09 18:31 ` Cong Wang
2014-05-11 11:29 ` Jamal Hadi Salim
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.