Re: [ovs-discuss] kernel panic receiving flooded VXLAN traffic with OVS

[Jay Vosburgh <jay.vosburgh-Z7WLFzj8eWMS+FvcfC7Uqw@public.gmane.org>]

Pravin Shelar <pshelar@nicira.com> wrote:

>On Fri, Nov 7, 2014 at 2:29 PM, Jesse Gross <jesse@nicira.com> wrote:
>> On Fri, Nov 7, 2014 at 1:13 PM, Jay Vosburgh <jay.vosburgh@canonical.com> wrote:
>>> Jesse Gross <jesse@nicira.com> wrote:
>>>
>>>>On Fri, Nov 7, 2014 at 10:34 AM, Jesse Gross <jesse@nicira.com> wrote:
>>>>> On Fri, Nov 7, 2014 at 9:40 AM, Pravin Shelar <pshelar@nicira.com> wrote:
>>>>>> On Thu, Nov 6, 2014 at 5:58 PM, Jay Vosburgh <jay.vosburgh@canonical.com> wrote:
>>> [...]
>>>>>>>         I'm not sure if this is an error on the part of the RX / GRO
>>>>>>> processing in assembling the GRO skb, or in how OVS calls skb_segment.
>>>>>>>
>>>>>>
>>>>>> I think this is related skb_segment() issue where it is not able to
>>>>>> handle this type of skb geometry. We need to fix skb-segmentation. I
>>>>>> will investigate it more.
>>>>>
>>>>> One problem that I see is that vxlan_gro_complete() doesn't add
>>>>> SKB_GSO_UDP_TUNNEL to gso_type. This causes us to attempt
>>>>> fragmentation as UDP rather than continuing down to do TCP
>>>>> segmentation. That probably screws up the skb geometry.
>>>>
>>>>I sent out a patch to fix this issue. I'm pretty sure that it is the
>>>>root cause of the originally reported case but I don't have a good way
>>>>to reproduce it so it would be great if you could test it Jay.
>>>
>>>         I'm having an issue there; when I set up my recreation on
>>> current net-next (3.18-rc2) without your new patch, I get the following
>>> oops when my ovs script does "ovs-vsctl --if-exists del-br br-ex":
>>
>> Hmm, that looks like a totally different problem. Pravin - any ideas?
>
>I am not able to reproduce with above command. Jay, Can you send me
>steps to reproduce this issue?

	Well, at the moment a 3.18.0-rc2 kernel panics in
ovs_flow_tbl_insert as soon as ovs-vswitchd starts up.  Booting to an
earlier kernel (3.17, for example) with no other changes doesn't panic.

	I moved /etc/openvswitch/conf.db away and rebooted the system
(which I think will eliminate the stored configuration), and the kernel
still hits this oops when ovs-vswitchd starts up.

	A bit of poking with crash 

[   22.180002] RIP: 0010:[<ffffffffa01a55ec>]  [<ffffffffa01a55ec>] ovs_flow_tbl_insert+0xdc/0x1f0 [openvswitch]
[   22.180002] RSP: 0018:ffff8801391a77a8  EFLAGS: 00010203
[   22.180002] RAX: 00000000076cc6f1 RBX: ffff8800b35c2020 RCX: 00000000fb994f19
[   22.180002] RDX: 000000009cc907e5 RSI: 00000000a490ff19 RDI: ffff8800b0c1c19c
[   22.180002] RBP: ffff8801391a77f8 R08: 000000006867223a R09: 00000000819d92a7
[   22.180002] R10: ffffffffa01a5696 R11: 0000000000000004 R12: ffff8800b35c2028
[   22.180002] R13: ffff8801391a7850 R14: ffff8800b35c2028 R15: ffff880134827800
[   22.180002] FS:  00007f0ef491a980(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000

0xffffffffa01a55e7 <ovs_flow_tbl_insert+0xd7>:	callq  0xffffffff813a75c0 <__jhash2>
0xffffffffa01a55ec <ovs_flow_tbl_insert+0xdc>:	mov    %eax,0x30(%r8)
0xffffffffa01a55f0 <ovs_flow_tbl_insert+0xe0>:	mov    (%rbx),%r13
0xffffffffa01a55f3 <ovs_flow_tbl_insert+0xe3>:	mov    %r8,%rsi
0xffffffffa01a55f6 <ovs_flow_tbl_insert+0xe6>:	mov    %r13,%rdi
0xffffffffa01a55f9 <ovs_flow_tbl_insert+0xe9>:	callq  0xffffffffa01a4ba0 <table_instance_insert>

	So it panics on return from __jhash2 because %r8 is invalid;
this is presumably

int ovs_flow_tbl_insert(struct flow_table *table, struct sw_flow *flow,
			struct sw_flow_mask *mask)
{
[...]
	flow->hash = flow_hash(&flow->key, flow->mask->range.start,
			flow->mask->range.end);
	ti = ovsl_dereference(table->ti);
	table_instance_insert(ti, flow);

	Looking at __jash2, it uses %r8 internally, but %r8 doesn't
appear to be saved and restored either around the call to, or within,
__jhash2.  In ovs_flow_tbl_insert %r8 appears to hold the "flow"
variable, so this panic might have nothing to do with ovs itself.

	I'll have to look at this further on Monday.

	-J

---
	-Jay Vosburgh, jay.vosburgh@canonical.com
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss