Stable request for 5882a07c72093d

Stable request for 5882a07c72093d

[Dave Chiluk]

Commit 5882a07c72093dc3a18e2d2b129fb200686bb6ee remedies a BUG_ON in
skb_segment.  This is hit reliably in clouds that are using openvswitch for
network capabilities (which is the default network used by Openstack) on
machines that have gso offload capabilities.  

This has already been reported multiple times to Canonical in numerous
openstack clouds.  The fix has been integrated and tested in those clouds, and
has been show to resolve their issues.

Please include this patch for stable release.

Thank you, 
Dave Chiluk

The patch follows for convenience.

[PATCH] net: fix UDP tunnel GSO of frag_list GRO packets

[Dave Chiluk]

From: Wei-Chun Chao <weichunc@plumgrid.com>

This patch fixes a kernel BUG_ON in skb_segment. It is hit when
testing two VMs on openvswitch with one VM acting as VXLAN gateway.

During VXLAN packet GSO, skb_segment is called with skb->data
pointing to inner TCP payload. skb_segment calls skb_network_protocol
to retrieve the inner protocol. skb_network_protocol actually expects
skb->data to point to MAC and it calls pskb_may_pull with ETH_HLEN.
This ends up pulling in ETH_HLEN data from header tail. As a result,
pskb_trim logic is skipped and BUG_ON is hit later.

Move skb_push in front of skb_network_protocol so that skb->data
lines up properly.

kernel BUG at net/core/skbuff.c:2999!
Call Trace:
[<ffffffff816ac412>] tcp_gso_segment+0x122/0x410
[<ffffffff816bc74c>] inet_gso_segment+0x13c/0x390
[<ffffffff8164b39b>] skb_mac_gso_segment+0x9b/0x170
[<ffffffff816b3658>] skb_udp_tunnel_segment+0xd8/0x390
[<ffffffff816b3c00>] udp4_ufo_fragment+0x120/0x140
[<ffffffff816bc74c>] inet_gso_segment+0x13c/0x390
[<ffffffff8109d742>] ? default_wake_function+0x12/0x20
[<ffffffff8164b39b>] skb_mac_gso_segment+0x9b/0x170
[<ffffffff8164b4d0>] __skb_gso_segment+0x60/0xc0
[<ffffffff8164b6b3>] dev_hard_start_xmit+0x183/0x550
[<ffffffff8166c91e>] sch_direct_xmit+0xfe/0x1d0
[<ffffffff8164bc94>] __dev_queue_xmit+0x214/0x4f0
[<ffffffff8164bf90>] dev_queue_xmit+0x10/0x20
[<ffffffff81687edb>] ip_finish_output+0x66b/0x890
[<ffffffff81688a58>] ip_output+0x58/0x90
[<ffffffff816c628f>] ? fib_table_lookup+0x29f/0x350
[<ffffffff816881c9>] ip_local_out_sk+0x39/0x50
[<ffffffff816cbfad>] iptunnel_xmit+0x10d/0x130
[<ffffffffa0212200>] vxlan_xmit_skb+0x1d0/0x330 [vxlan]
[<ffffffffa02a3919>] vxlan_tnl_send+0x129/0x1a0 [openvswitch]
[<ffffffffa02a2cd6>] ovs_vport_send+0x26/0xa0 [openvswitch]
[<ffffffffa029931e>] do_output+0x2e/0x50 [openvswitch]

Signed-off-by: Wei-Chun Chao <weichunc@plumgrid.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 net/core/skbuff.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 8383b2b..9433047 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -2881,12 +2881,13 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
 	int pos;
 	int dummy;
 
+	__skb_push(head_skb, doffset);
 	proto = skb_network_protocol(head_skb, &dummy);
 	if (unlikely(!proto))
 		return ERR_PTR(-EINVAL);
 
 	csum = !!can_checksum_protocol(features, proto);
-	__skb_push(head_skb, doffset);
+
 	headroom = skb_headroom(head_skb);
 	pos = skb_headlen(head_skb);
 
-- 
1.9.1

Re: Stable request for 5882a07c72093d

[Eric Dumazet]

On Mon, 2014-06-23 at 10:56 -0500, Dave Chiluk wrote:
> Commit 5882a07c72093dc3a18e2d2b129fb200686bb6ee remedies a BUG_ON in
> skb_segment.  This is hit reliably in clouds that are using openvswitch for
> network capabilities (which is the default network used by Openstack) on
> machines that have gso offload capabilities.  
> 
> This has already been reported multiple times to Canonical in numerous
> openstack clouds.  The fix has been integrated and tested in those clouds, and
> has been show to resolve their issues.
> 
> Please include this patch for stable release.
> 
> Thank you, 
> Dave Chiluk
> 
> The patch follows for convenience.
> --

Both original commit and your request lack the bug origin.

I wish bug submissions are more explicit in this regard, using

Fixes: sha1-12digits ("title")

It would save a lot of time for stable teams, and ease future bug
hunting as well.

Re: Stable request for 5882a07c72093d

[Dave Chiluk]

On 06/23/2014 11:18 AM, Eric Dumazet wrote:
> On Mon, 2014-06-23 at 10:56 -0500, Dave Chiluk wrote:
>> Commit 5882a07c72093dc3a18e2d2b129fb200686bb6ee remedies a BUG_ON in
>> skb_segment.  This is hit reliably in clouds that are using openvswitch for
>> network capabilities (which is the default network used by Openstack) on
>> machines that have gso offload capabilities.  
>>
>> This has already been reported multiple times to Canonical in numerous
>> openstack clouds.  The fix has been integrated and tested in those clouds, and
>> has been show to resolve their issues.
>>
>> Please include this patch for stable release.
>>
>> Thank you, 
>> Dave Chiluk
>>
>> The patch follows for convenience.
>> --
> 
> Both original commit and your request lack the bug origin.
> 
> I wish bug submissions are more explicit in this regard, using
> 
> Fixes: sha1-12digits ("title")
> 
> It would save a lot of time for stable teams, and ease future bug
> hunting as well.
> 
> 
According to the initial thread, Wei-Chun determined it originated with
19acc327258a.  It has existed since 3.10-rc1.

Fixes: 19acc327258a ("gso: Handle Trans-Ether-Bridging protocol in
skb_network_protocol()")

Re: Stable request for 5882a07c72093d

[Wei-Chun Chao]

On Mon, Jun 23, 2014 at 10:35 AM, Dave Chiluk <chiluk@canonical.com> wrote:
> On 06/23/2014 11:18 AM, Eric Dumazet wrote:
>> On Mon, 2014-06-23 at 10:56 -0500, Dave Chiluk wrote:
>>> Commit 5882a07c72093dc3a18e2d2b129fb200686bb6ee remedies a BUG_ON in
>>> skb_segment.  This is hit reliably in clouds that are using openvswitch for
>>> network capabilities (which is the default network used by Openstack) on
>>> machines that have gso offload capabilities.
>>>
>>> This has already been reported multiple times to Canonical in numerous
>>> openstack clouds.  The fix has been integrated and tested in those clouds, and
>>> has been show to resolve their issues.
>>>
>>> Please include this patch for stable release.
>>>
>>> Thank you,
>>> Dave Chiluk
>>>
>>> The patch follows for convenience.
>>> --
>>
>> Both original commit and your request lack the bug origin.
>>
>> I wish bug submissions are more explicit in this regard, using
>>
>> Fixes: sha1-12digits ("title")
>>
>> It would save a lot of time for stable teams, and ease future bug
>> hunting as well.
>>
>>
> According to the initial thread, Wei-Chun determined it originated with
> 19acc327258a.  It has existed since 3.10-rc1.
>
> Fixes: 19acc327258a ("gso: Handle Trans-Ether-Bridging protocol in
> skb_network_protocol()")
>

Sorry I didn't include the bug origin in the original patch.

The issue was introduced by the combination of these two commits -
ec5f06156423 ("net: Kill link between CSUM and SG features.)
19acc327258a ("gso: Handle Trans-Ether-Bridging protocol in
skb_network_protocol()")

Thanks,
Weichun

Re: Stable request for 5882a07c72093d

[David Miller]

From: Dave Chiluk <chiluk@canonical.com>
Date: Mon, 23 Jun 2014 10:56:55 -0500

> Commit 5882a07c72093dc3a18e2d2b129fb200686bb6ee remedies a BUG_ON in
> skb_segment.  This is hit reliably in clouds that are using openvswitch for
> network capabilities (which is the default network used by Openstack) on
> machines that have gso offload capabilities.  
> 
> This has already been reported multiple times to Canonical in numerous
> openstack clouds.  The fix has been integrated and tested in those clouds, and
> has been show to resolve their issues.
> 
> Please include this patch for stable release.

Queued up for -stable submission, thank you.