Xen bug or sky2 device driver bug?

Xen bug or sky2 device driver bug?

[Jae-Wan Jang]

Hello,

I'm using ASUS P5WD2-E Premium mother board.
It has two Marvell Yukon II gigabit Ethernet and sky2 device driver is
used for them.

When domU receives lots of lots of packets (about 10Kb each),
dom0 crashes.
Unfortunately, my machine doesn't have serial port, I can't capture all
the oops messages.
Following is backtrace of the oops messages.

When domU receives lots of packets from other physical machine, kernel
panic occurs.

Backtrace

br_handle_frame
br_nf_pre_routing_finish
br_nf_pre_routing
nf_iterate
nf_hook_slow
br_handle_frame
netif_receive_skb
sky2_poll
net_rx_action
__do_softirq
irq_exit
do_IRQ
evtchn_do_upcall
hypervisor_callback
xen_idle
cpu_idle
rest_init
start_kernel
0xc010006f

Is it bug of Xen or sky2 device?

Thanks in advance.

-- 

Jae-Wan Jang
jwjang@camars.kaist.ac.kr
+82-42-869-3559
http://camars.kaist.ac.kr/~jwjang

RE: Xen bug or sky2 device driver bug?

[Ian Pratt]

> I'm using ASUS P5WD2-E Premium mother board.
> It has two Marvell Yukon II gigabit Ethernet and sky2 device driver is
> used for them.
> 
> When domU receives lots of lots of packets (about 10Kb each),
> dom0 crashes.
> Unfortunately, my machine doesn't have serial port, I can't capture
all
> the oops messages.
> Following is backtrace of the oops messages.

What xen version? 
Are you talking about 10KB UDP datagrams fragemented into 1500byte
Ethernet frames?
Have you any iptables or ebtables rules?
Ian

> When domU receives lots of packets from other physical machine, kernel
> panic occurs.
> 
> Backtrace
> 
> br_handle_frame
> br_nf_pre_routing_finish
> br_nf_pre_routing
> nf_iterate
> nf_hook_slow
> br_handle_frame
> netif_receive_skb
> sky2_poll
> net_rx_action
> __do_softirq
> irq_exit
> do_IRQ
> evtchn_do_upcall
> hypervisor_callback
> xen_idle
> cpu_idle
> rest_init
> start_kernel
> 0xc010006f
> 
> Is it bug of Xen or sky2 device?
> 
> Thanks in advance.
> 
> --
> 
> Jae-Wan Jang
> jwjang@camars.kaist.ac.kr
> +82-42-869-3559
> http://camars.kaist.ac.kr/~jwjang
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen bug or sky2 device driver bug?

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

We had determined this to be a problem in the bridge netfilter code, in that it can't
deal with skb-s that have no headroom set up (for alignment purposes), as the
sky2 driver appears to do.

I'm attaching the patch we use to address this in SLE10.

Jan

>>> Jae-Wan Jang <jwjang@camars.kaist.ac.kr> 18.08.06 09:50 >>>
Hello,

I'm using ASUS P5WD2-E Premium mother board.
It has two Marvell Yukon II gigabit Ethernet and sky2 device driver is
used for them.

When domU receives lots of lots of packets (about 10Kb each),
dom0 crashes.
Unfortunately, my machine doesn't have serial port, I can't capture all
the oops messages.
Following is backtrace of the oops messages.

When domU receives lots of packets from other physical machine, kernel
panic occurs.

Backtrace

br_handle_frame
br_nf_pre_routing_finish
br_nf_pre_routing
nf_iterate
nf_hook_slow
br_handle_frame
netif_receive_skb
sky2_poll
net_rx_action
__do_softirq
irq_exit
do_IRQ
evtchn_do_upcall
hypervisor_callback
xen_idle
cpu_idle
rest_init
start_kernel
0xc010006f

Is it bug of Xen or sky2 device?

Thanks in advance.

-- 

Jae-Wan Jang
jwjang@camars.kaist.ac.kr
+82-42-869-3559
http://camars.kaist.ac.kr/~jwjang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #2: nf_bridge-header-size --]
[-- Type: application/octet-stream, Size: 1279 bytes --]

Subject: nf_bridge: ethernet header is 14 not 16 bytes
From: jbeulich@novell.com
Acked-by: kraxel@suse.de
References: 150410

The bridge netfilter code saves two more bytes that it should.
In most cases it doesn't hurt because many drivers use NET_IP_ALIGN
to make the IP header aligned, so there are two extra bytes head room
available.

Some drivers don't do that though (sky2 for example), so copying
accesses data outside the skbuff data allocation.  On xen kernels
this can kill the machine with a page fault due to the way how
skbuffs are allocated and the memory is managed.

Index: linux-2.6.16/include/linux/netfilter_bridge.h
===================================================================
--- linux-2.6.16.orig/include/linux/netfilter_bridge.h
+++ linux-2.6.16/include/linux/netfilter_bridge.h
@@ -73,14 +73,14 @@ void nf_bridge_maybe_copy_header(struct 
 			memcpy(skb->data - 18, skb->nf_bridge->data, 18);
 			skb_push(skb, 4);
 		} else
-			memcpy(skb->data - 16, skb->nf_bridge->data, 16);
+			memcpy(skb->data - 14, skb->nf_bridge->data, 14);
 	}
 }
 
 static inline
 void nf_bridge_save_header(struct sk_buff *skb)
 {
-        int header_size = 16;
+        int header_size = 14;
 
 	if (skb->protocol == __constant_htons(ETH_P_8021Q))
 		header_size = 18;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen bug or sky2 device driver bug?

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 255 bytes --]

> Is it bug of Xen or sky2 device?

Most likely bridging code, try the patch below (reminds /me that I
wanted to forward that one to the networking folks ...)

cheers,
  Gerd

-- 
Gerd Hoffmann <kraxel@suse.de>
http://www.suse.de/~kraxel/julika-dora.jpeg

[-- Attachment #2: nf_bridge-header-size --]
[-- Type: text/plain, Size: 1441 bytes --]

Subject: nf_bridge: ethernet header is 14 not 16 bytes
From: jbeulich@novell.com
Acked-by: kraxel@suse.de
References: 150410

The bridge netfilter code saves two more bytes that it should.
In most cases it doesn't hurt because many drivers use NET_IP_ALIGN
to make the IP header aligned, so there are two extra bytes head room
available.

Some drivers don't do that though (sky2 for example), so copying
accesses data outside the skbuff data allocation.  On xen kernels
this can kill the machine with a page fault due to the way how
skbuffs are allocated and the memory is managed.

---
 include/linux/netfilter_bridge.h |    2 +-
 net/bridge/br_netfilter.c        |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

--- linux-2.6.17.orig/include/linux/netfilter_bridge.h
+++ linux-2.6.17/include/linux/netfilter_bridge.h
@@ -57,7 +57,7 @@ void nf_bridge_maybe_copy_header(struct 
 			memcpy(skb->data - 18, skb->nf_bridge->data, 18);
 			skb_push(skb, 4);
 		} else
-			memcpy(skb->data - 16, skb->nf_bridge->data, 16);
+			memcpy(skb->data - 14, skb->nf_bridge->data, 14);
 	}
 }
 
--- linux-2.6.17.orig/net/bridge/br_netfilter.c
+++ linux-2.6.17/net/bridge/br_netfilter.c
@@ -124,7 +124,7 @@ static inline struct nf_bridge_info *nf_
 
 static inline void nf_bridge_save_header(struct sk_buff *skb)
 {
-        int header_size = 16;
+        int header_size = 14;
 
 	if (skb->protocol == htons(ETH_P_8021Q))
 		header_size = 18;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen bug or sky2 device driver bug?

[Marco Gerards]

Jae-Wan Jang <jwjang@camars.kaist.ac.kr> writes:

Hi,

> I'm using ASUS P5WD2-E Premium mother board.
> It has two Marvell Yukon II gigabit Ethernet and sky2 device driver is
> used for them.
>
> When domU receives lots of lots of packets (about 10Kb each),
> dom0 crashes.
> Unfortunately, my machine doesn't have serial port, I can't capture all
> the oops messages.
> Following is backtrace of the oops messages.

May I assume this is only when you are using it in gigabit mode?

This same bug is mentioned here:
http://www.scl.ameslab.gov/Projects/mini-xen/index.html

I have a MacBook with the Marvell Yukon II and I am using it as
100MBit only.  It works perfectly for me, although I tried hard to
reproduce this specific bug.

--
Marco

Re: Xen bug or sky2 device driver bug?

[Jae-Wan Jang]

[-- Attachment #1: Type: text/html, Size: 3026 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen bug or sky2 device driver bug?

[Jan De Landtsheer]

Bug exists in linux-vanilla too... sky2 driver is crap (or the NIC
itself for that matter).

On Fri, 2006-08-18 at 10:16 +0200, Gerd Hoffmann wrote:
> > Is it bug of Xen or sky2 device?
> 
> Most likely bridging code, try the patch below (reminds /me that I
> wanted to forward that one to the networking folks ...)
> 
> cheers,
>   Gerd
> 
> plain text document attachment (nf_bridge-header-size)
> Subject: nf_bridge: ethernet header is 14 not 16 bytes
> From: jbeulich@novell.com
> Acked-by: kraxel@suse.de
> References: 150410
> 
> The bridge netfilter code saves two more bytes that it should.
> In most cases it doesn't hurt because many drivers use NET_IP_ALIGN
> to make the IP header aligned, so there are two extra bytes head room
> available.
> 
> Some drivers don't do that though (sky2 for example), so copying
> accesses data outside the skbuff data allocation.  On xen kernels
> this can kill the machine with a page fault due to the way how
> skbuffs are allocated and the memory is managed.
> 
> ---
>  include/linux/netfilter_bridge.h |    2 +-
>  net/bridge/br_netfilter.c        |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> --- linux-2.6.17.orig/include/linux/netfilter_bridge.h
> +++ linux-2.6.17/include/linux/netfilter_bridge.h
> @@ -57,7 +57,7 @@ void nf_bridge_maybe_copy_header(struct 
>  			memcpy(skb->data - 18, skb->nf_bridge->data, 18);
>  			skb_push(skb, 4);
>  		} else
> -			memcpy(skb->data - 16, skb->nf_bridge->data, 16);
> +			memcpy(skb->data - 14, skb->nf_bridge->data, 14);
>  	}
>  }
>  
> --- linux-2.6.17.orig/net/bridge/br_netfilter.c
> +++ linux-2.6.17/net/bridge/br_netfilter.c
> @@ -124,7 +124,7 @@ static inline struct nf_bridge_info *nf_
>  
>  static inline void nf_bridge_save_header(struct sk_buff *skb)
>  {
> -        int header_size = 16;
> +        int header_size = 14;
>  
>  	if (skb->protocol == htons(ETH_P_8021Q))
>  		header_size = 18;
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen bug or sky2 device driver bug?

[Stephen Hemminger]

On Fri, 18 Aug 2006 17:59:23 +0200
Jan De Landtsheer <Jan@qlayer.com> wrote:

> Bug exists in linux-vanilla too... sky2 driver is crap (or the NIC
> itself for that matter).
> 

The driver is new, and Xen is following several kernel versions behind.
So you are seeing the worst end of the dog. 

Re: Xen bug or sky2 device driver bug?

[Stephen Hemminger]

On Fri, 18 Aug 2006 16:50:18 +0900
Jae-Wan Jang <jwjang@camars.kaist.ac.kr> wrote:

> Hello,
> 
> I'm using ASUS P5WD2-E Premium mother board.
> It has two Marvell Yukon II gigabit Ethernet and sky2 device driver is
> used for them.
> 
> When domU receives lots of lots of packets (about 10Kb each),
> dom0 crashes.
> Unfortunately, my machine doesn't have serial port, I can't capture all
> the oops messages.
> Following is backtrace of the oops messages.
> 
> When domU receives lots of packets from other physical machine, kernel
> panic occurs.
> 
> Backtrace
> 
> br_handle_frame
> br_nf_pre_routing_finish
> br_nf_pre_routing
> nf_iterate
> nf_hook_slow
> br_handle_frame
> netif_receive_skb
> sky2_poll
> net_rx_action
> __do_softirq
> irq_exit
> do_IRQ
> evtchn_do_upcall
> hypervisor_callback
> xen_idle
> cpu_idle
> rest_init
> start_kernel
> 0xc010006f
> 
> Is it bug of Xen or sky2 device?
> 
> Thanks in advance.
> 

Did you ever think or reading the MAINTAINERS file and sending
a bug report?