[Bridge] Out of memory problem

[Bridge] Out of memory problem

[Tekale Sharad-FHJN78]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

Hi, I'm using linux 2.6.21.5 and our kernel is freeze.
 
The problem is, if I create a Software bridge using $brctl command. and
add two interfaces say, eth0.0 and eth0.1 using

$brctl addbr br-lan
$brctl addif br-lan eth0.0
$brctl addif br-lan eth0.1
 
and when i send traffic from a host connected to one port to host
connected at other end, soon all the memory is dried up and and kernel
crashes, 
then oom-killer gets invoked which kills all the processes finally
system reboots.
 
I googled for solution or patch, I saw the same defect was reported by
some people, but in between only the threaded ended, with no exact
solution. Don't know Y???
 
Can any one help me to refer to some patch to point some location in
code from where memory is failed to deallocate.
 
Thanks,
Sharad.

[-- Attachment #2: Type: text/html, Size: 2896 bytes --]

[Bridge] Out of memory problem

[Tekale Sharad-FHJN78]

[-- Attachment #1.1: Type: text/plain, Size: 642 bytes --]

Hi, I'm using linux 2.6.21.5 and our kernel is freeze.
 
The problem is, if I create a Software bridge using $brctl command. and
add two interfaces say, eth0.0 and eth0.1 using

$brctl addbr br-lan
$brctl addif br-lan eth0.0
$brctl addif br-lan eth0.1
 
and when i send traffic from a host connected to one port to host
connected at other end, soon all the memory is dried up and and kernel
crashes, 
then oom-killer gets invoked which kills all the processes finally
system reboots.
 
Can any one help me to refer to some patch  or point some location in
code from where memory is failed to deallocate.
 
Thanks,
Sharad.

[-- Attachment #1.2: Type: text/html, Size: 2607 bytes --]

[-- Attachment #2: ATT5660628.txt --]
[-- Type: text/plain, Size: 163 bytes --]

_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge

Re: [Bridge] Out of memory problem

[Stephen Hemminger]

On Mon, 13 Apr 2009 16:58:20 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

> Hi, I'm using linux 2.6.21.5 and our kernel is freeze.
>  
> The problem is, if I create a Software bridge using $brctl command. and
> add two interfaces say, eth0.0 and eth0.1 using
> 
> $brctl addbr br-lan
> $brctl addif br-lan eth0.0
> $brctl addif br-lan eth0.1
>  
> and when i send traffic from a host connected to one port to host
> connected at other end, soon all the memory is dried up and and kernel
> crashes, 
> then oom-killer gets invoked which kills all the processes finally
> system reboots.
>  
> Can any one help me to refer to some patch  or point some location in
> code from where memory is failed to deallocate.
>  
> Thanks,
> Sharad.

Most likely, you have created a loop and are not running spanning
tree. When a loop happens, packets multiply until all resources are
exhausted.

Re: [Bridge] Out of memory problem

[Tekale Sharad-FHJN78]

 

Hi Stephen 
> Most likely, you have created a loop and are not running spanning
tree. When a loop happens, packets multiply until all > resources are
exhausted.

> you have created a loop.
Ya, I have bridge named br-lan created by($brctl), and I have attached
two interfaces to it, and I'm sending traffic from PC1 to PC2. Some
thing like this

 	    		 ----------------------- 
	   		|    ------------       |
	   		|   |   br-lan   |      |    <--- AP
	   		|    ------------       |      
	   		|  eth0.0       eth0.1  | 
	    		 ----|------------|-----

	      	     |	      |
	      	     |	      |
  	     		    PC1          PC2

Previously:
root@none:/$ brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          8000.001570d8d8fd       no              eth0.0
                                                        eth0.1

>are not running spanning tree.
Is it always necessary to enable spanning tree when traffic is sent over
loop?

I have enabled spanning tree using

root@none:/$ brctl stp br-lan on
root@none:/$ brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan          8000.001570d8d8fd       yes             eth0.0
                                                        eth0.1

But again, when huge traffic is pumped from PC1 to PC2, The
skbuff_head_cache value is increased rapidly and even after stop pumping
the traffic, this value doesn't come down, and system goes for crash or
reboot because of out of memory.

root@none:/$ cat /proc/slabinfo | grep skbuff_head
skbuff_head_cache   4560   4560    192   20    1 : tunables  120   60
0 : slabdata    228    228      0

One more point, say I have each port attached to there own bridge, and
when I route traffic from one host to other, I'll not face this
behavior, This behavior is relevant only in above case.

Do I have to do any other setting to overcome out of memory problem? 
Also, Please let me know if any other information is required to isolate
this behavior.

Thanks,
Sharad.

-----Original Message-----
From: Stephen Hemminger [mailto:shemminger@vyatta.com] 
Sent: Monday, April 13, 2009 10:00 AM
To: Tekale Sharad-FHJN78
Cc: bridge@lists.linux-foundation.org
Subject: Re: [Bridge] Out of memory problem

On Mon, 13 Apr 2009 16:58:20 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

> Hi, I'm using linux 2.6.21.5 and our kernel is freeze.
>  
> The problem is, if I create a Software bridge using $brctl command. 
> and add two interfaces say, eth0.0 and eth0.1 using
> 
> $brctl addbr br-lan
> $brctl addif br-lan eth0.0
> $brctl addif br-lan eth0.1
>  
> and when i send traffic from a host connected to one port to host 
> connected at other end, soon all the memory is dried up and and kernel

> crashes, then oom-killer gets invoked which kills all the processes 
> finally system reboots.
>  
> Can any one help me to refer to some patch  or point some location in 
> code from where memory is failed to deallocate.
>  
> Thanks,
> Sharad.

Re: [Bridge] Out of memory problem

[Stephen Hemminger]

On Tue, 14 Apr 2009 13:21:13 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

>  
> 
> Hi Stephen 
> > Most likely, you have created a loop and are not running spanning
> tree. When a loop happens, packets multiply until all > resources are
> exhausted.
> 
> > you have created a loop.
> Ya, I have bridge named br-lan created by($brctl), and I have attached
> two interfaces to it, and I'm sending traffic from PC1 to PC2. Some
> thing like this
> 
>  	    		 ----------------------- 
> 	   		|    ------------       |
> 	   		|   |   br-lan   |      |    <--- AP
> 	   		|    ------------       |      
> 	   		|  eth0.0       eth0.1  | 
> 	    		 ----|------------|-----
> 
> 	      	     |	      |
> 	      	     |	      |
>   	     		    PC1          PC2
> 
> Previously:
> root@none:/$ brctl show
> bridge name     bridge id               STP enabled     interfaces
> br-lan          8000.001570d8d8fd       no              eth0.0
>                                                         eth0.1
> 
> >are not running spanning tree.
> Is it always necessary to enable spanning tree when traffic is sent over
> loop?
> 
> I have enabled spanning tree using
> 
> root@none:/$ brctl stp br-lan on
> root@none:/$ brctl show
> bridge name     bridge id               STP enabled     interfaces
> br-lan          8000.001570d8d8fd       yes             eth0.0
>                                                         eth0.1
> 
> But again, when huge traffic is pumped from PC1 to PC2, The
> skbuff_head_cache value is increased rapidly and even after stop pumping
> the traffic, this value doesn't come down, and system goes for crash or
> reboot because of out of memory.
> 
> root@none:/$ cat /proc/slabinfo | grep skbuff_head
> skbuff_head_cache   4560   4560    192   20    1 : tunables  120   60
> 0 : slabdata    228    228      0
> 
> One more point, say I have each port attached to there own bridge, and
> when I route traffic from one host to other, I'll not face this
> behavior, This behavior is relevant only in above case.
> 
> Do I have to do any other setting to overcome out of memory problem? 
> Also, Please let me know if any other information is required to isolate
> this behavior.

What network device driver, that is probably the problem?

Re: [Bridge] Out of memory problem

[Tekale Sharad-FHJN78]

> What network device driver, that is probably the problem?
We are using Openwrt on our AP, and the driver is mvswitch.c its an
marvell driver.

If possible, can you point me to some piece of code in the driver to
check for memory leak?

Thanks Again,
Sharad.

-----Original Message-----
From: Stephen Hemminger [mailto:shemminger@vyatta.com] 
Sent: Tuesday, April 14, 2009 1:05 AM
To: Tekale Sharad-FHJN78
Cc: bridge@lists.linux-foundation.org
Subject: Re: [Bridge] Out of memory problem

On Tue, 14 Apr 2009 13:21:13 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

>  
> 
> Hi Stephen
> > Most likely, you have created a loop and are not running spanning
> tree. When a loop happens, packets multiply until all > resources are 
> exhausted.
> 
> > you have created a loop.
> Ya, I have bridge named br-lan created by($brctl), and I have attached

> two interfaces to it, and I'm sending traffic from PC1 to PC2. Some 
> thing like this
> 
>  	    		 ----------------------- 
> 	   		|    ------------       |
> 	   		|   |   br-lan   |      |    <--- AP
> 	   		|    ------------       |      
> 	   		|  eth0.0       eth0.1  | 
> 	    		 ----|------------|-----
> 
> 	      	     |	      |
> 	      	     |	      |
>   	     		    PC1          PC2
> 
> Previously:
> root@none:/$ brctl show
> bridge name     bridge id               STP enabled     interfaces
> br-lan          8000.001570d8d8fd       no              eth0.0
>                                                         eth0.1
> 
> >are not running spanning tree.
> Is it always necessary to enable spanning tree when traffic is sent 
> over loop?
> 
> I have enabled spanning tree using
> 
> root@none:/$ brctl stp br-lan on
> root@none:/$ brctl show
> bridge name     bridge id               STP enabled     interfaces
> br-lan          8000.001570d8d8fd       yes             eth0.0
>                                                         eth0.1
> 
> But again, when huge traffic is pumped from PC1 to PC2, The 
> skbuff_head_cache value is increased rapidly and even after stop 
> pumping the traffic, this value doesn't come down, and system goes for

> crash or reboot because of out of memory.
> 
> root@none:/$ cat /proc/slabinfo | grep skbuff_head
> skbuff_head_cache   4560   4560    192   20    1 : tunables  120   60
> 0 : slabdata    228    228      0
> 
> One more point, say I have each port attached to there own bridge, and

> when I route traffic from one host to other, I'll not face this 
> behavior, This behavior is relevant only in above case.
> 
> Do I have to do any other setting to overcome out of memory problem? 
> Also, Please let me know if any other information is required to 
> isolate this behavior.

What network device driver, that is probably the problem?

Re: [Bridge] Out of memory problem

[Stephen Hemminger]

On Tue, 14 Apr 2009 14:10:01 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

> > What network device driver, that is probably the problem?
> We are using Openwrt on our AP, and the driver is mvswitch.c its an
> marvell driver.
> 
> If possible, can you point me to some piece of code in the driver to
> check for memory leak?
> 
> Thanks Again,
> Sharad.

If the driver was open source and in one of the standard distributions,
it could be reviewed. But since you are on an older kernel with a non-standard
driver there really isn't a lot of interest on my part to help. Sorry.

Re: [Bridge] Out of memory problem

[Tekale Sharad-FHJN78]

 
>If the driver was open source and in one of the standard distributions,
it could be reviewed. But since you are on an 
>older kernel with a non-standard driver there really isn't a lot of
interest on my part to help. Sorry.

That's fine and thanks for guiding till now.

But, Please look at this case once:
Now each of ports are attached to there own bridge instead of one, and
when I route traffic from one host to other, We will never face this
problem.

root@none:/$ brctl addbr br-lan1
root@none:/$ brctl addif br-lan1 eth0.0
root@none:/$ brctl stp br-lan1 on

root@none:/$ brctl addbr br-lan2
root@none:/$ brctl addif br-lan2 eth0.1
root@none:/$ brctl stp br-lan2 on

root@none:/$ brctl show
bridge name     bridge id               STP enabled     interfaces
br-lan1         8000.001570d8d8fd       yes             eth0.0
br-lan2         8000.001570d8d8fd       yes             eth0.1

In this case, even if I send huge traffic the throughput is decreased
but device is not going out of memory.

Hence I feel there is some leak in bridge instead of driver

root@OpenWrt:/$ brctl --version
bridge-utils, 1.0.6.

If possible, please check this.

Thanks,
Sharad.



-----Original Message-----
From: Stephen Hemminger [mailto:shemminger@vyatta.com] 
Sent: Tuesday, April 14, 2009 10:08 AM
To: Tekale Sharad-FHJN78
Cc: bridge@lists.linux-foundation.org
Subject: Re: [Bridge] Out of memory problem

On Tue, 14 Apr 2009 14:10:01 +0800
"Tekale Sharad-FHJN78" <FHJN78@motorola.com> wrote:

> > What network device driver, that is probably the problem?
> We are using Openwrt on our AP, and the driver is mvswitch.c its an 
> marvell driver.
> 
> If possible, can you point me to some piece of code in the driver to 
> check for memory leak?
> 
> Thanks Again,
> Sharad.

If the driver was open source and in one of the standard distributions,
it could be reviewed. But since you are on an older kernel with a
non-standard driver there really isn't a lot of interest on my part to
help. Sorry.