[Bridge] Strange problem, please help

[Bridge] Strange problem, please help

[Jaime Nebrera]

  Hi all,

  We are experiencing a very strange problem and would need some help.
We have a Leaf based box (actually a Lince box kernel 2.4.26) running as
a bridge with 8 gigabit ethernets, PIV 3Ghz, 2GB RAM. 4 of them share
the same PCI Express and the other 4 a different PCI bus. We have NAPI
enabled on all ethernets and IRQ moderation enabled (dynamic)

  Some ASCII art before proceeding.

     Router 1               Router 2
        |                       |
        --------- Switch --------
                     |
                     |
                  Firewall

 
   WAN  LAN Empty Empty Empty Empty Empty Empty
    |    |     |     |     |     |     |     |
   eth0 eth1 eth2  eth3  eth4   eth5  eth6  eth7
    -----------------      -------------------
          PCI-X                     PCI

  Both routers use HSRP from Cisco to share information about who is
alive. This app uses multicast UDP packets to 224.0.0.1 address, port
1985.

  The problem is, after a while (1 or 2 minutes) the CPU reaches 100%
(0.99 load 99% System) with the process ksoftirqd_CPU0 reaching 99%.
Using iptraf we discover ethernets 4 to 7 (the ones that share the PCI
bus) are at full speed. The traffic is on port 1985 and comes from the 2
virtual IP from the redundant routers. It seems they enter an infinite
loop and completely kill the system. BTW, the only used ethernets are 0
and 1, both on the PCI-X bus, and eth2 and eth3 seem unaffected (no
traffic). Bear in mind, real traffic on eth0 and eth1 doesnt surpass
1Mbps. Also, no service is provided at this point, not even firewalling.

  The problem appears with and without STP activated and we have
verified there is not a loop in the network.

  If we disable ethernets from 4 to 7 (ip link set ethx down) the
problem seems to disappear, but we are not sure as we didnt want to
disturb the client more time (actually, for 15 minutes the problem didnt
appear, while the other way it appeared in much less than 5 minutes). In
this case, even activating things like a Netflow probe in eth0 didnt
disturb at all the system.

  The same problem seems to appear with a Via 1Ghz box with 4 realtek
ethernets and around 4Mbps of traffic (this system was placed under
heavier load, and as the problem appeared, we tested with the big box 
the same afternoon). When the problem appeared this box was so slow 
we could not even make a ssh session so we dont know if this is the 
same problem (but bet it is).

  So, some questions:

  1) Is this related to running as a bridge? Would this problem
disappear if we used a pseudo bridge (proxy ARP)?

  2) Can such a beast sustain 8 ethernets as a single bridge? Bear in
mind they dont have gigabit traffic, they just use gigabit ethernets :)
Whats the limit for a linux bridge? Would be better to break it into two
bridges?

  3) As this traffic is only needed on both routers but doesnt need to
pass trough the firewall, will dropping it on eth0 solve the problem?
(That way there is no way the packets enter into other ethernet ports)
What would happen with other multicast based apps? Would they need to be
dropped too?

  Very thankful in advance. Regards.

-- 
Jaime Nebrera - jnebrera@eneotecnologia.com
Consultor TI - ENEO Tecnologia SL
Telf.- 95 455 40 62 - 619 04 55 18

Re: [Bridge] Strange problem, please help

[Ryan McConigley]

At 09:25 AM 30/05/2005 +0200, Jaime Nebrera wrote:
>   Hi all,

         <snip>

>   So, some questions:
>
>   1) Is this related to running as a bridge? Would this problem
>disappear if we used a pseudo bridge (proxy ARP)?
>
>   2) Can such a beast sustain 8 ethernets as a single bridge? Bear in
>mind they dont have gigabit traffic, they just use gigabit ethernets :)
>Whats the limit for a linux bridge? Would be better to break it into two
>bridges?

         Just my $0.02 worth, no solutions I'm afraid, just an 
observation.  The behavour you describe is virtually identical to the 
behavour I had on the first bridge I constructed which was using tulip 
network cards.  The system would work wonderfully in test, but put it in 
situ on the network it would last a few minutes, then lock up with the CPU 
maxed out.  We ended up changing the tulip cards to Intels which worked 
perfectly.

         The weird thing was on their own, the tulip cards worked fine, but 
couldn't handle a bridge config.  At the time folks suggested that it was a 
combined interrupt/timing/buffering problem, but I didn't have the skills 
or time to track it down.  From what you've said about the problem going 
away when the other network ports are disabled, I wouldn't mind betting its 
a related issue.  8 Gigabit ports would be a substantial number of 
interrupts, so I wouldn't be surprised if you're starting to max out the 
PCI bus, but I don't have any hard numbers to test that theory.

         Cheers,
                 Ryan.
--
           Ryan McConigley - Systems Administrator                  _.-,
      Computer Science   University of Western Australia        .--'  '-._
        Tel: (+61 8) 6488 7082 - Fax: (+61 8) 6488 1089       _/`-  _      '.
Ryan[@]csse.uwa.edu.au - http://www.csse.uwa.edu.au/~ryan  '----'._`.----. \
                                                                      `     \;
  "You're just jealous because the voices are talking to me"                ;_\

Re: [Bridge] Strange problem, please help

[Jaime Nebrera]

  Hi Ryan,

>          Just my $0.02 worth, no solutions I'm afraid, just an 
> observation.  The behavour you describe is virtually identical to the 
> behavour I had on the first bridge I constructed which was using tulip 
> network cards.  The system would work wonderfully in test, but put it in 
> situ on the network it would last a few minutes, then lock up with the CPU 
> maxed out.  We ended up changing the tulip cards to Intels which worked 
> perfectly.
> 
>          The weird thing was on their own, the tulip cards worked fine, but 
> couldn't handle a bridge config.  At the time folks suggested that it was a 
> combined interrupt/timing/buffering problem, but I didn't have the skills 
> or time to track it down.  From what you've said about the problem going 
> away when the other network ports are disabled, I wouldn't mind betting its 
> a related issue.  8 Gigabit ports would be a substantial number of 
> interrupts, so I wouldn't be surprised if you're starting to max out the 
> PCI bus, but I don't have any hard numbers to test that theory.

  IMHO I dont agree. We are using chipsets from Marvel, a well
stablished ones. Also, this might be a problem if the traffic was really
important, say 200Mbps or even 100Mbps, but this is not the case for
0,5Mbps. Even those cheap Realtek 100Mbps ethernets can sustain that
traffic as a bridge.

  The problem is WHY suddenly some traffic appears on ethernet ports
that are not in use (and why not in two others) and WHY this traffic
keeps accelerating until it reaches a speed the hardware wont support
(2400Mbps total) Of course when such traffic is reached the system blows
away, the point is, why that traffic appears?

  Regarding this point, we are using the same software solution with a
much cheaper box (Via 1Ghz, 4x Realtek 100 Ethernets) working as a
bridge and have been able to mange more than 20Mbps without a problem.
The difference is they dont have that HSRP set up.

  So it is related to HSRP or something related to IRQ as somebody has
suggested me on other email. The problem here is why this IRQ problem
didnt appear in our test environment (without HSRP) with more traffic
and open sessions than the client had?

  Still, thanks for your 2 cents :)

  Regards

-- 
Jaime Nebrera - jnebrera@eneotecnologia.com
Consultor TI - ENEO Tecnologia SL
Telf.- 95 455 40 62 - 619 04 55 18

[Bridge] Source code bug or doubt - Not freeing port struct

[Jorge Lanza]

Hi all.

During the last few days I've been going through the code in order to 
understand how can I hack it to make the things I want to a packet on layer 2.
First of all, congratulations. I have learnt a lot.

However I have a doubt. In n<http://lxr.linux.no/ident?i=new_nbp>ew_nbp() 
funtion in br_if.c you make a kalloc for the port. While reading the code I 
haven't seen any kfree for that memory, which should be done in 
<http://lxr.linux.no/ident?i=del_nbp>del_nbp or similar functions. In 
<http://lxr.linux.no/ident?i=destroy_nbp>destroy_nbp you set the pointer to 
null, but I think that's not enough. I don't know if it is a bug or is sth 
that is done somewhere else.

I would appreciate if you can give me some hint on that.
TA.

Re: [Bridge] Strange problem, please help

[Stephen Hemminger]

On Tue, 31 May 2005 09:06:47 +0200
Jaime Nebrera <jnebrera@eneotecnologia.com> wrote:

>   Hi Ryan,
> 
> >          Just my $0.02 worth, no solutions I'm afraid, just an 
> > observation.  The behavour you describe is virtually identical to the 
> > behavour I had on the first bridge I constructed which was using tulip 
> > network cards.  The system would work wonderfully in test, but put it in 
> > situ on the network it would last a few minutes, then lock up with the CPU 
> > maxed out.  We ended up changing the tulip cards to Intels which worked 
> > perfectly.
> > 
> >          The weird thing was on their own, the tulip cards worked fine, but 
> > couldn't handle a bridge config.  At the time folks suggested that it was a 
> > combined interrupt/timing/buffering problem, but I didn't have the skills 
> > or time to track it down.  From what you've said about the problem going 
> > away when the other network ports are disabled, I wouldn't mind betting its 
> > a related issue.  8 Gigabit ports would be a substantial number of 
> > interrupts, so I wouldn't be surprised if you're starting to max out the 
> > PCI bus, but I don't have any hard numbers to test that theory.
> 
>   IMHO I dont agree. We are using chipsets from Marvel, a well
> stablished ones. Also, this might be a problem if the traffic was really
> important, say 200Mbps or even 100Mbps, but this is not the case for
> 0,5Mbps. Even those cheap Realtek 100Mbps ethernets can sustain that
> traffic as a bridge.

If you use the sk98lin driver it wants to do it's own spanning tree type
failover so it multicasts STP packets all the time.

>   The problem is WHY suddenly some traffic appears on ethernet ports
> that are not in use (and why not in two others) and WHY this traffic
> keeps accelerating until it reaches a speed the hardware wont support
> (2400Mbps total) Of course when such traffic is reached the system blows
> away, the point is, why that traffic appears?

You need to somehow figure out what those packets are (ethereal, tcpdump, ...)
and go from there.

>   Regarding this point, we are using the same software solution with a
> much cheaper box (Via 1Ghz, 4x Realtek 100 Ethernets) working as a
> bridge and have been able to mange more than 20Mbps without a problem.
> The difference is they dont have that HSRP set up.

You need to use a driver with NAPI or the interrupt per packet overhead
will swamp the box.

>   So it is related to HSRP or something related to IRQ as somebody has
> suggested me on other email. The problem here is why this IRQ problem
> didnt appear in our test environment (without HSRP) with more traffic
> and open sessions than the client had?
> 
>   Still, thanks for your 2 cents :)
> 
>   Regards
> 
> -- 
> Jaime Nebrera - jnebrera@eneotecnologia.com
> Consultor TI - ENEO Tecnologia SL
> Telf.- 95 455 40 62 - 619 04 55 18
> 

Re: [Bridge] Source code bug or doubt - Not freeing port struct

[Stephen Hemminger]

On Tue, 31 May 2005 10:45:28 +0200
Jorge Lanza <jlanza@tlmat.unican.es> wrote:

> Hi all.
> 
> During the last few days I've been going through the code in order to 
> understand how can I hack it to make the things I want to a packet on layer 2.
> First of all, congratulations. I have learnt a lot.
> 
> However I have a doubt. In n<http://lxr.linux.no/ident?i=new_nbp>ew_nbp() 
> funtion in br_if.c you make a kalloc for the port. While reading the code I 
> haven't seen any kfree for that memory, which should be done in 
> <http://lxr.linux.no/ident?i=del_nbp>del_nbp or similar functions. In 
> <http://lxr.linux.no/ident?i=destroy_nbp>destroy_nbp you set the pointer to 
> null, but I think that's not enough. I don't know if it is a bug or is sth 
> that is done somewhere else.

What kernel version?
	For 2.6
		kmalloc of port data is in br_if:new_nbp
		kfree is done by:
		del_nbp calls destroy_nbp (via rcu)
		   calls br_sysfs_freeif which does kobject_put(of port kobj)
			when ref == 0 calls brport_release (through brport_ktype)

This complexity comes because application could have /sys/class/net/ethX/brif/port_id
or other file open, in which case the bridge port can't be deleted. Until last
reference is zero.
		

Re: [Bridge] Strange problem, please help

[Jaime Nebrera]

  Hi,

> If you use the sk98lin driver it wants to do it's own spanning tree type
> failover so it multicasts STP packets all the time.

  Ok, I will ask my teammate

> You need to somehow figure out what those packets are (ethereal, tcpdump, ...)
> and go from there.

  They were HSRP packages going to IP multicast address 224.0.0.1 or
224.0.0.2 cant recall from the real IP of the two routers and going to
port 1985. These were clearly HSRP related packages.

> >   Regarding this point, we are using the same software solution with a
> > much cheaper box (Via 1Ghz, 4x Realtek 100 Ethernets) working as a
> > bridge and have been able to mange more than 20Mbps without a problem.
> > The difference is they dont have that HSRP set up.
> 
> You need to use a driver with NAPI or the interrupt per packet overhead
> will swamp the box.

  Yes, we are using NAPI with those Intel (100Mbps) and Marvel (GE).
Also, we discovered there is a port to Realtek with NAPI support for
kernel 2.4.x but we are just evaluating that and we are just trying to
figure if we are loading and using it right
(ftp://ftp.ovh.net/made-in-ovh/kernel/) Do you know how to know if its
using it or not? Yes, I know Realtek is not a pro driver, but this is
for an embedded system and cant be replaced.

  Regards

-- 
Jaime Nebrera - jnebrera@eneotecnologia.com
Consultor TI - ENEO Tecnologia SL
Telf.- 95 455 40 62 - 619 04 55 18