[Bridge] Freezing bridges

[Bridge] Freezing bridges

[Jeroen Hoek]

I have the following situation:

A Linux 2.4.22 server:

Physical ethernet:
lan1 - Ethernet port (Internet)
lan2 - Ethernet port (Cisco Switch, Trunk port)

Tunnel:
tap1 - OpenVPN in tcp-server mode

VLAN's:
lan2.201
lan2.202
lan2.203
tap1.201
tap1.202
tap1.203

Bridges:
br201 (lan2.201, tap1.201)
br202 (lan2.202, tap1.202)
br203 (lan2.203, tap1.203)

On the other end, the same type of server:

Physical ethernet:
lan1 - Ethernet port (Internet)
lan2 - Ethernet port (LAN)
lan3 - Ethernet port (LAN)
lan4 - Ethernet port (LAN)

Tunnel:
tap1 - OpenVPN in tcp-client mode

VLAN's:
tap1.201
tap1.202
tap1.203

Bridges:
br201 (lan2, tap1.201)
br202 (lan3, tap1.202)
br203 (lan4, tap1.203)

Both servers are connected over Internet. This setup works. The server
with the trunk port actually has more bridges; there are 3 servers that
connect to it thru an OpenVPN tunnel, and all 3 server have 3 unique
VLAN's bridged to it.

The problem is, that although in testing the setup everything runs
stable, in reality, it doesn't. The first server crashes irregularly,
but usually after a day or 2/3. The other servers have the exact same
hardware, and crash aswell, albeit after a longer period of time. There
is no indication as to what goes wrong. The servers still reply to ping
and xinetd-daytime, but ssh is gone, as are the tunnels. The log shows
nothing. The hardware watchdog does not reset and seems to keep
receiving signals from the kernel module to prevent it from rebooting.

I don't know anymore. I replaced vtun with OpenVPN for the ethernet
tunneling, but that does not seem to matter.

Please advice, I am lost. Would upgrading to kernel 2.6.1 be a valid
solution? Am I overreaching the possibilities of the bridging
functionality? Am I overlooking some pitfall?

With kind and desperate regards,

ing. Jeroen Hoek

Re: [Bridge] Freezing bridges

[Stephen Hemminger]

On Wed, 14 Jan 2004 17:09:54 +0100
Jeroen Hoek <jeroen.hoek@dcf.nl> wrote:

> I have the following situation:
> 
> A Linux 2.4.22 server:
> 
> Physical ethernet:
> lan1 - Ethernet port (Internet)
> lan2 - Ethernet port (Cisco Switch, Trunk port)
> 
> Tunnel:
> tap1 - OpenVPN in tcp-server mode
> 
> VLAN's:
> lan2.201
> lan2.202
> lan2.203
> tap1.201
> tap1.202
> tap1.203
> 
> Bridges:
> br201 (lan2.201, tap1.201)
> br202 (lan2.202, tap1.202)
> br203 (lan2.203, tap1.203)
> 
> On the other end, the same type of server:
> 
> Physical ethernet:
> lan1 - Ethernet port (Internet)
> lan2 - Ethernet port (LAN)
> lan3 - Ethernet port (LAN)
> lan4 - Ethernet port (LAN)
> 
> Tunnel:
> tap1 - OpenVPN in tcp-client mode
> 
> VLAN's:
> tap1.201
> tap1.202
> tap1.203
> 
> Bridges:
> br201 (lan2, tap1.201)
> br202 (lan3, tap1.202)
> br203 (lan4, tap1.203)
> 
> Both servers are connected over Internet. This setup works. The server
> with the trunk port actually has more bridges; there are 3 servers that
> connect to it thru an OpenVPN tunnel, and all 3 server have 3 unique
> VLAN's bridged to it.
> 
> The problem is, that although in testing the setup everything runs
> stable, in reality, it doesn't. The first server crashes irregularly,
> but usually after a day or 2/3. The other servers have the exact same
> hardware, and crash aswell, albeit after a longer period of time. There
> is no indication as to what goes wrong. The servers still reply to ping
> and xinetd-daytime, but ssh is gone, as are the tunnels. The log shows
> nothing. The hardware watchdog does not reset and seems to keep
> receiving signals from the kernel module to prevent it from rebooting.
> 
> I don't know anymore. I replaced vtun with OpenVPN for the ethernet
> tunneling, but that does not seem to matter.
> 
> Please advice, I am lost. Would upgrading to kernel 2.6.1 be a valid
> solution? Am I overreaching the possibilities of the bridging
> functionality? Am I overlooking some pitfall?
> 
> With kind and desperate regards,
> 
> ing. Jeroen Hoek
> 
> _______________________________________________
> Bridge mailing list
> Bridge@lists.osdl.org
> http://lists.osdl.org/mailman/listinfo/bridge

Sounds like the bridge freezes, but the server doesn't really crash.
Is there any console output or message log when this occurs?

Are you running spanning tree protocol?
Are the machines SMP or UP?
What is the output of brctl show and ifconfig when the network freezes?

Re: [Bridge] Freezing bridges

[Jeroen Hoek]

On Wed, 2004-01-14 at 21:43, Stephen Hemminger wrote:
> Sounds like the bridge freezes, but the server doesn't really crash.
> Is there any console output or message log when this occurs?
> 
> Are you running spanning tree protocol?
> Are the machines SMP or UP?
> What is the output of brctl show and ifconfig when the network freezes?

Well, the servers become unapproachable. Ssh fails (although ping and
daytime/xinetd work) and attaching a monitor and keyboard to the
physical machine shows only a blank screen instead of a loginprompt. The
logs show nothing at all afterwards. There seems to be some life left in
the server during these freezes, but no ssh or tunnels.

I have STP turned on on the bridges, although there is no chance of a
networkloop. Should this be turned off?

All machines have a single Pentium III 1GHz CPU and 256mb of SDRAM. Each
has four Intel NICs, using the e100 driver. The eepro driver has
significant MTU issues so it is not an option in this case.

Kind regards,

ing. Jeroen Hoek

Re: [Bridge] Freezing bridges

[Stephen Hemminger]

On Thu, 15 Jan 2004 09:29:11 +0100
Jeroen Hoek <jeroen.hoek@dcf.nl> wrote:

> On Wed, 2004-01-14 at 21:43, Stephen Hemminger wrote:
> > Sounds like the bridge freezes, but the server doesn't really crash.
> > Is there any console output or message log when this occurs?
> > 
> > Are you running spanning tree protocol?
> > Are the machines SMP or UP?
> > What is the output of brctl show and ifconfig when the network freezes?
> 
> Well, the servers become unapproachable. Ssh fails (although ping and
> daytime/xinetd work) and attaching a monitor and keyboard to the
> physical machine shows only a blank screen instead of a loginprompt. The
> logs show nothing at all afterwards. There seems to be some life left in
> the server during these freezes, but no ssh or tunnels.
> 
> I have STP turned on on the bridges, although there is no chance of a
> networkloop. Should this be turned off?

Unless, you need to discover other bridges turn it off.
Just looking for what could disturb your system after it is running.
In 2.6 (and later 2.4) kernels STP defaults to off, to avoid potential
problems with those that don't need it.

You could try 2.6 (2.6.1) on one machine, it would at least provide information
about whether this problem has already been fixed.

Re: [Bridge] Freezing bridges

[Jeroen Hoek]

On Fri, 2004-01-16 at 20:07, Stephen Hemminger wrote:
> On Thu, 15 Jan 2004 09:29:11 +0100
> Jeroen Hoek <jeroen.hoek@dcf.nl> wrote:
> 
> > On Wed, 2004-01-14 at 21:43, Stephen Hemminger wrote:
> > > Sounds like the bridge freezes, but the server doesn't really crash.
> > > Is there any console output or message log when this occurs?
> > > 
> > > Are you running spanning tree protocol?
> > > Are the machines SMP or UP?
> > > What is the output of brctl show and ifconfig when the network freezes?
> > 
> > Well, the servers become unapproachable. Ssh fails (although ping and
> > daytime/xinetd work) and attaching a monitor and keyboard to the
> > physical machine shows only a blank screen instead of a loginprompt. The
> > logs show nothing at all afterwards. There seems to be some life left in
> > the server during these freezes, but no ssh or tunnels.
> > 
> > I have STP turned on on the bridges, although there is no chance of a
> > networkloop. Should this be turned off?
> 
> Unless, you need to discover other bridges turn it off.
> Just looking for what could disturb your system after it is running.
> In 2.6 (and later 2.4) kernels STP defaults to off, to avoid potential
> problems with those that don't need it.
> 
> You could try 2.6 (2.6.1) on one machine, it would at least provide information
> about whether this problem has already been fixed.

So far the central machine and the three satellite machines have been
running stable since thursday after disabling STP. I'm hoping that
solved it. :)

I'm keeping the 2.6 option in case the problems aren't gone yet, I have
the systems ready to switch to kernel 2.6 in a moments notice.