[Bridge] Bridging vSwitches in VMwares ESXi

[Bridge] Bridging vSwitches in VMwares ESXi

[Ryan Whelan]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

I'm having an issue bridging 2 virtual switches in VMwares ESXi.  I've made
a post on the VMware forums describing the issue (
http://communities.vmware.com/message/1507261#1507261).

I have searched the internet and found a post (
http://archives.free.net.ph/message/20100108.174704.efbb18cc.ja.html) by
someone having the exact same issue- it looks like that post was on this
list?

In short, I have a Linux VM in ESXi with 2 vNICs- one in each of 2 different
vSwitches.  A client (in my case a windows machine) on the second vSwitch
can't get the MAC address of the default gateway on the first vSwitch.
 Sniffing the traffic shows the arp broadcast from the windows machine
making if over the linux bridge and getting responded to by the cisco
gateway but the response never makes it back over the bridge.  Watching the
mac table in the linux bridge shows it mistakenly associates the mac address
of the windows machine to the wrong port (eth0 in my case, eth1 is the vNIC
plugged into the switch with the windows box)

Im not sure where the issue is; its really pretty simple setup.  Am I
missing something simple? Is there really a bug here?

Thanks!

[-- Attachment #2: Type: text/html, Size: 1465 bytes --]

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Robert LeBlanc]

On Tue, Apr 6, 2010 at 3:02 PM, Ryan Whelan <ryan.whelan@tbamerica.com> wrote:
>
> I'm having an issue bridging 2 virtual switches in VMwares ESXi.  I've made a post on the VMware forums describing the issue (http://communities.vmware.com/message/1507261#1507261).
> I have searched the internet and found a post (http://archives.free.net.ph/message/20100108.174704.efbb18cc.ja.html) by someone having the exact same issue- it looks like that post was on this list?
> In short, I have a Linux VM in ESXi with 2 vNICs- one in each of 2 different vSwitches.  A client (in my case a windows machine) on the second vSwitch can't get the MAC address of the default gateway on the first vSwitch.  Sniffing the traffic shows the arp broadcast from the windows machine making if over the linux bridge and getting responded to by the cisco gateway but the response never makes it back over the bridge.  Watching the mac table in the linux bridge shows it mistakenly associates the mac address of the windows machine to the wrong port (eth0 in my case, eth1 is the vNIC plugged into the switch with the windows box)
> Im not sure where the issue is; its really pretty simple setup.  Am I missing something simple? Is there really a bug here?
> Thanks!

You do not wan to bridge in a VMWare environment, it will only drive
you to an early grave. I've blogged my experience with this problem at
http://robert.leblancnet.us/ you will need a google wave account to
view it. In short use proxy arp instead if you can.

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Ryan King]

On Tue, Apr 6, 2010 at 3:02 PM, Ryan Whelan <ryan.whelan@tbamerica.com>
wrote:
>
> I'm having an issue bridging 2 virtual switches in VMwares ESXi.  I've
made a post on the VMware forums describing the issue
(http://communities.vmware.com/message/1507261#1507261).
> I have searched the internet and found a post
(http://archives.free.net.ph/message/20100108.174704.efbb18cc.ja.html) by
someone having the exact same issue- it looks like that post was on this
list?
> In short, I have a Linux VM in ESXi with 2 vNICs- one in each of 2
different vSwitches.  A client (in my case a windows machine) on the second
vSwitch can't get the MAC address of the default gateway on the first
vSwitch.  Sniffing the traffic shows the arp broadcast from the windows
machine making if over the linux bridge and getting responded to by the
cisco gateway but the response never makes it back over the bridge.
 Watching the mac table in the linux bridge shows it mistakenly associates
the mac address of the windows machine to the wrong port (eth0 in my case,
eth1 is the vNIC plugged into the switch with the windows box)
> Im not sure where the issue is; its really pretty simple setup.  Am I
missing something simple? Is there really a bug here?
> Thanks!

You do not wan to bridge in a VMWare environment, it will only drive
you to an early grave. I've blogged my experience with this problem at
http://robert.leblancnet.us/ you will need a google wave account to
view it. In short use proxy arp instead if you can.

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University




I've also had the exact same issues.  However, if I move one of the vNICs to
a vSwitch using a different physical nic, then the issue seems to go away.

Ryan King

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Robert LeBlanc]

On Tue, Apr 6, 2010 at 5:57 PM, Ryan King <ryank@globaldial.com> wrote:

> I've also had the exact same issues.  However, if I move one of the vNICs to
> a vSwitch using a different physical nic, then the issue seems to go away.

In my experience, as long as you only have on pNIC per vSwitch, it
works just fine. As soon as you add a second pNIC to the vSwitch, it
creates a loop and confuses the linux bridge (you can watch the MAC
bounce between ports if you do a standard ping from one machine and a
broadcast ping from another machine). We were using Broadcom NICs.


Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Ryan Whelan]

[-- Attachment #1: Type: text/plain, Size: 1533 bytes --]

oh good- glad to find out im not crazy. we do have 2 pNICs in the external
facing vSwitch.  Even when setting them as active/passive, its still an
issue.

so its a confirmed issue with vmware- do they have any intention on
correcting it? do we know?

On Tue, Apr 6, 2010 at 9:08 PM, Robert LeBlanc <robert@leblancnet.us> wrote:

> On Tue, Apr 6, 2010 at 5:57 PM, Ryan King <ryank@globaldial.com> wrote:
>
> > I've also had the exact same issues.  However, if I move one of the vNICs
> to
> > a vSwitch using a different physical nic, then the issue seems to go
> away.
>
> In my experience, as long as you only have on pNIC per vSwitch, it
> works just fine. As soon as you add a second pNIC to the vSwitch, it
> creates a loop and confuses the linux bridge (you can watch the MAC
> bounce between ports if you do a standard ping from one machine and a
> broadcast ping from another machine). We were using Broadcom NICs.
>
>
> Robert LeBlanc
> Life Sciences & Undergraduate Education Computer Support
> Brigham Young University
>
>
> _____________________________________
> This e-mail and any attachments from Toyota Boshoku America
> (TBA), TrimMasters Inc. (TMI), or other affiliated companies may
> contain confidential and privileged information.
>
> If you are not the intended recipient, please notify the sender
> immediately by return e-mail, delete this e-mail and destroy any
> copies.
>
> Any dissemination or use of this information by a person other
> than the intended recipient is unauthorized and may be illegal.
>

[-- Attachment #2: Type: text/html, Size: 2037 bytes --]

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Robert LeBlanc]

On Tue, Apr 6, 2010 at 8:41 PM, Ryan Whelan <ryan.whelan@tbamerica.com> wrote:
> oh good- glad to find out im not crazy. we do have 2 pNICs in the external
> facing vSwitch.  Even when setting them as active/passive, its still an
> issue.
>
> so its a confirmed issue with vmware- do they have any intention on
> correcting it? do we know?
>

Yes as long as the pNIC is attached, even if in standby mode it causes
a problem.

It is an issue, but they won't fix it for two reasons. 1. They will
never create a bridge between two vSwitches/VLANs, and 2. a fix would
introduce more overhead and reduce flexibility and since they adhere
to #1, it doesn't make sense. The flexibility they lose is multiple
pNICs to one or more switches without having to have any trunking
protocols.

You can get around the problem by buying the Cisco Nexus virtual
switch, it's a real layer 3 switch, but it's a pricy option.

If you want a bridge in a VM, then only pNIC per switch (no
redundancy). The other option, try to make the bridge a router
instead. For us we wanted a transparent firewall, so it was easy to
change the configuration to proxyarp for a transparent router instead.

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Ryan King]

> -----Original Message-----
> From: bridge-bounces@lists.linux-foundation.org [mailto:bridge-
> bounces@lists.linux-foundation.org] On Behalf Of Robert LeBlanc
> Sent: Wednesday, 7 April 2010 11:02 AM
> To: Ryan Whelan
> Cc: bridge@lists.linux-foundation.org
> Subject: Re: [Bridge] Bridging vSwitches in VMwares ESXi
> 
> On Tue, Apr 6, 2010 at 8:41 PM, Ryan Whelan <ryan.whelan@tbamerica.com>
> wrote:
> > oh good- glad to find out im not crazy. we do have 2 pNICs in the
> external
> > facing vSwitch.  Even when setting them as active/passive, its still
> an
> > issue.
> >
> > so its a confirmed issue with vmware- do they have any intention on
> > correcting it? do we know?
> >
> 
> Yes as long as the pNIC is attached, even if in standby mode it causes
> a problem.
> 
> It is an issue, but they won't fix it for two reasons. 1. They will
> never create a bridge between two vSwitches/VLANs, and 2. a fix would
> introduce more overhead and reduce flexibility and since they adhere
> to #1, it doesn't make sense. The flexibility they lose is multiple
> pNICs to one or more switches without having to have any trunking
> protocols.
> 
> You can get around the problem by buying the Cisco Nexus virtual
> switch, it's a real layer 3 switch, but it's a pricy option.
> 
> If you want a bridge in a VM, then only pNIC per switch (no
> redundancy). The other option, try to make the bridge a router
> instead. For us we wanted a transparent firewall, so it was easy to
> change the configuration to proxyarp for a transparent router instead.
> 

Just to clarify our setup:

The physical server has 4 physical nics.  vswitch1 uses 2 pnics in
active/active.  vswitch2 uses 2 pnics in active/active.

The VM (running openvpn) has a bridge with one vnic on vswitch1 and one vnic
on vswitch2.  Since moving the 2nd interface to vswitch2, I have not
experienced this issue any more (ie: previously, both vnics were on vswitch1
- this was where we saw this issue).

Ryan King

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Robert LeBlanc]

On Tue, Apr 6, 2010 at 9:21 PM, Ryan King <ryank@globaldial.com> wrote:
> Just to clarify our setup:
>
> The physical server has 4 physical nics.  vswitch1 uses 2 pnics in
> active/active.  vswitch2 uses 2 pnics in active/active.
>
> The VM (running openvpn) has a bridge with one vnic on vswitch1 and one vnic
> on vswitch2.  Since moving the 2nd interface to vswitch2, I have not
> experienced this issue any more (ie: previously, both vnics were on vswitch1
> - this was where we saw this issue).

We were using VLANs and bridging across VLANs, this may have
compounded our problem. We we put openvpn on it to connect two
datacenters, we got similar broadcasts storms over the VPN connection
as well.

I was going to do something similar to connect family networks using
openvpn at home until we ran into the problem. I think disbanded the
project, maybe I'll have to look into it again if VLANs were the
problem.

Robert LeBlanc
Life Sciences & Undergraduate Education Computer Support
Brigham Young University

Re: [Bridge] Bridging vSwitches in VMwares ESXi

[Ryan Whelan]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

it seems the fix for this would be for vmware to do a better job 'fencing'
the secondary pNIC when in an active/passive setup

im really disappointed to find out this isn't an issue i can work around
(without compromising redundancy, or replacing the vSwitch etc)

On Tue, Apr 6, 2010 at 11:01 PM, Robert LeBlanc <robert@leblancnet.us>wrote:

> On Tue, Apr 6, 2010 at 8:41 PM, Ryan Whelan <ryan.whelan@tbamerica.com>
> wrote:
> > oh good- glad to find out im not crazy. we do have 2 pNICs in the
> external
> > facing vSwitch.  Even when setting them as active/passive, its still an
> > issue.
> >
> > so its a confirmed issue with vmware- do they have any intention on
> > correcting it? do we know?
> >
>
> Yes as long as the pNIC is attached, even if in standby mode it causes
> a problem.
>
> It is an issue, but they won't fix it for two reasons. 1. They will
> never create a bridge between two vSwitches/VLANs, and 2. a fix would
> introduce more overhead and reduce flexibility and since they adhere
> to #1, it doesn't make sense. The flexibility they lose is multiple
> pNICs to one or more switches without having to have any trunking
> protocols.
>
> You can get around the problem by buying the Cisco Nexus virtual
> switch, it's a real layer 3 switch, but it's a pricy option.
>
> If you want a bridge in a VM, then only pNIC per switch (no
> redundancy). The other option, try to make the bridge a router
> instead. For us we wanted a transparent firewall, so it was easy to
> change the configuration to proxyarp for a transparent router instead.
>
> Robert LeBlanc
> Life Sciences & Undergraduate Education Computer Support
> Brigham Young University
>
>
> _____________________________________
> This e-mail and any attachments from Toyota Boshoku America
> (TBA), TrimMasters Inc. (TMI), or other affiliated companies may
> contain confidential and privileged information.
>
> If you are not the intended recipient, please notify the sender
> immediately by return e-mail, delete this e-mail and destroy any
> copies.
>
> Any dissemination or use of this information by a person other
> than the intended recipient is unauthorized and may be illegal.
>

[-- Attachment #2: Type: text/html, Size: 2737 bytes --]