[Bridge] How to connect more than 200 interfaces to a bridge

[Bridge] How to connect more than 200 interfaces to a bridge

[Ali Shirvani]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

Hello everyone,

It seems we reach the Linux bridge limitation on the number of interfaces in a single bridge. Currently, we have 210 tap interface in a bridge, and we suffer from more than 50% packet loss when we ping the IP address of the virtual machine that uses one of the tap interfaces in the bridge.
Do you know how we can connect more than 200 VMs virtual interfaces to a bridge?

Best regards,
Ali

Sent with [Proton Mail](https://proton.me/) secure email.

[-- Attachment #2: Type: text/html, Size: 1775 bytes --]

Re: [Bridge] How to connect more than 200 interfaces to a bridge

[Stephen Hemminger]

On Wed, 09 Nov 2022 10:51:27 +0000
Ali Shirvani via Bridge <bridge@lists.linux-foundation.org> wrote:

> Hello everyone,
> 
> It seems we reach the Linux bridge limitation on the number of interfaces in a single bridge. Currently, we have 210 tap interface in a bridge, and we suffer from more than 50% packet loss when we ping the IP address of the virtual machine that uses one of the tap interfaces in the bridge.
> Do you know how we can connect more than 200 VMs virtual interfaces to a bridge?
> 
> Best regards,
> Ali
> 
> Sent with [Proton Mail](https://proton.me/) secure email.

The upper limit on interfaces per bridge should be 1023.
That limitation comes from spanning tree.

You might bet able to improve performance by disabling flooding to those tap devices.
Normally, any broadcast/unknown/multicast must be copied and flooded to each interface.

Re: [Bridge] How to connect more than 200 interfaces to a bridge

[Ali Shirvani]

Sent with Proton Mail secure email.

------- Original Message -------
On Wednesday, November 9th, 2022 at 8:21 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:


> On Wed, 09 Nov 2022 10:51:27 +0000
> Ali Shirvani via Bridge bridge@lists.linux-foundation.org wrote:
> 
> > Hello everyone,
> > 
> > It seems we reach the Linux bridge limitation on the number of interfaces in a single bridge. Currently, we have 210 tap interface in a bridge, and we suffer from more than 50% packet loss when we ping the IP address of the virtual machine that uses one of the tap interfaces in the bridge.
> > Do you know how we can connect more than 200 VMs virtual interfaces to a bridge?
> > 
> > Best regards,
> > Ali
> > 
> > Sent with Proton Mail secure email.
> 
> 
> The upper limit on interfaces per bridge should be 1023.
> That limitation comes from spanning tree.
> 
> You might bet able to improve performance by disabling flooding to those tap devices.
> Normally, any broadcast/unknown/multicast must be copied and flooded to each interface.

Thanks a lot for your guidance. I disabled the spanning tree on the bridge with `brctl stp br0 off` but the issue does not resolve. Would you please elaborate more about disabling flooding on tap devices, I don't know how I should disable flooding on tap devices.

Re: [Bridge] How to connect more than 200 interfaces to a bridge

[Stephen Hemminger]

On Wed, 09 Nov 2022 19:25:32 +0000
Ali Shirvani <alishirv@protonmail.com> wrote:

> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Wednesday, November 9th, 2022 at 8:21 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> 
> > On Wed, 09 Nov 2022 10:51:27 +0000
> > Ali Shirvani via Bridge bridge@lists.linux-foundation.org wrote:
> >   
> > > Hello everyone,
> > > 
> > > It seems we reach the Linux bridge limitation on the number of interfaces in a single bridge. Currently, we have 210 tap interface in a bridge, and we suffer from more than 50% packet loss when we ping the IP address of the virtual machine that uses one of the tap interfaces in the bridge.
> > > Do you know how we can connect more than 200 VMs virtual interfaces to a bridge?
> > > 
> > > Best regards,
> > > Ali
> > > 
> > > Sent with Proton Mail secure email.  
> > 
> > 
> > The upper limit on interfaces per bridge should be 1023.
> > That limitation comes from spanning tree.
> > 
> > You might bet able to improve performance by disabling flooding to those tap devices.
> > Normally, any broadcast/unknown/multicast must be copied and flooded to each interface.  
> 
> Thanks a lot for your guidance. I disabled the spanning tree on the bridge with `brctl stp br0 off` but the issue does not resolve. Would you please elaborate more about disabling flooding on tap devices, I don't know how I should disable flooding on tap devices.

It is not a spanning tree issue, in fact STP can protect you from bad VM's.
It is more about configuring the bridge ports after setup.

Re: [Bridge] How to connect more than 200 interfaces to a bridge

[Stephen Hemminger]

On Wed, 09 Nov 2022 19:25:32 +0000
Ali Shirvani <alishirv@protonmail.com> wrote:

> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Wednesday, November 9th, 2022 at 8:21 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> 
> > On Wed, 09 Nov 2022 10:51:27 +0000
> > Ali Shirvani via Bridge bridge@lists.linux-foundation.org wrote:
> >   
> > > Hello everyone,
> > > 
> > > It seems we reach the Linux bridge limitation on the number of interfaces in a single bridge. Currently, we have 210 tap interface in a bridge, and we suffer from more than 50% packet loss when we ping the IP address of the virtual machine that uses one of the tap interfaces in the bridge.
> > > Do you know how we can connect more than 200 VMs virtual interfaces to a bridge?
> > > 
> > > Best regards,
> > > Ali
> > > 
> > > Sent with Proton Mail secure email.  
> > 
> > 
> > The upper limit on interfaces per bridge should be 1023.
> > That limitation comes from spanning tree.
> > 
> > You might bet able to improve performance by disabling flooding to those tap devices.
> > Normally, any broadcast/unknown/multicast must be copied and flooded to each interface.  
> 
> Thanks a lot for your guidance. I disabled the spanning tree on the bridge with `brctl stp br0 off` but the issue does not resolve. Would you please elaborate more about disabling flooding on tap devices, I don't know how I should disable flooding on tap devices.

Look at the documentation of the bridge command which describes per-port options:
https://man7.org/linux/man-pages/man8/bridge.8.html

You do want to leave flooding on for the downstream bridge port.

You can also add some security by limiting where/when STP comes from and disable
learning on the TAP devices so that if VM sends bogus packets, the bridge won't get DoS.