Rx issues with Linux Bridge and thunderbolt-net

Rx issues with Linux Bridge and thunderbolt-net

[Arjun Mehta]

Hi there, I’d like to report what I believe to be a bug with either Linux Bridge (maybe and/or thunderbolt-net as well).

Problem: Rx on bridged Thunderbolt interface are blocked

Reported Behavior:
Tested on Proxmox host via iperf3, between B550 Vision D-P and MacBook Pro (2019 intel). On a direct interface, thunderbolt bridge Tx and Rx speeds are equal and full speed (in my case 9GB/s each). However, when a thunderbolt bridge is passed through via Linux Bridge to a VM or container (in my case a Proxmox LXC container or VM) the bridge achieves full Tx speeds, but Rx speeds are reporting limited to ~30kb/s

Expected:
The VM/CT should have the same general performance for Tx AND Rx as the host

Reproducing:
- Setup for the bridge was done by following this guide: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
- Both devices on Thunderbolt interfaces have static IPs
- VM is given the same IP, but unique MAC address
- BIOS has Thunderbolt security mode set to “No security”

Further reading:
The problem is outlined more with screenshots and further details in this Reddit post: https://www.reddit.com/r/Proxmox/comments/17kq5st/slow_rx_speed_from_thunderbolt_3_port_to_vm_over/.

Please let me know if there is any further action I can do to help investigate or where else I can direct the bug/concern

Thanks!
Arjun

Re: Rx issues with Linux Bridge and thunderbolt-net

[Stephen Hemminger]

On Thu, 7 Dec 2023 12:57:08 -0700
Arjun Mehta <arjunmeht@gmail.com> wrote:

> Hi there, I’d like to report what I believe to be a bug with either Linux Bridge (maybe and/or thunderbolt-net as well).
> 
> Problem: Rx on bridged Thunderbolt interface are blocked
> 
> Reported Behavior:
> Tested on Proxmox host via iperf3, between B550 Vision D-P and MacBook Pro (2019 intel). On a direct interface, thunderbolt bridge Tx and Rx speeds are equal and full speed (in my case 9GB/s each). However, when a thunderbolt bridge is passed through via Linux Bridge to a VM or container (in my case a Proxmox LXC container or VM) the bridge achieves full Tx speeds, but Rx speeds are reporting limited to ~30kb/s
> 
> Expected:
> The VM/CT should have the same general performance for Tx AND Rx as the host
> 
> Reproducing:
> - Setup for the bridge was done by following this guide: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
> - Both devices on Thunderbolt interfaces have static IPs
> - VM is given the same IP, but unique MAC address
> - BIOS has Thunderbolt security mode set to “No security”
> 
> Further reading:
> The problem is outlined more with screenshots and further details in this Reddit post: https://www.reddit.com/r/Proxmox/comments/17kq5st/slow_rx_speed_from_thunderbolt_3_port_to_vm_over/.
> 
> Please let me know if there is any further action I can do to help investigate or where else I can direct the bug/concern

Most likely this is a hardware issue on the thunderbolt interface where it will not
allow sending with a different source MAC address.  Some Wifi interfaces have this
problem.

Is Promox using a kernel from upstream Linux repository directly.
Netdev developers are unwilling to assist if there are any non-upstream kernel modules in use.

Re: Rx issues with Linux Bridge and thunderbolt-net

[Arjun Mehta]

Hi Stephen, thank you for the reply.

Proxmox does use a kernel derived from Ubuntu I believe (eg. kernel for Proxmox 8.1 which is what I'm using is 6.5.11-6-pve derived from Ubuntu 23.10). Not sure if there have been any modifications to the Linux Bridge in it.
https://pve.proxmox.com/wiki/Proxmox_VE_Kernel#Proxmox_VE_8.x

Long shot, but do you happen to know of any workarounds with the Wifi interface issue you mentioned that would mitigate this issue? Maybe they would apply here.

I will also post to the Proxmox forums about this issue to flag for them.

Arjun


> On Dec 7, 2023, at 3:37 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> On Thu, 7 Dec 2023 12:57:08 -0700
> Arjun Mehta <arjunmeht@gmail.com> wrote:
> 
>> Hi there, I’d like to report what I believe to be a bug with either Linux Bridge (maybe and/or thunderbolt-net as well).
>> 
>> Problem: Rx on bridged Thunderbolt interface are blocked
>> 
>> Reported Behavior:
>> Tested on Proxmox host via iperf3, between B550 Vision D-P and MacBook Pro (2019 intel). On a direct interface, thunderbolt bridge Tx and Rx speeds are equal and full speed (in my case 9GB/s each). However, when a thunderbolt bridge is passed through via Linux Bridge to a VM or container (in my case a Proxmox LXC container or VM) the bridge achieves full Tx speeds, but Rx speeds are reporting limited to ~30kb/s
>> 
>> Expected:
>> The VM/CT should have the same general performance for Tx AND Rx as the host
>> 
>> Reproducing:
>> - Setup for the bridge was done by following this guide: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
>> - Both devices on Thunderbolt interfaces have static IPs
>> - VM is given the same IP, but unique MAC address
>> - BIOS has Thunderbolt security mode set to “No security”
>> 
>> Further reading:
>> The problem is outlined more with screenshots and further details in this Reddit post: https://www.reddit.com/r/Proxmox/comments/17kq5st/slow_rx_speed_from_thunderbolt_3_port_to_vm_over/.
>> 
>> Please let me know if there is any further action I can do to help investigate or where else I can direct the bug/concern
> 
> Most likely this is a hardware issue on the thunderbolt interface where it will not
> allow sending with a different source MAC address.  Some Wifi interfaces have this
> problem.
> 
> Is Promox using a kernel from upstream Linux repository directly.
> Netdev developers are unwilling to assist if there are any non-upstream kernel modules in use.

Re: Rx issues with Linux Bridge and thunderbolt-net

[Arjun Mehta]

Also I want to clarify that the issue. You state "Most likely this is a hardware issue on the thunderbolt interface where it will not allow sending with a different source MAC address."

Actually the issue is that the interface will not _receive_. Transmission to another device from the interface works at full speed. Receiving is not working as expected.


> On Dec 8, 2023, at 10:30 AM, Arjun Mehta <arjunmeht@gmail.com> wrote:
> 
> Hi Stephen, thank you for the reply.
> 
> Proxmox does use a kernel derived from Ubuntu I believe (eg. kernel for Proxmox 8.1 which is what I'm using is 6.5.11-6-pve derived from Ubuntu 23.10). Not sure if there have been any modifications to the Linux Bridge in it.
> https://pve.proxmox.com/wiki/Proxmox_VE_Kernel#Proxmox_VE_8.x
> 
> Long shot, but do you happen to know of any workarounds with the Wifi interface issue you mentioned that would mitigate this issue? Maybe they would apply here.
> 
> I will also post to the Proxmox forums about this issue to flag for them.
> 
> Arjun
> 
> 
>> On Dec 7, 2023, at 3:37 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
>> 
>> On Thu, 7 Dec 2023 12:57:08 -0700
>> Arjun Mehta <arjunmeht@gmail.com> wrote:
>> 
>>> Hi there, I’d like to report what I believe to be a bug with either Linux Bridge (maybe and/or thunderbolt-net as well).
>>> 
>>> Problem: Rx on bridged Thunderbolt interface are blocked
>>> 
>>> Reported Behavior:
>>> Tested on Proxmox host via iperf3, between B550 Vision D-P and MacBook Pro (2019 intel). On a direct interface, thunderbolt bridge Tx and Rx speeds are equal and full speed (in my case 9GB/s each). However, when a thunderbolt bridge is passed through via Linux Bridge to a VM or container (in my case a Proxmox LXC container or VM) the bridge achieves full Tx speeds, but Rx speeds are reporting limited to ~30kb/s
>>> 
>>> Expected:
>>> The VM/CT should have the same general performance for Tx AND Rx as the host
>>> 
>>> Reproducing:
>>> - Setup for the bridge was done by following this guide: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
>>> - Both devices on Thunderbolt interfaces have static IPs
>>> - VM is given the same IP, but unique MAC address
>>> - BIOS has Thunderbolt security mode set to “No security”
>>> 
>>> Further reading:
>>> The problem is outlined more with screenshots and further details in this Reddit post: https://www.reddit.com/r/Proxmox/comments/17kq5st/slow_rx_speed_from_thunderbolt_3_port_to_vm_over/.
>>> 
>>> Please let me know if there is any further action I can do to help investigate or where else I can direct the bug/concern
>> 
>> Most likely this is a hardware issue on the thunderbolt interface where it will not
>> allow sending with a different source MAC address.  Some Wifi interfaces have this
>> problem.
>> 
>> Is Promox using a kernel from upstream Linux repository directly.
>> Netdev developers are unwilling to assist if there are any non-upstream kernel modules in use.
> 
> 

Re: Rx issues with Linux Bridge and thunderbolt-net

[Stephen Hemminger]

On Fri, 8 Dec 2023 10:30:39 -0700
Arjun Mehta <arjunmeht@gmail.com> wrote:

> Hi Stephen, thank you for the reply.
> 
> Proxmox does use a kernel derived from Ubuntu I believe (eg. kernel for Proxmox 8.1 which is what I'm using is 6.5.11-6-pve derived from Ubuntu 23.10). Not sure if there have been any modifications to the Linux Bridge in it.
> https://pve.proxmox.com/wiki/Proxmox_VE_Kernel#Proxmox_VE_8.x
> 
> Long shot, but do you happen to know of any workarounds with the Wifi interface issue you mentioned that would mitigate this issue? Maybe they would apply here.
> 
> I will also post to the Proxmox forums about this issue to flag for them.
> 
> Arjun

With VM's the issue is that many hosts have source address protection in
either the SW or HW for VF's to prevent address spoofing. 
This means you can't run a L2 bridge in the guest.

The only mitigation is to do some form of Layer 2 NAT.
Possible, but I have never done it, and likely to have scaling issues.