* status of rate adaptation @ 2022-11-11 20:44 Tim Harvey 2022-11-11 20:57 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-11 20:44 UTC (permalink / raw) To: netdev, Sean Anderson, Russell King, David S. Miller Greetings, I've noticed some recent commits that appear to add rate adaptation support: 3c42563b3041 net: phy: aquantia: Add support for rate matching 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces b7e9294885b6 net: phylink: Adjust advertisement based on rate matching ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching 0c3e10cb4423 net: phy: Add support for rate matching I have a board with an AQR113C PHY over XFI that functions properly at 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 Should I expect this to work now at those lower rates and if so what kind of debug information or testing can I provide? Best Regards, Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 20:44 status of rate adaptation Tim Harvey @ 2022-11-11 20:57 ` Sean Anderson 2022-11-11 20:58 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Sean Anderson @ 2022-11-11 20:57 UTC (permalink / raw) To: Tim Harvey, netdev, Russell King, David S. Miller Hi Tim, On 11/11/22 15:44, Tim Harvey wrote: > Greetings, > > I've noticed some recent commits that appear to add rate adaptation support: > 3c42563b3041 net: phy: aquantia: Add support for rate matching > 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > 0c3e10cb4423 net: phy: Add support for rate matching > > I have a board with an AQR113C PHY over XFI that functions properly at > 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > > Should I expect this to work now at those lower rates Yes. > and if so what kind of debug information or testing can I provide? Please send - Your test procedure (how do you select 1G?) - Device tree node for the interface - Output of ethtool (on both ends if possible). - Kernel logs with debug enabled for drivers/phylink.c That should be enough to get us started. --Sean ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 20:57 ` Sean Anderson @ 2022-11-11 20:58 ` Sean Anderson 2022-11-11 21:20 ` Tim Harvey 0 siblings, 1 reply; 21+ messages in thread From: Sean Anderson @ 2022-11-11 20:58 UTC (permalink / raw) To: Tim Harvey, netdev, Russell King, David S. Miller On 11/11/22 15:57, Sean Anderson wrote: > Hi Tim, > > On 11/11/22 15:44, Tim Harvey wrote: >> Greetings, >> >> I've noticed some recent commits that appear to add rate adaptation support: >> 3c42563b3041 net: phy: aquantia: Add support for rate matching >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching >> 0c3e10cb4423 net: phy: Add support for rate matching >> >> I have a board with an AQR113C PHY over XFI that functions properly at >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 >> >> Should I expect this to work now at those lower rates > > Yes. > >> and if so what kind of debug information or testing can I provide? > > Please send > > - Your test procedure (how do you select 1G?) > - Device tree node for the interface > - Output of ethtool (on both ends if possible). > - Kernel logs with debug enabled for drivers/phylink.c Sorry, this should be drivers/net/phy/phylink.c > > That should be enough to get us started. > > --Sean ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 20:58 ` Sean Anderson @ 2022-11-11 21:20 ` Tim Harvey 2022-11-11 21:54 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-11 21:20 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > > On 11/11/22 15:57, Sean Anderson wrote: > > Hi Tim, > > > > On 11/11/22 15:44, Tim Harvey wrote: > >> Greetings, > >> > >> I've noticed some recent commits that appear to add rate adaptation support: > >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > >> 0c3e10cb4423 net: phy: Add support for rate matching > >> > >> I have a board with an AQR113C PHY over XFI that functions properly at > >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > >> > >> Should I expect this to work now at those lower rates > > > > Yes. Sean, Good to hear - thank you for your work on this feature! > > > >> and if so what kind of debug information or testing can I provide? > > > > Please send > > > > - Your test procedure (how do you select 1G?) > > - Device tree node for the interface > > - Output of ethtool (on both ends if possible). > > - Kernel logs with debug enabled for drivers/phylink.c > > Sorry, this should be drivers/net/phy/phylink.c > > > > > That should be enough to get us started. > > > > --Sean > I'm currently testing by bringing up the network interface while connected to a 10gbe switch, verifying link and traffic, then forcing the switch port to 1000mbps. The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: #include "cn9130.dtsi" /* include SoC device tree */ &cp0_xmdio { pinctrl-names = "default"; pinctrl-0 = <&cp0_xsmi_pins>; status = "okay"; phy1: ethernet-phy@8 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <8>; }; }; &cp0_ethernet { status = "okay"; }; /* 10GbE XFI AQR113C */ &cp0_eth0 { status = "okay"; phy = <&phy1>; phy-mode = "10gbase-r"; phys = <&cp0_comphy4 0>; }; Here are some logs with debug enabled in drivers/net/phy/phylink.c and some additional debug in mvpp2.c and aquantia_main.c: # ifconfig eth0 192.168.1.22 [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 speed=-1 26:10gbase-r [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 [ 8.898165] aqr107_resume [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 duplex=255 speed=-1 26:10gbase-r 0: [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting supported 00000000,00018000,000e706f advertising 00000000,00018000,000e706f [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for phy/10gbase-r link mode [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 pause=00 link=0 an=0 [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down [ 8.976267] aqr107_resume [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down 10gbase-r/10Gbps/Full/none/off [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 duplex=1 speed=10000 26:10gbase-r [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control off [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up 10gbase-r/10Gbps/Full/none/off [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 duplex=1 speed=10000 26:10gbase-r # ethtool eth0 Settings for eth0: Supported ports: [ ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 10000baseT/Full 1000baseKX/Full 10000baseKX4/Full 10000baseKR/Full 2500baseT/Full 5000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 10000baseT/Full 1000baseKX/Full 10000baseKX4/Full 10000baseKR/Full 2500baseT/Full 5000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 8 Transceiver: external Auto-negotiation: on MDI-X: Unknown Link detected: yes # ping 192.168.1.146 -c5 PING 192.168.1.146 (192.168.1.146): 56 data bytes 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms --- 192.168.1.146 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 0.267/0.416/0.991 ms # # force switch port to 1G [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down 10gbase-r/Unknown/Unknown/none/off [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 duplex=255 speed=-1 26:10gbase-r [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off [ 197.472504] mvpp2 f2000000.ethernet eth0: major config [ 197.472614] mvpp2 f2000000.ethernet eth0: phylink_mac_config: mode=phy//1Gbps/Full/pause adv=00000000,00000000,00000000 pause=00 link=1 an=0 [ 197.479561] aqr107_link_change_notify state=4:running an=1 link=1 duplex=1 speed=1000 0: [ 197.484972] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off # ethtool eth0 Settings for eth0: Supported ports: [ ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 10000baseT/Full 1000baseKX/Full 10000baseKX4/Full 10000baseKR/Full 2500baseT/Full 5000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 10000baseT/Full 1000baseKX/Full 10000baseKX4/Full 10000baseKR/Full 2500baseT/Full 5000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 1000baseT/Half 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 8 Transceiver: external Auto-negotiation: on MDI-X: Unknown Link detected: yes # ping 192.168.1.146 -c5 PING 192.168.1.146 (192.168.1.146): 56 data bytes --- 192.168.1.146 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss Best Regards, Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 21:20 ` Tim Harvey @ 2022-11-11 21:54 ` Sean Anderson 2022-11-11 22:14 ` Tim Harvey ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Sean Anderson @ 2022-11-11 21:54 UTC (permalink / raw) To: Tim Harvey; +Cc: netdev, Russell King, David S. Miller On 11/11/22 16:20, Tim Harvey wrote: > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> On 11/11/22 15:57, Sean Anderson wrote: >> > Hi Tim, >> > >> > On 11/11/22 15:44, Tim Harvey wrote: >> >> Greetings, >> >> >> >> I've noticed some recent commits that appear to add rate adaptation support: >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching >> >> 0c3e10cb4423 net: phy: Add support for rate matching >> >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 >> >> >> >> Should I expect this to work now at those lower rates >> > >> > Yes. > > Sean, > > Good to hear - thank you for your work on this feature! > >> > >> >> and if so what kind of debug information or testing can I provide? >> > >> > Please send >> > >> > - Your test procedure (how do you select 1G?) >> > - Device tree node for the interface >> > - Output of ethtool (on both ends if possible). >> > - Kernel logs with debug enabled for drivers/phylink.c >> >> Sorry, this should be drivers/net/phy/phylink.c >> >> > >> > That should be enough to get us started. >> > >> > --Sean >> > > I'm currently testing by bringing up the network interface while > connected to a 10gbe switch, verifying link and traffic, then forcing > the switch port to 1000mbps. > > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > > #include "cn9130.dtsi" /* include SoC device tree */ > > &cp0_xmdio { > pinctrl-names = "default"; > pinctrl-0 = <&cp0_xsmi_pins>; > status = "okay"; > > phy1: ethernet-phy@8 { > compatible = "ethernet-phy-ieee802.3-c45"; > reg = <8>; > }; > }; > > &cp0_ethernet { > status = "okay"; > }; > > /* 10GbE XFI AQR113C */ > &cp0_eth0 { > status = "okay"; > phy = <&phy1>; > phy-mode = "10gbase-r"; > phys = <&cp0_comphy4 0>; > }; > > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > some additional debug in mvpp2.c and aquantia_main.c: > # ifconfig eth0 192.168.1.22 > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > speed=-1 26:10gbase-r > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > [ 8.898165] aqr107_resume > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > duplex=255 speed=-1 26:10gbase-r 0: > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > supported 00000000,00018000,000e706f advertising > 00000000,00018000,000e706f > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > phy/10gbase-r link mode > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > pause=00 link=0 an=0 > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > [ 8.976267] aqr107_resume > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > 10gbase-r/10Gbps/Full/none/off > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > duplex=1 speed=10000 26:10gbase-r > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > - flow control off > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > 10gbase-r/10Gbps/Full/none/off > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > duplex=1 speed=10000 26:10gbase-r > > # ethtool eth0 > Settings for eth0: > Supported ports: [ ] > Supported link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full 10/100 half duplex aren't achievable with rate matching (and we avoid turning them on), so they must be coming from somewhere else. I wonder if this is because PHY_INTERFACE_MODE_SGMII is set in supported_interfaces. I wonder if you could enable USXGMII? Seems like mvpp2 with comphy should support it. I'm not sure if the aquantia driver is set up for it. > 1000baseT/Full > 10000baseT/Full > 1000baseKX/Full > 10000baseKX4/Full > 10000baseKR/Full > 2500baseT/Full > 5000baseT/Full > Supported pause frame use: Symmetric Receive-only > Supports auto-negotiation: Yes > Supported FEC modes: Not reported > Advertised link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > 10000baseT/Full > 1000baseKX/Full > 10000baseKX4/Full > 10000baseKR/Full > 2500baseT/Full > 5000baseT/Full > Advertised pause frame use: Symmetric Receive-only > Advertised auto-negotiation: Yes > Advertised FEC modes: Not reported > Link partner advertised link modes: 100baseT/Half 100baseT/Full > 1000baseT/Half 1000baseT/Full > 10000baseT/Full > 2500baseT/Full > 5000baseT/Full > Link partner advertised pause frame use: No > Link partner advertised auto-negotiation: Yes > Link partner advertised FEC modes: Not reported > Speed: 10000Mb/s > Duplex: Full > Port: Twisted Pair > PHYAD: 8 > Transceiver: external > Auto-negotiation: on > MDI-X: Unknown > Link detected: yes > # ping 192.168.1.146 -c5 > PING 192.168.1.146 (192.168.1.146): 56 data bytes > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > > --- 192.168.1.146 ping statistics --- > 5 packets transmitted, 5 packets received, 0% packet loss > round-trip min/avg/max = 0.267/0.416/0.991 ms > # # force switch port to 1G > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > 10gbase-r/Unknown/Unknown/none/off > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > duplex=255 speed=-1 26:10gbase-r > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) and the vendor provisioning registers (dev 4, reg 0xc440 through 0xc449). It's possible that your firmware doesn't support rate adaptation... I'm not sure what we can do about that. --Sean > [ 197.472504] mvpp2 f2000000.ethernet eth0: major config > [ 197.472614] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > mode=phy//1Gbps/Full/pause adv=00000000,00000000,00000000 pause=00 > link=1 an=0 > [ 197.479561] aqr107_link_change_notify state=4:running an=1 link=1 > duplex=1 speed=1000 0: > [ 197.484972] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - > flow control off > # ethtool eth0 > Settings for eth0: > Supported ports: [ ] > Supported link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > 10000baseT/Full > 1000baseKX/Full > 10000baseKX4/Full > 10000baseKR/Full > 2500baseT/Full > 5000baseT/Full > Supported pause frame use: Symmetric Receive-only > Supports auto-negotiation: Yes > Supported FEC modes: Not reported > Advertised link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > 10000baseT/Full > 1000baseKX/Full > 10000baseKX4/Full > 10000baseKR/Full > 2500baseT/Full > 5000baseT/Full > Advertised pause frame use: Symmetric Receive-only > Advertised auto-negotiation: Yes > Advertised FEC modes: Not reported > Link partner advertised link modes: 1000baseT/Half 1000baseT/Full > Link partner advertised pause frame use: No > Link partner advertised auto-negotiation: Yes > Link partner advertised FEC modes: Not reported > Speed: 1000Mb/s > Duplex: Full > Port: Twisted Pair > PHYAD: 8 > Transceiver: external > Auto-negotiation: on > MDI-X: Unknown > Link detected: yes > # ping 192.168.1.146 -c5 > PING 192.168.1.146 (192.168.1.146): 56 data bytes > > --- 192.168.1.146 ping statistics --- > 5 packets transmitted, 0 packets received, 100% packet loss > > Best Regards, > > Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 21:54 ` Sean Anderson @ 2022-11-11 22:14 ` Tim Harvey 2022-11-11 22:38 ` Sean Anderson 2022-11-11 22:33 ` Russell King (Oracle) 2022-11-12 13:15 ` Russell King (Oracle) 2 siblings, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-11 22:14 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: > > On 11/11/22 16:20, Tim Harvey wrote: > > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> > >> On 11/11/22 15:57, Sean Anderson wrote: > >> > Hi Tim, > >> > > >> > On 11/11/22 15:44, Tim Harvey wrote: > >> >> Greetings, > >> >> > >> >> I've noticed some recent commits that appear to add rate adaptation support: > >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > >> >> 0c3e10cb4423 net: phy: Add support for rate matching > >> >> > >> >> I have a board with an AQR113C PHY over XFI that functions properly at > >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > >> >> > >> >> Should I expect this to work now at those lower rates > >> > > >> > Yes. > > > > Sean, > > > > Good to hear - thank you for your work on this feature! > > > >> > > >> >> and if so what kind of debug information or testing can I provide? > >> > > >> > Please send > >> > > >> > - Your test procedure (how do you select 1G?) > >> > - Device tree node for the interface > >> > - Output of ethtool (on both ends if possible). > >> > - Kernel logs with debug enabled for drivers/phylink.c > >> > >> Sorry, this should be drivers/net/phy/phylink.c > >> > >> > > >> > That should be enough to get us started. > >> > > >> > --Sean > >> > > > > I'm currently testing by bringing up the network interface while > > connected to a 10gbe switch, verifying link and traffic, then forcing > > the switch port to 1000mbps. > > > > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > > > > #include "cn9130.dtsi" /* include SoC device tree */ > > > > &cp0_xmdio { > > pinctrl-names = "default"; > > pinctrl-0 = <&cp0_xsmi_pins>; > > status = "okay"; > > > > phy1: ethernet-phy@8 { > > compatible = "ethernet-phy-ieee802.3-c45"; > > reg = <8>; > > }; > > }; > > > > &cp0_ethernet { > > status = "okay"; > > }; > > > > /* 10GbE XFI AQR113C */ > > &cp0_eth0 { > > status = "okay"; > > phy = <&phy1>; > > phy-mode = "10gbase-r"; > > phys = <&cp0_comphy4 0>; > > }; > > > > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > > some additional debug in mvpp2.c and aquantia_main.c: > > # ifconfig eth0 192.168.1.22 > > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > > speed=-1 26:10gbase-r > > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > > [ 8.898165] aqr107_resume > > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > > duplex=255 speed=-1 26:10gbase-r 0: > > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > > supported 00000000,00018000,000e706f advertising > > 00000000,00018000,000e706f > > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > > phy/10gbase-r link mode > > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > > pause=00 link=0 an=0 > > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > > [ 8.976267] aqr107_resume > > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > > 10gbase-r/10Gbps/Full/none/off > > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > > duplex=1 speed=10000 26:10gbase-r > > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > > - flow control off > > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > > 10gbase-r/10Gbps/Full/none/off > > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > > duplex=1 speed=10000 26:10gbase-r > > > > # ethtool eth0 > > Settings for eth0: > > Supported ports: [ ] > > Supported link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 10/100 half duplex aren't achievable with rate matching (and we avoid > turning them on), so they must be coming from somewhere else. I wonder > if this is because PHY_INTERFACE_MODE_SGMII is set in > supported_interfaces. > > I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > should support it. I'm not sure if the aquantia driver is set up for it. This appears to trigger an issue from mvpp2: mvpp2 f2000000.ethernet eth0: validation of usxgmii with support 00000000,00018000,000e706f and advertisement 00000000,00018000,000e706f failed: -EINVAL > > > 1000baseT/Full > > 10000baseT/Full > > 1000baseKX/Full > > 10000baseKX4/Full > > 10000baseKR/Full > > 2500baseT/Full > > 5000baseT/Full > > Supported pause frame use: Symmetric Receive-only > > Supports auto-negotiation: Yes > > Supported FEC modes: Not reported > > Advertised link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full > > 10000baseT/Full > > 1000baseKX/Full > > 10000baseKX4/Full > > 10000baseKR/Full > > 2500baseT/Full > > 5000baseT/Full > > Advertised pause frame use: Symmetric Receive-only > > Advertised auto-negotiation: Yes > > Advertised FEC modes: Not reported > > Link partner advertised link modes: 100baseT/Half 100baseT/Full > > 1000baseT/Half 1000baseT/Full > > 10000baseT/Full > > 2500baseT/Full > > 5000baseT/Full > > Link partner advertised pause frame use: No > > Link partner advertised auto-negotiation: Yes > > Link partner advertised FEC modes: Not reported > > Speed: 10000Mb/s > > Duplex: Full > > Port: Twisted Pair > > PHYAD: 8 > > Transceiver: external > > Auto-negotiation: on > > MDI-X: Unknown > > Link detected: yes > > # ping 192.168.1.146 -c5 > > PING 192.168.1.146 (192.168.1.146): 56 data bytes > > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > > > > --- 192.168.1.146 ping statistics --- > > 5 packets transmitted, 5 packets received, 0% packet loss > > round-trip min/avg/max = 0.267/0.416/0.991 ms > > # # force switch port to 1G > > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > > 10gbase-r/Unknown/Unknown/none/off > > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > > duplex=255 speed=-1 26:10gbase-r > > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off > > Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send > the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also > send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) > and the vendor provisioning registers (dev 4, reg 0xc440 through > 0xc449). yes, this is what I've been looking at as well. When forced to 1000m the register shows a phy type of 11 which according to the aqr113 datasheet is XFI 5G: aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 link=1 duplex=1 speed=1000 interface=0 > > It's possible that your firmware doesn't support rate adaptation... I'm > not sure what we can do about that. > I will enquire with my Aquantia FAE to see what they say about rate adaptation support Something interesting is that when I configured the xmdio node with an interrupt I ended up in a mode where 5g,2.5g and 1g all worked for at least 1 test. There was something wrong with my interrupt configuration (i'm not clear if the AQR113C's interrupt should be IRQ_TYPE_LEVEL_LOW, IRQ_TYPE_EDGE_FALLING or something different). While I can't reliably reproduce this and I believe I was on the 6.0 kernel at the time without the rate adaptation support a debug log when I was in this mode shows the following: [ 27.700221] aqr107_config_init state=1 an=1 link=0 duplex=255 speed=-1 26:10gbase-r [ 27.709694] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 [ 27.716457] aqr107_resume [ 27.723551] aqr107_get_rate_matching state=1 an=1 link=0 duplex=255 speed=-1 26:10gbase-r 0: [ 27.733075] mvpp2 f2000000.ethernet eth0: PHY [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=40) [ 27.752939] mvpp2 f2000000.ethernet eth0: configuring for phy/10gbase-r link mode [ 27.760508] aqr107_resume [ 27.769781] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 speed=10000 26:10gbase-r [ 32.670293] aqr107_read_status state=5 an=1 link=1 duplex=1 speed=10000 0: [ 32.678642] aqr107_read_rate state=5 an=1 link=1 duplex=1 speed=10000 0: [ 32.686405] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 speed=10000 0: [ 32.686628] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full - flow control off [ 32.702981] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready ^^^ 10gbe link; ping ok # force port to 1Gbe [ 945.918132] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 speed=10000 26:10gbase-r [ 945.918193] mvpp2_port_isr 10gbase-r [ 945.919186] mvpp2_port_disable 10gbase-r [ 945.935304] mvpp2 f2000000.ethernet eth0: Link is Down [ 949.509595] aqr107_read_status state=5 an=1 link=1 duplex=1 speed=1000 26:10gbase-r [ 949.518562] aqr107_read_rate state=5 an=1 link=1 duplex=1 speed=1000 26:10gbase-r [ 949.527112] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 speed=1000 26:10gbase-r [ 949.527166] mvpp2_port_isr 10gbase-r [ 949.527176] mvpp2_port_enable 10gbase-r [ 949.527306] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off ^^^ 1gbe link; ping ok # force port to 2.5Gbe [ 1024.518112] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 speed=1000 26:10gbase-r [ 1024.518187] mvpp2_port_isr 10gbase-r [ 1024.532897] mvpp2_port_disable 10gbase-r [ 1024.536880] mvpp2 f2000000.ethernet eth0: Link is Down [ 1029.295136] aqr107_read_status state=5 an=1 link=1 duplex=1 speed=2500 26:10gbase-r [ 1029.304070] aqr107_read_rate state=5 an=1 link=1 duplex=1 speed=2500 26:10gbase-r [ 1029.312611] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 speed=2500 26:10gbase-r [ 1029.312638] mvpp2_port_isr 10gbase-r [ 1029.325584] mvpp2_port_enable 10gbase-r [ 1029.329564] mvpp2 f2000000.ethernet eth0: Link is Up - 2.5Gbps/Full - flow control off ^^^ 2.5gbe link; ping ok # force port to 5gbe [ 1060.401209] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 speed=2500 26:10gbase-r [ 1060.401272] mvpp2_port_isr 10gbase-r [ 1060.402274] mvpp2_port_disable 10gbase-r [ 1060.419006] mvpp2 f2000000.ethernet eth0: Link is Down [ 1065.167937] aqr107_read_status state=5 an=1 link=1 duplex=1 speed=5000 26:10gbase-r [ 1065.176865] aqr107_read_rate state=5 an=1 link=1 duplex=1 speed=5000 26:10gbase-r [ 1065.185415] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 speed=5000 26:10gbase-r [ 1065.185456] mvpp2_port_isr 10gbase-r [ 1065.185474] mvpp2_port_enable 10gbase-r [ 1065.185597] mvpp2 f2000000.ethernet eth0: Link is Up - 5Gbps/Full - flow control off ^^^ 5gpbe link; ping ok Thanks, Tim > --Sean > > > [ 197.472504] mvpp2 f2000000.ethernet eth0: major config > > [ 197.472614] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > > mode=phy//1Gbps/Full/pause adv=00000000,00000000,00000000 pause=00 > > link=1 an=0 > > [ 197.479561] aqr107_link_change_notify state=4:running an=1 link=1 > > duplex=1 speed=1000 0: > > [ 197.484972] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - > > flow control off > > # ethtool eth0 > > Settings for eth0: > > Supported ports: [ ] > > Supported link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full > > 10000baseT/Full > > 1000baseKX/Full > > 10000baseKX4/Full > > 10000baseKR/Full > > 2500baseT/Full > > 5000baseT/Full > > Supported pause frame use: Symmetric Receive-only > > Supports auto-negotiation: Yes > > Supported FEC modes: Not reported > > Advertised link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full > > 10000baseT/Full > > 1000baseKX/Full > > 10000baseKX4/Full > > 10000baseKR/Full > > 2500baseT/Full > > 5000baseT/Full > > Advertised pause frame use: Symmetric Receive-only > > Advertised auto-negotiation: Yes > > Advertised FEC modes: Not reported > > Link partner advertised link modes: 1000baseT/Half 1000baseT/Full > > Link partner advertised pause frame use: No > > Link partner advertised auto-negotiation: Yes > > Link partner advertised FEC modes: Not reported > > Speed: 1000Mb/s > > Duplex: Full > > Port: Twisted Pair > > PHYAD: 8 > > Transceiver: external > > Auto-negotiation: on > > MDI-X: Unknown > > Link detected: yes > > # ping 192.168.1.146 -c5 > > PING 192.168.1.146 (192.168.1.146): 56 data bytes > > > > --- 192.168.1.146 ping statistics --- > > 5 packets transmitted, 0 packets received, 100% packet loss > > > > Best Regards, > > > > Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 22:14 ` Tim Harvey @ 2022-11-11 22:38 ` Sean Anderson 2022-11-12 0:48 ` Vladimir Oltean 2022-11-14 19:33 ` Tim Harvey 0 siblings, 2 replies; 21+ messages in thread From: Sean Anderson @ 2022-11-11 22:38 UTC (permalink / raw) To: Tim Harvey; +Cc: netdev, Russell King, David S. Miller On 11/11/22 17:14, Tim Harvey wrote: > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> On 11/11/22 16:20, Tim Harvey wrote: >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> >> >> On 11/11/22 15:57, Sean Anderson wrote: >> >> > Hi Tim, >> >> > >> >> > On 11/11/22 15:44, Tim Harvey wrote: >> >> >> Greetings, >> >> >> >> >> >> I've noticed some recent commits that appear to add rate adaptation support: >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching >> >> >> >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 >> >> >> >> >> >> Should I expect this to work now at those lower rates >> >> > >> >> > Yes. >> > >> > Sean, >> > >> > Good to hear - thank you for your work on this feature! >> > >> >> > >> >> >> and if so what kind of debug information or testing can I provide? >> >> > >> >> > Please send >> >> > >> >> > - Your test procedure (how do you select 1G?) >> >> > - Device tree node for the interface >> >> > - Output of ethtool (on both ends if possible). >> >> > - Kernel logs with debug enabled for drivers/phylink.c >> >> >> >> Sorry, this should be drivers/net/phy/phylink.c >> >> >> >> > >> >> > That should be enough to get us started. >> >> > >> >> > --Sean >> >> >> > >> > I'm currently testing by bringing up the network interface while >> > connected to a 10gbe switch, verifying link and traffic, then forcing >> > the switch port to 1000mbps. >> > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: >> > >> > #include "cn9130.dtsi" /* include SoC device tree */ >> > >> > &cp0_xmdio { >> > pinctrl-names = "default"; >> > pinctrl-0 = <&cp0_xsmi_pins>; >> > status = "okay"; >> > >> > phy1: ethernet-phy@8 { >> > compatible = "ethernet-phy-ieee802.3-c45"; >> > reg = <8>; >> > }; >> > }; >> > >> > &cp0_ethernet { >> > status = "okay"; >> > }; >> > >> > /* 10GbE XFI AQR113C */ >> > &cp0_eth0 { >> > status = "okay"; >> > phy = <&phy1>; >> > phy-mode = "10gbase-r"; >> > phys = <&cp0_comphy4 0>; >> > }; >> > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and >> > some additional debug in mvpp2.c and aquantia_main.c: >> > # ifconfig eth0 192.168.1.22 >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 >> > speed=-1 26:10gbase-r >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 >> > [ 8.898165] aqr107_resume >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 >> > duplex=255 speed=-1 26:10gbase-r 0: >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting >> > supported 00000000,00018000,000e706f advertising >> > 00000000,00018000,000e706f >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for >> > phy/10gbase-r link mode >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 >> > pause=00 link=0 an=0 >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down >> > [ 8.976267] aqr107_resume >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down >> > 10gbase-r/10Gbps/Full/none/off >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 >> > duplex=1 speed=10000 26:10gbase-r >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full >> > - flow control off >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up >> > 10gbase-r/10Gbps/Full/none/off >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 >> > duplex=1 speed=10000 26:10gbase-r >> > >> > # ethtool eth0 >> > Settings for eth0: >> > Supported ports: [ ] >> > Supported link modes: 10baseT/Half 10baseT/Full >> > 100baseT/Half 100baseT/Full >> >> 10/100 half duplex aren't achievable with rate matching (and we avoid >> turning them on), so they must be coming from somewhere else. I wonder >> if this is because PHY_INTERFACE_MODE_SGMII is set in >> supported_interfaces. >> >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy >> should support it. I'm not sure if the aquantia driver is set up for it. > > This appears to trigger an issue from mvpp2: > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support > 00000000,00018000,000e706f and advertisement > 00000000,00018000,000e706f failed: -EINVAL Ah, I forgot this was a separate phy mode. Disregard this. >> >> > 1000baseT/Full >> > 10000baseT/Full >> > 1000baseKX/Full >> > 10000baseKX4/Full >> > 10000baseKR/Full >> > 2500baseT/Full >> > 5000baseT/Full >> > Supported pause frame use: Symmetric Receive-only >> > Supports auto-negotiation: Yes >> > Supported FEC modes: Not reported >> > Advertised link modes: 10baseT/Half 10baseT/Full >> > 100baseT/Half 100baseT/Full >> > 1000baseT/Full >> > 10000baseT/Full >> > 1000baseKX/Full >> > 10000baseKX4/Full >> > 10000baseKR/Full >> > 2500baseT/Full >> > 5000baseT/Full >> > Advertised pause frame use: Symmetric Receive-only >> > Advertised auto-negotiation: Yes >> > Advertised FEC modes: Not reported >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full >> > 1000baseT/Half 1000baseT/Full >> > 10000baseT/Full >> > 2500baseT/Full >> > 5000baseT/Full >> > Link partner advertised pause frame use: No >> > Link partner advertised auto-negotiation: Yes >> > Link partner advertised FEC modes: Not reported >> > Speed: 10000Mb/s >> > Duplex: Full >> > Port: Twisted Pair >> > PHYAD: 8 >> > Transceiver: external >> > Auto-negotiation: on >> > MDI-X: Unknown >> > Link detected: yes >> > # ping 192.168.1.146 -c5 >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms >> > >> > --- 192.168.1.146 ping statistics --- >> > 5 packets transmitted, 5 packets received, 0% packet loss >> > round-trip min/avg/max = 0.267/0.416/0.991 ms >> > # # force switch port to 1G >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down >> > 10gbase-r/Unknown/Unknown/none/off >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 >> > duplex=255 speed=-1 26:10gbase-r >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off >> >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) >> and the vendor provisioning registers (dev 4, reg 0xc440 through >> 0xc449). > > yes, this is what I've been looking at as well. When forced to 1000m > the register shows a phy type of 11 which according to the aqr113 > datasheet is XFI 5G: > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 > link=1 duplex=1 speed=1000 interface=0 That's pretty strange. Seems like it's rate adapting from 5g instead of 10g. Is SERDES Mode in the Global System Configuration For 1G register set to XFI? >> >> It's possible that your firmware doesn't support rate adaptation... I'm >> not sure what we can do about that. >> > > I will enquire with my Aquantia FAE to see what they say about rate > adaptation support > > Something interesting is that when I configured the xmdio node with an > interrupt I ended up in a mode where 5g,2.5g and 1g all worked for at > least 1 test. There was something wrong with my interrupt > configuration (i'm not clear if the AQR113C's interrupt should be > IRQ_TYPE_LEVEL_LOW, IRQ_TYPE_EDGE_FALLING or something different). NXP use IRQ_TYPE_LEVEL_HIGH on the LS1046ARDB. --Sean > While I can't reliably reproduce this and I believe I was on the 6.0 > kernel at the time without the rate adaptation support a debug log > when I was in this mode shows the following: > [ 27.700221] aqr107_config_init state=1 an=1 link=0 duplex=255 > speed=-1 26:10gbase-r > [ 27.709694] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > [ 27.716457] aqr107_resume > [ 27.723551] aqr107_get_rate_matching state=1 an=1 link=0 duplex=255 > speed=-1 26:10gbase-r 0: > [ 27.733075] mvpp2 f2000000.ethernet eth0: PHY > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=40) > [ 27.752939] mvpp2 f2000000.ethernet eth0: configuring for > phy/10gbase-r link mode > [ 27.760508] aqr107_resume > [ 27.769781] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 > speed=10000 26:10gbase-r > [ 32.670293] aqr107_read_status state=5 an=1 link=1 duplex=1 speed=10000 0: > [ 32.678642] aqr107_read_rate state=5 an=1 link=1 duplex=1 speed=10000 0: > [ 32.686405] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 > speed=10000 0: > [ 32.686628] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > - flow control off > [ 32.702981] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > ^^^ 10gbe link; ping ok > # force port to 1Gbe > [ 945.918132] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 > speed=10000 26:10gbase-r > [ 945.918193] mvpp2_port_isr 10gbase-r > [ 945.919186] mvpp2_port_disable 10gbase-r > [ 945.935304] mvpp2 f2000000.ethernet eth0: Link is Down > [ 949.509595] aqr107_read_status state=5 an=1 link=1 duplex=1 > speed=1000 26:10gbase-r > [ 949.518562] aqr107_read_rate state=5 an=1 link=1 duplex=1 > speed=1000 26:10gbase-r > [ 949.527112] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 > speed=1000 26:10gbase-r > [ 949.527166] mvpp2_port_isr 10gbase-r > [ 949.527176] mvpp2_port_enable 10gbase-r > [ 949.527306] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - > flow control off > ^^^ 1gbe link; ping ok > # force port to 2.5Gbe > [ 1024.518112] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 > speed=1000 26:10gbase-r > [ 1024.518187] mvpp2_port_isr 10gbase-r > [ 1024.532897] mvpp2_port_disable 10gbase-r > [ 1024.536880] mvpp2 f2000000.ethernet eth0: Link is Down > [ 1029.295136] aqr107_read_status state=5 an=1 link=1 duplex=1 > speed=2500 26:10gbase-r > [ 1029.304070] aqr107_read_rate state=5 an=1 link=1 duplex=1 > speed=2500 26:10gbase-r > [ 1029.312611] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 > speed=2500 26:10gbase-r > [ 1029.312638] mvpp2_port_isr 10gbase-r > [ 1029.325584] mvpp2_port_enable 10gbase-r > [ 1029.329564] mvpp2 f2000000.ethernet eth0: Link is Up - 2.5Gbps/Full > - flow control off > ^^^ 2.5gbe link; ping ok > # force port to 5gbe > [ 1060.401209] aqr107_link_change_notify state=5 an=1 link=0 duplex=1 > speed=2500 26:10gbase-r > [ 1060.401272] mvpp2_port_isr 10gbase-r > [ 1060.402274] mvpp2_port_disable 10gbase-r > [ 1060.419006] mvpp2 f2000000.ethernet eth0: Link is Down > [ 1065.167937] aqr107_read_status state=5 an=1 link=1 duplex=1 > speed=5000 26:10gbase-r > [ 1065.176865] aqr107_read_rate state=5 an=1 link=1 duplex=1 > speed=5000 26:10gbase-r > [ 1065.185415] aqr107_link_change_notify state=4 an=1 link=1 duplex=1 > speed=5000 26:10gbase-r > [ 1065.185456] mvpp2_port_isr 10gbase-r > [ 1065.185474] mvpp2_port_enable 10gbase-r > [ 1065.185597] mvpp2 f2000000.ethernet eth0: Link is Up - 5Gbps/Full - > flow control off > ^^^ 5gpbe link; ping ok > > Thanks, > > Tim > >> --Sean >> >> > [ 197.472504] mvpp2 f2000000.ethernet eth0: major config >> > [ 197.472614] mvpp2 f2000000.ethernet eth0: phylink_mac_config: >> > mode=phy//1Gbps/Full/pause adv=00000000,00000000,00000000 pause=00 >> > link=1 an=0 >> > [ 197.479561] aqr107_link_change_notify state=4:running an=1 link=1 >> > duplex=1 speed=1000 0: >> > [ 197.484972] mvpp2 f2000000.ethernet eth0: Link is Up - 1Gbps/Full - >> > flow control off >> > # ethtool eth0 >> > Settings for eth0: >> > Supported ports: [ ] >> > Supported link modes: 10baseT/Half 10baseT/Full >> > 100baseT/Half 100baseT/Full >> > 1000baseT/Full >> > 10000baseT/Full >> > 1000baseKX/Full >> > 10000baseKX4/Full >> > 10000baseKR/Full >> > 2500baseT/Full >> > 5000baseT/Full >> > Supported pause frame use: Symmetric Receive-only >> > Supports auto-negotiation: Yes >> > Supported FEC modes: Not reported >> > Advertised link modes: 10baseT/Half 10baseT/Full >> > 100baseT/Half 100baseT/Full >> > 1000baseT/Full >> > 10000baseT/Full >> > 1000baseKX/Full >> > 10000baseKX4/Full >> > 10000baseKR/Full >> > 2500baseT/Full >> > 5000baseT/Full >> > Advertised pause frame use: Symmetric Receive-only >> > Advertised auto-negotiation: Yes >> > Advertised FEC modes: Not reported >> > Link partner advertised link modes: 1000baseT/Half 1000baseT/Full >> > Link partner advertised pause frame use: No >> > Link partner advertised auto-negotiation: Yes >> > Link partner advertised FEC modes: Not reported >> > Speed: 1000Mb/s >> > Duplex: Full >> > Port: Twisted Pair >> > PHYAD: 8 >> > Transceiver: external >> > Auto-negotiation: on >> > MDI-X: Unknown >> > Link detected: yes >> > # ping 192.168.1.146 -c5 >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes >> > >> > --- 192.168.1.146 ping statistics --- >> > 5 packets transmitted, 0 packets received, 100% packet loss >> > >> > Best Regards, >> > >> > Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 22:38 ` Sean Anderson @ 2022-11-12 0:48 ` Vladimir Oltean 2022-11-12 16:08 ` Andrew Lunn 2022-11-14 15:08 ` Sean Anderson 2022-11-14 19:33 ` Tim Harvey 1 sibling, 2 replies; 21+ messages in thread From: Vladimir Oltean @ 2022-11-12 0:48 UTC (permalink / raw) To: Sean Anderson; +Cc: Tim Harvey, netdev, Russell King, David S. Miller On Fri, Nov 11, 2022 at 05:38:12PM -0500, Sean Anderson wrote: > > Something interesting is that when I configured the xmdio node with an > > interrupt I ended up in a mode where 5g,2.5g and 1g all worked for at > > least 1 test. There was something wrong with my interrupt > > configuration (i'm not clear if the AQR113C's interrupt should be > > IRQ_TYPE_LEVEL_LOW, IRQ_TYPE_EDGE_FALLING or something different). > > NXP use IRQ_TYPE_LEVEL_HIGH on the LS1046ARDB. Partly true, but mostly false. What is described in fsl-ls1046a-rdb.dts as: interrupts = <0 131 4>; should really have been described as: interrupts-extended = <&extirq 0 IRQ_TYPE_LEVEL_LOW>; There's a polarity inverter which inverts the signal by default, changing what the GIC sees. The first description bypasses it. So that's not what the problem is in Tim's case. As to LEVEL_LOW vs EDGE_FALLING, I suppose the only real difference is if the interrupt line is shared with other peripherals? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-12 0:48 ` Vladimir Oltean @ 2022-11-12 16:08 ` Andrew Lunn 2022-11-14 15:08 ` Sean Anderson 1 sibling, 0 replies; 21+ messages in thread From: Andrew Lunn @ 2022-11-12 16:08 UTC (permalink / raw) To: Vladimir Oltean Cc: Sean Anderson, Tim Harvey, netdev, Russell King, David S. Miller > As to LEVEL_LOW vs EDGE_FALLING, I suppose the only real difference is > if the interrupt line is shared with other peripherals? It pretty much always is, on the PHY side. The PHY is an interrupt controller, with lots of different interrupt sources within the PHY coming together to trigger one external interrupt. It is unlikely to produce another edge if the hardware has another interrupt source trigger an interrupt while the interrupt handler is running. With a level interrupt, the interrupt handler will exit, the interrupt will get enabled in the parent interrupt controller, and immediately fire again. I have seen some boards using edge, but that is only because the interrupt controller does not support level. They mostly get away with it because generally PHYs are slow things, interrupts tend to be few and infrequency and the race window is small. Andrew ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-12 0:48 ` Vladimir Oltean 2022-11-12 16:08 ` Andrew Lunn @ 2022-11-14 15:08 ` Sean Anderson 1 sibling, 0 replies; 21+ messages in thread From: Sean Anderson @ 2022-11-14 15:08 UTC (permalink / raw) To: Vladimir Oltean; +Cc: Tim Harvey, netdev, Russell King, David S. Miller On 11/11/22 19:48, Vladimir Oltean wrote: > On Fri, Nov 11, 2022 at 05:38:12PM -0500, Sean Anderson wrote: >> > Something interesting is that when I configured the xmdio node with an >> > interrupt I ended up in a mode where 5g,2.5g and 1g all worked for at >> > least 1 test. There was something wrong with my interrupt >> > configuration (i'm not clear if the AQR113C's interrupt should be >> > IRQ_TYPE_LEVEL_LOW, IRQ_TYPE_EDGE_FALLING or something different). >> >> NXP use IRQ_TYPE_LEVEL_HIGH on the LS1046ARDB. > > Partly true, but mostly false. What is described in fsl-ls1046a-rdb.dts as: > > interrupts = <0 131 4>; > > should really have been described as: > > interrupts-extended = <&extirq 0 IRQ_TYPE_LEVEL_LOW>; > > There's a polarity inverter which inverts the signal by default, > changing what the GIC sees. The first description bypasses it. Ah. I missed that they described it as going straight to the GIC, skipping the extirq. Thanks for pointing that out. --Sean > So that's not what the problem is in Tim's case. > > As to LEVEL_LOW vs EDGE_FALLING, I suppose the only real difference is > if the interrupt line is shared with other peripherals? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 22:38 ` Sean Anderson 2022-11-12 0:48 ` Vladimir Oltean @ 2022-11-14 19:33 ` Tim Harvey 2022-11-16 22:37 ` Tim Harvey 1 sibling, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-14 19:33 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: > > On 11/11/22 17:14, Tim Harvey wrote: > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> > >> On 11/11/22 16:20, Tim Harvey wrote: > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> >> > >> >> On 11/11/22 15:57, Sean Anderson wrote: > >> >> > Hi Tim, > >> >> > > >> >> > On 11/11/22 15:44, Tim Harvey wrote: > >> >> >> Greetings, > >> >> >> > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching > >> >> >> > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > >> >> >> > >> >> >> Should I expect this to work now at those lower rates > >> >> > > >> >> > Yes. > >> > > >> > Sean, > >> > > >> > Good to hear - thank you for your work on this feature! > >> > > >> >> > > >> >> >> and if so what kind of debug information or testing can I provide? > >> >> > > >> >> > Please send > >> >> > > >> >> > - Your test procedure (how do you select 1G?) > >> >> > - Device tree node for the interface > >> >> > - Output of ethtool (on both ends if possible). > >> >> > - Kernel logs with debug enabled for drivers/phylink.c > >> >> > >> >> Sorry, this should be drivers/net/phy/phylink.c > >> >> > >> >> > > >> >> > That should be enough to get us started. > >> >> > > >> >> > --Sean > >> >> > >> > > >> > I'm currently testing by bringing up the network interface while > >> > connected to a 10gbe switch, verifying link and traffic, then forcing > >> > the switch port to 1000mbps. > >> > > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > >> > > >> > #include "cn9130.dtsi" /* include SoC device tree */ > >> > > >> > &cp0_xmdio { > >> > pinctrl-names = "default"; > >> > pinctrl-0 = <&cp0_xsmi_pins>; > >> > status = "okay"; > >> > > >> > phy1: ethernet-phy@8 { > >> > compatible = "ethernet-phy-ieee802.3-c45"; > >> > reg = <8>; > >> > }; > >> > }; > >> > > >> > &cp0_ethernet { > >> > status = "okay"; > >> > }; > >> > > >> > /* 10GbE XFI AQR113C */ > >> > &cp0_eth0 { > >> > status = "okay"; > >> > phy = <&phy1>; > >> > phy-mode = "10gbase-r"; > >> > phys = <&cp0_comphy4 0>; > >> > }; > >> > > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > >> > some additional debug in mvpp2.c and aquantia_main.c: > >> > # ifconfig eth0 192.168.1.22 > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > >> > speed=-1 26:10gbase-r > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > >> > [ 8.898165] aqr107_resume > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > >> > duplex=255 speed=-1 26:10gbase-r 0: > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > >> > supported 00000000,00018000,000e706f advertising > >> > 00000000,00018000,000e706f > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > >> > phy/10gbase-r link mode > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > >> > pause=00 link=0 an=0 > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > >> > [ 8.976267] aqr107_resume > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > >> > 10gbase-r/10Gbps/Full/none/off > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> > duplex=1 speed=10000 26:10gbase-r > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > >> > - flow control off > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > >> > 10gbase-r/10Gbps/Full/none/off > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > >> > duplex=1 speed=10000 26:10gbase-r > >> > > >> > # ethtool eth0 > >> > Settings for eth0: > >> > Supported ports: [ ] > >> > Supported link modes: 10baseT/Half 10baseT/Full > >> > 100baseT/Half 100baseT/Full > >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid > >> turning them on), so they must be coming from somewhere else. I wonder > >> if this is because PHY_INTERFACE_MODE_SGMII is set in > >> supported_interfaces. > >> > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > >> should support it. I'm not sure if the aquantia driver is set up for it. > > > > This appears to trigger an issue from mvpp2: > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support > > 00000000,00018000,000e706f and advertisement > > 00000000,00018000,000e706f failed: -EINVAL > > Ah, I forgot this was a separate phy mode. Disregard this. > > >> > >> > 1000baseT/Full > >> > 10000baseT/Full > >> > 1000baseKX/Full > >> > 10000baseKX4/Full > >> > 10000baseKR/Full > >> > 2500baseT/Full > >> > 5000baseT/Full > >> > Supported pause frame use: Symmetric Receive-only > >> > Supports auto-negotiation: Yes > >> > Supported FEC modes: Not reported > >> > Advertised link modes: 10baseT/Half 10baseT/Full > >> > 100baseT/Half 100baseT/Full > >> > 1000baseT/Full > >> > 10000baseT/Full > >> > 1000baseKX/Full > >> > 10000baseKX4/Full > >> > 10000baseKR/Full > >> > 2500baseT/Full > >> > 5000baseT/Full > >> > Advertised pause frame use: Symmetric Receive-only > >> > Advertised auto-negotiation: Yes > >> > Advertised FEC modes: Not reported > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full > >> > 1000baseT/Half 1000baseT/Full > >> > 10000baseT/Full > >> > 2500baseT/Full > >> > 5000baseT/Full > >> > Link partner advertised pause frame use: No > >> > Link partner advertised auto-negotiation: Yes > >> > Link partner advertised FEC modes: Not reported > >> > Speed: 10000Mb/s > >> > Duplex: Full > >> > Port: Twisted Pair > >> > PHYAD: 8 > >> > Transceiver: external > >> > Auto-negotiation: on > >> > MDI-X: Unknown > >> > Link detected: yes > >> > # ping 192.168.1.146 -c5 > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > >> > > >> > --- 192.168.1.146 ping statistics --- > >> > 5 packets transmitted, 5 packets received, 0% packet loss > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms > >> > # # force switch port to 1G > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > >> > 10gbase-r/Unknown/Unknown/none/off > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> > duplex=255 speed=-1 26:10gbase-r > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off > >> > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) > >> and the vendor provisioning registers (dev 4, reg 0xc440 through > >> 0xc449). > > > > yes, this is what I've been looking at as well. When forced to 1000m > > the register shows a phy type of 11 which according to the aqr113 > > datasheet is XFI 5G: > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 > > link=1 duplex=1 speed=1000 interface=0 > > That's pretty strange. Seems like it's rate adapting from 5g instead of > 10g. Is SERDES Mode in the Global System Configuration For 1G register > set to XFI? 1E.31C=0x0106: Rate Adaptation Method: 2=Pause Rate Adaptation SERDES Mode: 6=XFI/2 (XFI 5G) Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-14 19:33 ` Tim Harvey @ 2022-11-16 22:37 ` Tim Harvey 2022-11-17 15:38 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-16 22:37 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Mon, Nov 14, 2022 at 11:33 AM Tim Harvey <tharvey@gateworks.com> wrote: > > On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: > > > > On 11/11/22 17:14, Tim Harvey wrote: > > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: > > >> > > >> On 11/11/22 16:20, Tim Harvey wrote: > > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > > >> >> > > >> >> On 11/11/22 15:57, Sean Anderson wrote: > > >> >> > Hi Tim, > > >> >> > > > >> >> > On 11/11/22 15:44, Tim Harvey wrote: > > >> >> >> Greetings, > > >> >> >> > > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: > > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching > > >> >> >> > > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at > > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > > >> >> >> > > >> >> >> Should I expect this to work now at those lower rates > > >> >> > > > >> >> > Yes. > > >> > > > >> > Sean, > > >> > > > >> > Good to hear - thank you for your work on this feature! > > >> > > > >> >> > > > >> >> >> and if so what kind of debug information or testing can I provide? > > >> >> > > > >> >> > Please send > > >> >> > > > >> >> > - Your test procedure (how do you select 1G?) > > >> >> > - Device tree node for the interface > > >> >> > - Output of ethtool (on both ends if possible). > > >> >> > - Kernel logs with debug enabled for drivers/phylink.c > > >> >> > > >> >> Sorry, this should be drivers/net/phy/phylink.c > > >> >> > > >> >> > > > >> >> > That should be enough to get us started. > > >> >> > > > >> >> > --Sean > > >> >> > > >> > > > >> > I'm currently testing by bringing up the network interface while > > >> > connected to a 10gbe switch, verifying link and traffic, then forcing > > >> > the switch port to 1000mbps. > > >> > > > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > > >> > > > >> > #include "cn9130.dtsi" /* include SoC device tree */ > > >> > > > >> > &cp0_xmdio { > > >> > pinctrl-names = "default"; > > >> > pinctrl-0 = <&cp0_xsmi_pins>; > > >> > status = "okay"; > > >> > > > >> > phy1: ethernet-phy@8 { > > >> > compatible = "ethernet-phy-ieee802.3-c45"; > > >> > reg = <8>; > > >> > }; > > >> > }; > > >> > > > >> > &cp0_ethernet { > > >> > status = "okay"; > > >> > }; > > >> > > > >> > /* 10GbE XFI AQR113C */ > > >> > &cp0_eth0 { > > >> > status = "okay"; > > >> > phy = <&phy1>; > > >> > phy-mode = "10gbase-r"; > > >> > phys = <&cp0_comphy4 0>; > > >> > }; > > >> > > > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > > >> > some additional debug in mvpp2.c and aquantia_main.c: > > >> > # ifconfig eth0 192.168.1.22 > > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > > >> > speed=-1 26:10gbase-r > > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > > >> > [ 8.898165] aqr107_resume > > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > > >> > duplex=255 speed=-1 26:10gbase-r 0: > > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > > >> > supported 00000000,00018000,000e706f advertising > > >> > 00000000,00018000,000e706f > > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > > >> > phy/10gbase-r link mode > > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > > >> > pause=00 link=0 an=0 > > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > > >> > [ 8.976267] aqr107_resume > > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > > >> > 10gbase-r/10Gbps/Full/none/off > > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > > >> > duplex=1 speed=10000 26:10gbase-r > > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > > >> > - flow control off > > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > > >> > 10gbase-r/10Gbps/Full/none/off > > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > > >> > duplex=1 speed=10000 26:10gbase-r > > >> > > > >> > # ethtool eth0 > > >> > Settings for eth0: > > >> > Supported ports: [ ] > > >> > Supported link modes: 10baseT/Half 10baseT/Full > > >> > 100baseT/Half 100baseT/Full > > >> > > >> 10/100 half duplex aren't achievable with rate matching (and we avoid > > >> turning them on), so they must be coming from somewhere else. I wonder > > >> if this is because PHY_INTERFACE_MODE_SGMII is set in > > >> supported_interfaces. > > >> > > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > > >> should support it. I'm not sure if the aquantia driver is set up for it. > > > > > > This appears to trigger an issue from mvpp2: > > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support > > > 00000000,00018000,000e706f and advertisement > > > 00000000,00018000,000e706f failed: -EINVAL > > > > Ah, I forgot this was a separate phy mode. Disregard this. > > > > >> > > >> > 1000baseT/Full > > >> > 10000baseT/Full > > >> > 1000baseKX/Full > > >> > 10000baseKX4/Full > > >> > 10000baseKR/Full > > >> > 2500baseT/Full > > >> > 5000baseT/Full > > >> > Supported pause frame use: Symmetric Receive-only > > >> > Supports auto-negotiation: Yes > > >> > Supported FEC modes: Not reported > > >> > Advertised link modes: 10baseT/Half 10baseT/Full > > >> > 100baseT/Half 100baseT/Full > > >> > 1000baseT/Full > > >> > 10000baseT/Full > > >> > 1000baseKX/Full > > >> > 10000baseKX4/Full > > >> > 10000baseKR/Full > > >> > 2500baseT/Full > > >> > 5000baseT/Full > > >> > Advertised pause frame use: Symmetric Receive-only > > >> > Advertised auto-negotiation: Yes > > >> > Advertised FEC modes: Not reported > > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full > > >> > 1000baseT/Half 1000baseT/Full > > >> > 10000baseT/Full > > >> > 2500baseT/Full > > >> > 5000baseT/Full > > >> > Link partner advertised pause frame use: No > > >> > Link partner advertised auto-negotiation: Yes > > >> > Link partner advertised FEC modes: Not reported > > >> > Speed: 10000Mb/s > > >> > Duplex: Full > > >> > Port: Twisted Pair > > >> > PHYAD: 8 > > >> > Transceiver: external > > >> > Auto-negotiation: on > > >> > MDI-X: Unknown > > >> > Link detected: yes > > >> > # ping 192.168.1.146 -c5 > > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes > > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > > >> > > > >> > --- 192.168.1.146 ping statistics --- > > >> > 5 packets transmitted, 5 packets received, 0% packet loss > > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms > > >> > # # force switch port to 1G > > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > > >> > 10gbase-r/Unknown/Unknown/none/off > > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > > >> > duplex=255 speed=-1 26:10gbase-r > > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off > > >> > > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send > > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also > > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) > > >> and the vendor provisioning registers (dev 4, reg 0xc440 through > > >> 0xc449). > > > > > > yes, this is what I've been looking at as well. When forced to 1000m > > > the register shows a phy type of 11 which according to the aqr113 > > > datasheet is XFI 5G: > > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 > > > link=1 duplex=1 speed=1000 interface=0 > > > > That's pretty strange. Seems like it's rate adapting from 5g instead of > > 10g. Is SERDES Mode in the Global System Configuration For 1G register > > set to XFI? > > 1E.31C=0x0106: > Rate Adaptation Method: 2=Pause Rate Adaptation > SERDES Mode: 6=XFI/2 (XFI 5G) > The SERDES mode here is not valid and it seems to always be set to XFI/2 unless I init/use the AQR113 in U-Boot. If I manually set SERDES Mode to 0 (XFI) in the driver I find that all rates 10g,5g,2.5g,1g,100m work fine both in Linux 6.0 and in Linux 6.1-rc5. I'm still trying to understand why I would need to set SERDES Mode manually (vs the XFI mode specific firmware setting this) but I am unclear what the rate adaptation in 6.1 provides in this case. Is it that perhaps the AQR113 is handling rate adaptation internally and that's why the non 10gbe rates are working on 6.0? Best Regards, Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-16 22:37 ` Tim Harvey @ 2022-11-17 15:38 ` Sean Anderson 2022-11-17 23:42 ` Tim Harvey 0 siblings, 1 reply; 21+ messages in thread From: Sean Anderson @ 2022-11-17 15:38 UTC (permalink / raw) To: Tim Harvey; +Cc: netdev, Russell King, David S. Miller On 11/16/22 17:37, Tim Harvey wrote: > On Mon, Nov 14, 2022 at 11:33 AM Tim Harvey <tharvey@gateworks.com> wrote: >> >> On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: >> > >> > On 11/11/22 17:14, Tim Harvey wrote: >> > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: >> > >> >> > >> On 11/11/22 16:20, Tim Harvey wrote: >> > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: >> > >> >> >> > >> >> On 11/11/22 15:57, Sean Anderson wrote: >> > >> >> > Hi Tim, >> > >> >> > >> > >> >> > On 11/11/22 15:44, Tim Harvey wrote: >> > >> >> >> Greetings, >> > >> >> >> >> > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: >> > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching >> > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces >> > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching >> > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching >> > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching >> > >> >> >> >> > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at >> > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 >> > >> >> >> >> > >> >> >> Should I expect this to work now at those lower rates >> > >> >> > >> > >> >> > Yes. >> > >> > >> > >> > Sean, >> > >> > >> > >> > Good to hear - thank you for your work on this feature! >> > >> > >> > >> >> > >> > >> >> >> and if so what kind of debug information or testing can I provide? >> > >> >> > >> > >> >> > Please send >> > >> >> > >> > >> >> > - Your test procedure (how do you select 1G?) >> > >> >> > - Device tree node for the interface >> > >> >> > - Output of ethtool (on both ends if possible). >> > >> >> > - Kernel logs with debug enabled for drivers/phylink.c >> > >> >> >> > >> >> Sorry, this should be drivers/net/phy/phylink.c >> > >> >> >> > >> >> > >> > >> >> > That should be enough to get us started. >> > >> >> > >> > >> >> > --Sean >> > >> >> >> > >> > >> > >> > I'm currently testing by bringing up the network interface while >> > >> > connected to a 10gbe switch, verifying link and traffic, then forcing >> > >> > the switch port to 1000mbps. >> > >> > >> > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: >> > >> > >> > >> > #include "cn9130.dtsi" /* include SoC device tree */ >> > >> > >> > >> > &cp0_xmdio { >> > >> > pinctrl-names = "default"; >> > >> > pinctrl-0 = <&cp0_xsmi_pins>; >> > >> > status = "okay"; >> > >> > >> > >> > phy1: ethernet-phy@8 { >> > >> > compatible = "ethernet-phy-ieee802.3-c45"; >> > >> > reg = <8>; >> > >> > }; >> > >> > }; >> > >> > >> > >> > &cp0_ethernet { >> > >> > status = "okay"; >> > >> > }; >> > >> > >> > >> > /* 10GbE XFI AQR113C */ >> > >> > &cp0_eth0 { >> > >> > status = "okay"; >> > >> > phy = <&phy1>; >> > >> > phy-mode = "10gbase-r"; >> > >> > phys = <&cp0_comphy4 0>; >> > >> > }; >> > >> > >> > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and >> > >> > some additional debug in mvpp2.c and aquantia_main.c: >> > >> > # ifconfig eth0 192.168.1.22 >> > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 >> > >> > speed=-1 26:10gbase-r >> > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 >> > >> > [ 8.898165] aqr107_resume >> > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 >> > >> > duplex=255 speed=-1 26:10gbase-r 0: >> > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY >> > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) >> > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting >> > >> > supported 00000000,00018000,000e706f advertising >> > >> > 00000000,00018000,000e706f >> > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down >> > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for >> > >> > phy/10gbase-r link mode >> > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r >> > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: >> > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 >> > >> > pause=00 link=0 an=0 >> > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down >> > >> > [ 8.976267] aqr107_resume >> > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down >> > >> > 10gbase-r/10Gbps/Full/none/off >> > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 >> > >> > duplex=1 speed=10000 26:10gbase-r >> > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up >> > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full >> > >> > - flow control off >> > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready >> > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up >> > >> > 10gbase-r/10Gbps/Full/none/off >> > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 >> > >> > duplex=1 speed=10000 26:10gbase-r >> > >> > >> > >> > # ethtool eth0 >> > >> > Settings for eth0: >> > >> > Supported ports: [ ] >> > >> > Supported link modes: 10baseT/Half 10baseT/Full >> > >> > 100baseT/Half 100baseT/Full >> > >> >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid >> > >> turning them on), so they must be coming from somewhere else. I wonder >> > >> if this is because PHY_INTERFACE_MODE_SGMII is set in >> > >> supported_interfaces. >> > >> >> > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy >> > >> should support it. I'm not sure if the aquantia driver is set up for it. >> > > >> > > This appears to trigger an issue from mvpp2: >> > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support >> > > 00000000,00018000,000e706f and advertisement >> > > 00000000,00018000,000e706f failed: -EINVAL >> > >> > Ah, I forgot this was a separate phy mode. Disregard this. >> > >> > >> >> > >> > 1000baseT/Full >> > >> > 10000baseT/Full >> > >> > 1000baseKX/Full >> > >> > 10000baseKX4/Full >> > >> > 10000baseKR/Full >> > >> > 2500baseT/Full >> > >> > 5000baseT/Full >> > >> > Supported pause frame use: Symmetric Receive-only >> > >> > Supports auto-negotiation: Yes >> > >> > Supported FEC modes: Not reported >> > >> > Advertised link modes: 10baseT/Half 10baseT/Full >> > >> > 100baseT/Half 100baseT/Full >> > >> > 1000baseT/Full >> > >> > 10000baseT/Full >> > >> > 1000baseKX/Full >> > >> > 10000baseKX4/Full >> > >> > 10000baseKR/Full >> > >> > 2500baseT/Full >> > >> > 5000baseT/Full >> > >> > Advertised pause frame use: Symmetric Receive-only >> > >> > Advertised auto-negotiation: Yes >> > >> > Advertised FEC modes: Not reported >> > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full >> > >> > 1000baseT/Half 1000baseT/Full >> > >> > 10000baseT/Full >> > >> > 2500baseT/Full >> > >> > 5000baseT/Full >> > >> > Link partner advertised pause frame use: No >> > >> > Link partner advertised auto-negotiation: Yes >> > >> > Link partner advertised FEC modes: Not reported >> > >> > Speed: 10000Mb/s >> > >> > Duplex: Full >> > >> > Port: Twisted Pair >> > >> > PHYAD: 8 >> > >> > Transceiver: external >> > >> > Auto-negotiation: on >> > >> > MDI-X: Unknown >> > >> > Link detected: yes >> > >> > # ping 192.168.1.146 -c5 >> > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes >> > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms >> > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms >> > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms >> > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms >> > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms >> > >> > >> > >> > --- 192.168.1.146 ping statistics --- >> > >> > 5 packets transmitted, 5 packets received, 0% packet loss >> > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms >> > >> > # # force switch port to 1G >> > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down >> > >> > 10gbase-r/Unknown/Unknown/none/off >> > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down >> > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down >> > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 >> > >> > duplex=255 speed=-1 26:10gbase-r >> > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off >> > >> >> > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send >> > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also >> > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) >> > >> and the vendor provisioning registers (dev 4, reg 0xc440 through >> > >> 0xc449). >> > > >> > > yes, this is what I've been looking at as well. When forced to 1000m >> > > the register shows a phy type of 11 which according to the aqr113 >> > > datasheet is XFI 5G: >> > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 >> > > link=1 duplex=1 speed=1000 interface=0 >> > >> > That's pretty strange. Seems like it's rate adapting from 5g instead of >> > 10g. Is SERDES Mode in the Global System Configuration For 1G register >> > set to XFI? >> >> 1E.31C=0x0106: >> Rate Adaptation Method: 2=Pause Rate Adaptation >> SERDES Mode: 6=XFI/2 (XFI 5G) >> > > The SERDES mode here is not valid and it seems to always be set to > XFI/2 unless I init/use the AQR113 in U-Boot. If I manually set SERDES > Mode to 0 (XFI) in the driver I find that all rates > 10g,5g,2.5g,1g,100m work fine both in Linux 6.0 and in Linux 6.1-rc5. > I'm still trying to understand why I would need to set SERDES Mode > manually (vs the XFI mode specific firmware setting this) but I am > unclear what the rate adaptation in 6.1 provides in this case. Is it > that perhaps the AQR113 is handling rate adaptation internally and > that's why the non 10gbe rates are working on 6.0? The changes in 6.1 are - We now always enable pause frame reception when doing rate adaptation. This is necessary for rate adaptation to work, but wasn't done automatically before. - We advertise lower speed modes which are enabled with rate adaptation, even if we would not otherwise be able to support them. I'm not sure why you'd have XFI/2 selected in 6.1 if it isn't selected in 6.0. --Sean ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-17 15:38 ` Sean Anderson @ 2022-11-17 23:42 ` Tim Harvey 2022-11-28 19:57 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Tim Harvey @ 2022-11-17 23:42 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Thu, Nov 17, 2022 at 7:38 AM Sean Anderson <sean.anderson@seco.com> wrote: > > On 11/16/22 17:37, Tim Harvey wrote: > > On Mon, Nov 14, 2022 at 11:33 AM Tim Harvey <tharvey@gateworks.com> wrote: > >> > >> On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> > > >> > On 11/11/22 17:14, Tim Harvey wrote: > >> > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> > >> > >> > >> On 11/11/22 16:20, Tim Harvey wrote: > >> > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> > >> >> > >> > >> >> On 11/11/22 15:57, Sean Anderson wrote: > >> > >> >> > Hi Tim, > >> > >> >> > > >> > >> >> > On 11/11/22 15:44, Tim Harvey wrote: > >> > >> >> >> Greetings, > >> > >> >> >> > >> > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: > >> > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > >> > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > >> > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > >> > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > >> > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching > >> > >> >> >> > >> > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at > >> > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > >> > >> >> >> > >> > >> >> >> Should I expect this to work now at those lower rates > >> > >> >> > > >> > >> >> > Yes. > >> > >> > > >> > >> > Sean, > >> > >> > > >> > >> > Good to hear - thank you for your work on this feature! > >> > >> > > >> > >> >> > > >> > >> >> >> and if so what kind of debug information or testing can I provide? > >> > >> >> > > >> > >> >> > Please send > >> > >> >> > > >> > >> >> > - Your test procedure (how do you select 1G?) > >> > >> >> > - Device tree node for the interface > >> > >> >> > - Output of ethtool (on both ends if possible). > >> > >> >> > - Kernel logs with debug enabled for drivers/phylink.c > >> > >> >> > >> > >> >> Sorry, this should be drivers/net/phy/phylink.c > >> > >> >> > >> > >> >> > > >> > >> >> > That should be enough to get us started. > >> > >> >> > > >> > >> >> > --Sean > >> > >> >> > >> > >> > > >> > >> > I'm currently testing by bringing up the network interface while > >> > >> > connected to a 10gbe switch, verifying link and traffic, then forcing > >> > >> > the switch port to 1000mbps. > >> > >> > > >> > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > >> > >> > > >> > >> > #include "cn9130.dtsi" /* include SoC device tree */ > >> > >> > > >> > >> > &cp0_xmdio { > >> > >> > pinctrl-names = "default"; > >> > >> > pinctrl-0 = <&cp0_xsmi_pins>; > >> > >> > status = "okay"; > >> > >> > > >> > >> > phy1: ethernet-phy@8 { > >> > >> > compatible = "ethernet-phy-ieee802.3-c45"; > >> > >> > reg = <8>; > >> > >> > }; > >> > >> > }; > >> > >> > > >> > >> > &cp0_ethernet { > >> > >> > status = "okay"; > >> > >> > }; > >> > >> > > >> > >> > /* 10GbE XFI AQR113C */ > >> > >> > &cp0_eth0 { > >> > >> > status = "okay"; > >> > >> > phy = <&phy1>; > >> > >> > phy-mode = "10gbase-r"; > >> > >> > phys = <&cp0_comphy4 0>; > >> > >> > }; > >> > >> > > >> > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > >> > >> > some additional debug in mvpp2.c and aquantia_main.c: > >> > >> > # ifconfig eth0 192.168.1.22 > >> > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > >> > >> > speed=-1 26:10gbase-r > >> > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > >> > >> > [ 8.898165] aqr107_resume > >> > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > >> > >> > duplex=255 speed=-1 26:10gbase-r 0: > >> > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > >> > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > >> > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > >> > >> > supported 00000000,00018000,000e706f advertising > >> > >> > 00000000,00018000,000e706f > >> > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > >> > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > >> > >> > phy/10gbase-r link mode > >> > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > >> > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > >> > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > >> > >> > pause=00 link=0 an=0 > >> > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > >> > >> > [ 8.976267] aqr107_resume > >> > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > >> > >> > 10gbase-r/10Gbps/Full/none/off > >> > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> > >> > duplex=1 speed=10000 26:10gbase-r > >> > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > >> > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > >> > >> > - flow control off > >> > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > >> > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > >> > >> > 10gbase-r/10Gbps/Full/none/off > >> > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > >> > >> > duplex=1 speed=10000 26:10gbase-r > >> > >> > > >> > >> > # ethtool eth0 > >> > >> > Settings for eth0: > >> > >> > Supported ports: [ ] > >> > >> > Supported link modes: 10baseT/Half 10baseT/Full > >> > >> > 100baseT/Half 100baseT/Full > >> > >> > >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid > >> > >> turning them on), so they must be coming from somewhere else. I wonder > >> > >> if this is because PHY_INTERFACE_MODE_SGMII is set in > >> > >> supported_interfaces. > >> > >> > >> > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > >> > >> should support it. I'm not sure if the aquantia driver is set up for it. > >> > > > >> > > This appears to trigger an issue from mvpp2: > >> > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support > >> > > 00000000,00018000,000e706f and advertisement > >> > > 00000000,00018000,000e706f failed: -EINVAL > >> > > >> > Ah, I forgot this was a separate phy mode. Disregard this. > >> > > >> > >> > >> > >> > 1000baseT/Full > >> > >> > 10000baseT/Full > >> > >> > 1000baseKX/Full > >> > >> > 10000baseKX4/Full > >> > >> > 10000baseKR/Full > >> > >> > 2500baseT/Full > >> > >> > 5000baseT/Full > >> > >> > Supported pause frame use: Symmetric Receive-only > >> > >> > Supports auto-negotiation: Yes > >> > >> > Supported FEC modes: Not reported > >> > >> > Advertised link modes: 10baseT/Half 10baseT/Full > >> > >> > 100baseT/Half 100baseT/Full > >> > >> > 1000baseT/Full > >> > >> > 10000baseT/Full > >> > >> > 1000baseKX/Full > >> > >> > 10000baseKX4/Full > >> > >> > 10000baseKR/Full > >> > >> > 2500baseT/Full > >> > >> > 5000baseT/Full > >> > >> > Advertised pause frame use: Symmetric Receive-only > >> > >> > Advertised auto-negotiation: Yes > >> > >> > Advertised FEC modes: Not reported > >> > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full > >> > >> > 1000baseT/Half 1000baseT/Full > >> > >> > 10000baseT/Full > >> > >> > 2500baseT/Full > >> > >> > 5000baseT/Full > >> > >> > Link partner advertised pause frame use: No > >> > >> > Link partner advertised auto-negotiation: Yes > >> > >> > Link partner advertised FEC modes: Not reported > >> > >> > Speed: 10000Mb/s > >> > >> > Duplex: Full > >> > >> > Port: Twisted Pair > >> > >> > PHYAD: 8 > >> > >> > Transceiver: external > >> > >> > Auto-negotiation: on > >> > >> > MDI-X: Unknown > >> > >> > Link detected: yes > >> > >> > # ping 192.168.1.146 -c5 > >> > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes > >> > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > >> > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > >> > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > >> > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > >> > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > >> > >> > > >> > >> > --- 192.168.1.146 ping statistics --- > >> > >> > 5 packets transmitted, 5 packets received, 0% packet loss > >> > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms > >> > >> > # # force switch port to 1G > >> > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > >> > >> > 10gbase-r/Unknown/Unknown/none/off > >> > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > >> > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > >> > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> > >> > duplex=255 speed=-1 26:10gbase-r > >> > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off > >> > >> > >> > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send > >> > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also > >> > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) > >> > >> and the vendor provisioning registers (dev 4, reg 0xc440 through > >> > >> 0xc449). > >> > > > >> > > yes, this is what I've been looking at as well. When forced to 1000m > >> > > the register shows a phy type of 11 which according to the aqr113 > >> > > datasheet is XFI 5G: > >> > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 > >> > > link=1 duplex=1 speed=1000 interface=0 > >> > > >> > That's pretty strange. Seems like it's rate adapting from 5g instead of > >> > 10g. Is SERDES Mode in the Global System Configuration For 1G register > >> > set to XFI? > >> > >> 1E.31C=0x0106: > >> Rate Adaptation Method: 2=Pause Rate Adaptation > >> SERDES Mode: 6=XFI/2 (XFI 5G) > >> > > > > The SERDES mode here is not valid and it seems to always be set to > > XFI/2 unless I init/use the AQR113 in U-Boot. If I manually set SERDES > > Mode to 0 (XFI) in the driver I find that all rates > > 10g,5g,2.5g,1g,100m work fine both in Linux 6.0 and in Linux 6.1-rc5. > > I'm still trying to understand why I would need to set SERDES Mode > > manually (vs the XFI mode specific firmware setting this) but I am > > unclear what the rate adaptation in 6.1 provides in this case. Is it > > that perhaps the AQR113 is handling rate adaptation internally and > > that's why the non 10gbe rates are working on 6.0? > > The changes in 6.1 are > > - We now always enable pause frame reception when doing rate adaptation. > This is necessary for rate adaptation to work, but wasn't done > automatically before. > - We advertise lower speed modes which are enabled with rate adaptation, > even if we would not otherwise be able to support them. > > I'm not sure why you'd have XFI/2 selected in 6.1 if it isn't selected > in 6.0. Sean, Thanks for the explanation. The issue I had which resulted in the wrong SERDES mode was simply that I was using the wrong aquantia firmware. They customize the firmware to default registers like SERDES mode specifically for customers and I was unknowingly using the firmware for XFI/2 instead of XFI. I suppose it would be worth putting something in the aquantia driver that verifies SERDES mode matches the phy-mode from dt to throw an error. Best Regards, Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-17 23:42 ` Tim Harvey @ 2022-11-28 19:57 ` Sean Anderson 2022-12-01 1:11 ` Tim Harvey 0 siblings, 1 reply; 21+ messages in thread From: Sean Anderson @ 2022-11-28 19:57 UTC (permalink / raw) To: Tim Harvey; +Cc: netdev, Russell King, David S. Miller Hi Tim, On 11/17/22 18:42, Tim Harvey wrote: > On Thu, Nov 17, 2022 at 7:38 AM Sean Anderson <sean.anderson@seco.com> wrote: >> >> On 11/16/22 17:37, Tim Harvey wrote: >> > On Mon, Nov 14, 2022 at 11:33 AM Tim Harvey <tharvey@gateworks.com> wrote: >> >> >> >> On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> > >> >> > On 11/11/22 17:14, Tim Harvey wrote: >> >> > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> > >> >> >> > >> On 11/11/22 16:20, Tim Harvey wrote: >> >> > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: >> >> > >> >> >> >> > >> >> On 11/11/22 15:57, Sean Anderson wrote: >> >> > >> >> > Hi Tim, >> >> > >> >> > >> >> > >> >> > On 11/11/22 15:44, Tim Harvey wrote: >> >> > >> >> >> Greetings, >> >> > >> >> >> >> >> > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: >> >> > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching >> >> > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces >> >> > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching >> >> > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching >> >> > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching >> >> > >> >> >> >> >> > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at >> >> > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 >> >> > >> >> >> >> >> > >> >> >> Should I expect this to work now at those lower rates >> >> > >> >> > >> >> > >> >> > Yes. >> >> > >> > >> >> > >> > Sean, >> >> > >> > >> >> > >> > Good to hear - thank you for your work on this feature! >> >> > >> > >> >> > >> >> > >> >> > >> >> >> and if so what kind of debug information or testing can I provide? >> >> > >> >> > >> >> > >> >> > Please send >> >> > >> >> > >> >> > >> >> > - Your test procedure (how do you select 1G?) >> >> > >> >> > - Device tree node for the interface >> >> > >> >> > - Output of ethtool (on both ends if possible). >> >> > >> >> > - Kernel logs with debug enabled for drivers/phylink.c >> >> > >> >> >> >> > >> >> Sorry, this should be drivers/net/phy/phylink.c >> >> > >> >> >> >> > >> >> > >> >> > >> >> > That should be enough to get us started. >> >> > >> >> > >> >> > >> >> > --Sean >> >> > >> >> >> >> > >> > >> >> > >> > I'm currently testing by bringing up the network interface while >> >> > >> > connected to a 10gbe switch, verifying link and traffic, then forcing >> >> > >> > the switch port to 1000mbps. >> >> > >> > >> >> > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: >> >> > >> > >> >> > >> > #include "cn9130.dtsi" /* include SoC device tree */ >> >> > >> > >> >> > >> > &cp0_xmdio { >> >> > >> > pinctrl-names = "default"; >> >> > >> > pinctrl-0 = <&cp0_xsmi_pins>; >> >> > >> > status = "okay"; >> >> > >> > >> >> > >> > phy1: ethernet-phy@8 { >> >> > >> > compatible = "ethernet-phy-ieee802.3-c45"; >> >> > >> > reg = <8>; >> >> > >> > }; >> >> > >> > }; >> >> > >> > >> >> > >> > &cp0_ethernet { >> >> > >> > status = "okay"; >> >> > >> > }; >> >> > >> > >> >> > >> > /* 10GbE XFI AQR113C */ >> >> > >> > &cp0_eth0 { >> >> > >> > status = "okay"; >> >> > >> > phy = <&phy1>; >> >> > >> > phy-mode = "10gbase-r"; >> >> > >> > phys = <&cp0_comphy4 0>; >> >> > >> > }; >> >> > >> > >> >> > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and >> >> > >> > some additional debug in mvpp2.c and aquantia_main.c: >> >> > >> > # ifconfig eth0 192.168.1.22 >> >> > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 >> >> > >> > speed=-1 26:10gbase-r >> >> > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 >> >> > >> > [ 8.898165] aqr107_resume >> >> > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 >> >> > >> > duplex=255 speed=-1 26:10gbase-r 0: >> >> > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY >> >> > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) >> >> > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting >> >> > >> > supported 00000000,00018000,000e706f advertising >> >> > >> > 00000000,00018000,000e706f >> >> > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down >> >> > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for >> >> > >> > phy/10gbase-r link mode >> >> > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r >> >> > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: >> >> > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 >> >> > >> > pause=00 link=0 an=0 >> >> > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down >> >> > >> > [ 8.976267] aqr107_resume >> >> > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down >> >> > >> > 10gbase-r/10Gbps/Full/none/off >> >> > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 >> >> > >> > duplex=1 speed=10000 26:10gbase-r >> >> > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up >> >> > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full >> >> > >> > - flow control off >> >> > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready >> >> > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up >> >> > >> > 10gbase-r/10Gbps/Full/none/off >> >> > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 >> >> > >> > duplex=1 speed=10000 26:10gbase-r >> >> > >> > >> >> > >> > # ethtool eth0 >> >> > >> > Settings for eth0: >> >> > >> > Supported ports: [ ] >> >> > >> > Supported link modes: 10baseT/Half 10baseT/Full >> >> > >> > 100baseT/Half 100baseT/Full >> >> > >> >> >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid >> >> > >> turning them on), so they must be coming from somewhere else. I wonder >> >> > >> if this is because PHY_INTERFACE_MODE_SGMII is set in >> >> > >> supported_interfaces. >> >> > >> >> >> > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy >> >> > >> should support it. I'm not sure if the aquantia driver is set up for it. >> >> > > >> >> > > This appears to trigger an issue from mvpp2: >> >> > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support >> >> > > 00000000,00018000,000e706f and advertisement >> >> > > 00000000,00018000,000e706f failed: -EINVAL >> >> > >> >> > Ah, I forgot this was a separate phy mode. Disregard this. >> >> > >> >> > >> >> >> > >> > 1000baseT/Full >> >> > >> > 10000baseT/Full >> >> > >> > 1000baseKX/Full >> >> > >> > 10000baseKX4/Full >> >> > >> > 10000baseKR/Full >> >> > >> > 2500baseT/Full >> >> > >> > 5000baseT/Full >> >> > >> > Supported pause frame use: Symmetric Receive-only >> >> > >> > Supports auto-negotiation: Yes >> >> > >> > Supported FEC modes: Not reported >> >> > >> > Advertised link modes: 10baseT/Half 10baseT/Full >> >> > >> > 100baseT/Half 100baseT/Full >> >> > >> > 1000baseT/Full >> >> > >> > 10000baseT/Full >> >> > >> > 1000baseKX/Full >> >> > >> > 10000baseKX4/Full >> >> > >> > 10000baseKR/Full >> >> > >> > 2500baseT/Full >> >> > >> > 5000baseT/Full >> >> > >> > Advertised pause frame use: Symmetric Receive-only >> >> > >> > Advertised auto-negotiation: Yes >> >> > >> > Advertised FEC modes: Not reported >> >> > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full >> >> > >> > 1000baseT/Half 1000baseT/Full >> >> > >> > 10000baseT/Full >> >> > >> > 2500baseT/Full >> >> > >> > 5000baseT/Full >> >> > >> > Link partner advertised pause frame use: No >> >> > >> > Link partner advertised auto-negotiation: Yes >> >> > >> > Link partner advertised FEC modes: Not reported >> >> > >> > Speed: 10000Mb/s >> >> > >> > Duplex: Full >> >> > >> > Port: Twisted Pair >> >> > >> > PHYAD: 8 >> >> > >> > Transceiver: external >> >> > >> > Auto-negotiation: on >> >> > >> > MDI-X: Unknown >> >> > >> > Link detected: yes >> >> > >> > # ping 192.168.1.146 -c5 >> >> > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes >> >> > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms >> >> > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms >> >> > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms >> >> > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms >> >> > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms >> >> > >> > >> >> > >> > --- 192.168.1.146 ping statistics --- >> >> > >> > 5 packets transmitted, 5 packets received, 0% packet loss >> >> > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms >> >> > >> > # # force switch port to 1G >> >> > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down >> >> > >> > 10gbase-r/Unknown/Unknown/none/off >> >> > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down >> >> > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down >> >> > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 >> >> > >> > duplex=255 speed=-1 26:10gbase-r >> >> > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off >> >> > >> >> >> > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send >> >> > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also >> >> > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) >> >> > >> and the vendor provisioning registers (dev 4, reg 0xc440 through >> >> > >> 0xc449). >> >> > > >> >> > > yes, this is what I've been looking at as well. When forced to 1000m >> >> > > the register shows a phy type of 11 which according to the aqr113 >> >> > > datasheet is XFI 5G: >> >> > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 >> >> > > link=1 duplex=1 speed=1000 interface=0 >> >> > >> >> > That's pretty strange. Seems like it's rate adapting from 5g instead of >> >> > 10g. Is SERDES Mode in the Global System Configuration For 1G register >> >> > set to XFI? >> >> >> >> 1E.31C=0x0106: >> >> Rate Adaptation Method: 2=Pause Rate Adaptation >> >> SERDES Mode: 6=XFI/2 (XFI 5G) >> >> >> > >> > The SERDES mode here is not valid and it seems to always be set to >> > XFI/2 unless I init/use the AQR113 in U-Boot. If I manually set SERDES >> > Mode to 0 (XFI) in the driver I find that all rates >> > 10g,5g,2.5g,1g,100m work fine both in Linux 6.0 and in Linux 6.1-rc5. >> > I'm still trying to understand why I would need to set SERDES Mode >> > manually (vs the XFI mode specific firmware setting this) but I am >> > unclear what the rate adaptation in 6.1 provides in this case. Is it >> > that perhaps the AQR113 is handling rate adaptation internally and >> > that's why the non 10gbe rates are working on 6.0? >> >> The changes in 6.1 are >> >> - We now always enable pause frame reception when doing rate adaptation. >> This is necessary for rate adaptation to work, but wasn't done >> automatically before. >> - We advertise lower speed modes which are enabled with rate adaptation, >> even if we would not otherwise be able to support them. >> >> I'm not sure why you'd have XFI/2 selected in 6.1 if it isn't selected >> in 6.0. > > Sean, > > Thanks for the explanation. The issue I had which resulted in the > wrong SERDES mode was simply that I was using the wrong aquantia > firmware. They customize the firmware to default registers like SERDES > mode specifically for customers and I was unknowingly using the > firmware for XFI/2 instead of XFI. > > I suppose it would be worth putting something in the aquantia driver > that verifies SERDES mode matches the phy-mode from dt to throw an > error. > > Best Regards, > > Tim Can you test the following series to see if it fixes your problem: https://lore.kernel.org/netdev/20221128195409.100873-1-sean.anderson@seco.com/ --Sean ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-28 19:57 ` Sean Anderson @ 2022-12-01 1:11 ` Tim Harvey 0 siblings, 0 replies; 21+ messages in thread From: Tim Harvey @ 2022-12-01 1:11 UTC (permalink / raw) To: Sean Anderson; +Cc: netdev, Russell King, David S. Miller On Mon, Nov 28, 2022 at 11:58 AM Sean Anderson <sean.anderson@seco.com> wrote: > > Hi Tim, > > On 11/17/22 18:42, Tim Harvey wrote: > > On Thu, Nov 17, 2022 at 7:38 AM Sean Anderson <sean.anderson@seco.com> wrote: > >> > >> On 11/16/22 17:37, Tim Harvey wrote: > >> > On Mon, Nov 14, 2022 at 11:33 AM Tim Harvey <tharvey@gateworks.com> wrote: > >> >> > >> >> On Fri, Nov 11, 2022 at 2:38 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> >> > > >> >> > On 11/11/22 17:14, Tim Harvey wrote: > >> >> > > On Fri, Nov 11, 2022 at 1:54 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> >> > >> > >> >> > >> On 11/11/22 16:20, Tim Harvey wrote: > >> >> > >> > On Fri, Nov 11, 2022 at 12:58 PM Sean Anderson <sean.anderson@seco.com> wrote: > >> >> > >> >> > >> >> > >> >> On 11/11/22 15:57, Sean Anderson wrote: > >> >> > >> >> > Hi Tim, > >> >> > >> >> > > >> >> > >> >> > On 11/11/22 15:44, Tim Harvey wrote: > >> >> > >> >> >> Greetings, > >> >> > >> >> >> > >> >> > >> >> >> I've noticed some recent commits that appear to add rate adaptation support: > >> >> > >> >> >> 3c42563b3041 net: phy: aquantia: Add support for rate matching > >> >> > >> >> >> 7de26bf144f6 net: phy: aquantia: Add some additional phy interfaces > >> >> > >> >> >> b7e9294885b6 net: phylink: Adjust advertisement based on rate matching > >> >> > >> >> >> ae0e4bb2a0e0 net: phylink: Adjust link settings based on rate matching > >> >> > >> >> >> 0c3e10cb4423 net: phy: Add support for rate matching > >> >> > >> >> >> > >> >> > >> >> >> I have a board with an AQR113C PHY over XFI that functions properly at > >> >> > >> >> >> 10Gbe links but still not at 1Gbe,2.5Gbe,5.0Gbe,100M with v6.1-rc4 > >> >> > >> >> >> > >> >> > >> >> >> Should I expect this to work now at those lower rates > >> >> > >> >> > > >> >> > >> >> > Yes. > >> >> > >> > > >> >> > >> > Sean, > >> >> > >> > > >> >> > >> > Good to hear - thank you for your work on this feature! > >> >> > >> > > >> >> > >> >> > > >> >> > >> >> >> and if so what kind of debug information or testing can I provide? > >> >> > >> >> > > >> >> > >> >> > Please send > >> >> > >> >> > > >> >> > >> >> > - Your test procedure (how do you select 1G?) > >> >> > >> >> > - Device tree node for the interface > >> >> > >> >> > - Output of ethtool (on both ends if possible). > >> >> > >> >> > - Kernel logs with debug enabled for drivers/phylink.c > >> >> > >> >> > >> >> > >> >> Sorry, this should be drivers/net/phy/phylink.c > >> >> > >> >> > >> >> > >> >> > > >> >> > >> >> > That should be enough to get us started. > >> >> > >> >> > > >> >> > >> >> > --Sean > >> >> > >> >> > >> >> > >> > > >> >> > >> > I'm currently testing by bringing up the network interface while > >> >> > >> > connected to a 10gbe switch, verifying link and traffic, then forcing > >> >> > >> > the switch port to 1000mbps. > >> >> > >> > > >> >> > >> > The board has a CN9130 on it (NIC is mvpp2) and the dt node snippets are: > >> >> > >> > > >> >> > >> > #include "cn9130.dtsi" /* include SoC device tree */ > >> >> > >> > > >> >> > >> > &cp0_xmdio { > >> >> > >> > pinctrl-names = "default"; > >> >> > >> > pinctrl-0 = <&cp0_xsmi_pins>; > >> >> > >> > status = "okay"; > >> >> > >> > > >> >> > >> > phy1: ethernet-phy@8 { > >> >> > >> > compatible = "ethernet-phy-ieee802.3-c45"; > >> >> > >> > reg = <8>; > >> >> > >> > }; > >> >> > >> > }; > >> >> > >> > > >> >> > >> > &cp0_ethernet { > >> >> > >> > status = "okay"; > >> >> > >> > }; > >> >> > >> > > >> >> > >> > /* 10GbE XFI AQR113C */ > >> >> > >> > &cp0_eth0 { > >> >> > >> > status = "okay"; > >> >> > >> > phy = <&phy1>; > >> >> > >> > phy-mode = "10gbase-r"; > >> >> > >> > phys = <&cp0_comphy4 0>; > >> >> > >> > }; > >> >> > >> > > >> >> > >> > Here are some logs with debug enabled in drivers/net/phy/phylink.c and > >> >> > >> > some additional debug in mvpp2.c and aquantia_main.c: > >> >> > >> > # ifconfig eth0 192.168.1.22 > >> >> > >> > [ 8.882437] aqr107_config_init state=1:ready an=1 link=0 duplex=255 > >> >> > >> > speed=-1 26:10gbase-r > >> >> > >> > [ 8.891391] aqr107_chip_info FW 5.6, Build 7, Provisioning 6 > >> >> > >> > [ 8.898165] aqr107_resume > >> >> > >> > [ 8.902853] aqr107_get_rate_matching state=1:ready an=1 link=0 > >> >> > >> > duplex=255 speed=-1 26:10gbase-r 0: > >> >> > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > >> >> > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > >> >> > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > >> >> > >> > supported 00000000,00018000,000e706f advertising > >> >> > >> > 00000000,00018000,000e706f > >> >> > >> > [ 8.934349] mvpp2 f2000000.ethernet eth0: mac link down > >> >> > >> > [ 8.948812] mvpp2 f2000000.ethernet eth0: configuring for > >> >> > >> > phy/10gbase-r link mode > >> >> > >> > [ 8.956350] mvpp2 f2000000.ethernet eth0: major config 10gbase-r > >> >> > >> > [ 8.962414] mvpp2 f2000000.ethernet eth0: phylink_mac_config: > >> >> > >> > mode=phy/10gbase-r/Unknown/Unknown/none adv=00000000,00000000,00000000 > >> >> > >> > pause=00 link=0 an=0 > >> >> > >> > [ 8.976252] mvpp2 f2000000.ethernet eth0: mac link down > >> >> > >> > [ 8.976267] aqr107_resume > >> >> > >> > [ 8.988970] mvpp2 f2000000.ethernet eth0: phy link down > >> >> > >> > 10gbase-r/10Gbps/Full/none/off > >> >> > >> > [ 8.997086] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> >> > >> > duplex=1 speed=10000 26:10gbase-r > >> >> > >> > [ 14.112540] mvpp2 f2000000.ethernet eth0: mac link up > >> >> > >> > [ 14.112594] mvpp2 f2000000.ethernet eth0: Link is Up - 10Gbps/Full > >> >> > >> > - flow control off > >> >> > >> > [ 14.112673] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > >> >> > >> > [ 14.118198] mvpp2 f2000000.ethernet eth0: phy link up > >> >> > >> > 10gbase-r/10Gbps/Full/none/off > >> >> > >> > [ 14.139808] aqr107_link_change_notify state=4:running an=1 link=1 > >> >> > >> > duplex=1 speed=10000 26:10gbase-r > >> >> > >> > > >> >> > >> > # ethtool eth0 > >> >> > >> > Settings for eth0: > >> >> > >> > Supported ports: [ ] > >> >> > >> > Supported link modes: 10baseT/Half 10baseT/Full > >> >> > >> > 100baseT/Half 100baseT/Full > >> >> > >> > >> >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid > >> >> > >> turning them on), so they must be coming from somewhere else. I wonder > >> >> > >> if this is because PHY_INTERFACE_MODE_SGMII is set in > >> >> > >> supported_interfaces. > >> >> > >> > >> >> > >> I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > >> >> > >> should support it. I'm not sure if the aquantia driver is set up for it. > >> >> > > > >> >> > > This appears to trigger an issue from mvpp2: > >> >> > > mvpp2 f2000000.ethernet eth0: validation of usxgmii with support > >> >> > > 00000000,00018000,000e706f and advertisement > >> >> > > 00000000,00018000,000e706f failed: -EINVAL > >> >> > > >> >> > Ah, I forgot this was a separate phy mode. Disregard this. > >> >> > > >> >> > >> > >> >> > >> > 1000baseT/Full > >> >> > >> > 10000baseT/Full > >> >> > >> > 1000baseKX/Full > >> >> > >> > 10000baseKX4/Full > >> >> > >> > 10000baseKR/Full > >> >> > >> > 2500baseT/Full > >> >> > >> > 5000baseT/Full > >> >> > >> > Supported pause frame use: Symmetric Receive-only > >> >> > >> > Supports auto-negotiation: Yes > >> >> > >> > Supported FEC modes: Not reported > >> >> > >> > Advertised link modes: 10baseT/Half 10baseT/Full > >> >> > >> > 100baseT/Half 100baseT/Full > >> >> > >> > 1000baseT/Full > >> >> > >> > 10000baseT/Full > >> >> > >> > 1000baseKX/Full > >> >> > >> > 10000baseKX4/Full > >> >> > >> > 10000baseKR/Full > >> >> > >> > 2500baseT/Full > >> >> > >> > 5000baseT/Full > >> >> > >> > Advertised pause frame use: Symmetric Receive-only > >> >> > >> > Advertised auto-negotiation: Yes > >> >> > >> > Advertised FEC modes: Not reported > >> >> > >> > Link partner advertised link modes: 100baseT/Half 100baseT/Full > >> >> > >> > 1000baseT/Half 1000baseT/Full > >> >> > >> > 10000baseT/Full > >> >> > >> > 2500baseT/Full > >> >> > >> > 5000baseT/Full > >> >> > >> > Link partner advertised pause frame use: No > >> >> > >> > Link partner advertised auto-negotiation: Yes > >> >> > >> > Link partner advertised FEC modes: Not reported > >> >> > >> > Speed: 10000Mb/s > >> >> > >> > Duplex: Full > >> >> > >> > Port: Twisted Pair > >> >> > >> > PHYAD: 8 > >> >> > >> > Transceiver: external > >> >> > >> > Auto-negotiation: on > >> >> > >> > MDI-X: Unknown > >> >> > >> > Link detected: yes > >> >> > >> > # ping 192.168.1.146 -c5 > >> >> > >> > PING 192.168.1.146 (192.168.1.146): 56 data bytes > >> >> > >> > 64 bytes from 192.168.1.146: seq=0 ttl=64 time=0.991 ms > >> >> > >> > 64 bytes from 192.168.1.146: seq=1 ttl=64 time=0.267 ms > >> >> > >> > 64 bytes from 192.168.1.146: seq=2 ttl=64 time=0.271 ms > >> >> > >> > 64 bytes from 192.168.1.146: seq=3 ttl=64 time=0.280 ms > >> >> > >> > 64 bytes from 192.168.1.146: seq=4 ttl=64 time=0.271 ms > >> >> > >> > > >> >> > >> > --- 192.168.1.146 ping statistics --- > >> >> > >> > 5 packets transmitted, 5 packets received, 0% packet loss > >> >> > >> > round-trip min/avg/max = 0.267/0.416/0.991 ms > >> >> > >> > # # force switch port to 1G > >> >> > >> > [ 193.343494] mvpp2 f2000000.ethernet eth0: phy link down > >> >> > >> > 10gbase-r/Unknown/Unknown/none/off > >> >> > >> > [ 193.343539] mvpp2 f2000000.ethernet eth0: mac link down > >> >> > >> > [ 193.344524] mvpp2 f2000000.ethernet eth0: Link is Down > >> >> > >> > [ 193.351973] aqr107_link_change_notify state=5:nolink an=1 link=0 > >> >> > >> > duplex=255 speed=-1 26:10gbase-r > >> >> > >> > [ 197.472489] mvpp2 f2000000.ethernet eth0: phy link up /1Gbps/Full/pause/off > >> >> > >> > >> >> > >> Well, it looks like we have selected PHY_INTERFACE_MODE_NA. Can you send > >> >> > >> the value of MDIO_PHYXS_VEND_IF_STATUS (dev 4, reg 0xe812)? Please also > >> >> > >> send the global config registers (dev 0x1e, reg 0x0310 through 0x031f) > >> >> > >> and the vendor provisioning registers (dev 4, reg 0xc440 through > >> >> > >> 0xc449). > >> >> > > > >> >> > > yes, this is what I've been looking at as well. When forced to 1000m > >> >> > > the register shows a phy type of 11 which according to the aqr113 > >> >> > > datasheet is XFI 5G: > >> >> > > aqr107_read_status STATUS=0x00001258 (type=11) state=4:running an=1 > >> >> > > link=1 duplex=1 speed=1000 interface=0 > >> >> > > >> >> > That's pretty strange. Seems like it's rate adapting from 5g instead of > >> >> > 10g. Is SERDES Mode in the Global System Configuration For 1G register > >> >> > set to XFI? > >> >> > >> >> 1E.31C=0x0106: > >> >> Rate Adaptation Method: 2=Pause Rate Adaptation > >> >> SERDES Mode: 6=XFI/2 (XFI 5G) > >> >> > >> > > >> > The SERDES mode here is not valid and it seems to always be set to > >> > XFI/2 unless I init/use the AQR113 in U-Boot. If I manually set SERDES > >> > Mode to 0 (XFI) in the driver I find that all rates > >> > 10g,5g,2.5g,1g,100m work fine both in Linux 6.0 and in Linux 6.1-rc5. > >> > I'm still trying to understand why I would need to set SERDES Mode > >> > manually (vs the XFI mode specific firmware setting this) but I am > >> > unclear what the rate adaptation in 6.1 provides in this case. Is it > >> > that perhaps the AQR113 is handling rate adaptation internally and > >> > that's why the non 10gbe rates are working on 6.0? > >> > >> The changes in 6.1 are > >> > >> - We now always enable pause frame reception when doing rate adaptation. > >> This is necessary for rate adaptation to work, but wasn't done > >> automatically before. > >> - We advertise lower speed modes which are enabled with rate adaptation, > >> even if we would not otherwise be able to support them. > >> > >> I'm not sure why you'd have XFI/2 selected in 6.1 if it isn't selected > >> in 6.0. > > > > Sean, > > > > Thanks for the explanation. The issue I had which resulted in the > > wrong SERDES mode was simply that I was using the wrong aquantia > > firmware. They customize the firmware to default registers like SERDES > > mode specifically for customers and I was unknowingly using the > > firmware for XFI/2 instead of XFI. > > > > I suppose it would be worth putting something in the aquantia driver > > that verifies SERDES mode matches the phy-mode from dt to throw an > > error. > > > > Best Regards, > > > > Tim > > Can you test the following series to see if it fixes your problem: > > https://lore.kernel.org/netdev/20221128195409.100873-1-sean.anderson@seco.com/ > > --Sean Sean, I believe the discussion is still ongoing regarding this being the correct approach but that series does in fact resolve the issue I had where my firmware was provisioned for something not compatible with the SERDES link I needed. Best Regards, Tim ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 21:54 ` Sean Anderson 2022-11-11 22:14 ` Tim Harvey @ 2022-11-11 22:33 ` Russell King (Oracle) 2022-11-12 13:15 ` Russell King (Oracle) 2 siblings, 0 replies; 21+ messages in thread From: Russell King (Oracle) @ 2022-11-11 22:33 UTC (permalink / raw) To: Sean Anderson; +Cc: Tim Harvey, netdev, David S. Miller On Fri, Nov 11, 2022 at 04:54:40PM -0500, Sean Anderson wrote: > I wonder if you could enable USXGMII? Seems like mvpp2 with comphy > should support it. I'm not sure if the aquantia driver is set up for it. mvpp2 doesn't support USXGMII. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-11 21:54 ` Sean Anderson 2022-11-11 22:14 ` Tim Harvey 2022-11-11 22:33 ` Russell King (Oracle) @ 2022-11-12 13:15 ` Russell King (Oracle) 2022-11-14 15:33 ` Sean Anderson 2 siblings, 1 reply; 21+ messages in thread From: Russell King (Oracle) @ 2022-11-12 13:15 UTC (permalink / raw) To: Sean Anderson; +Cc: Tim Harvey, netdev, David S. Miller On Fri, Nov 11, 2022 at 04:54:40PM -0500, Sean Anderson wrote: > > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > > supported 00000000,00018000,000e706f advertising > > 00000000,00018000,000e706f > > # ethtool eth0 > > Settings for eth0: > > Supported ports: [ ] > > Supported link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 10/100 half duplex aren't achievable with rate matching (and we avoid > turning them on), so they must be coming from somewhere else. I wonder > if this is because PHY_INTERFACE_MODE_SGMII is set in > supported_interfaces. The reason is due to the way phylink_bringup_phy() works. This is being called with interface = 10GBASE-R, and the PHY is a C45 PHY, which means we call phy_get_rate_matching() with PHY_INTERFACE_MODE_NA as we don't know whether the PHY will be switching its interface or not. Looking at the Aquanta PHY driver, this will return that pause mode rate matching will be used, so config.rate_matching will be RATE_MATCH_PAUSE. phylink_validate() will be called for PHY_INTERFACE_MODE_NA, which causes it to scan all supported interface modes (as again, we don't know which will be used by the PHY [*]) and the union of those results will be used. So when we e.g. try SGMII mode, caps & mac_capabilities will allow the half duplex modes through. Now for the bit marked with [*] - at this point, if rate matching is will be used, we in fact know which interface mode is going to be in operation, and it isn't going to change. So maybe we need this instead in phylink_bringup_phy(): - if (phy->is_c45 && + config.rate_matching = phy_get_rate_matching(phy, interface); + if (phy->is_c45 && config.rate_matching == RATE_MATCH_NONE && interface != PHY_INTERFACE_MODE_RXAUI && interface != PHY_INTERFACE_MODE_XAUI && interface != PHY_INTERFACE_MODE_USXGMII) config.interface = PHY_INTERFACE_MODE_NA; else config.interface = interface; - config.rate_matching = phy_get_rate_matching(phy, config.interface); ret = phylink_validate(pl, supported, &config); ? -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-12 13:15 ` Russell King (Oracle) @ 2022-11-14 15:33 ` Sean Anderson 2022-11-14 16:13 ` Russell King (Oracle) 0 siblings, 1 reply; 21+ messages in thread From: Sean Anderson @ 2022-11-14 15:33 UTC (permalink / raw) To: Russell King (Oracle); +Cc: Tim Harvey, netdev, David S. Miller On 11/12/22 08:15, Russell King (Oracle) wrote: > On Fri, Nov 11, 2022 at 04:54:40PM -0500, Sean Anderson wrote: >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting >> > supported 00000000,00018000,000e706f advertising >> > 00000000,00018000,000e706f > >> > # ethtool eth0 >> > Settings for eth0: >> > Supported ports: [ ] >> > Supported link modes: 10baseT/Half 10baseT/Full >> > 100baseT/Half 100baseT/Full >> >> 10/100 half duplex aren't achievable with rate matching (and we avoid >> turning them on), so they must be coming from somewhere else. I wonder >> if this is because PHY_INTERFACE_MODE_SGMII is set in >> supported_interfaces. > > The reason is due to the way phylink_bringup_phy() works. This is > being called with interface = 10GBASE-R, and the PHY is a C45 PHY, > which means we call phy_get_rate_matching() with > PHY_INTERFACE_MODE_NA as we don't know whether the PHY will be > switching its interface or not. > > Looking at the Aquanta PHY driver, this will return that pause mode > rate matching will be used, so config.rate_matching will be > RATE_MATCH_PAUSE. > > phylink_validate() will be called for PHY_INTERFACE_MODE_NA, which > causes it to scan all supported interface modes (as again, we don't > know which will be used by the PHY [*]) and the union of those > results will be used. > > So when we e.g. try SGMII mode, caps & mac_capabilities will allow > the half duplex modes through. > > Now for the bit marked with [*] - at this point, if rate matching is > will be used, we in fact know which interface mode is going to be in > operation, and it isn't going to change. So maybe we need this instead > in phylink_bringup_phy(): > > - if (phy->is_c45 && > + config.rate_matching = phy_get_rate_matching(phy, interface); > + if (phy->is_c45 && config.rate_matching == RATE_MATCH_NONE && > interface != PHY_INTERFACE_MODE_RXAUI && > interface != PHY_INTERFACE_MODE_XAUI && > interface != PHY_INTERFACE_MODE_USXGMII) > config.interface = PHY_INTERFACE_MODE_NA; > else > config.interface = interface; > - config.rate_matching = phy_get_rate_matching(phy, config.interface); > > ret = phylink_validate(pl, supported, &config); > > ? Yeah, that sounds reasonable. Actually, this was the logic I was thinking of when I asked Tim to try USXGMII earlier. The funny thing is that the comment above this implies that the link mode is never actually (R)XAUI or USXGMII. On another subject, if setting the SERDES mode field above fixes the issue, then the Aquantia driver should be modified to set that field to use a supported interface. Will host_interfaces work for this? It seems to be set only when there's an SFP module. That said, imagine if Tim was using a MAC without pause support, but which supported SGMII and 10GBASE-R. Currently, we would just advertise 10G modes. But 1G could be supported by switching the phy interface. I don't think this is supported right now. But if it were, we would need a way to tell the phy to use a lower phy interface mode, instead of rate adapting. We might also need a way to let the board tell us what interfaces are supported (like [1]). --Sean [1] https://lore.kernel.org/netdev/20211117225050.18395-1-kabel@kernel.org/ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-14 15:33 ` Sean Anderson @ 2022-11-14 16:13 ` Russell King (Oracle) 2022-11-18 18:16 ` Sean Anderson 0 siblings, 1 reply; 21+ messages in thread From: Russell King (Oracle) @ 2022-11-14 16:13 UTC (permalink / raw) To: Sean Anderson; +Cc: Tim Harvey, netdev, David S. Miller On Mon, Nov 14, 2022 at 10:33:52AM -0500, Sean Anderson wrote: > On 11/12/22 08:15, Russell King (Oracle) wrote: > > On Fri, Nov 11, 2022 at 04:54:40PM -0500, Sean Anderson wrote: > >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY > >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) > >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting > >> > supported 00000000,00018000,000e706f advertising > >> > 00000000,00018000,000e706f > > > >> > # ethtool eth0 > >> > Settings for eth0: > >> > Supported ports: [ ] > >> > Supported link modes: 10baseT/Half 10baseT/Full > >> > 100baseT/Half 100baseT/Full > >> > >> 10/100 half duplex aren't achievable with rate matching (and we avoid > >> turning them on), so they must be coming from somewhere else. I wonder > >> if this is because PHY_INTERFACE_MODE_SGMII is set in > >> supported_interfaces. > > > > The reason is due to the way phylink_bringup_phy() works. This is > > being called with interface = 10GBASE-R, and the PHY is a C45 PHY, > > which means we call phy_get_rate_matching() with > > PHY_INTERFACE_MODE_NA as we don't know whether the PHY will be > > switching its interface or not. > > > > Looking at the Aquanta PHY driver, this will return that pause mode > > rate matching will be used, so config.rate_matching will be > > RATE_MATCH_PAUSE. > > > > phylink_validate() will be called for PHY_INTERFACE_MODE_NA, which > > causes it to scan all supported interface modes (as again, we don't > > know which will be used by the PHY [*]) and the union of those > > results will be used. > > > > So when we e.g. try SGMII mode, caps & mac_capabilities will allow > > the half duplex modes through. > > > > Now for the bit marked with [*] - at this point, if rate matching is > > will be used, we in fact know which interface mode is going to be in > > operation, and it isn't going to change. So maybe we need this instead > > in phylink_bringup_phy(): > > > > - if (phy->is_c45 && > > + config.rate_matching = phy_get_rate_matching(phy, interface); > > + if (phy->is_c45 && config.rate_matching == RATE_MATCH_NONE && > > interface != PHY_INTERFACE_MODE_RXAUI && > > interface != PHY_INTERFACE_MODE_XAUI && > > interface != PHY_INTERFACE_MODE_USXGMII) > > config.interface = PHY_INTERFACE_MODE_NA; > > else > > config.interface = interface; > > - config.rate_matching = phy_get_rate_matching(phy, config.interface); > > > > ret = phylink_validate(pl, supported, &config); > > > > ? > > Yeah, that sounds reasonable. Actually, this was the logic I was > thinking of when I asked Tim to try USXGMII earlier. The funny thing is > that the comment above this implies that the link mode is never actually > (R)XAUI or USXGMII. I think you're misunderstanding the comment... If a clause 45 PHY is using USXGMII, then it is highly likely that the PHY will not switch between different interface modes depending on the media side negotiation. If a clause 45 PHY is using RXAUI or XAUI, then I believe according to the information available to me at the time, that there is no possibility of different interface modes being used. If any other interface type is specified (e.g. 10GBASE-R etc) then there is the possibility that the PHY will be switching between different interface modes, and we have no idea what so ever at this point what modes the PHY will be making use of - so the best we can do is to validate _all_ possible modes. This is what is done by setting the interface mode to _NA. Obviously, if we are using rate matching with a particular interface mode (e.g. 10GBASE-R) then we know that we are only going to be using 10GBASE-R, so we can validate just the single interface mode. > On another subject, if setting the SERDES mode field above fixes the > issue, then the Aquantia driver should be modified to set that field to > use a supported interface. Will host_interfaces work for this? It seems > to be set only when there's an SFP module. The reason I didn't push host_interfaces upstream myself is that I was unconvinced that it was the proper approach - and I still have my reservations with it. This can only tell the PHY driver what the MAC driver supports, and it means the PHY driver is then free to do its own choosing of what group of interface modes it wants to use. However, think about what I've said above about phylink not having any clue about what interface modes the PHY is going to be using - having the PHY driver decide on its own which group of interface modes should be used adds even more complexity in a completely different chunk of code, one where driver authors are free to make whatever decisions they deem (and we know that wildly different solutions will happen.) I had been toying with the idea of doing this differently, and had dropped most of the host_interfaces approach from my git tree, instead having PHYs provide a bitmap of the interface modes they support and having them initialise in their config_init function which interface modes they're going to be making use of given their resulting configuration. I never properly finished this though. > That said, imagine if Tim was using a MAC without pause support, but > which supported SGMII and 10GBASE-R. Currently, we would just advertise > 10G modes. But 1G could be supported by switching the phy interface. Note that we already have boards that make use of interface switching. Macchiatobin has switched between 10GBASE-R, 5GBASE-R, 2500BASE-X and SGMII depending on the negotiated media speed. In fact, that switching is rather enforced by the 3310 PHY firmware. We could force 10GBASE-R and enable rate matching, but then we get into the problems that the 3310 on these boards does not have MACSEC therefore can't send pause frames to the host MAC (and the host MAC doesn't support pause frames - eek) and we have not come up with an implementation for extending the IPG, although I believe mvpp2 hardware is capable of it. However, there's also the BCM84881 PHY which does the same dynamic switching which we can't prevent (we don't know how to!) -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: status of rate adaptation 2022-11-14 16:13 ` Russell King (Oracle) @ 2022-11-18 18:16 ` Sean Anderson 0 siblings, 0 replies; 21+ messages in thread From: Sean Anderson @ 2022-11-18 18:16 UTC (permalink / raw) To: Russell King (Oracle); +Cc: Tim Harvey, netdev, David S. Miller On 11/14/22 11:13, Russell King (Oracle) wrote: > On Mon, Nov 14, 2022 at 10:33:52AM -0500, Sean Anderson wrote: >> On 11/12/22 08:15, Russell King (Oracle) wrote: >> > On Fri, Nov 11, 2022 at 04:54:40PM -0500, Sean Anderson wrote: >> >> > [ 8.911932] mvpp2 f2000000.ethernet eth0: PHY >> >> > [f212a600.mdio-mii:08] driver [Aquantia AQR113C] (irq=POLL) >> >> > [ 8.921577] mvpp2 f2000000.ethernet eth0: phy: 10gbase-r setting >> >> > supported 00000000,00018000,000e706f advertising >> >> > 00000000,00018000,000e706f >> > >> >> > # ethtool eth0 >> >> > Settings for eth0: >> >> > Supported ports: [ ] >> >> > Supported link modes: 10baseT/Half 10baseT/Full >> >> > 100baseT/Half 100baseT/Full >> >> >> >> 10/100 half duplex aren't achievable with rate matching (and we avoid >> >> turning them on), so they must be coming from somewhere else. I wonder >> >> if this is because PHY_INTERFACE_MODE_SGMII is set in >> >> supported_interfaces. >> > >> > The reason is due to the way phylink_bringup_phy() works. This is >> > being called with interface = 10GBASE-R, and the PHY is a C45 PHY, >> > which means we call phy_get_rate_matching() with >> > PHY_INTERFACE_MODE_NA as we don't know whether the PHY will be >> > switching its interface or not. >> > >> > Looking at the Aquanta PHY driver, this will return that pause mode >> > rate matching will be used, so config.rate_matching will be >> > RATE_MATCH_PAUSE. >> > >> > phylink_validate() will be called for PHY_INTERFACE_MODE_NA, which >> > causes it to scan all supported interface modes (as again, we don't >> > know which will be used by the PHY [*]) and the union of those >> > results will be used. >> > >> > So when we e.g. try SGMII mode, caps & mac_capabilities will allow >> > the half duplex modes through. >> > >> > Now for the bit marked with [*] - at this point, if rate matching is >> > will be used, we in fact know which interface mode is going to be in >> > operation, and it isn't going to change. So maybe we need this instead >> > in phylink_bringup_phy(): >> > >> > - if (phy->is_c45 && >> > + config.rate_matching = phy_get_rate_matching(phy, interface); >> > + if (phy->is_c45 && config.rate_matching == RATE_MATCH_NONE && >> > interface != PHY_INTERFACE_MODE_RXAUI && >> > interface != PHY_INTERFACE_MODE_XAUI && >> > interface != PHY_INTERFACE_MODE_USXGMII) >> > config.interface = PHY_INTERFACE_MODE_NA; >> > else >> > config.interface = interface; >> > - config.rate_matching = phy_get_rate_matching(phy, config.interface); >> > >> > ret = phylink_validate(pl, supported, &config); >> > >> > ? >> >> Yeah, that sounds reasonable. Actually, this was the logic I was >> thinking of when I asked Tim to try USXGMII earlier. The funny thing is >> that the comment above this implies that the link mode is never actually >> (R)XAUI or USXGMII. > > I think you're misunderstanding the comment... > > If a clause 45 PHY is using USXGMII, then it is highly likely that the > PHY will not switch between different interface modes depending on the > media side negotiation. > > If a clause 45 PHY is using RXAUI or XAUI, then I believe according to > the information available to me at the time, that there is no > possibility of different interface modes being used. > > If any other interface type is specified (e.g. 10GBASE-R etc) then there > is the possibility that the PHY will be switching between different > interface modes, and we have no idea what so ever at this point what > modes the PHY will be making use of - so the best we can do is to > validate _all_ possible modes. This is what is done by setting the > interface mode to _NA. > > Obviously, if we are using rate matching with a particular interface > mode (e.g. 10GBASE-R) then we know that we are only going to be using > 10GBASE-R, so we can validate just the single interface mode. Ah, you're right, I was reading this backwards. >> On another subject, if setting the SERDES mode field above fixes the >> issue, then the Aquantia driver should be modified to set that field to >> use a supported interface. Will host_interfaces work for this? It seems >> to be set only when there's an SFP module. > > The reason I didn't push host_interfaces upstream myself is that I was > unconvinced that it was the proper approach - and I still have my > reservations with it. This can only tell the PHY driver what the MAC > driver supports, and it means the PHY driver is then free to do its > own choosing of what group of interface modes it wants to use. Well, this is what we have already. The Aquantia phys are initialized by the firmware to use particular interface for a particular link speed. Rate adaptation may or may not be involved. If it picks an interface the MAC doesn't support, you're SOL (until you get a new firmware). Ideally, we'd be able to configure the phy to always select an interface the MAC supports. > However, think about what I've said above about phylink not having any > clue about what interface modes the PHY is going to be using - having > the PHY driver decide on its own which group of interface modes should > be used adds even more complexity in a completely different chunk of > code, one where driver authors are free to make whatever decisions > they deem (and we know that wildly different solutions will happen.) > > I had been toying with the idea of doing this differently, and had > dropped most of the host_interfaces approach from my git tree, instead > having PHYs provide a bitmap of the interface modes they support and > having them initialise in their config_init function which interface > modes they're going to be making use of given their resulting > configuration. I never properly finished this though. That sounds like a good start. >> That said, imagine if Tim was using a MAC without pause support, but >> which supported SGMII and 10GBASE-R. Currently, we would just advertise >> 10G modes. But 1G could be supported by switching the phy interface. > > Note that we already have boards that make use of interface switching. > Macchiatobin has switched between 10GBASE-R, 5GBASE-R, 2500BASE-X and > SGMII depending on the negotiated media speed. In fact, that switching > is rather enforced by the 3310 PHY firmware. > > We could force 10GBASE-R and enable rate matching, but then we get > into the problems that the 3310 on these boards does not have MACSEC > therefore can't send pause frames to the host MAC (and the host MAC > doesn't support pause frames - eek) and we have not come up with an > implementation for extending the IPG, although I believe mvpp2 > hardware is capable of it. The DPAA hardware is as well. As I understand it, the 10GBASE-W standard specifies linear scaling of the IPG with the packet size, not A simple implementation could be to have MACs expose something like mac_rate_match_ipg_numerator, mac_rate_match_ipg_denominator, With the intention that they'd do something like numerator = SPEED_10000, denominator = 9500, and then we could multiply the speed to adjust. We could also do something like int mac_minimum_speed(int base_speed); But AIUI we are trying to move away from these sorts of things. FWIW I don't have any 10GBASE-W hardware. --Sean > However, there's also the BCM84881 PHY which does the same dynamic > switching which we can't prevent (we don't know how to!) ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2022-12-01 1:11 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-11-11 20:44 status of rate adaptation Tim Harvey 2022-11-11 20:57 ` Sean Anderson 2022-11-11 20:58 ` Sean Anderson 2022-11-11 21:20 ` Tim Harvey 2022-11-11 21:54 ` Sean Anderson 2022-11-11 22:14 ` Tim Harvey 2022-11-11 22:38 ` Sean Anderson 2022-11-12 0:48 ` Vladimir Oltean 2022-11-12 16:08 ` Andrew Lunn 2022-11-14 15:08 ` Sean Anderson 2022-11-14 19:33 ` Tim Harvey 2022-11-16 22:37 ` Tim Harvey 2022-11-17 15:38 ` Sean Anderson 2022-11-17 23:42 ` Tim Harvey 2022-11-28 19:57 ` Sean Anderson 2022-12-01 1:11 ` Tim Harvey 2022-11-11 22:33 ` Russell King (Oracle) 2022-11-12 13:15 ` Russell King (Oracle) 2022-11-14 15:33 ` Sean Anderson 2022-11-14 16:13 ` Russell King (Oracle) 2022-11-18 18:16 ` Sean Anderson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).