* RE: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
@ 2018-09-26 19:29 Chris Preimesberger
2018-09-26 19:44 ` Andrew Lunn
2018-09-26 21:34 ` Neil Horman
0 siblings, 2 replies; 16+ messages in thread
From: Chris Preimesberger @ 2018-09-26 19:29 UTC (permalink / raw)
To: linville@tuxdriver.com, netdev@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 24434 bytes --]
Hello,
I'm re-sending in plain text per the auto-reply from a spam filter. I have attached some text files this time, which explain the situation below, in case the below email's font & formatting is now too messed up for easy comprehension.
Thank you and best regards.
Chris Preimesberger | Test & Validation Engineer
Transition Networks, Inc.
chrisp@transition.com
direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
________________________________________
From: Chris Preimesberger
Sent: Wednesday, September 26, 2018 2:14 PM
To: 'linville@tuxdriver.com'; 'netdev@vger.kernel.org'
Subject: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
Hello John, All,
I think I may have found a bug or two in ethtool, with respect to its reporting of a QSFP28 transceiver's diagnostic information. Ethtool seems to correctly report all diagnostic information about QSFP28 transceivers, except for the transceiver's warning and alarm thresholds. I'm not sure whether the spurious warning and alarm values that get reported are the fault of ethtool or my NIC/driver, and I have no other models of 100GbE NICs to test with. I've contacted Mellanox support about this, and they point the finger at ethtool. Can these issues be investigated by ethtool developers? Here is some background information about the equipment and software used when I observe these issues:
Equipment used:
NIC: Mellanox ConnectX-4 100GbE, part number MCX415A-CCAT
Transceiver: Any 40Gb or 100Gb QSFP28 transceiver installed in the NIC (Intel, Mellanox, Transition Networks, etc..)
Software used:
Ubuntu 18.04 with the distro's packaged NIC driver and ethtool v4.15
also tested were ethtool v4.18 compiled from source and the current Mellanox OFED driver.
All test scenarios produced the same bugs.
Bug #1. Ethtool's reporting of the installed transceiver's alarm and warning thresholds will differ, depending on whether or not ethtool is piped to another command. Example commands are below, with their respective differing output values highlighted:
tech1@D8:~$ sudo ethtool -m enp1s0
Identifier : 0x11 (QSFP28)
Extended identifier : 0xfc
Extended identifier description : 3.5W max. Power consumption
Extended identifier description : CDR present in TX, CDR present in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
Encoding : 0x03 (NRZ)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 2km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 47.500nm
Vendor name : TRANSITION
Vendor OUI : 00:c0:f2
Vendor PN : TNQSFP100GCWDM4
Vendor rev : 1A
Vendor SN : TN02000302
Date code : 180919
Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
Module temperature : 39.53 degrees C / 103.15 degrees F
Module voltage : 3.3241 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 34.432 mA
Laser tx bias current (Channel 2) : 34.432 mA
Laser tx bias current (Channel 3) : 33.408 mA
Laser tx bias current (Channel 4) : 33.920 mA
Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 0.000 mA
Laser bias current low alarm threshold : 0.000 mA
Laser bias current high warning threshold : 0.000 mA
Laser bias current low warning threshold : 0.000 mA
Laser output power high alarm threshold : 0.0000 mW / -inf dBm
Laser output power low alarm threshold : 0.0000 mW / -inf dBm
Laser output power high warning threshold : 0.0000 mW / -inf dBm
Laser output power low warning threshold : 0.0000 mW / -inf dBm
Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
Module voltage high alarm threshold : 0.0000 V
Module voltage low alarm threshold : 0.0000 V
Module voltage high warning threshold : 0.0000 V
Module voltage low warning threshold : 0.0000 V
Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
Laser rx power high warning threshold : 0.0000 mW / -inf dBm
Laser rx power low warning threshold : 0.0000 mW / -inf dBm
tech1@D8:~$ sudo ethtool -m enp1s0 | cat
Identifier : 0x11 (QSFP28)
Extended identifier : 0xfc
Extended identifier description : 3.5W max. Power consumption
Extended identifier description : CDR present in TX, CDR present in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
Encoding : 0x03 (NRZ)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 2km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 47.500nm
Vendor name : TRANSITION
Vendor OUI : 00:c0:f2
Vendor PN : TNQSFP100GCWDM4
Vendor rev : 1A
Vendor SN : TN02000302
Date code : 180919
Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
Module temperature : 39.53 degrees C / 103.15 degrees F
Module voltage : 3.3249 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 34.432 mA
Laser tx bias current (Channel 2) : 34.432 mA
Laser tx bias current (Channel 3) : 33.408 mA
Laser tx bias current (Channel 4) : 33.920 mA
Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 16.448 mA
Laser bias current low alarm threshold : 16.448 mA
Laser bias current high warning threshold : 16.448 mA
Laser bias current low warning threshold : 16.448 mA
Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
Module voltage high alarm threshold : 2.9728 V
Module voltage low alarm threshold : 2.6990 V
Module voltage high warning threshold : 0.8274 V
Module voltage low warning threshold : 2.2538 V
Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
Bug # 2. All of the alarm and warning threshold values reported in the above commands are spurious.
At first glance, one would assume that the threshold values reported by the piped ethtool command are correct, but they're not. I know the programmed values for the above transceiver, so that makes it easy for me to spot the spurious values, but even without knowing the programmed values of a given transceiver, one can use logic to detect when the ethtool displayed values don't make sense.
For example, lets scrutinize the values for voltage warnings and alarms reported by ethtool on this transceiver. We will look at each voltage threshold, and scrutinize that value relative to the other voltage thresholds, and look for contradictions to determine whether the reported values seem legit.
Known ethtool
Actual Reported
Values Values
High Voltage Alarm 3.70V 2.9728 V
High Voltage Warning 3.59V 0.8274 V
(Operating spec = 3.30V)
Low Voltage Warning 3.00V 2.2538 V
Low Voltage Alarm 2.90V 2.6990 V
Contradictions for the ethtool reported voltage warning and alarm thresholds:
1. The high voltage alarm should occur at higher voltage than the operating voltage, but ethtool didn't report that.
2. The high voltage warning should occur at higher voltage than the low voltage warning and alarm, but ethtool didn't report that.
3. The low voltage warning should occur at higher voltage than the low voltage alarm, but ethtool didn't report that.
4. The low voltage alarm should occur at a lower voltage than any of the other voltage warnings and alarms, but ethtool didn't report that.
5. The current voltage value was reported as 3.3249V, which should trigger high voltage warning and alarm, according to the reported thresholds, but no warnings or alarms are indicated.
Each of the 4 voltage thresholds reported by ethtool have contradictions, so we know something is not right. This same kind of logic can be applied to the thresholds for temperature, laser TX power, etc.. to find that those values are also spurious.
Installing the above transceiver in a Cisco switch reveals that the Cisco correctly retrieves the true warning and alarm threshold values from the transceiver's EEPROM, so we trust that the transceiver has been correctly programmed. Cisco CLI output for that transceiver shown here:
switch# show interface ethernet 1/3 transceiver details
Ethernet1/3
transceiver is present
type is QSFP-100G-CWDM4-MSA-FEC
name is TRANSITION
part number is TNQSFP100GCWDM4
revision is 1A
serial number is TN02000302
nominal bitrate is 25500 MBit/sec per channel
Link length supported for 9/125um fiber is 2 km
cisco id is 17
cisco extended id number is 252
Lane Number:1 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:2 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:3 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:4 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
switch#
Any help with these issues is greatly appreciated. If you have any questions or advice, please let me know. I'll be glad to continue troubleshooting this until it's resolved. Thank you.
Chris Preimesberger | Test & Validation Engineer
Transition Networks, Inc.
chrisp@transition.com
direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
________________________________________
[-- Attachment #2: ethtoolQSFP28thresholdsCiscoComparison.txt --]
[-- Type: text/plain, Size: 4602 bytes --]
For comparison to ethtool's output that shows incorrect threshold values, when installing the same transceiver in a Cisco Nexus switch, and issuing the Cisco command "show interface ethernet 1/3 transceiver details", the switch correctly correctly reads/displays the transceiver's Alarm and Warning thresholds, as shown below:
switch# show interface ethernet 1/3 transceiver details
Ethernet1/3
transceiver is present
type is QSFP-100G-CWDM4-MSA-FEC
name is TRANSITION
part number is TNQSFP100GCWDM4
revision is 1A
serial number is TN02000302
nominal bitrate is 25500 MBit/sec per channel
Link length supported for 9/125um fiber is 2 km
cisco id is 17
cisco extended id number is 252
Lane Number:1 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:2 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:3 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
Lane Number:4 Network Lane
SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
switch#
[-- Attachment #3: ethtoolQSFP28thresholdsExpectedOutput.txt --]
[-- Type: text/plain, Size: 7258 bytes --]
Look at each line in the ethtool output below that includes the word "threshold". This file has been hand-edited to show the threshold values that have been programmed into the transceiver, which should be displayed by ethtool. The threshold values shown below are copied and pasted from the output of the Cisco NX-OS command "show interface ethernet 1/3 transceiver details", while the transceiver was installed in a Cisco Nexus switch.
Note - I only copied the threshold values in the units that were displayed by the Cisco switch. The "?" symbols are just a placeholder for the converted values; I was too lazy to do conversions between dBm and mW, or between degrees C and degrees F. Ethtool would be expected to report the true / converted values.
tech1@D8:~$ sudo ethtool -m enp1s0
Identifier : 0x11 (QSFP28)
Extended identifier : 0xfc
Extended identifier description : 3.5W max. Power consumption
Extended identifier description : CDR present in TX, CDR present in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
Encoding : 0x03 (NRZ)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 2km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 47.500nm
Vendor name : TRANSITION
Vendor OUI : 00:c0:f2
Vendor PN : TNQSFP100GCWDM4
Vendor rev : 1A
Vendor SN : TN02000302
Date code : 180919
Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
Module temperature : 39.53 degrees C / 103.15 degrees F
Module voltage : 3.3233 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 34.432 mA
Laser tx bias current (Channel 2) : 34.432 mA
Laser tx bias current (Channel 3) : 33.408 mA
Laser tx bias current (Channel 4) : 33.920 mA
Transmit avg optical power (Channel 1) : 0.9052 mW / -0.43 dBm
Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
Rcvr signal avg optical power(Channel 4) : 0.6948 mW / -1.58 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 75.000 mA
Laser bias current low alarm threshold : 10.000 mA
Laser bias current high warning threshold : 70.000 mA
Laser bias current low warning threshold : 15.000 mA
Laser output power high alarm threshold : ? mW / 4.49 dBm
Laser output power low alarm threshold : ? mW / -8.50 dBm
Laser output power high warning threshold : ? mW / 3.49 dBm
Laser output power low warning threshold : ? mW / -7.52 dBm
Module temperature high alarm threshold : 80.00 degrees C / ? degrees F
Module temperature low alarm threshold : -10.00 degrees C / ? degrees F
Module temperature high warning threshold : 75.00 degrees C / ? degrees F
Module temperature low warning threshold : -5.00 degrees C / ? degrees F
Module voltage high alarm threshold : 3.7000 V
Module voltage low alarm threshold : 2.9000 V
Module voltage high warning threshold : 3.5900 V
Module voltage low warning threshold : 3.0000 V
Laser rx power high alarm threshold : ? mW / 4.49 dBm
Laser rx power low alarm threshold : ? mW / -14.55 dBm
Laser rx power high warning threshold : ? mW / 3.49 dBm
Laser rx power low warning threshold : ? mW / -12.51 dBm
[-- Attachment #4: ethtoolQSFP28thresholdsSpuriousOutput1of2.txt --]
[-- Type: text/plain, Size: 6843 bytes --]
Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the output is not piped to another command. Notice that all of the displayed threshold values are 0 (which is incorrect), while other values report as expected.
tech1@D8:~$ sudo ethtool -m enp1s0
Identifier : 0x11 (QSFP28)
Extended identifier : 0xfc
Extended identifier description : 3.5W max. Power consumption
Extended identifier description : CDR present in TX, CDR present in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
Encoding : 0x03 (NRZ)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 2km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 47.500nm
Vendor name : TRANSITION
Vendor OUI : 00:c0:f2
Vendor PN : TNQSFP100GCWDM4
Vendor rev : 1A
Vendor SN : TN02000302
Date code : 180919
Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
Module temperature : 39.53 degrees C / 103.15 degrees F
Module voltage : 3.3241 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 34.432 mA
Laser tx bias current (Channel 2) : 34.432 mA
Laser tx bias current (Channel 3) : 33.408 mA
Laser tx bias current (Channel 4) : 33.920 mA
Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 0.000 mA
Laser bias current low alarm threshold : 0.000 mA
Laser bias current high warning threshold : 0.000 mA
Laser bias current low warning threshold : 0.000 mA
Laser output power high alarm threshold : 0.0000 mW / -inf dBm
Laser output power low alarm threshold : 0.0000 mW / -inf dBm
Laser output power high warning threshold : 0.0000 mW / -inf dBm
Laser output power low warning threshold : 0.0000 mW / -inf dBm
Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
Module voltage high alarm threshold : 0.0000 V
Module voltage low alarm threshold : 0.0000 V
Module voltage high warning threshold : 0.0000 V
Module voltage low warning threshold : 0.0000 V
Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
Laser rx power high warning threshold : 0.0000 mW / -inf dBm
Laser rx power low warning threshold : 0.0000 mW / -inf dBm
[-- Attachment #5: ethtoolQSFP28thresholdsSpuriousOutput2of2.txt --]
[-- Type: text/plain, Size: 6866 bytes --]
Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the ethtool output is piped to another command. Notice that all of the displayed threshold values are spurious while other values report as expected.
tech1@D8:~$ sudo ethtool -m enp1s0 | cat
Identifier : 0x11 (QSFP28)
Extended identifier : 0xfc
Extended identifier description : 3.5W max. Power consumption
Extended identifier description : CDR present in TX, CDR present in RX
Extended identifier description : High Power Class (> 3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
Encoding : 0x03 (NRZ)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 2km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1310.000nm
Laser wavelength tolerance : 47.500nm
Vendor name : TRANSITION
Vendor OUI : 00:c0:f2
Vendor PN : TNQSFP100GCWDM4
Vendor rev : 1A
Vendor SN : TN02000302
Date code : 180919
Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
Module temperature : 39.53 degrees C / 103.15 degrees F
Module voltage : 3.3249 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 34.432 mA
Laser tx bias current (Channel 2) : 34.432 mA
Laser tx bias current (Channel 3) : 33.408 mA
Laser tx bias current (Channel 4) : 33.920 mA
Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 16.448 mA
Laser bias current low alarm threshold : 16.448 mA
Laser bias current high warning threshold : 16.448 mA
Laser bias current low warning threshold : 16.448 mA
Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
Module voltage high alarm threshold : 2.9728 V
Module voltage low alarm threshold : 2.6990 V
Module voltage high warning threshold : 0.8274 V
Module voltage low warning threshold : 2.2538 V
Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
tech1@D8:~$
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 19:29 bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers Chris Preimesberger
@ 2018-09-26 19:44 ` Andrew Lunn
2018-09-26 20:47 ` Chris Preimesberger
2018-09-26 21:34 ` Neil Horman
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Lunn @ 2018-09-26 19:44 UTC (permalink / raw)
To: Chris Preimesberger; +Cc: linville@tuxdriver.com, netdev@vger.kernel.org
On Wed, Sep 26, 2018 at 07:29:23PM +0000, Chris Preimesberger wrote:
> Hello,
>
> I'm re-sending in plain text per the auto-reply from a spam filter.
Yep. no html obfustication accepted here. Please ASCII only please :-)
Please can you also wrap your lines at about 75 characters.
> I have attached some text files this time, which explain the situation below, in case the below email's font & formatting is now too messed up for easy comprehension.
> Bug #1. Ethtool's reporting of the installed transceiver's alarm and warning thresholds will differ, depending on whether or not ethtool is piped to another command. Example commands are below, with their respective differing output values highlighted:
Could you dump the raw values. That will make it easier for us to
reproduce this issue, assuming it is ethtool, and not the kernel
driver.
Thanks
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 19:44 ` Andrew Lunn
@ 2018-09-26 20:47 ` Chris Preimesberger
2018-09-26 21:46 ` Andrew Lunn
0 siblings, 1 reply; 16+ messages in thread
From: Chris Preimesberger @ 2018-09-26 20:47 UTC (permalink / raw)
To: Andrew Lunn; +Cc: linville@tuxdriver.com, netdev@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 3140 bytes --]
Hello Andrew,
Thank you for the quick response!!
Apologies in advance for my use of outlook and top-posting, etc...
I've run the raw option and the hex option, and pasted the results below.
Since the raw option printed strange characters on the CLI, I re-ran it,
Sending the output to a file (raw.txt) and attached that file as well.
Pasted from Ubuntu CLI:
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$ sudo ethtool -m enp1s0 raw on
\x11UU$��pA`?�@�G\x10#
�\x12v\x01\x11��\x03�\x02@TRANSITION ��TNQSFP100GCWDM4 1AfX%\x1cF?\x06?�TN02000301 180919
h�\x02I��_��'\x16��Ri=\x02`��Zntech1@D7:~$
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$ sudo ethtool -m enp1s0 hex on
Offset Values
------ ------
0x0000: 11 00 00 0f 00 00 00 00 00 55 55 00 00 00 00 00
0x0010: 00 00 00 00 00 00 24 e2 00 00 81 68 00 00 00 00
0x0020: 00 00 00 00 00 00 00 00 00 00 41 60 3f e0 40 e0
0x0030: 47 00 1f 10 0e 1e 0b f7 12 76 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
0x0060: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 11 fc 07 80 00 00 00 00 00 00 00 03 ff 00 02 00
0x0090: 00 00 00 40 54 52 41 4e 53 49 54 49 4f 4e 20 20
0x00a0: 20 20 20 20 00 00 c0 f2 54 4e 51 53 46 50 31 30
0x00b0: 30 47 43 57 44 4d 34 20 31 41 66 58 25 1c 46 3f
0x00c0: 06 00 3f d6 54 4e 30 32 30 30 30 33 30 31 20 20
0x00d0: 20 20 20 20 31 38 30 39 31 39 20 20 0c 00 68 f3
0x00e0: 00 00 02 49 80 a0 5f 1f de c9 27 16 f8 ae 52 69
0x00f0: 3d 02 60 00 00 00 00 00 00 00 00 00 83 f4 5a 6e
tech1@D7:~$
tech1@D7:~$
Chris Preimesberger | Test & Validation Engineer
Transition Networks, Inc.
chrisp@transition.com
direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
-----Original Message-----
From: Andrew Lunn [mailto:andrew@lunn.ch]
Sent: Wednesday, September 26, 2018 2:45 PM
To: Chris Preimesberger
Cc: linville@tuxdriver.com; netdev@vger.kernel.org
Subject: Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
On Wed, Sep 26, 2018 at 07:29:23PM +0000, Chris Preimesberger wrote:
> Hello,
>
> I'm re-sending in plain text per the auto-reply from a spam filter.
Yep. no html obfustication accepted here. Please ASCII only please :-)
Please can you also wrap your lines at about 75 characters.
> I have attached some text files this time, which explain the situation below, in case the below email's font & formatting is now too messed up for easy comprehension.
> Bug #1. Ethtool's reporting of the installed transceiver's alarm and warning thresholds will differ, depending on whether or not ethtool is piped to another command. Example commands are below, with their respective differing output values highlighted:
Could you dump the raw values. That will make it easier for us to
reproduce this issue, assuming it is ethtool, and not the kernel
driver.
Thanks
Andrew
[-- Attachment #2: raw.txt --]
[-- Type: text/plain, Size: 256 bytes --]
\x11\0\0\x0f\0\0\0\0\0UU\0\0\0\0\0\0\0\0\0\0\0$ò\0\0h\0\0\0\0\0\0\0\0\0\0\0\0\0\0A`?à@àG\0\x1f\x10\x0e\x1e\v÷\x12{\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\x1f\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x11ü\a€\0\0\0\0\0\0\0\x03ÿ\0\x02\0\0\0\0@TRANSITION \0\0ÀòTNQSFP100GCWDM4 1AfX%\x1cF?\x06\0?ÖTN02000301 180919 \f\0hó\0\0\x02I€ _\x1fÞÉ'\x16ø®Ri=\x02`\0\0\0\0\0\0\0\0\0ƒôZn
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 19:29 bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers Chris Preimesberger
2018-09-26 19:44 ` Andrew Lunn
@ 2018-09-26 21:34 ` Neil Horman
2018-09-26 21:58 ` Andrew Lunn
2018-09-27 13:25 ` Eran Ben Elisha
1 sibling, 2 replies; 16+ messages in thread
From: Neil Horman @ 2018-09-26 21:34 UTC (permalink / raw)
To: Chris Preimesberger; +Cc: linville@tuxdriver.com, netdev@vger.kernel.org
On Wed, Sep 26, 2018 at 07:29:23PM +0000, Chris Preimesberger wrote:
> Hello,
>
> I'm re-sending in plain text per the auto-reply from a spam filter. I have attached some text files this time, which explain the situation below, in case the below email's font & formatting is now too messed up for easy comprehension.
>
> Thank you and best regards.
>
>
> Chris Preimesberger | Test & Validation Engineer
> Transition Networks, Inc.
>
> chrisp@transition.com
> direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
> ________________________________________
>
>
>
This is just a drive by guess, but I think this is a driver issue.
Issue 1 seems like a red herring, cat doesn't modify output, nor does ethtool
know if its output is going to a console or a pipe, its all the same. And given
issue 2 (that the output of the thresholds, etc are spurriously changing and
wrong), suggests that they are spurriously changing and wrong regardless of what
cat does.
That said, I think issue two is a problem with the mlx4 driver. Specifically
that the driver is copying garbage data.
The three ethtool functions at work here are:
mlx4_en_get_module_info
mlx4_en_get_module_eeprom
mlx4_get_module_info
When you run ethtool -m on this driver, the kernel calls mlx4_en_get_module_info
to determine the length of the eeprom, and that value will be either 256 or 512
bytes. Lets assume that the value is 256 for the sake of argument
Next it calls mlx4_en_get_module_eeprom, passing in that size 256 to actually
read the eeprom data, which in turn calls mlx4_get_module_info to fetch the data
from hardware, again, passing in 256 as the size for the first call (theres a
loop, but it will only get executed once in this scenario)
mlx4_get_module_info then issues the appropriate mailbox commands to dump the
eeprom. Here it starts to go sideways. The mailbox buffer allocated for the
return data is of type mlx4_mad_ifc, which has some front matter information and
a data buffer that is 192 bytes long!
A little further down in the function, size gets restricted if the buffer
crosses a page boundary, but given that the size is 256 on the first call here,
and offset is zero on the first call, we're not crossing anything, so size
remains unchanged.
The output mailbox buffer outmad->data (a 192 byte array), then gets cast to a
sturt mlx4_cable_info structure, which has its own internal data buffer that is
only 48 bytes long.
The memcpy in this functionthen copies cable_info->data to the buffer that gets
returned to ethtool, but it copies size bytes (256), even though the source data
buffer is only 48 bytes long. That 48 byte array is embedded in the larger 192
byte structure, so there won't be a panic on the overrun, but theres no telling
what garbage is in the buffer beyond those first 48 bytes. Even if the
remaining 144 bytes have valid eeprom data, its less than the required 256
bytes. The additional copy may cause a panic, but if the buffer commonly bumps
up against other allocated memory, that will go unnoticed.
after the memcpy, mlx4_get_module_info just returns the size of the passed in
buffer (256), and so the calling function thinks its work is done, and lets the
kernel send back the buffer with garbage data to ethtool.
I think the mlx4 guys have some work to do here.
My $0.02
Neil
>
>
> From: Chris Preimesberger
> Sent: Wednesday, September 26, 2018 2:14 PM
> To: 'linville@tuxdriver.com'; 'netdev@vger.kernel.org'
> Subject: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
>
> Hello John, All,
>
>
> I think I may have found a bug or two in ethtool, with respect to its reporting of a QSFP28 transceiver's diagnostic information. Ethtool seems to correctly report all diagnostic information about QSFP28 transceivers, except for the transceiver's warning and alarm thresholds. I'm not sure whether the spurious warning and alarm values that get reported are the fault of ethtool or my NIC/driver, and I have no other models of 100GbE NICs to test with. I've contacted Mellanox support about this, and they point the finger at ethtool. Can these issues be investigated by ethtool developers? Here is some background information about the equipment and software used when I observe these issues:
>
> Equipment used:
> NIC: Mellanox ConnectX-4 100GbE, part number MCX415A-CCAT
> Transceiver: Any 40Gb or 100Gb QSFP28 transceiver installed in the NIC (Intel, Mellanox, Transition Networks, etc..)
>
> Software used:
> Ubuntu 18.04 with the distro's packaged NIC driver and ethtool v4.15
> also tested were ethtool v4.18 compiled from source and the current Mellanox OFED driver.
>
> All test scenarios produced the same bugs.
>
>
> Bug #1. Ethtool's reporting of the installed transceiver's alarm and warning thresholds will differ, depending on whether or not ethtool is piped to another command. Example commands are below, with their respective differing output values highlighted:
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3241 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 0.000 mA
> Laser bias current low alarm threshold : 0.000 mA
> Laser bias current high warning threshold : 0.000 mA
> Laser bias current low warning threshold : 0.000 mA
> Laser output power high alarm threshold : 0.0000 mW / -inf dBm
> Laser output power low alarm threshold : 0.0000 mW / -inf dBm
> Laser output power high warning threshold : 0.0000 mW / -inf dBm
> Laser output power low warning threshold : 0.0000 mW / -inf dBm
> Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
> Module voltage high alarm threshold : 0.0000 V
> Module voltage low alarm threshold : 0.0000 V
> Module voltage high warning threshold : 0.0000 V
> Module voltage low warning threshold : 0.0000 V
> Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power high warning threshold : 0.0000 mW / -inf dBm
> Laser rx power low warning threshold : 0.0000 mW / -inf dBm
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0 | cat
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3249 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 16.448 mA
> Laser bias current low alarm threshold : 16.448 mA
> Laser bias current high warning threshold : 16.448 mA
> Laser bias current low warning threshold : 16.448 mA
> Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
> Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
> Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
> Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
> Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
> Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
> Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
> Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
> Module voltage high alarm threshold : 2.9728 V
> Module voltage low alarm threshold : 2.6990 V
> Module voltage high warning threshold : 0.8274 V
> Module voltage low warning threshold : 2.2538 V
> Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
> Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
> Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
> Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
>
>
> Bug # 2. All of the alarm and warning threshold values reported in the above commands are spurious.
> At first glance, one would assume that the threshold values reported by the piped ethtool command are correct, but they're not. I know the programmed values for the above transceiver, so that makes it easy for me to spot the spurious values, but even without knowing the programmed values of a given transceiver, one can use logic to detect when the ethtool displayed values don't make sense.
> For example, lets scrutinize the values for voltage warnings and alarms reported by ethtool on this transceiver. We will look at each voltage threshold, and scrutinize that value relative to the other voltage thresholds, and look for contradictions to determine whether the reported values seem legit.
> Known ethtool
> Actual Reported
> Values Values
> High Voltage Alarm 3.70V 2.9728 V
> High Voltage Warning 3.59V 0.8274 V
> (Operating spec = 3.30V)
> Low Voltage Warning 3.00V 2.2538 V
> Low Voltage Alarm 2.90V 2.6990 V
>
> Contradictions for the ethtool reported voltage warning and alarm thresholds:
> 1. The high voltage alarm should occur at higher voltage than the operating voltage, but ethtool didn't report that.
> 2. The high voltage warning should occur at higher voltage than the low voltage warning and alarm, but ethtool didn't report that.
> 3. The low voltage warning should occur at higher voltage than the low voltage alarm, but ethtool didn't report that.
> 4. The low voltage alarm should occur at a lower voltage than any of the other voltage warnings and alarms, but ethtool didn't report that.
> 5. The current voltage value was reported as 3.3249V, which should trigger high voltage warning and alarm, according to the reported thresholds, but no warnings or alarms are indicated.
>
> Each of the 4 voltage thresholds reported by ethtool have contradictions, so we know something is not right. This same kind of logic can be applied to the thresholds for temperature, laser TX power, etc.. to find that those values are also spurious.
>
>
> Installing the above transceiver in a Cisco switch reveals that the Cisco correctly retrieves the true warning and alarm threshold values from the transceiver's EEPROM, so we trust that the transceiver has been correctly programmed. Cisco CLI output for that transceiver shown here:
>
> switch# show interface ethernet 1/3 transceiver details
> Ethernet1/3
> transceiver is present
> type is QSFP-100G-CWDM4-MSA-FEC
> name is TRANSITION
> part number is TNQSFP100GCWDM4
> revision is 1A
> serial number is TN02000302
> nominal bitrate is 25500 MBit/sec per channel
> Link length supported for 9/125um fiber is 2 km
> cisco id is 17
> cisco extended id number is 252
>
> Lane Number:1 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:2 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:3 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:4 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> switch#
>
>
> Any help with these issues is greatly appreciated. If you have any questions or advice, please let me know. I'll be glad to continue troubleshooting this until it's resolved. Thank you.
>
>
> Chris Preimesberger | Test & Validation Engineer
> Transition Networks, Inc.
>
> chrisp@transition.com
> direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
> ________________________________________
>
>
>
>
>
>
>
> For comparison to ethtool's output that shows incorrect threshold values, when installing the same transceiver in a Cisco Nexus switch, and issuing the Cisco command "show interface ethernet 1/3 transceiver details", the switch correctly correctly reads/displays the transceiver's Alarm and Warning thresholds, as shown below:
>
>
> switch# show interface ethernet 1/3 transceiver details
> Ethernet1/3
> transceiver is present
> type is QSFP-100G-CWDM4-MSA-FEC
> name is TRANSITION
> part number is TNQSFP100GCWDM4
> revision is 1A
> serial number is TN02000302
> nominal bitrate is 25500 MBit/sec per channel
> Link length supported for 9/125um fiber is 2 km
> cisco id is 17
> cisco extended id number is 252
>
> Lane Number:1 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.44 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:2 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 34.24 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.20 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:3 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.21 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -0.96 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> Lane Number:4 Network Lane
> SFP Detail Diagnostics Information (internal calibration)
> ----------------------------------------------------------------------------
> Current Alarms Warnings
> Measurement High Low High Low
> ----------------------------------------------------------------------------
> Temperature 38.08 C 80.00 C -10.00 C 75.00 C -5.00 C
> Voltage 3.34 V 3.70 V 2.90 V 3.59 V 3.00 V
> Current 33.72 mA 75.00 mA 10.00 mA 70.00 mA 15.00 mA
> Tx Power -1.59 dBm 4.49 dBm -8.50 dBm 3.49 dBm -7.52 dBm
> Rx Power N/A 4.49 dBm -14.55 dBm 3.49 dBm -12.51 dBm
> Transmit Fault Count = 0
> ----------------------------------------------------------------------------
> Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
>
> switch#
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file has been hand-edited to show the threshold values that have been programmed into the transceiver, which should be displayed by ethtool. The threshold values shown below are copied and pasted from the output of the Cisco NX-OS command "show interface ethernet 1/3 transceiver details", while the transceiver was installed in a Cisco Nexus switch.
>
> Note - I only copied the threshold values in the units that were displayed by the Cisco switch. The "?" symbols are just a placeholder for the converted values; I was too lazy to do conversions between dBm and mW, or between degrees C and degrees F. Ethtool would be expected to report the true / converted values.
>
>
>
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3233 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9052 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6948 mW / -1.58 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 75.000 mA
> Laser bias current low alarm threshold : 10.000 mA
> Laser bias current high warning threshold : 70.000 mA
> Laser bias current low warning threshold : 15.000 mA
> Laser output power high alarm threshold : ? mW / 4.49 dBm
> Laser output power low alarm threshold : ? mW / -8.50 dBm
> Laser output power high warning threshold : ? mW / 3.49 dBm
> Laser output power low warning threshold : ? mW / -7.52 dBm
> Module temperature high alarm threshold : 80.00 degrees C / ? degrees F
> Module temperature low alarm threshold : -10.00 degrees C / ? degrees F
> Module temperature high warning threshold : 75.00 degrees C / ? degrees F
> Module temperature low warning threshold : -5.00 degrees C / ? degrees F
> Module voltage high alarm threshold : 3.7000 V
> Module voltage low alarm threshold : 2.9000 V
> Module voltage high warning threshold : 3.5900 V
> Module voltage low warning threshold : 3.0000 V
> Laser rx power high alarm threshold : ? mW / 4.49 dBm
> Laser rx power low alarm threshold : ? mW / -14.55 dBm
> Laser rx power high warning threshold : ? mW / 3.49 dBm
> Laser rx power low warning threshold : ? mW / -12.51 dBm
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the output is not piped to another command. Notice that all of the displayed threshold values are 0 (which is incorrect), while other values report as expected.
>
> tech1@D8:~$ sudo ethtool -m enp1s0
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3241 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9048 mW / -0.43 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7014 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 0.000 mA
> Laser bias current low alarm threshold : 0.000 mA
> Laser bias current high warning threshold : 0.000 mA
> Laser bias current low warning threshold : 0.000 mA
> Laser output power high alarm threshold : 0.0000 mW / -inf dBm
> Laser output power low alarm threshold : 0.0000 mW / -inf dBm
> Laser output power high warning threshold : 0.0000 mW / -inf dBm
> Laser output power low warning threshold : 0.0000 mW / -inf dBm
> Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
> Module voltage high alarm threshold : 0.0000 V
> Module voltage low alarm threshold : 0.0000 V
> Module voltage high warning threshold : 0.0000 V
> Module voltage low warning threshold : 0.0000 V
> Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
> Laser rx power high warning threshold : 0.0000 mW / -inf dBm
> Laser rx power low warning threshold : 0.0000 mW / -inf dBm
>
>
>
> Look at each line in the ethtool output below that includes the word "threshold". This file shows the actual output from ethtool v4.18, when the ethtool output is piped to another command. Notice that all of the displayed threshold values are spurious while other values report as expected.
>
> tech1@D8:~$ sudo ethtool -m enp1s0 | cat
> Identifier : 0x11 (QSFP28)
> Extended identifier : 0xfc
> Extended identifier description : 3.5W max. Power consumption
> Extended identifier description : CDR present in TX, CDR present in RX
> Extended identifier description : High Power Class (> 3.5 W) not enabled
> Connector : 0x07 (LC)
> Transceiver codes : 0x80 0x00 0x00 0x00 0x00 0x00 0x00 0x00
> Transceiver type : 100G Ethernet: 100G CWDM4 MSA with FEC
> Encoding : 0x03 (NRZ)
> BR, Nominal : 25500Mbps
> Rate identifier : 0x00
> Length (SMF,km) : 2km
> Length (OM3 50um) : 0m
> Length (OM2 50um) : 0m
> Length (OM1 62.5um) : 0m
> Length (Copper or Active cable) : 0m
> Transmitter technology : 0x40 (1310 nm DFB)
> Laser wavelength : 1310.000nm
> Laser wavelength tolerance : 47.500nm
> Vendor name : TRANSITION
> Vendor OUI : 00:c0:f2
> Vendor PN : TNQSFP100GCWDM4
> Vendor rev : 1A
> Vendor SN : TN02000302
> Date code : 180919
> Revision Compliance : SFF-8636 Rev 2.5/2.6/2.7
> Module temperature : 39.53 degrees C / 103.15 degrees F
> Module voltage : 3.3249 V
> Alarm/warning flags implemented : Yes
> Laser tx bias current (Channel 1) : 34.432 mA
> Laser tx bias current (Channel 2) : 34.432 mA
> Laser tx bias current (Channel 3) : 33.408 mA
> Laser tx bias current (Channel 4) : 33.920 mA
> Transmit avg optical power (Channel 1) : 0.9043 mW / -0.44 dBm
> Transmit avg optical power (Channel 2) : 0.7832 mW / -1.06 dBm
> Transmit avg optical power (Channel 3) : 0.8057 mW / -0.94 dBm
> Transmit avg optical power (Channel 4) : 0.7009 mW / -1.54 dBm
> Rcvr signal avg optical power(Channel 1) : 0.7378 mW / -1.32 dBm
> Rcvr signal avg optical power(Channel 2) : 0.7553 mW / -1.22 dBm
> Rcvr signal avg optical power(Channel 3) : 0.6529 mW / -1.85 dBm
> Rcvr signal avg optical power(Channel 4) : 0.6847 mW / -1.64 dBm
> Laser bias current high alarm (Chan 1) : Off
> Laser bias current low alarm (Chan 1) : Off
> Laser bias current high warning (Chan 1) : Off
> Laser bias current low warning (Chan 1) : Off
> Laser bias current high alarm (Chan 2) : Off
> Laser bias current low alarm (Chan 2) : Off
> Laser bias current high warning (Chan 2) : Off
> Laser bias current low warning (Chan 2) : Off
> Laser bias current high alarm (Chan 3) : Off
> Laser bias current low alarm (Chan 3) : Off
> Laser bias current high warning (Chan 3) : Off
> Laser bias current low warning (Chan 3) : Off
> Laser bias current high alarm (Chan 4) : Off
> Laser bias current low alarm (Chan 4) : Off
> Laser bias current high warning (Chan 4) : Off
> Laser bias current low warning (Chan 4) : Off
> Module temperature high alarm : Off
> Module temperature low alarm : Off
> Module temperature high warning : Off
> Module temperature low warning : Off
> Module voltage high alarm : Off
> Module voltage low alarm : Off
> Module voltage high warning : Off
> Module voltage low warning : Off
> Laser tx power high alarm (Channel 1) : Off
> Laser tx power low alarm (Channel 1) : Off
> Laser tx power high warning (Channel 1) : Off
> Laser tx power low warning (Channel 1) : Off
> Laser tx power high alarm (Channel 2) : Off
> Laser tx power low alarm (Channel 2) : Off
> Laser tx power high warning (Channel 2) : Off
> Laser tx power low warning (Channel 2) : Off
> Laser tx power high alarm (Channel 3) : Off
> Laser tx power low alarm (Channel 3) : Off
> Laser tx power high warning (Channel 3) : Off
> Laser tx power low warning (Channel 3) : Off
> Laser tx power high alarm (Channel 4) : Off
> Laser tx power low alarm (Channel 4) : Off
> Laser tx power high warning (Channel 4) : Off
> Laser tx power low warning (Channel 4) : Off
> Laser rx power high alarm (Channel 1) : Off
> Laser rx power low alarm (Channel 1) : Off
> Laser rx power high warning (Channel 1) : Off
> Laser rx power low warning (Channel 1) : Off
> Laser rx power high alarm (Channel 2) : Off
> Laser rx power low alarm (Channel 2) : Off
> Laser rx power high warning (Channel 2) : Off
> Laser rx power low warning (Channel 2) : Off
> Laser rx power high alarm (Channel 3) : Off
> Laser rx power low alarm (Channel 3) : Off
> Laser rx power high warning (Channel 3) : Off
> Laser rx power low warning (Channel 3) : Off
> Laser rx power high alarm (Channel 4) : Off
> Laser rx power low alarm (Channel 4) : Off
> Laser rx power high warning (Channel 4) : Off
> Laser rx power low warning (Channel 4) : Off
> Laser bias current high alarm threshold : 16.448 mA
> Laser bias current low alarm threshold : 16.448 mA
> Laser bias current high warning threshold : 16.448 mA
> Laser bias current low warning threshold : 16.448 mA
> Laser output power high alarm threshold : 0.8224 mW / -0.85 dBm
> Laser output power low alarm threshold : 0.8250 mW / -0.84 dBm
> Laser output power high warning threshold : 0.8264 mW / -0.83 dBm
> Laser output power low warning threshold : 2.6983 mW / 4.31 dBm
> Module temperature high alarm threshold : 110.12 degrees C / 230.22 degrees F
> Module temperature low alarm threshold : 84.34 degrees C / 183.82 degrees F
> Module temperature high warning threshold : 44.12 degrees C / 111.42 degrees F
> Module temperature low warning threshold : 67.27 degrees C / 153.08 degrees F
> Module voltage high alarm threshold : 2.9728 V
> Module voltage low alarm threshold : 2.6990 V
> Module voltage high warning threshold : 0.8274 V
> Module voltage low warning threshold : 2.2538 V
> Laser rx power high alarm threshold : 2.5458 mW / 4.06 dBm
> Laser rx power low alarm threshold : 2.6992 mW / 4.31 dBm
> Laser rx power high warning threshold : 2.9801 mW / 4.74 dBm
> Laser rx power low warning threshold : 2.8526 mW / 4.55 dBm
> tech1@D8:~$
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 20:47 ` Chris Preimesberger
@ 2018-09-26 21:46 ` Andrew Lunn
0 siblings, 0 replies; 16+ messages in thread
From: Andrew Lunn @ 2018-09-26 21:46 UTC (permalink / raw)
To: Chris Preimesberger; +Cc: linville@tuxdriver.com, netdev@vger.kernel.org
On Wed, Sep 26, 2018 at 08:47:34PM +0000, Chris Preimesberger wrote:
> Hello Andrew,
>
> Thank you for the quick response!!
> Apologies in advance for my use of outlook and top-posting, etc...
>
> I've run the raw option and the hex option, and pasted the results below.
> Since the raw option printed strange characters on the CLI, I re-ran it,
> Sending the output to a file (raw.txt) and attached that file as well.
>
> Pasted from Ubuntu CLI:
>
> tech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$ sudo ethtool -m enp1s0 raw on
> \x11UU$��pA`?�@�G\x10#
> �\x12v\x01\x11��\x03�\x02@TRANSITION ��TNQSFP100GCWDM4 1AfX%\x1cF?\x06?�TN02000301 180919
> h�\x02I��_��'\x16��Ri=\x02`��Zntech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$
> tech1@D7:~$ sudo ethtool -m enp1s0 hex on
> Offset Values
> ------ ------
> 0x0000: 11 00 00 0f 00 00 00 00 00 55 55 00 00 00 00 00
> 0x0010: 00 00 00 00 00 00 24 e2 00 00 81 68 00 00 00 00
> 0x0020: 00 00 00 00 00 00 00 00 00 00 41 60 3f e0 40 e0
> 0x0030: 47 00 1f 10 0e 1e 0b f7 12 76 00 00 00 00 00 00
> 0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
> 0x0060: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 00 00
> 0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0080: 11 fc 07 80 00 00 00 00 00 00 00 03 ff 00 02 00
> 0x0090: 00 00 00 40 54 52 41 4e 53 49 54 49 4f 4e 20 20
> 0x00a0: 20 20 20 20 00 00 c0 f2 54 4e 51 53 46 50 31 30
> 0x00b0: 30 47 43 57 44 4d 34 20 31 41 66 58 25 1c 46 3f
> 0x00c0: 06 00 3f d6 54 4e 30 32 30 30 30 33 30 31 20 20
> 0x00d0: 20 20 20 20 31 38 30 39 31 39 20 20 0c 00 68 f3
> 0x00e0: 00 00 02 49 80 a0 5f 1f de c9 27 16 f8 ae 52 69
> 0x00f0: 3d 02 60 00 00 00 00 00 00 00 00 00 83 f4 5a 6e
Hi Chris
I've only recently got involved with SFP modules. ethtool says this is
a SFF-8636 SFP. So a QSFP. It has multiple pages, each 128 bytes in
length, which should be returned in a concatenated form. Here we see
256 bytes, meaning there are two pages. There can be up to 5 pages.
ethtool is looking for the temperature alarms at offset 0x200. So that
does not exist in this hex dump. But the raw dump you provided has
more bytes, 0x400 of them.
So i would say the first bug is that ethtool dumps different amounts
of data in hex than raw.
The fact you get different alarm thresholds on different runs suggests
to me we might only be getting two pages from the kernel?
Can you build ethtool from source and run it inside a debugger?
ethtool makes two IOCTL calls. The first is ETHTOOL_GMODULEINFO.
Could you print out the modinfo which is returned. It then does a
ETHTOOL_GMODULEEEPROM. Can you print out eeprom after the second
IOCTL.
Thanks
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 21:34 ` Neil Horman
@ 2018-09-26 21:58 ` Andrew Lunn
2018-09-27 13:23 ` Neil Horman
2018-09-27 13:25 ` Eran Ben Elisha
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Lunn @ 2018-09-26 21:58 UTC (permalink / raw)
To: Neil Horman
Cc: Chris Preimesberger, linville@tuxdriver.com,
netdev@vger.kernel.org
> When you run ethtool -m on this driver, the kernel calls mlx4_en_get_module_info
> to determine the length of the eeprom, and that value will be either 256 or 512
> bytes.
So it sounds like QSFP modules using 8636 are not supported. You would
expect a size to be one of 256, 384, 512 or 640.
> Next it calls mlx4_en_get_module_eeprom, passing in that size 256 to actually
> read the eeprom data, which in turn calls mlx4_get_module_info to fetch the data
> from hardware, again, passing in 256 as the size for the first call (theres a
> loop, but it will only get executed once in this scenario)
>
> mlx4_get_module_info then issues the appropriate mailbox commands to dump the
> eeprom. Here it starts to go sideways. The mailbox buffer allocated for the
> return data is of type mlx4_mad_ifc, which has some front matter information and
> a data buffer that is 192 bytes long!
Which suggests all SFP dumps are broken as well, not just QSFP.
Oh dear.
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 21:58 ` Andrew Lunn
@ 2018-09-27 13:23 ` Neil Horman
0 siblings, 0 replies; 16+ messages in thread
From: Neil Horman @ 2018-09-27 13:23 UTC (permalink / raw)
To: Andrew Lunn
Cc: Chris Preimesberger, linville@tuxdriver.com,
netdev@vger.kernel.org
On Wed, Sep 26, 2018 at 11:58:12PM +0200, Andrew Lunn wrote:
> > When you run ethtool -m on this driver, the kernel calls mlx4_en_get_module_info
> > to determine the length of the eeprom, and that value will be either 256 or 512
> > bytes.
>
> So it sounds like QSFP modules using 8636 are not supported. You would
> expect a size to be one of 256, 384, 512 or 640.
>
> > Next it calls mlx4_en_get_module_eeprom, passing in that size 256 to actually
> > read the eeprom data, which in turn calls mlx4_get_module_info to fetch the data
> > from hardware, again, passing in 256 as the size for the first call (theres a
> > loop, but it will only get executed once in this scenario)
> >
> > mlx4_get_module_info then issues the appropriate mailbox commands to dump the
> > eeprom. Here it starts to go sideways. The mailbox buffer allocated for the
> > return data is of type mlx4_mad_ifc, which has some front matter information and
> > a data buffer that is 192 bytes long!
>
> Which suggests all SFP dumps are broken as well, not just QSFP.
>
No, not at all. Each driver that implements a get_eeprom ethtool method, is
capable of doing multiple reads at various offsets, and filling up the user
buffer with real data. The bug here is that the mellanox data structures are
not sized properly vis a vis the amount of eeprom data that user space might
expect, or more specifically that the driver isn't smart enough to do several
small reads to fill up the full sized request buffer
Neil
> Oh dear.
>
> Andrew
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-26 21:34 ` Neil Horman
2018-09-26 21:58 ` Andrew Lunn
@ 2018-09-27 13:25 ` Eran Ben Elisha
2018-09-27 14:52 ` Andrew Lunn
1 sibling, 1 reply; 16+ messages in thread
From: Eran Ben Elisha @ 2018-09-27 13:25 UTC (permalink / raw)
To: Neil Horman, Chris Preimesberger
Cc: linville@tuxdriver.com, netdev@vger.kernel.org
> This is just a drive by guess, but I think this is a driver issue.
>
>
> Issue 1 seems like a red herring, cat doesn't modify output, nor does ethtool
> know if its output is going to a console or a pipe, its all the same. And given
> issue 2 (that the output of the thresholds, etc are spurriously changing and
> wrong), suggests that they are spurriously changing and wrong regardless of what
> cat does.
>
> That said, I think issue two is a problem with the mlx4 driver. Specifically
> that the driver is copying garbage data.
>
> The three ethtool functions at work here are:
> mlx4_en_get_module_info
> mlx4_en_get_module_eeprom
> mlx4_get_module_info
>
> When you run ethtool -m on this driver, the kernel calls mlx4_en_get_module_info
> to determine the length of the eeprom, and that value will be either 256 or 512
> bytes. Lets assume that the value is 256 for the sake of argument
>
> Next it calls mlx4_en_get_module_eeprom, passing in that size 256 to actually
> read the eeprom data, which in turn calls mlx4_get_module_info to fetch the data
> from hardware, again, passing in 256 as the size for the first call (theres a
> loop, but it will only get executed once in this scenario)
>
> mlx4_get_module_info then issues the appropriate mailbox commands to dump the
> eeprom. Here it starts to go sideways. The mailbox buffer allocated for the
> return data is of type mlx4_mad_ifc, which has some front matter information and
> a data buffer that is 192 bytes long!
>
> A little further down in the function, size gets restricted if the buffer
> crosses a page boundary, but given that the size is 256 on the first call here,
> and offset is zero on the first call, we're not crossing anything, so size
> remains unchanged.
>
> The output mailbox buffer outmad->data (a 192 byte array), then gets cast to a
> sturt mlx4_cable_info structure, which has its own internal data buffer that is
> only 48 bytes long.
Hi guys,
Thanks for digging into it.
Here are some observations I found:
1. Chris system has CX4 (which is served by mlx5 driver), all analysis
by Neil was done over mlx4 driver (which serves the older generation of
NICs. e.h CX3Pro).
2. In general, MAD commands are limited to 192 bytes of data.
3. CableInfo MAD command info is limited to 48 Bytes.
4. First check that mlx4_get_module_info is having:
if (size > MODULE_INFO_MAX_READ)
size = MODULE_INFO_MAX_READ;
So this is the info that were missing in the analysis. x <= 48 is
also returned by this function. No trash copy or overrun. It is
expected from the caller(also inside mlx4) to recall with new offset in
order to fetch more data.
5. I reviewed mlx5 driver, and it have reading mechanism (small diff:
via MCIA register and not via MAD)
Both drivers read up to 256 bytes. 0-127 (from page 0). and 128-256
(from page 0). Driver is not capable of reading over 256 bytes currently.
looking on qsfp.c parser in ethtool.c (user space), I see an
uninitialized bug issue that have caused bug #1 + #2.
Applied it locally solved the issue (Not showing alarm data, which
should be expected as driver do not fill it).
diff --git a/qsfp.c b/qsfp.c
index 32e195d12dc0..d196aa1753de 100644
--- a/qsfp.c
+++ b/qsfp.c
@@ -671,7 +671,7 @@ static void sff8636_dom_parse(const __u8 *id, struct
sff_diags *sd)
static void sff8636_show_dom(const __u8 *id, __u32 eeprom_len)
{
- struct sff_diags sd;
+ struct sff_diags sd = {0};
char *rx_power_string = NULL;
char power_string[MAX_DESC_SIZE];
int i;
I will soon post a fix for it.
Thanks,
Eran
Thanks,
Eran
>
> The memcpy in this functionthen copies cable_info->data to the buffer that gets
> returned to ethtool, but it copies size bytes (256), even though the source data
> buffer is only 48 bytes long. That 48 byte array is embedded in the larger 192
> byte structure, so there won't be a panic on the overrun, but theres no telling
> what garbage is in the buffer beyond those first 48 bytes. Even if the
> remaining 144 bytes have valid eeprom data, its less than the required 256
> bytes. The additional copy may cause a panic, but if the buffer commonly bumps
> up against other allocated memory, that will go unnoticed.
>
> after the memcpy, mlx4_get_module_info just returns the size of the passed in
> buffer (256), and so the calling function thinks its work is done, and lets the
> kernel send back the buffer with garbage data to ethtool.
>
> I think the mlx4 guys have some work to do here.
>
> My $0.02
> Neil
>
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 13:25 ` Eran Ben Elisha
@ 2018-09-27 14:52 ` Andrew Lunn
2018-09-27 15:20 ` Eran Ben Elisha
0 siblings, 1 reply; 16+ messages in thread
From: Andrew Lunn @ 2018-09-27 14:52 UTC (permalink / raw)
To: Eran Ben Elisha
Cc: Neil Horman, Chris Preimesberger, linville@tuxdriver.com,
netdev@vger.kernel.org
> Both drivers read up to 256 bytes. 0-127 (from page 0). and 128-256 (from
> page 0). Driver is not capable of reading over 256 bytes currently.
Hi Erin
There should not be any need to read more than 256 bytes. For older
SFP devices, two addresses on the i2c bus are used, each with 256
bytes. For QSFP, one address is used, and you swap page by writing to
offset 127.
> looking on qsfp.c parser in ethtool.c (user space), I see an uninitialized
> bug issue that have caused bug #1 + #2.
> Applied it locally solved the issue (Not showing alarm data, which should be
> expected as driver do not fill it).
There appears to be a second bug somewhere. dumping the module info
using HEX returned 256 bytes. But the binary dump had more bytes.
Since you have the hardware, could you look into this?
Thanks
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 14:52 ` Andrew Lunn
@ 2018-09-27 15:20 ` Eran Ben Elisha
2018-09-27 15:32 ` Andrew Lunn
0 siblings, 1 reply; 16+ messages in thread
From: Eran Ben Elisha @ 2018-09-27 15:20 UTC (permalink / raw)
To: Andrew Lunn
Cc: Neil Horman, Chris Preimesberger, linville@tuxdriver.com,
netdev@vger.kernel.org
On 9/27/2018 5:52 PM, Andrew Lunn wrote:
>> Both drivers read up to 256 bytes. 0-127 (from page 0). and 128-256 (from
>> page 0). Driver is not capable of reading over 256 bytes currently.
>
> Hi Erin
>
> There should not be any need to read more than 256 bytes. For older
> SFP devices, two addresses on the i2c bus are used, each with 256
> bytes. For QSFP, one address is used, and you swap page by writing to
> offset 127.
>
>> looking on qsfp.c parser in ethtool.c (user space), I see an uninitialized
>> bug issue that have caused bug #1 + #2.
>> Applied it locally solved the issue (Not showing alarm data, which should be
>> expected as driver do not fill it).
>
> There appears to be a second bug somewhere. dumping the module info
> using HEX returned 256 bytes. But the binary dump had more bytes.
> Since you have the hardware, could you look into this?
See fix I posted few minutes ago.
title: [PATCH ethtool] ethtool: Fix uninitialized variable use at qsfp dump
This is HEX dump, similar for both with/without the fix:
Offset Values
------ ------
0x0000: 11 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 00 2a 2a 00 00 7f 0b 00 00 00 00
0x0020: 00 00 38 b6 3e 50 2b e9 40 0d 47 0d 47 ac 48 58
0x0030: 49 0f 3a 09 36 77 39 c9 3a 6a 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 aa aa 00 00 00 00 01 00 00
0x0060: 00 00 ff 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0080: 11 cc 07 80 00 00 00 00 00 00 00 05 ff 00 0a 00
0x0090: 00 00 00 44 4d 65 6c 6c 61 6e 6f 78 20 20 20 20
0x00a0: 20 20 20 20 00 00 02 c9 4d 4d 41 31 4c 31 30 2d
0x00b0: 43 52 20 20 20 20 20 20 41 31 65 bf 00 ce 00 60
0x00c0: 03 07 ff de 4d 54 31 36 33 39 44 4d 30 30 30 32
0x00d0: 36 20 20 20 31 36 30 39 32 36 20 20 0c 10 68 40
0x00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x00f0: 00 00 00 00 00 00 00 00 00 00 00 00 14 31 00 00
This is parsed output before the fix:
Identifier : 0x11 (QSFP28)
Extended identifier : 0xcc
Extended identifier description : 3.5W max. Power
consumption
Extended identifier description : CDR present in TX,
CDR present in RX
Extended identifier description : High Power Class (>
3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00
0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G
Base-LR4
Encoding : 0x05 (64B/66B)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 10km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1302.350nm
Laser wavelength tolerance : 1.030nm
Vendor name : Mellanox
Vendor OUI : 00:02:c9
Vendor PN : MMA1L10-CR
Vendor rev : A1
Vendor SN : MT1639DM00026
Date code : 160926
Revision Compliance : SFF-8636 Rev
2.5/2.6/2.7
Module temperature : 42.16 degrees C /
107.90 degrees F
Module voltage : 3.2523 V
Alarm/warning flags implemented : Yes
Laser tx bias current (Channel 1) : 36.454 mA
Laser tx bias current (Channel 2) : 36.696 mA
Laser tx bias current (Channel 3) : 37.006 mA
Laser tx bias current (Channel 4) : 37.404 mA
Transmit avg optical power (Channel 1) : 1.4812 mW / 1.71 dBm
Transmit avg optical power (Channel 2) : 1.3942 mW / 1.44 dBm
Transmit avg optical power (Channel 3) : 1.4793 mW / 1.70 dBm
Transmit avg optical power (Channel 4) : 1.4949 mW / 1.75 dBm
Rcvr signal avg optical power(Channel 1) : 1.4489 mW / 1.61 dBm
Rcvr signal avg optical power(Channel 2) : 1.5911 mW / 2.02 dBm
Rcvr signal avg optical power(Channel 3) : 1.1196 mW / 0.49 dBm
Rcvr signal avg optical power(Channel 4) : 1.6397 mW / 2.15 dBm
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
Laser bias current high alarm threshold : 0.000 mA
Laser bias current low alarm threshold : 0.000 mA
Laser bias current high warning threshold : 0.000 mA
Laser bias current low warning threshold : 0.000 mA
Laser output power high alarm threshold : 0.0000 mW / -inf dBm
Laser output power low alarm threshold : 0.0000 mW / -inf dBm
Laser output power high warning threshold : 0.0000 mW / -inf dBm
Laser output power low warning threshold : 0.0000 mW / -inf dBm
Module temperature high alarm threshold : 0.00 degrees C /
32.00 degrees F
Module temperature low alarm threshold : 0.00 degrees C /
32.00 degrees F
Module temperature high warning threshold : 0.00 degrees C /
32.00 degrees F
Module temperature low warning threshold : 0.00 degrees C /
32.00 degrees F
Module voltage high alarm threshold : 0.0000 V
Module voltage low alarm threshold : 0.0000 V
Module voltage high warning threshold : 0.0000 V
Module voltage low warning threshold : 0.0000 V
Laser rx power high alarm threshold : 0.0000 mW / -inf dBm
Laser rx power low alarm threshold : 0.0000 mW / -inf dBm
Laser rx power high warning threshold : 0.0000 mW / -inf dBm
Laser rx power low warning threshold : 0.0000 mW / -inf dBm
This is parsed output after the fix:
Identifier : 0x11 (QSFP28)
Extended identifier : 0xcc
Extended identifier description : 3.5W max. Power
consumption
Extended identifier description : CDR present in TX,
CDR present in RX
Extended identifier description : High Power Class (>
3.5 W) not enabled
Connector : 0x07 (LC)
Transceiver codes : 0x80 0x00 0x00 0x00
0x00 0x00 0x00 0x00
Transceiver type : 100G Ethernet: 100G
Base-LR4
Encoding : 0x05 (64B/66B)
BR, Nominal : 25500Mbps
Rate identifier : 0x00
Length (SMF,km) : 10km
Length (OM3 50um) : 0m
Length (OM2 50um) : 0m
Length (OM1 62.5um) : 0m
Length (Copper or Active cable) : 0m
Transmitter technology : 0x40 (1310 nm DFB)
Laser wavelength : 1302.350nm
Laser wavelength tolerance : 1.030nm
Vendor name : Mellanox
Vendor OUI : 00:02:c9
Vendor PN : MMA1L10-CR
Vendor rev : A1
Vendor SN : MT1639DM00026
Date code : 160926
Revision Compliance : SFF-8636 Rev
2.5/2.6/2.7
Module temperature : 42.16 degrees C /
107.90 degrees F
Module voltage : 3.2523 V
Alarm/warning flags implemented : No
Laser tx bias current (Channel 1) : 36.462 mA
Laser tx bias current (Channel 2) : 36.668 mA
Laser tx bias current (Channel 3) : 37.000 mA
Laser tx bias current (Channel 4) : 37.416 mA
Transmit avg optical power (Channel 1) : 1.4812 mW / 1.71 dBm
Transmit avg optical power (Channel 2) : 1.3940 mW / 1.44 dBm
Transmit avg optical power (Channel 3) : 1.4829 mW / 1.71 dBm
Transmit avg optical power (Channel 4) : 1.4866 mW / 1.72 dBm
Rcvr signal avg optical power(Channel 1) : 1.4518 mW / 1.62 dBm
Rcvr signal avg optical power(Channel 2) : 1.5938 mW / 2.02 dBm
Rcvr signal avg optical power(Channel 3) : 1.1211 mW / 0.50 dBm
Rcvr signal avg optical power(Channel 4) : 1.6378 mW / 2.14 dBm
Major diff:
* Alarm/warning flags implemented : No
* All alarm data is not presented.
Driver return 256 bytes (reading it correctly, I verified it, no
overruns), however the extra bytes are presented due to this bug
(expecting to parse 640 bytes).
Do you see another bug here? Am I missing something?
Eran
>
> Thanks
> Andrew
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 15:20 ` Eran Ben Elisha
@ 2018-09-27 15:32 ` Andrew Lunn
2018-09-27 16:08 ` Chris Preimesberger
2018-10-02 7:10 ` Eran Ben Elisha
0 siblings, 2 replies; 16+ messages in thread
From: Andrew Lunn @ 2018-09-27 15:32 UTC (permalink / raw)
To: Eran Ben Elisha
Cc: Neil Horman, Chris Preimesberger, linville@tuxdriver.com,
netdev@vger.kernel.org
> Driver return 256 bytes (reading it correctly, I verified it, no overruns),
> however the extra bytes are presented due to this bug (expecting to parse
> 640 bytes).
>
> Do you see another bug here? Am I missing something?
Hi Erin
Please could you try ethtool -m raw on so you get a binary dump. The
file which Chris provided had more bytes in it than 256.
Thanks
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 15:32 ` Andrew Lunn
@ 2018-09-27 16:08 ` Chris Preimesberger
2018-09-27 16:38 ` Andrew Lunn
2018-10-02 7:10 ` Eran Ben Elisha
1 sibling, 1 reply; 16+ messages in thread
From: Chris Preimesberger @ 2018-09-27 16:08 UTC (permalink / raw)
To: Andrew Lunn, Eran Ben Elisha
Cc: Neil Horman, linville@tuxdriver.com, netdev@vger.kernel.org
Please correct me if I'm wrong, but...
It looks like Eran's proposed fix would remove all warning and
alarm indications from ethtool's output. It's worth mentioning
that for me, the following fields always reported correctly
as Off while no alarm condition was present
and On while alarm condition(s) were present
*per the QSFP's true/programmed threshold values*
*not per the incorrectly reported threshold values*
Laser bias current high alarm (Chan 1) : Off
Laser bias current low alarm (Chan 1) : Off
Laser bias current high warning (Chan 1) : Off
Laser bias current low warning (Chan 1) : Off
Laser bias current high alarm (Chan 2) : Off
Laser bias current low alarm (Chan 2) : Off
Laser bias current high warning (Chan 2) : Off
Laser bias current low warning (Chan 2) : Off
Laser bias current high alarm (Chan 3) : Off
Laser bias current low alarm (Chan 3) : Off
Laser bias current high warning (Chan 3) : Off
Laser bias current low warning (Chan 3) : Off
Laser bias current high alarm (Chan 4) : Off
Laser bias current low alarm (Chan 4) : Off
Laser bias current high warning (Chan 4) : Off
Laser bias current low warning (Chan 4) : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser tx power high alarm (Channel 1) : Off
Laser tx power low alarm (Channel 1) : Off
Laser tx power high warning (Channel 1) : Off
Laser tx power low warning (Channel 1) : Off
Laser tx power high alarm (Channel 2) : Off
Laser tx power low alarm (Channel 2) : Off
Laser tx power high warning (Channel 2) : Off
Laser tx power low warning (Channel 2) : Off
Laser tx power high alarm (Channel 3) : Off
Laser tx power low alarm (Channel 3) : Off
Laser tx power high warning (Channel 3) : Off
Laser tx power low warning (Channel 3) : Off
Laser tx power high alarm (Channel 4) : Off
Laser tx power low alarm (Channel 4) : Off
Laser tx power high warning (Channel 4) : Off
Laser tx power low warning (Channel 4) : Off
Laser rx power high alarm (Channel 1) : Off
Laser rx power low alarm (Channel 1) : Off
Laser rx power high warning (Channel 1) : Off
Laser rx power low warning (Channel 1) : Off
Laser rx power high alarm (Channel 2) : Off
Laser rx power low alarm (Channel 2) : Off
Laser rx power high warning (Channel 2) : Off
Laser rx power low warning (Channel 2) : Off
Laser rx power high alarm (Channel 3) : Off
Laser rx power low alarm (Channel 3) : Off
Laser rx power high warning (Channel 3) : Off
Laser rx power low warning (Channel 3) : Off
Laser rx power high alarm (Channel 4) : Off
Laser rx power low alarm (Channel 4) : Off
Laser rx power high warning (Channel 4) : Off
Laser rx power low warning (Channel 4) : Off
I would like to request that any fix keeps the above information
included in the ethtool -m output because it is working and valuable.
The only values that report incorrectly can be seen
by issuing the command:
ethtool -m interfaceXXX | grep threshold
Ideally, any fix would display the thresholds correctly
instead of omit them.
Thank you and best regards,
Chris
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 16:08 ` Chris Preimesberger
@ 2018-09-27 16:38 ` Andrew Lunn
2018-09-27 18:56 ` Chris Preimesberger
0 siblings, 1 reply; 16+ messages in thread
From: Andrew Lunn @ 2018-09-27 16:38 UTC (permalink / raw)
To: Chris Preimesberger
Cc: Eran Ben Elisha, Neil Horman, linville@tuxdriver.com,
netdev@vger.kernel.org
On Thu, Sep 27, 2018 at 04:08:24PM +0000, Chris Preimesberger wrote:
> Please correct me if I'm wrong, but...
> It looks like Eran's proposed fix would remove all warning and
> alarm indications from ethtool's output. It's worth mentioning
> that for me, the following fields always reported correctly
> as Off while no alarm condition was present
> and On while alarm condition(s) were present
> *per the QSFP's true/programmed threshold values*
> *not per the incorrectly reported threshold values*
These alarm values are in the first page. So the information the
driver returns does contain this information. What is missing is the
thresholds, which are not provided by the driver.
But there is a comment in the code:
/*
* There is no clear identifier to signify the existence of
* optical diagnostics similar to SFF-8472. So checking existence
* of page 3, will provide the gurantee for existence of alarms
* and thresholds
* If pagging support exists, then supports_alarms is marked as 1
*/
These alarm values are optional. The spec says so. So in order to
decide if they are implemented, ethtool looks to see if the thresholds
are available. If there are thresholds, it makes sense the alarms are
implemented.
Unfortunately, the driver never returns the thresholds. So ethtool has
no real choice and won't display the alarms since it cannot determine
if they are valid.
In order to get alarms, the driver needs to be extended to return all
the pages.
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 16:38 ` Andrew Lunn
@ 2018-09-27 18:56 ` Chris Preimesberger
2018-09-27 20:17 ` Chris Preimesberger
0 siblings, 1 reply; 16+ messages in thread
From: Chris Preimesberger @ 2018-09-27 18:56 UTC (permalink / raw)
To: 'Andrew Lunn', Eran Ben Elisha
Cc: Neil Horman, linville@tuxdriver.com, netdev@vger.kernel.org
I greatly appreciate everyone's work on this. Thank you to all.
I've had Mellanox support case # 00508027 open for this issue,
and just now requested an updated driver from them to resolve,
explaining that really smart ethtool developers figured out this
was due to the Mellanox driver not reporting thresholds to ethtool.
I intend to post back here for posterity if/when I get an updated
driver that fixes the issue.
Thanks again!!
Chris Preimesberger | Test & Validation Engineer
Transition Networks, Inc.
chrisp@transition.com
direct: +1.952.996.1509 | fax: +1.952.941.2322 | www.transition.com
-----Original Message-----
From: Andrew Lunn [mailto:andrew@lunn.ch]
Sent: Thursday, September 27, 2018 11:38 AM
To: Chris Preimesberger
Cc: Eran Ben Elisha; Neil Horman; linville@tuxdriver.com; netdev@vger.kernel.org
Subject: Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
On Thu, Sep 27, 2018 at 04:08:24PM +0000, Chris Preimesberger wrote:
> Please correct me if I'm wrong, but...
> It looks like Eran's proposed fix would remove all warning and alarm
> indications from ethtool's output. It's worth mentioning that for me,
> the following fields always reported correctly as Off while no alarm
> condition was present and On while alarm condition(s) were present
> *per the QSFP's true/programmed threshold values* *not per the
> incorrectly reported threshold values*
These alarm values are in the first page. So the information the driver returns does contain this information. What is missing is the thresholds, which are not provided by the driver.
But there is a comment in the code:
/*
* There is no clear identifier to signify the existence of
* optical diagnostics similar to SFF-8472. So checking existence
* of page 3, will provide the gurantee for existence of alarms
* and thresholds
* If pagging support exists, then supports_alarms is marked as 1
*/
These alarm values are optional. The spec says so. So in order to decide if they are implemented, ethtool looks to see if the thresholds are available. If there are thresholds, it makes sense the alarms are implemented.
Unfortunately, the driver never returns the thresholds. So ethtool has no real choice and won't display the alarms since it cannot determine if they are valid.
In order to get alarms, the driver needs to be extended to return all the pages.
Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 18:56 ` Chris Preimesberger
@ 2018-09-27 20:17 ` Chris Preimesberger
0 siblings, 0 replies; 16+ messages in thread
From: Chris Preimesberger @ 2018-09-27 20:17 UTC (permalink / raw)
To: 'Andrew Lunn', 'Eran Ben Elisha'
Cc: 'Neil Horman', 'linville@tuxdriver.com',
'netdev@vger.kernel.org'
Update for posterity-
Mellanox support provided a work-around of using mlxcables instead of
ethtool to read alarm/warning info for an installed transceiver.
I was told that a couple of their engineers are currently looking into the
discrepancy between threshold reporting by mlxcables and ethtool, and
that they are deciding what to do about it...
Work-around steps:
1. add a cable with "sudo mst cable add".
2. find the cable name with "sudo mlxcables". The name of my cable is
01:00.0_cable_0 so I copy that name for insertion into the next command.
3. probe the cable for DDM with "sudo mlxcables -d 01:00.0_cable_0 --DDM".
Example copied/pasted from my CLI here.
All reported thresholds appear to be correct.
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$ sudo mst cable add
-I- Added 1 cable devices ..
tech1@D7:~$ sudo mlxcables
Querying Cables ....
Cable #1:
---------
Cable name : 01:00.0_cable_0
>> No FW data to show
-------- Cable EEPROM --------
Identifier : QSFP28 (11h)
Technology : 850 nm VCSEL (00h)
Compliance : Extended Specification Compliance is valid, 100GBASE-SR4 or 25GBASE-SR
Wavelength : 850 nm
OUI : 0x00c0f2
Vendor : TRANSITION
Serial number : TN02000263
Part number : TN-QSFP-100G-SR4
Revision : 02
Temperature : 34 C
Length : 50 m
tech1@D7:~$ sudo mlxcables -d 01:00.0_cable_0 --DDM
Cable DDM:
----------
Temperature : 34C
Voltage : 3.2918V
Channel 1:
RX Power : 0.1695dBm
TX Power : 0.8622dBm
TX Bias : 7.0720mA
Channel 2:
RX Power : 0.1355dBm
TX Power : 1.1042dBm
TX Bias : 6.9240mA
Channel 3:
RX Power : -0.1592dBm
TX Power : 0.6547dBm
TX Bias : 6.9420mA
Channel 4:
RX Power : -0.1300dBm
TX Power : 0.4653dBm
TX Bias : 6.9120mA
----- Thresholds -----
Temperature:
High Warning : 70C
Low Warning : 0C
High Alarm : 75C
Low Alarm : -5C
Warning mask : 0
Alarm mask : 0
Voltage:
High Warning : 3.4600V
Low Warning : 3.1300V
High Alarm : 3.6300V
Low Alarm : 2.9700V
Warning mask : 0
Alarm mask : 0
Channel 1:
RX Power high warn : 2.4000dBm
RX Power low warn : -9.5001dBm
RX Power high alarm : 5.4103dBm
RX Power low alarm : -12.5104dBm
RX Power Warning mask: 0
RX Power Alarm mask : 0
TX Power high warn : 2.4000dBm
TX Power low warn : -7.6020dBm
TX Power high alarm : 3.1917dBm
TX Power low alarm : -8.5699dBm
TX Power Warning mask: 0
TX Power Alarm mask : 0
TX Bias high warn : 12.0000mA
TX Bias low warn : 2.0000mA
TX Bias high alarm : 15.0000mA
TX Bias low alarm : 1.0000mA
TX Bias Warning mask : 0
TX Bias Alarm mask : 0
Channel 2:
RX Power high warn : 2.4000dBm
RX Power low warn : -9.5001dBm
RX Power high alarm : 5.4103dBm
RX Power low alarm : -12.5104dBm
RX Power Warning mask: 0
RX Power Alarm mask : 0
TX Power high warn : 2.4000dBm
TX Power low warn : -7.6020dBm
TX Power high alarm : 3.1917dBm
TX Power low alarm : -8.5699dBm
TX Power Warning mask: 0
TX Power Alarm mask : 0
TX Bias high warn : 12.0000mA
TX Bias low warn : 2.0000mA
TX Bias high alarm : 15.0000mA
TX Bias low alarm : 1.0000mA
TX Bias Warning mask : 0
TX Bias Alarm mask : 0
Channel 3:
RX Power high warn : 2.4000dBm
RX Power low warn : -9.5001dBm
RX Power high alarm : 5.4103dBm
RX Power low alarm : -12.5104dBm
RX Power Warning mask: 0
RX Power Alarm mask : 0
TX Power high warn : 2.4000dBm
TX Power low warn : -7.6020dBm
TX Power high alarm : 3.1917dBm
TX Power low alarm : -8.5699dBm
TX Power Warning mask: 0
TX Power Alarm mask : 0
TX Bias high warn : 12.0000mA
TX Bias low warn : 2.0000mA
TX Bias high alarm : 15.0000mA
TX Bias low alarm : 1.0000mA
TX Bias Warning mask : 0
TX Bias Alarm mask : 0
Channel 4:
RX Power high warn : 2.4000dBm
RX Power low warn : -9.5001dBm
RX Power high alarm : 5.4103dBm
RX Power low alarm : -12.5104dBm
RX Power Warning mask: 0
RX Power Alarm mask : 0
TX Power high warn : 2.4000dBm
TX Power low warn : -7.6020dBm
TX Power high alarm : 3.1917dBm
TX Power low alarm : -8.5699dBm
TX Power Warning mask: 0
TX Power Alarm mask : 0
TX Bias high warn : 12.0000mA
TX Bias low warn : 2.0000mA
TX Bias high alarm : 15.0000mA
TX Bias low alarm : 1.0000mA
TX Bias Warning mask : 0
TX Bias Alarm mask : 0
tech1@D7:~$
tech1@D7:~$
tech1@D7:~$
Chris Preimesberger
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers
2018-09-27 15:32 ` Andrew Lunn
2018-09-27 16:08 ` Chris Preimesberger
@ 2018-10-02 7:10 ` Eran Ben Elisha
1 sibling, 0 replies; 16+ messages in thread
From: Eran Ben Elisha @ 2018-10-02 7:10 UTC (permalink / raw)
To: Andrew Lunn
Cc: Eran Ben Elisha, nhorman, chrisp, John W. Linville,
Linux Netdev List
On Thu, Sep 27, 2018 at 6:34 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > Driver return 256 bytes (reading it correctly, I verified it, no overruns),
> > however the extra bytes are presented due to this bug (expecting to parse
> > 640 bytes).
> >
> > Do you see another bug here? Am I missing something?
>
> Hi Erin
Eran...
>
> Please could you try ethtool -m raw on so you get a binary dump. The
> file which Chris provided had more bytes in it than 256.
I ran '-m raw on' on QSFP28.
File size is 256 bytes.
(with and without my suggested patch...)
Eran
>
> Thanks
> Andrew
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2018-10-02 13:52 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-09-26 19:29 bug: 'ethtool -m' reports spurious alarm & warning threshold values for QSFP28 transceivers Chris Preimesberger
2018-09-26 19:44 ` Andrew Lunn
2018-09-26 20:47 ` Chris Preimesberger
2018-09-26 21:46 ` Andrew Lunn
2018-09-26 21:34 ` Neil Horman
2018-09-26 21:58 ` Andrew Lunn
2018-09-27 13:23 ` Neil Horman
2018-09-27 13:25 ` Eran Ben Elisha
2018-09-27 14:52 ` Andrew Lunn
2018-09-27 15:20 ` Eran Ben Elisha
2018-09-27 15:32 ` Andrew Lunn
2018-09-27 16:08 ` Chris Preimesberger
2018-09-27 16:38 ` Andrew Lunn
2018-09-27 18:56 ` Chris Preimesberger
2018-09-27 20:17 ` Chris Preimesberger
2018-10-02 7:10 ` Eran Ben Elisha
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).