The difference between linux kernel driver and FreeBSD's with Intel X533

The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

Hi there,

It seems that for reasons unknown to me, my Intel X533 based 10G SFP+ 
doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but it 
does in OPNsense which is based on FreeBSD 13.2.

How would I go about debugging this properly? Both sides see light, but 
no link unless I'm using FreeBSD.

For reference the thread on VyOS forums regarding this: 
https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253

P.S Sorry for the double posting, but now it should go to the Intel list 
too properly.

Best regards,
Skyler Mäntysaari

Re: The difference between linux kernel driver and FreeBSD's with Intel X533

[Jesse Brandeburg]

On 10/3/2023 4:25 PM, Skyler Mäntysaari wrote:
> Hi there,
> 
> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+ 
> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but it 
> does in OPNsense which is based on FreeBSD 13.2.
> 
> How would I go about debugging this properly? Both sides see light, but 
> no link unless I'm using FreeBSD.

have you tried setting the allow_unsupported_sfp=1 module parameter?
parm:           allow_unsupported_sfp:Allow unsupported and untested 
SFP+ modules on 82599-based adapters (bool)

> For reference the thread on VyOS forums regarding this: 
> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253

Also, for troubleshooting it would be useful to have the output of 
ethtool -m eth0 from your device.

- Jesse

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Jesse Brandeburg]

On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>> Hi there,
>>>
>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>
>>> How would I go about debugging this properly? Both sides see light,
>>> but no link unless I'm using FreeBSD.
> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537

Hi Skyler,

Response from Intel team:

In the ethtool -m output pasted I see TX and RX optical power is fine.
Could you request fault status on both sides? Just want to check if link
is down because we are at local-fault or link partner is at local-fault.

rmmod ixgbe
modprobe ixgbe
ethtool -S eth0 | grep fault

Since it is 10G, if our side TX is ON (power level says it is) then we
should expect link partner RX to be locked so cannot be at Local Fault.

Skyler, please gather that ethtool -S data for us.

Jesse

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>> Hi there,
>>>>
>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>
>>>> How would I go about debugging this properly? Both sides see light,
>>>> but no link unless I'm using FreeBSD.
>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>
> Hi Skyler,
>
> Response from Intel team:
>
> In the ethtool -m output pasted I see TX and RX optical power is fine.
> Could you request fault status on both sides? Just want to check if link
> is down because we are at local-fault or link partner is at local-fault.
>
> rmmod ixgbe
> modprobe ixgbe
> ethtool -S eth0 | grep fault
>
> Since it is 10G, if our side TX is ON (power level says it is) then we
> should expect link partner RX to be locked so cannot be at Local Fault.
>
> Skyler, please gather that ethtool -S data for us.
>
> Jesse
>
>
>
> 

Hi Jesse,

As the other side of the link is an Juniper, I'm not quite sure how I would gather the same data from it as it doesn't have ethtool?

I have also somewhat given up hope on it working on VyOS and instead I am using OPNsense for the moment but I still have VyOS installed as well.

Skyler

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>> Hi there,
>>>>>
>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>
>>>>> How would I go about debugging this properly? Both sides see light,
>>>>> but no link unless I'm using FreeBSD.
>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>>
>> Hi Skyler,
>>
>> Response from Intel team:
>>
>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>> Could you request fault status on both sides? Just want to check if link
>> is down because we are at local-fault or link partner is at local-fault.
>>
>> rmmod ixgbe
>> modprobe ixgbe
>> ethtool -S eth0 | grep fault
>>
>> Since it is 10G, if our side TX is ON (power level says it is) then we
>> should expect link partner RX to be locked so cannot be at Local Fault.
>>
>> Skyler, please gather that ethtool -S data for us.
>>
>> Jesse
>>
>>
>>
>> 
>
> Hi Jesse,
>
> As the other side of the link is an Juniper, I'm not quite sure how I 
> would gather the same data from it as it doesn't have ethtool?
>
> I have also somewhat given up hope on it working on VyOS and instead I 
> am using OPNsense for the moment but I still have VyOS installed as 
> well.
>
> Skyler

Hi Jesse,

I did verify that the grep doesn't yield any results on the VyOS box and all of the data returned has an value of 0. Paste of which is here: https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x

I really think something weird is going on with the driver in Linux as otherwise the same exact config on Juniper wouldn't work there either. The VyOS box also says that it's unable to modify autoneg settings, or speed/duplex of the interface.

Skyler

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
> On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
>> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>>> Hi there,
>>>>>>
>>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>>
>>>>>> How would I go about debugging this properly? Both sides see light,
>>>>>> but no link unless I'm using FreeBSD.
>>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>>>
>>> Hi Skyler,
>>>
>>> Response from Intel team:
>>>
>>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>>> Could you request fault status on both sides? Just want to check if link
>>> is down because we are at local-fault or link partner is at local-fault.
>>>
>>> rmmod ixgbe
>>> modprobe ixgbe
>>> ethtool -S eth0 | grep fault
>>>
>>> Since it is 10G, if our side TX is ON (power level says it is) then we
>>> should expect link partner RX to be locked so cannot be at Local Fault.
>>>
>>> Skyler, please gather that ethtool -S data for us.
>>>
>>> Jesse
>>>
>>>
>>>
>>> 
>>
>> Hi Jesse,
>>
>> As the other side of the link is an Juniper, I'm not quite sure how I 
>> would gather the same data from it as it doesn't have ethtool?
>>
>> I have also somewhat given up hope on it working on VyOS and instead I 
>> am using OPNsense for the moment but I still have VyOS installed as 
>> well.
>>
>> Skyler
>
> Hi Jesse,
>
> I did verify that the grep doesn't yield any results on the VyOS box 
> and all of the data returned has an value of 0. Paste of which is here: 
> https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
>
> I really think something weird is going on with the driver in Linux as 
> otherwise the same exact config on Juniper wouldn't work there either. 
> The VyOS box also says that it's unable to modify autoneg settings, or 
> speed/duplex of the interface.
>
> Skyler

It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46

I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.

Please share these findings with the correct Intel team so that this could be fixed.

- Skyler

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Ivan Pang]

On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
> > On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
> >> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
> >>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
> >>>>>> Hi there,
> >>>>>>
> >>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
> >>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
> >>>>>> it does in OPNsense which is based on FreeBSD 13.2.
> >>>>>>
> >>>>>> How would I go about debugging this properly? Both sides see light,
> >>>>>> but no link unless I'm using FreeBSD.
> >>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
> >>>
> >>> Hi Skyler,
> >>>
> >>> Response from Intel team:
> >>>
> >>> In the ethtool -m output pasted I see TX and RX optical power is fine.
> >>> Could you request fault status on both sides? Just want to check if link
> >>> is down because we are at local-fault or link partner is at local-fault.
> >>>
> >>> rmmod ixgbe
> >>> modprobe ixgbe
> >>> ethtool -S eth0 | grep fault
> >>>
> >>> Since it is 10G, if our side TX is ON (power level says it is) then we
> >>> should expect link partner RX to be locked so cannot be at Local Fault.
> >>>
> >>> Skyler, please gather that ethtool -S data for us.
> >>>
> >>> Jesse
> >>>
> >>>
> >>>
> >>> 
> >>
> >> Hi Jesse,
> >>
> >> As the other side of the link is an Juniper, I'm not quite sure how I 
> >> would gather the same data from it as it doesn't have ethtool?
> >>
> >> I have also somewhat given up hope on it working on VyOS and instead I 
> >> am using OPNsense for the moment but I still have VyOS installed as 
> >> well.
> >>
> >> Skyler
> >
> > Hi Jesse,
> >
> > I did verify that the grep doesn't yield any results on the VyOS box 
> > and all of the data returned has an value of 0. Paste of which is here: 
> > https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
> >
> > I really think something weird is going on with the driver in Linux as 
> > otherwise the same exact config on Juniper wouldn't work there either. 
> > The VyOS box also says that it's unable to modify autoneg settings, or 
> > speed/duplex of the interface.
> >
> > Skyler
> 
> It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
> 
> I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.
> 
> Please share these findings with the correct Intel team so that this could be fixed.
> 
> - Skyler
> 

Hi Skyler, Jesse,

I came across this very similar issue when upgrading our networking gear
from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
driver but works with the out-of-tree 5.x versions. I found that my link
issues were related to this commit:

ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d

Specifically, our 10G link works when both sides of the fiber are
running the in-tree 6.1 ixgbe driver with this autonegotiation change.
Our link also works when both sides are running the 5.x ixgbe drivers
without this commit. It fails, however, when only one side has this
commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
this commit reverted, for compatibility with our varying hardware.

I would appreciate it if anyone can cross-check my claim with their
hardware as well. Also, would anyone be able to help explain what some
of those registers and reg_val being written are doing?

-Ivan

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

On 11/28/23 04:20, Ivan Pang wrote:
> On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
>> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
>>> On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
>>>> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>>>>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>>>>
>>>>>>>> How would I go about debugging this properly? Both sides see light,
>>>>>>>> but no link unless I'm using FreeBSD.
>>>>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>>>>>
>>>>> Hi Skyler,
>>>>>
>>>>> Response from Intel team:
>>>>>
>>>>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>>>>> Could you request fault status on both sides? Just want to check if link
>>>>> is down because we are at local-fault or link partner is at local-fault.
>>>>>
>>>>> rmmod ixgbe
>>>>> modprobe ixgbe
>>>>> ethtool -S eth0 | grep fault
>>>>>
>>>>> Since it is 10G, if our side TX is ON (power level says it is) then we
>>>>> should expect link partner RX to be locked so cannot be at Local Fault.
>>>>>
>>>>> Skyler, please gather that ethtool -S data for us.
>>>>>
>>>>> Jesse
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Hi Jesse,
>>>>
>>>> As the other side of the link is an Juniper, I'm not quite sure how I
>>>> would gather the same data from it as it doesn't have ethtool?
>>>>
>>>> I have also somewhat given up hope on it working on VyOS and instead I
>>>> am using OPNsense for the moment but I still have VyOS installed as
>>>> well.
>>>>
>>>> Skyler
>>>
>>> Hi Jesse,
>>>
>>> I did verify that the grep doesn't yield any results on the VyOS box
>>> and all of the data returned has an value of 0. Paste of which is here:
>>> https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
>>>
>>> I really think something weird is going on with the driver in Linux as
>>> otherwise the same exact config on Juniper wouldn't work there either.
>>> The VyOS box also says that it's unable to modify autoneg settings, or
>>> speed/duplex of the interface.
>>>
>>> Skyler
>>
>> It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
>>
>> I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.
>>
>> Please share these findings with the correct Intel team so that this could be fixed.
>>
>> - Skyler
>>
> 
> Hi Skyler, Jesse,
> 
> I came across this very similar issue when upgrading our networking gear
> from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
> driver but works with the out-of-tree 5.x versions. I found that my link
> issues were related to this commit:
> 
> ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d
> 
> Specifically, our 10G link works when both sides of the fiber are
> running the in-tree 6.1 ixgbe driver with this autonegotiation change.
> Our link also works when both sides are running the 5.x ixgbe drivers
> without this commit. It fails, however, when only one side has this
> commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
> this commit reverted, for compatibility with our varying hardware.
> 
> I would appreciate it if anyone can cross-check my claim with their
> hardware as well. Also, would anyone be able to help explain what some
> of those registers and reg_val being written are doing?
> 
> -Ivan
> 

Hi Ivan,

It seems that for whatever reason, your reply does not appear on the lists?

- Skyler

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Skyler Mäntysaari]

On 11/28/23 04:20, Ivan Pang wrote:
> On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
>> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
>>> On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
>>>> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>>>>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>>>>
>>>>>>>> How would I go about debugging this properly? Both sides see light,
>>>>>>>> but no link unless I'm using FreeBSD.
>>>>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537
>>>>>
>>>>> Hi Skyler,
>>>>>
>>>>> Response from Intel team:
>>>>>
>>>>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>>>>> Could you request fault status on both sides? Just want to check if link
>>>>> is down because we are at local-fault or link partner is at local-fault.
>>>>>
>>>>> rmmod ixgbe
>>>>> modprobe ixgbe
>>>>> ethtool -S eth0 | grep fault
>>>>>
>>>>> Since it is 10G, if our side TX is ON (power level says it is) then we
>>>>> should expect link partner RX to be locked so cannot be at Local Fault.
>>>>>
>>>>> Skyler, please gather that ethtool -S data for us.
>>>>>
>>>>> Jesse
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Hi Jesse,
>>>>
>>>> As the other side of the link is an Juniper, I'm not quite sure how I
>>>> would gather the same data from it as it doesn't have ethtool?
>>>>
>>>> I have also somewhat given up hope on it working on VyOS and instead I
>>>> am using OPNsense for the moment but I still have VyOS installed as
>>>> well.
>>>>
>>>> Skyler
>>>
>>> Hi Jesse,
>>>
>>> I did verify that the grep doesn't yield any results on the VyOS box
>>> and all of the data returned has an value of 0. Paste of which is here:
>>> https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
>>>
>>> I really think something weird is going on with the driver in Linux as
>>> otherwise the same exact config on Juniper wouldn't work there either.
>>> The VyOS box also says that it's unable to modify autoneg settings, or
>>> speed/duplex of the interface.
>>>
>>> Skyler
>>
>> It has been  verified that the driver in kernel version 5.4.255 seems to work aka 1.3 VyOS.  Post from another user in the same thread about it: https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
>>
>> I have also verified that the out-of-tree ixgbe driver does work, but in-kernel doesn't in kernel 6.1.58.
>>
>> Please share these findings with the correct Intel team so that this could be fixed.
>>
>> - Skyler
>>
> 
> Hi Skyler, Jesse,
> 
> I came across this very similar issue when upgrading our networking gear
> from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
> driver but works with the out-of-tree 5.x versions. I found that my link
> issues were related to this commit:
> 
> ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d
> 
> Specifically, our 10G link works when both sides of the fiber are
> running the in-tree 6.1 ixgbe driver with this autonegotiation change.
> Our link also works when both sides are running the 5.x ixgbe drivers
> without this commit. It fails, however, when only one side has this
> commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
> this commit reverted, for compatibility with our varying hardware.
> 
> I would appreciate it if anyone can cross-check my claim with their
> hardware as well. Also, would anyone be able to help explain what some
> of those registers and reg_val being written are doing?
> 
> -Ivan
> 


Hi Ivan,

It seems that for whatever reason, your reply does not appear on the lists?

P.S Sorry for the double posting if you see this twice. It wasn't sent 
to Ivan correctly the first try.

- Skyler

Re: [Intel-wired-lan] The difference between linux kernel driver and FreeBSD's with Intel X533

[Paul Menzel]

[Cc: +<jeffd@silicom-usa.com>, +regressions@lists.linux.dev]

#regzbot ^introduced: 565736048bd5f9888990569993c6b6bfdf6dcb6d

Dear Ivan,


Am 28.11.23 um 03:20 schrieb Ivan Pang:
> On Wed, Oct 18, 2023 at 09:50:35PM +0300, Skyler Mäntysaari wrote:
>> On Tue, Oct 10, 2023, at 03:39, Skyler Mäntysaari wrote:
>>> On Tue, Oct 10, 2023, at 02:50, Skyler Mäntysaari wrote:
>>>> On Mon, Oct 9, 2023, at 18:33, Jesse Brandeburg wrote:
>>>>> On 10/4/2023 10:08 AM, Skyler Mäntysaari wrote:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> It seems that for reasons unknown to me, my Intel X533 based 10G SFP+
>>>>>>>> doesn't want to work with kernel 6.1.55 in VyOS 1.4 nor Debian 12 but
>>>>>>>> it does in OPNsense which is based on FreeBSD 13.2.
>>>>>>>>
>>>>>>>> How would I go about debugging this properly? Both sides see light,
>>>>>>>> but no link unless I'm using FreeBSD.
>>>>>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/11?u=samip537

>>>>> Response from Intel team:
>>>>>
>>>>> In the ethtool -m output pasted I see TX and RX optical power is fine.
>>>>> Could you request fault status on both sides? Just want to check if link
>>>>> is down because we are at local-fault or link partner is at local-fault.
>>>>>
>>>>> rmmod ixgbe
>>>>> modprobe ixgbe
>>>>> ethtool -S eth0 | grep fault
>>>>>
>>>>> Since it is 10G, if our side TX is ON (power level says it is) then we
>>>>> should expect link partner RX to be locked so cannot be at Local Fault.
>>>>>
>>>>> Skyler, please gather that ethtool -S data for us.

>>>> As the other side of the link is an Juniper, I'm not quite sure how I
>>>> would gather the same data from it as it doesn't have ethtool?
>>>>
>>>> I have also somewhat given up hope on it working on VyOS and instead I
>>>> am using OPNsense for the moment but I still have VyOS installed as
>>>> well.

>>> I did verify that the grep doesn't yield any results on the VyOS box
>>> and all of the data returned has an value of 0. Paste of which is here:
>>> https://p.kapsi.fi/?4a82cedb4f4801ec#DcEgFMFK7cH13EqypsY4ZaHS5taeA1zXevmmTSVW3P9x
>>>
>>> I really think something weird is going on with the driver in Linux as
>>> otherwise the same exact config on Juniper wouldn't work there either.
>>> The VyOS box also says that it's unable to modify autoneg settings, or
>>> speed/duplex of the interface.

>> It has been  verified that the driver in kernel version 5.4.255
>> seems to work aka 1.3 VyOS.  Post from another user in the same
>> thread about it:
>> https://forum.vyos.io/t/10g-sfp-trouble-with-linking-intel-x553/12253/46
>> 
>> I have also verified that the out-of-tree ixgbe driver does work,
>> but in-kernel doesn't in kernel 6.1.58.
>> 
>> Please share these findings with the correct Intel team so that
>> this could be fixed.

> I came across this very similar issue when upgrading our networking gear
> from kernel 5.15 to 6.1. Our 10G link fails with the in-tree 6.1 ixgbe
> driver but works with the out-of-tree 5.x versions. I found that my link
> issues were related to this commit:
> 
> ixgbe: Manual AN-37 for troublesome link partners for X550 SFI
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.1.63&id=565736048bd5f9888990569993c6b6bfdf6dcb6d
> 
> Specifically, our 10G link works when both sides of the fiber are
> running the in-tree 6.1 ixgbe driver with this autonegotiation change.
> Our link also works when both sides are running the 5.x ixgbe drivers
> without this commit. It fails, however, when only one side has this
> commit. Our current setup compiles the in-tree 6.1 ixgbe driver with
> this commit reverted, for compatibility with our varying hardware.
> 
> I would appreciate it if anyone can cross-check my claim with their
> hardware as well. Also, would anyone be able to help explain what some
> of those registers and reg_val being written are doing?

Thank you for mentioning the culprit. That commit is present since 
v6.1-rc1. I am adding the regression folks.


Kind regards,

Paul