Re: [PATCH net 3/3] igc: return early when failing to read EECD register

Re: [PATCH net 3/3] igc: return early when failing to read EECD register

[Bjorn Helgaas]

[+cc Kalesh, Przemek, linux-pci]

On Tue, Jan 07, 2025 at 11:01:47AM -0800, Tony Nguyen wrote:
> From: En-Wei Wu <en-wei.wu@canonical.com>
> 
> When booting with a dock connected, the igc driver may get stuck for ~40
> seconds if PCIe link is lost during initialization.
> 
> This happens because the driver access device after EECD register reads
> return all F's, indicating failed reads. Consequently, hw->hw_addr is set
> to NULL, which impacts subsequent rd32() reads. This leads to the driver
> hanging in igc_get_hw_semaphore_i225(), as the invalid hw->hw_addr
> prevents retrieving the expected value.
> 
> To address this, a validation check and a corresponding return value
> catch is added for the EECD register read result. If all F's are
> returned, indicating PCIe link loss, the driver will return -ENXIO
> immediately. This avoids the 40-second hang and significantly improves
> boot time when using a dock with an igc NIC.
> 
> Log before the patch:
> [    0.911913] igc 0000:70:00.0: enabling device (0000 -> 0002)
> [    0.912386] igc 0000:70:00.0: PTM enabled, 4ns granularity
> [    1.571098] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
> [   43.449095] igc_get_hw_semaphore_i225: igc 0000:70:00.0 (unnamed net_device) (uninitialized): Driver can't access device - SMBI bit is set.
> [   43.449186] igc 0000:70:00.0: probe with driver igc failed with error -13
> [   46.345701] igc 0000:70:00.0: enabling device (0000 -> 0002)
> [   46.345777] igc 0000:70:00.0: PTM enabled, 4ns granularity
> 
> Log after the patch:
> [    1.031000] igc 0000:70:00.0: enabling device (0000 -> 0002)
> [    1.032097] igc 0000:70:00.0: PTM enabled, 4ns granularity
> [    1.642291] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
> [    5.480490] igc 0000:70:00.0: enabling device (0000 -> 0002)
> [    5.480516] igc 0000:70:00.0: PTM enabled, 4ns granularity
> 
> Fixes: ab4056126813 ("igc: Add NVM support")
> Cc: Chia-Lin Kao (AceLan) <acelan.kao@canonical.com>
> Signed-off-by: En-Wei Wu <en-wei.wu@canonical.com>
> Reviewed-by: Vitaly Lifshits <vitaly.lifshits@intel.com>
> Tested-by: Mor Bar-Gabay <morx.bar.gabay@intel.com>
> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
> ---
>  drivers/net/ethernet/intel/igc/igc_base.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/igc/igc_base.c b/drivers/net/ethernet/intel/igc/igc_base.c
> index 9fae8bdec2a7..1613b562d17c 100644
> --- a/drivers/net/ethernet/intel/igc/igc_base.c
> +++ b/drivers/net/ethernet/intel/igc/igc_base.c
> @@ -68,6 +68,10 @@ static s32 igc_init_nvm_params_base(struct igc_hw *hw)
>  	u32 eecd = rd32(IGC_EECD);
>  	u16 size;
>  
> +	/* failed to read reg and got all F's */
> +	if (!(~eecd))
> +		return -ENXIO;

I don't understand this.  It looks like a band-aid that makes boot
faster but doesn't solve the real problem.

In its defense, I guess that with this patch, the first igc probe
fails, and then for some reason we attempt another a few seconds
later, and the second igc probe works fine, so the NIC actually does
end up working correct, right?

I think the PCI core has some issues with configuring ASPM L1.2, and I
wonder if those are relevant here.  If somebody can repro the problem
(i.e., without this patch, which looks like it appeared in v6.13 as
bd2776e39c2a ("igc: return early when failing to read EECD
register")), I wonder if you could try booting with "pcie_port_pm=off
pcie_aspm.policy=performance" and see if that also avoids the problem?

If so, I'd like to see the dmesg log with "pci=earlydump" and the
"sudo lspci -vv" output when booted with and without "pcie_port_pm=off
pcie_aspm.policy=performance".

>  	size = FIELD_GET(IGC_EECD_SIZE_EX_MASK, eecd);
>  
>  	/* Added to a constant, "size" becomes the left-shift value
> @@ -221,6 +225,8 @@ static s32 igc_get_invariants_base(struct igc_hw *hw)
>  
>  	/* NVM initialization */
>  	ret_val = igc_init_nvm_params_base(hw);
> +	if (ret_val)
> +		goto out;
>  	switch (hw->mac.type) {
>  	case igc_i225:
>  		ret_val = igc_init_nvm_params_i225(hw);
> -- 
> 2.47.1
> 

Re: [PATCH net 3/3] igc: return early when failing to read EECD register

[Tony Nguyen]

On 4/18/2025 12:34 PM, Bjorn Helgaas wrote:
> [+cc Kalesh, Przemek, linux-pci]
> 
> On Tue, Jan 07, 2025 at 11:01:47AM -0800, Tony Nguyen wrote:
>> From: En-Wei Wu <en-wei.wu@canonical.com>
>>
>> When booting with a dock connected, the igc driver may get stuck for ~40
>> seconds if PCIe link is lost during initialization.
>>
>> This happens because the driver access device after EECD register reads
>> return all F's, indicating failed reads. Consequently, hw->hw_addr is set
>> to NULL, which impacts subsequent rd32() reads. This leads to the driver
>> hanging in igc_get_hw_semaphore_i225(), as the invalid hw->hw_addr
>> prevents retrieving the expected value.
>>
>> To address this, a validation check and a corresponding return value
>> catch is added for the EECD register read result. If all F's are
>> returned, indicating PCIe link loss, the driver will return -ENXIO
>> immediately. This avoids the 40-second hang and significantly improves
>> boot time when using a dock with an igc NIC.
>>
>> Log before the patch:
>> [    0.911913] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> [    0.912386] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> [    1.571098] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
>> [   43.449095] igc_get_hw_semaphore_i225: igc 0000:70:00.0 (unnamed net_device) (uninitialized): Driver can't access device - SMBI bit is set.
>> [   43.449186] igc 0000:70:00.0: probe with driver igc failed with error -13
>> [   46.345701] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> [   46.345777] igc 0000:70:00.0: PTM enabled, 4ns granularity
>>
>> Log after the patch:
>> [    1.031000] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> [    1.032097] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> [    1.642291] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
>> [    5.480490] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> [    5.480516] igc 0000:70:00.0: PTM enabled, 4ns granularity
>>
>> Fixes: ab4056126813 ("igc: Add NVM support")
>> Cc: Chia-Lin Kao (AceLan) <acelan.kao@canonical.com>
>> Signed-off-by: En-Wei Wu <en-wei.wu@canonical.com>
>> Reviewed-by: Vitaly Lifshits <vitaly.lifshits@intel.com>
>> Tested-by: Mor Bar-Gabay <morx.bar.gabay@intel.com>
>> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
>> ---
>>   drivers/net/ethernet/intel/igc/igc_base.c | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/intel/igc/igc_base.c b/drivers/net/ethernet/intel/igc/igc_base.c
>> index 9fae8bdec2a7..1613b562d17c 100644
>> --- a/drivers/net/ethernet/intel/igc/igc_base.c
>> +++ b/drivers/net/ethernet/intel/igc/igc_base.c
>> @@ -68,6 +68,10 @@ static s32 igc_init_nvm_params_base(struct igc_hw *hw)
>>   	u32 eecd = rd32(IGC_EECD);
>>   	u16 size;
>>   
>> +	/* failed to read reg and got all F's */
>> +	if (!(~eecd))
>> +		return -ENXIO;
> 
> I don't understand this.  It looks like a band-aid that makes boot
> faster but doesn't solve the real problem.
> 
> In its defense, I guess that with this patch, the first igc probe
> fails, and then for some reason we attempt another a few seconds
> later, and the second igc probe works fine, so the NIC actually does
> end up working correct, right?
> 
> I think the PCI core has some issues with configuring ASPM L1.2, and I
> wonder if those are relevant here.  If somebody can repro the problem
> (i.e., without this patch, which looks like it appeared in v6.13 as
> bd2776e39c2a ("igc: return early when failing to read EECD
> register")), I wonder if you could try booting with "pcie_port_pm=off
> pcie_aspm.policy=performance" and see if that also avoids the problem?
> 
> If so, I'd like to see the dmesg log with "pci=earlydump" and the
> "sudo lspci -vv" output when booted with and without "pcie_port_pm=off
> pcie_aspm.policy=performance".

We weren't able to get a repro here.

En-Wei would you be able to provide this to Bjorn?

Thanks,
Tony

>>   	size = FIELD_GET(IGC_EECD_SIZE_EX_MASK, eecd);
>>   
>>   	/* Added to a constant, "size" becomes the left-shift value
>> @@ -221,6 +225,8 @@ static s32 igc_get_invariants_base(struct igc_hw *hw)
>>   
>>   	/* NVM initialization */
>>   	ret_val = igc_init_nvm_params_base(hw);
>> +	if (ret_val)
>> +		goto out;
>>   	switch (hw->mac.type) {
>>   	case igc_i225:
>>   		ret_val = igc_init_nvm_params_i225(hw);
>> -- 
>> 2.47.1
>>

Re: [PATCH net 3/3] igc: return early when failing to read EECD register

[En-Wei WU]

Sure, I'll access the machine and do the debugging next week.

Thanks,
En-Wei.


On Wed, 23 Apr 2025 at 17:10, En-Wei WU <en-wei.wu@canonical.com> wrote:
>
> Sure, I'll access the machine and do the debugging next week.
>
> Thanks,
> En-Wei.
>
> On Wed, 23 Apr 2025 at 01:16, Tony Nguyen <anthony.l.nguyen@intel.com> wrote:
>>
>>
>>
>> On 4/18/2025 12:34 PM, Bjorn Helgaas wrote:
>> > [+cc Kalesh, Przemek, linux-pci]
>> >
>> > On Tue, Jan 07, 2025 at 11:01:47AM -0800, Tony Nguyen wrote:
>> >> From: En-Wei Wu <en-wei.wu@canonical.com>
>> >>
>> >> When booting with a dock connected, the igc driver may get stuck for ~40
>> >> seconds if PCIe link is lost during initialization.
>> >>
>> >> This happens because the driver access device after EECD register reads
>> >> return all F's, indicating failed reads. Consequently, hw->hw_addr is set
>> >> to NULL, which impacts subsequent rd32() reads. This leads to the driver
>> >> hanging in igc_get_hw_semaphore_i225(), as the invalid hw->hw_addr
>> >> prevents retrieving the expected value.
>> >>
>> >> To address this, a validation check and a corresponding return value
>> >> catch is added for the EECD register read result. If all F's are
>> >> returned, indicating PCIe link loss, the driver will return -ENXIO
>> >> immediately. This avoids the 40-second hang and significantly improves
>> >> boot time when using a dock with an igc NIC.
>> >>
>> >> Log before the patch:
>> >> [    0.911913] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> >> [    0.912386] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> >> [    1.571098] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
>> >> [   43.449095] igc_get_hw_semaphore_i225: igc 0000:70:00.0 (unnamed net_device) (uninitialized): Driver can't access device - SMBI bit is set.
>> >> [   43.449186] igc 0000:70:00.0: probe with driver igc failed with error -13
>> >> [   46.345701] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> >> [   46.345777] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> >>
>> >> Log after the patch:
>> >> [    1.031000] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> >> [    1.032097] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> >> [    1.642291] igc 0000:70:00.0 (unnamed net_device) (uninitialized): PCIe link lost, device now detached
>> >> [    5.480490] igc 0000:70:00.0: enabling device (0000 -> 0002)
>> >> [    5.480516] igc 0000:70:00.0: PTM enabled, 4ns granularity
>> >>
>> >> Fixes: ab4056126813 ("igc: Add NVM support")
>> >> Cc: Chia-Lin Kao (AceLan) <acelan.kao@canonical.com>
>> >> Signed-off-by: En-Wei Wu <en-wei.wu@canonical.com>
>> >> Reviewed-by: Vitaly Lifshits <vitaly.lifshits@intel.com>
>> >> Tested-by: Mor Bar-Gabay <morx.bar.gabay@intel.com>
>> >> Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
>> >> ---
>> >>   drivers/net/ethernet/intel/igc/igc_base.c | 6 ++++++
>> >>   1 file changed, 6 insertions(+)
>> >>
>> >> diff --git a/drivers/net/ethernet/intel/igc/igc_base.c b/drivers/net/ethernet/intel/igc/igc_base.c
>> >> index 9fae8bdec2a7..1613b562d17c 100644
>> >> --- a/drivers/net/ethernet/intel/igc/igc_base.c
>> >> +++ b/drivers/net/ethernet/intel/igc/igc_base.c
>> >> @@ -68,6 +68,10 @@ static s32 igc_init_nvm_params_base(struct igc_hw *hw)
>> >>      u32 eecd = rd32(IGC_EECD);
>> >>      u16 size;
>> >>
>> >> +    /* failed to read reg and got all F's */
>> >> +    if (!(~eecd))
>> >> +            return -ENXIO;
>> >
>> > I don't understand this.  It looks like a band-aid that makes boot
>> > faster but doesn't solve the real problem.
>> >
>> > In its defense, I guess that with this patch, the first igc probe
>> > fails, and then for some reason we attempt another a few seconds
>> > later, and the second igc probe works fine, so the NIC actually does
>> > end up working correct, right?
>> >
>> > I think the PCI core has some issues with configuring ASPM L1.2, and I
>> > wonder if those are relevant here.  If somebody can repro the problem
>> > (i.e., without this patch, which looks like it appeared in v6.13 as
>> > bd2776e39c2a ("igc: return early when failing to read EECD
>> > register")), I wonder if you could try booting with "pcie_port_pm=off
>> > pcie_aspm.policy=performance" and see if that also avoids the problem?
>> >
>> > If so, I'd like to see the dmesg log with "pci=earlydump" and the
>> > "sudo lspci -vv" output when booted with and without "pcie_port_pm=off
>> > pcie_aspm.policy=performance".
>>
>> We weren't able to get a repro here.
>>
>> En-Wei would you be able to provide this to Bjorn?
>>
>> Thanks,
>> Tony
>>
>> >>      size = FIELD_GET(IGC_EECD_SIZE_EX_MASK, eecd);
>> >>
>> >>      /* Added to a constant, "size" becomes the left-shift value
>> >> @@ -221,6 +225,8 @@ static s32 igc_get_invariants_base(struct igc_hw *hw)
>> >>
>> >>      /* NVM initialization */
>> >>      ret_val = igc_init_nvm_params_base(hw);
>> >> +    if (ret_val)
>> >> +            goto out;
>> >>      switch (hw->mac.type) {
>> >>      case igc_i225:
>> >>              ret_val = igc_init_nvm_params_i225(hw);
>> >> --
>> >> 2.47.1
>> >>
>>