Re: watchdog timeout panic in e1000 driver

[Auke Kok <auke-jan.h.kok@intel.com>]

Kenzo Iwami wrote:
> Hi,
> 
> Thank you for your comment.
> 
>>>>>> Anyway as I said in the same e-mail, we're working on reducing the lock timeout to a 
>>>>>> reasonable time. This will unfortunately take some time, as we need to change some major 
>>>>>> components in the driver to make sure this doesn't happen.
>>>>> How about the following approach?
 >>>>>
>>>>> If acquiring semaphore fails inside the interrupt handler, acquiring semaphore
>>>>> is abandoned immediately without waiting for timeout.
>>>>> However, I don't know whether this method affects other processes.
 >>>>
>>>> with the current hardware being accessed simultaneously from several users in the 
>>>> kernel, that would lead to large problems - the watchdog task accesses it every 2 
>>>> seconds as it reads the PHY link status, so when one of those fails the driver would 
>>>> have no choice but to reset the entire device.
 >>>
>>> This problem occurs because interrupt handler is executed while the
>>> interrupted code is still holding the semaphore. Acquiring the semaphore
>>> fails regardless of the timeout period.
>>>
>>> I think the watchdog task will fail trying to read the PHY link status,
>>> even if the lock timeout period has been reduced.
 >>
>> correct, we're not looking into reducing the lock timeout but towards reducing the total 
>> lock time. Once we have reduced that to something acceptable, we can reduce the timout 
>> accordingly.
> 
> Even if the total lock time can be reduced, it's possible that interrupt
> handler is executed while the interrupted code is still holding the semaphore.
> I think your method only decrease the frequency of this problem.
> Why does reducing the lock time solve this problem?

there are several problems here that need addressing. It's not acceptable for our driver 
to wait up to 15 seconds, and we can (presumably) reduce it to milliseconds, so that 
would help a lot. We should in no case at all hold it for any period longer than (give 
or take) half a second, so working towards that is a very good step in the right direction.

Adding the timer task back may also help, as we are no longer trying to aqcuire the 
sw_fw_semaphore in interrupt context, but we removed it for a reason, and I need to dig 
up what reason this exactly was before we can revert it. Jesse might know, so I'll talk 
to him. But this will not fix the fact that the semaphore is held for a long time :)

so, this needs two fixes. much to do.

Cheers,

Auke