Unhandled LM90 irq 308 on Dalmore?

Unhandled LM90 irq 308 on Dalmore?

[Paul Walmsley]

Just FYI, the Tegra114 Dalmore board here reports an unhandled IRQ about 
two minutes after boot:

[  120.950839] irq 308: nobody cared (try booting with the "irqpoll" option)
[  120.957654] CPU: 1 PID: 74 Comm: irq/308-lm90 Not tainted 
3.13.0-rc4-next-20131218-30442-g28522bc #1
[  120.966816] [<c0015c44>] (unwind_backtrace) from [<c0011898>] 
(show_stack+0x10/0x14)
[  120.974571] [<c0011898>] (show_stack) from [<c0565370>] 
(dump_stack+0x80/0xcc)
[  120.981804] [<c0565370>] (dump_stack) from [<c0066030>] 
(__report_bad_irq+0x20/0xc0)
[  120.989543] [<c0066030>] (__report_bad_irq) from [<c0066550>] 
(note_interrupt+0x1f8/0x254)
[  120.997811] [<c0066550>] (note_interrupt) from [<c0064fc0>] 
(irq_thread+0x12c/0x158)
[  121.005613] [<c0064fc0>] (irq_thread) from [<c003fcac>] 
(kthread+0xc4/0xe0)
[  121.012614] [<c003fcac>] (kthread) from [<c000e738>] 
(ret_from_fork+0x14/0x3c)
[  121.019825] handlers:
[  121.022117] [<c0064408>] irq_default_primary_handler threaded 
[<c0384764>] lm90_irq_thread
[  121.030418] Disabling IRQ #308

This is on next-20131218.

- Paul

Re: Unhandled LM90 irq 308 on Dalmore?

[Jean Delvare]

Hi Paul,

Adding Wei who added interrupt support to the lm90 driver, and moving
to the appropriate list.

On Thu, 19 Dec 2013 02:08:45 -0800, Paul Walmsley wrote:
> Just FYI, the Tegra114 Dalmore board here reports an unhandled IRQ about 
> two minutes after boot:
> 
> [  120.950839] irq 308: nobody cared (try booting with the "irqpoll" option)
> [  120.957654] CPU: 1 PID: 74 Comm: irq/308-lm90 Not tainted 
> 3.13.0-rc4-next-20131218-30442-g28522bc #1
> [  120.966816] [<c0015c44>] (unwind_backtrace) from [<c0011898>] 
> (show_stack+0x10/0x14)
> [  120.974571] [<c0011898>] (show_stack) from [<c0565370>] 
> (dump_stack+0x80/0xcc)
> [  120.981804] [<c0565370>] (dump_stack) from [<c0066030>] 
> (__report_bad_irq+0x20/0xc0)
> [  120.989543] [<c0066030>] (__report_bad_irq) from [<c0066550>] 
> (note_interrupt+0x1f8/0x254)
> [  120.997811] [<c0066550>] (note_interrupt) from [<c0064fc0>] 
> (irq_thread+0x12c/0x158)
> [  121.005613] [<c0064fc0>] (irq_thread) from [<c003fcac>] 
> (kthread+0xc4/0xe0)
> [  121.012614] [<c003fcac>] (kthread) from [<c000e738>] 
> (ret_from_fork+0x14/0x3c)
> [  121.019825] handlers:
> [  121.022117] [<c0064408>] irq_default_primary_handler threaded 
> [<c0384764>] lm90_irq_thread
> [  121.030418] Disabling IRQ #308
> 
> This is on next-20131218.

Which temperature chip is the Tegra114 Dalmore board using?

Is the interrupt shared with something else?

Is there any monitoring script, application or daemon polling for
temperatures on this system?

Wei, I think there is a race condition between lm90_update_device and
lm90_irq_thread. The values in registers LM90_REG_R_STATUS and
MAX6696_REG_R_STATUS2 are cleared on read, and lm90_update_device reads
these registers. So if lm90_update_device runs (caused by someone
reading any value from the sysfs interface) between the interrupt
firing and lm90_irq_thread being run, then lm90_is_tripped will return
false and consequently lm90_irq_thread will return IRQ_NONE.

Best would be if we could lock data->update_lock when the interrupt
fires, but I'm afraid there is no way to do that in a race-free way.

The next best thing I can think of is that lm90_is_tripped should check
for cache validity and read from the cache (instead of or additionally
to reading from the device registers directly.) If the cache is hot
then there's a chance that someone called lm90_update_device and was
able to read the status registers before the interrupt handler did.

In fact we probably have to do both to be completely safe.
data->last_updated is updated by lm90_update_device _after_ the status
registers have been read, so we can't rely on it unless we are also
holding data->update_lock.

-- 
Jean Delvare

Re: Unhandled LM90 irq 308 on Dalmore?

[Paul Walmsley]

Hi Jean,

On Thu, 19 Dec 2013, Jean Delvare wrote:

> Adding Wei who added interrupt support to the lm90 driver, and moving
> to the appropriate list.

Thanks for the speedy response and list correction.

> On Thu, 19 Dec 2013 02:08:45 -0800, Paul Walmsley wrote:
>> Just FYI, the Tegra114 Dalmore board here reports an unhandled IRQ about
>> two minutes after boot:
>>
>> [  120.950839] irq 308: nobody cared (try booting with the "irqpoll" option)

...

>> [  121.019825] handlers:
>> [  121.022117] [<c0064408>] irq_default_primary_handler threaded
>> [<c0384764>] lm90_irq_thread
>> [  121.030418] Disabling IRQ #308
>>
>> This is on next-20131218.
>
> Which temperature chip is the Tegra114 Dalmore board using?

It's an NCT72.

> Is the interrupt shared with something else?

Doesn't look like it.  From /proc/interrupts:

308:      74181          0          0          0      GPIO 116  lm90

That's from shortly after boot.  From successive 'cat's of 
/proc/interrupts, I see that the interrupt count is increasing very 
quickly.  So something is definitely not right.

> Is there any monitoring script, application or daemon polling for
> temperatures on this system?

This is a totally quiet system, booted with 'init=/bin/bash' on the kernel 
command line - nothing else running at all.

regards

- Paul

Re: Unhandled LM90 irq 308 on Dalmore?

[Wei Ni]

On 12/20/2013 06:36 AM, Paul Walmsley wrote:
> Hi Jean,
> 
> On Thu, 19 Dec 2013, Jean Delvare wrote:
> 
>> Adding Wei who added interrupt support to the lm90 driver, and moving
>> to the appropriate list.
> 
> Thanks for the speedy response and list correction.
> 
>> On Thu, 19 Dec 2013 02:08:45 -0800, Paul Walmsley wrote:
>>> Just FYI, the Tegra114 Dalmore board here reports an unhandled IRQ about
>>> two minutes after boot:
>>>
>>> [  120.950839] irq 308: nobody cared (try booting with the "irqpoll" option)
> 
> ...
> 
>>> [  121.019825] handlers:
>>> [  121.022117] [<c0064408>] irq_default_primary_handler threaded
>>> [<c0384764>] lm90_irq_thread
>>> [  121.030418] Disabling IRQ #308
>>>
>>> This is on next-20131218.
>>
>> Which temperature chip is the Tegra114 Dalmore board using?
> 
> It's an NCT72.
> 
>> Is the interrupt shared with something else?
> 
> Doesn't look like it.  From /proc/interrupts:
> 
> 308:      74181          0          0          0      GPIO 116  lm90

Hi, Paul
When I developed it, my Dalmore board used GPIO 116 (GPIO_PO4) as
interrupt line, but according to our downsttream kernel, there also have
other Dalmore board used GPIO 190 (GPIO_PX6). So I think your
problems was caused by it. Please try to change the interrupt line in
dts file.

Thanks.
Wei.

> 
> That's from shortly after boot.  From successive 'cat's of 
> /proc/interrupts, I see that the interrupt count is increasing very 
> quickly.  So something is definitely not right.
> 
>> Is there any monitoring script, application or daemon polling for
>> temperatures on this system?
> 
> This is a totally quiet system, booted with 'init=/bin/bash' on the kernel 
> command line - nothing else running at all.
> 
> regards
> 
> - Paul
> 

Re: Unhandled LM90 irq 308 on Dalmore?

[Stephen Warren]

On 12/19/2013 08:16 PM, Wei Ni wrote:
> On 12/20/2013 06:36 AM, Paul Walmsley wrote:
>> Hi Jean,
>>
>> On Thu, 19 Dec 2013, Jean Delvare wrote:
>>
>>> Adding Wei who added interrupt support to the lm90 driver, and moving
>>> to the appropriate list.
>>
>> Thanks for the speedy response and list correction.
>>
>>> On Thu, 19 Dec 2013 02:08:45 -0800, Paul Walmsley wrote:
>>>> Just FYI, the Tegra114 Dalmore board here reports an unhandled IRQ about
>>>> two minutes after boot:
>>>>
>>>> [  120.950839] irq 308: nobody cared (try booting with the "irqpoll" option)
>>
>> ...
>>
>>>> [  121.019825] handlers:
>>>> [  121.022117] [<c0064408>] irq_default_primary_handler threaded
>>>> [<c0384764>] lm90_irq_thread
>>>> [  121.030418] Disabling IRQ #308
>>>>
>>>> This is on next-20131218.
>>>
>>> Which temperature chip is the Tegra114 Dalmore board using?
>>
>> It's an NCT72.
>>
>>> Is the interrupt shared with something else?
>>
>> Doesn't look like it.  From /proc/interrupts:
>>
>> 308:      74181          0          0          0      GPIO 116  lm90
> 
> Hi, Paul
> When I developed it, my Dalmore board used GPIO 116 (GPIO_PO4) as
> interrupt line, but according to our downsttream kernel, there also have
> other Dalmore board used GPIO 190 (GPIO_PX6). So I think your
> problems was caused by it. Please try to change the interrupt line in
> dts file.

Paul, which board revision of Dalmore do you have?

According to the schematics, Dalmore A02 and A03 had TEMP_ALERT routed
to GPIO PX6, and that was the only option. Dalmore A04 and A05 can route
TEMP_ALERT to either GPIO PO4, or GPIO PX6 using 0-Ohm resistors, with
GPIO PO4 being the default stuffing option.

Note that upstream Linux *only* supports Dalmore A04, and no other
version. If you have a different board revision, it's not expected to
work with upstream. IIRC, Eric Brower volunteered to track down the
correct board revision for people working on upstream.

Re: Unhandled LM90 irq 308 on Dalmore?

[Paul Walmsley]

On 12/20/2013 08:26 AM, Stephen Warren wrote:
> On 12/19/2013 08:16 PM, Wei Ni wrote:
>>> Hi, Paul
>>> When I developed it, my Dalmore board used GPIO 116 (GPIO_PO4) as
>>> interrupt line, but according to our downsttream kernel, there also have
>>> other Dalmore board used GPIO 190 (GPIO_PX6). So I think your
>>> problems was caused by it. Please try to change the interrupt line in
>>> dts file.
> Paul, which board revision of Dalmore do you have?

Looks like an A03.

> According to the schematics, Dalmore A02 and A03 had TEMP_ALERT routed
> to GPIO PX6, and that was the only option. Dalmore A04 and A05 can route
> TEMP_ALERT to either GPIO PO4, or GPIO PX6 using 0-Ohm resistors, with
> GPIO PO4 being the default stuffing option.
>
> Note that upstream Linux *only* supports Dalmore A04, and no other
> version. If you have a different board revision, it's not expected to
> work with upstream. IIRC, Eric Brower volunteered to track down the
> correct board revision for people working on upstream.

Indeed, that's probably the problem, then

It would be good if this was documented somewhere in the upstream kernel 
tree.  And even better if the kernel was able to read the Dalmore EEPROM 
and print out a warning upon kernel boot...

thanks,

- Paul

Re: Unhandled LM90 irq 308 on Dalmore?

[Stephen Warren]

On 12/20/2013 10:47 PM, Paul Walmsley wrote:
> 
> On 12/20/2013 08:26 AM, Stephen Warren wrote:
...
>> Note that upstream Linux *only* supports Dalmore A04, and no other
>> version. If you have a different board revision, it's not expected to
>> work with upstream. IIRC, Eric Brower volunteered to track down the
>> correct board revision for people working on upstream.
> 
> Indeed, that's probably the problem, then
> 
> It would be good if this was documented somewhere in the upstream kernel
> tree.

I'll send a patch to document that in the DTS file.

> And even better if the kernel was able to read the Dalmore EEPROM
> and print out a warning upon kernel boot...

That would be painful, since it'd be board-specific code that isn't
really coupled with a specific piece of HW for which there's a driver.
I'd rather avoid that kind of thing.

Re: Unhandled LM90 irq 308 on Dalmore?

[Guenter Roeck]

On Mon, Jan 06, 2014 at 12:13:12PM -0700, Stephen Warren wrote:
> On 12/20/2013 10:47 PM, Paul Walmsley wrote:
> > 
> > On 12/20/2013 08:26 AM, Stephen Warren wrote:
> ...
> >> Note that upstream Linux *only* supports Dalmore A04, and no other
> >> version. If you have a different board revision, it's not expected to
> >> work with upstream. IIRC, Eric Brower volunteered to track down the
> >> correct board revision for people working on upstream.
> > 
> > Indeed, that's probably the problem, then
> > 
> > It would be good if this was documented somewhere in the upstream kernel
> > tree.
> 
> I'll send a patch to document that in the DTS file.
> 
> > And even better if the kernel was able to read the Dalmore EEPROM
> > and print out a warning upon kernel boot...
> 
> That would be painful, since it'd be board-specific code that isn't
> really coupled with a specific piece of HW for which there's a driver.
> I'd rather avoid that kind of thing.
> 
Another option might be for someone to add support for other board revisions,
and create appropriate dts files.

On a side note, if the board revisions are incompatible to each other,
shouldn't the 'compatible' property be different ? Just wondering ...

Thanks,
Guenter

Re: Unhandled LM90 irq 308 on Dalmore?

[Stephen Warren]

On 01/06/2014 12:40 PM, Guenter Roeck wrote:
> On Mon, Jan 06, 2014 at 12:13:12PM -0700, Stephen Warren wrote:
>> On 12/20/2013 10:47 PM, Paul Walmsley wrote:
>>>
>>> On 12/20/2013 08:26 AM, Stephen Warren wrote:
>> ...
>>>> Note that upstream Linux *only* supports Dalmore A04, and no other
>>>> version. If you have a different board revision, it's not expected to
>>>> work with upstream. IIRC, Eric Brower volunteered to track down the
>>>> correct board revision for people working on upstream.
>>>
>>> Indeed, that's probably the problem, then
>>>
>>> It would be good if this was documented somewhere in the upstream kernel
>>> tree.
>>
>> I'll send a patch to document that in the DTS file.
>>
>>> And even better if the kernel was able to read the Dalmore EEPROM
>>> and print out a warning upon kernel boot...
>>
>> That would be painful, since it'd be board-specific code that isn't
>> really coupled with a specific piece of HW for which there's a driver.
>> I'd rather avoid that kind of thing.
>>
> Another option might be for someone to add support for other board revisions,
> and create appropriate dts files.

We specifically decided not to support other board revisions to reduce
the support load.

> On a side note, if the board revisions are incompatible to each other,
> shouldn't the 'compatible' property be different ? Just wondering ...

Yes, if we did end up supporting other board revisions, we'd use
different compatible properties to differentiate them.