Spurious interrupt warning

Spurious interrupt warning

[Shah, Hardik]

I have ported the V4L2 driver on top of Tomi's DSS library.  DSS library has requested irq number 25.  It is never freeing it.  But whenever I get interrupt from the DSS I get this warning intermittently. 

<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25



Please note it's not continuous.  Driver is also working fine. Is it something to do with the latest spurious interrupt patch.  
http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git;a=commitdiff;h=ea153a1765dc754be688013192e8c83c40e008dc

Is there something to do in the driver to solve this interrupt warning issue. 






Thanks and Regards,
Hardik Shah

 

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Spurious interrupt warning

[Tony Lindgren]

Hi,

* Shah, Hardik <hardik.shah@ti.com> [090106 11:54]:
> I have ported the V4L2 driver on top of Tomi's DSS library.  DSS library has requested irq number 25.  It is never freeing it.  But whenever I get interrupt from the DSS I get this warning intermittently. 
> 
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> 
> 
> 
> Please note it's not continuous.  Driver is also working fine. Is it something to do with the latest spurious interrupt patch.  
> http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-2.6.git;a=commitdiff;h=ea153a1765dc754be688013192e8c83c40e008dc
> 
> Is there something to do in the driver to solve this interrupt warning issue. 

Interesting. Can you please try adding a read-back of the interrupt
status register (or revision register) at the end of the interrupt
handler for irq 25?

I guess it's INT_24XX_DSS_IRQ, which is the same for 34xx as 24xx?

Most likely this spurious interrupt means that the write does not
get posted all the way to the interrupt controller for device interrupt
handler for irq 25 before irq 25 is unmasked again.

Doing a read back in the irq 25 handler after the last write forces
the write to get posted.

Let me know if that does not help.

Regards,

Tony

RE: Spurious interrupt warning

[Hiremath, Vaibhav]

For capture driver I am also getting similar messages

<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
<4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
Spurious irq 95: 0xffffffdf, please flush posted write for irq 24


Thanks,
Vaibhav Hiremath

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
> owner@vger.kernel.org] On Behalf Of Tony Lindgren
> Sent: Tuesday, January 06, 2009 4:39 PM
> To: Shah, Hardik
> Cc: linux-omap@vger.kernel.org
> Subject: Re: Spurious interrupt warning
> 
> Hi,
> 
> * Shah, Hardik <hardik.shah@ti.com> [090106 11:54]:
> > I have ported the V4L2 driver on top of Tomi's DSS library.  DSS
> library has requested irq number 25.  It is never freeing it.  But
> whenever I get interrupt from the DSS I get this warning
> intermittently.
> >
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> 25
> >
> >
> >
> > Please note it's not continuous.  Driver is also working fine. Is
> it something to do with the latest spurious interrupt patch.
> > http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-
> 2.6.git;a=commitdiff;h=ea153a1765dc754be688013192e8c83c40e008dc
> >
> > Is there something to do in the driver to solve this interrupt
> warning issue.
> 
> Interesting. Can you please try adding a read-back of the interrupt
> status register (or revision register) at the end of the interrupt
> handler for irq 25?
> 
> I guess it's INT_24XX_DSS_IRQ, which is the same for 34xx as 24xx?
> 
> Most likely this spurious interrupt means that the write does not
> get posted all the way to the interrupt controller for device
> interrupt
> handler for irq 25 before irq 25 is unmasked again.
> 
> Doing a read back in the irq 25 handler after the last write forces
> the write to get posted.
> 
> Let me know if that does not help.
> 
> Regards,
> 
> Tony
> --
> To unsubscribe from this list: send the line "unsubscribe linux-
> omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Spurious interrupt warning

[Tony Lindgren]

* Hiremath, Vaibhav <hvaibhav@ti.com> [090106 13:13]:
> For capture driver I am also getting similar messages
> 
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> Spurious irq 95: 0xffffffdf, please flush posted write for irq 24

The same solution should work in the irq 24 handler as well. Just do
a read-back of the last written register in the irq 24 handler.

Or a read back of some safe register in the same device in case
you cannot read back the interrupt status register without clearing
the device interrupts :)

Regards,

Tony

> 
> 
> Thanks,
> Vaibhav Hiremath
> 
> > -----Original Message-----
> > From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
> > owner@vger.kernel.org] On Behalf Of Tony Lindgren
> > Sent: Tuesday, January 06, 2009 4:39 PM
> > To: Shah, Hardik
> > Cc: linux-omap@vger.kernel.org
> > Subject: Re: Spurious interrupt warning
> > 
> > Hi,
> > 
> > * Shah, Hardik <hardik.shah@ti.com> [090106 11:54]:
> > > I have ported the V4L2 driver on top of Tomi's DSS library.  DSS
> > library has requested irq number 25.  It is never freeing it.  But
> > whenever I get interrupt from the DSS I get this warning
> > intermittently.
> > >
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 25
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq
> > 25
> > >
> > >
> > >
> > > Please note it's not continuous.  Driver is also working fine. Is
> > it something to do with the latest spurious interrupt patch.
> > > http://git.kernel.org/?p=linux/kernel/git/tmlind/linux-omap-
> > 2.6.git;a=commitdiff;h=ea153a1765dc754be688013192e8c83c40e008dc
> > >
> > > Is there something to do in the driver to solve this interrupt
> > warning issue.
> > 
> > Interesting. Can you please try adding a read-back of the interrupt
> > status register (or revision register) at the end of the interrupt
> > handler for irq 25?
> > 
> > I guess it's INT_24XX_DSS_IRQ, which is the same for 34xx as 24xx?
> > 
> > Most likely this spurious interrupt means that the write does not
> > get posted all the way to the interrupt controller for device
> > interrupt
> > handler for irq 25 before irq 25 is unmasked again.
> > 
> > Doing a read back in the irq 25 handler after the last write forces
> > the write to get posted.
> > 
> > Let me know if that does not help.
> > 
> > Regards,
> > 
> > Tony
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-
> > omap" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: Spurious interrupt warning

[Woodruff, Richard]

Hi,

> * Hiremath, Vaibhav <hvaibhav@ti.com> [090106 13:13]:
> > For capture driver I am also getting similar messages
> >
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
>
> The same solution should work in the irq 24 handler as well. Just do
> a read-back of the last written register in the irq 24 handler.
>
> Or a read back of some safe register in the same device in case
> you cannot read back the interrupt status register without clearing
> the device interrupts :)

I'll re-raise the opinion that SO should be used for device mappings on OMAP.

This is not just an IRQ issue.  It's a generic one to programming a device and knowing when what you have written has taken.

It just happens to be the IRQ path has a very slow dissertation path when you take into account timing domains that it shows up.

Shrinking the race window with SO then using a read back to get the last one or two is most conservative and safe.  Probably 97% for IO accesses are ok on OMAP as device, but that is probably 99.9% with SO.  In our testing we may be able to make that 97 a 99.  Still I'd rather not risk the 1% for production.

Regards,
Richard W.

Re: Spurious interrupt warning

[Tony Lindgren]

* Woodruff, Richard <r-woodruff2@ti.com> [090106 16:06]:
> Hi,
> 
> > * Hiremath, Vaibhav <hvaibhav@ti.com> [090106 13:13]:
> > > For capture driver I am also getting similar messages
> > >
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > > <4>Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> > > Spurious irq 95: 0xffffffdf, please flush posted write for irq 24
> >
> > The same solution should work in the irq 24 handler as well. Just do
> > a read-back of the last written register in the irq 24 handler.
> >
> > Or a read back of some safe register in the same device in case
> > you cannot read back the interrupt status register without clearing
> > the device interrupts :)
> 
> I'll re-raise the opinion that SO should be used for device mappings on OMAP.
> 
> This is not just an IRQ issue.  It's a generic one to programming a device and knowing when what you have written has taken.
> 
> It just happens to be the IRQ path has a very slow dissertation path when you take into account timing domains that it shows up.
> 
> Shrinking the race window with SO then using a read back to get the last one or two is most conservative and safe.  Probably 97% for IO accesses are ok on OMAP as device, but that is probably 99.9% with SO.  In our testing we may be able to make that 97 a 99.  Still I'd rather not risk the 1% for production.

Sure you can patch the SO back if you want to. However, it's a more
generic problem, and Russell has stated that he does not want to
merge it.

At least the interrupt problems are easy to debug now, so I'd say
those are easy one liners to fix. The other places where we need
to flush posted writes may be harder to track down, but should be
possible to spot by reading the code of the problem driver.

Regards,

Tony

RE: Spurious interrupt warning

[Woodruff, Richard]

> From: Tony Lindgren [mailto:tony@atomide.com]
> Sent: Wednesday, January 07, 2009 2:26 AM
> > It just happens to be the IRQ path has a very slow dissertation path when
> you take into account timing domains that it shows up.
> >
> > Shrinking the race window with SO then using a read back to get the last one
> or two is most conservative and safe.  Probably 97% for IO accesses are ok on
> OMAP as device, but that is probably 99.9% with SO.  In our testing we may be
> able to make that 97 a 99.  Still I'd rather not risk the 1% for production.
>
> Sure you can patch the SO back if you want to. However, it's a more
> generic problem, and Russell has stated that he does not want to
> merge it.

Yes Russell doesn't like it.  However, I didn't see any viable alternate solution offered.  If ARM or any hardware vendor does something badly in hardware punishing the current generation in hopes of changing the next is not so good. Especially if some work around exists.

In the end vendors will carry some out of tree patch.  All that will happen is a new person to the lists will be put off.

> At least the interrupt problems are easy to debug now, so I'd say
> those are easy one liners to fix. The other places where we need
> to flush posted writes may be harder to track down, but should be
> possible to spot by reading the code of the problem driver.

The code you added to spotlight issues and the fixes it generates should stay in place.  SO does not solve the whole problem it just shrinks it.

I don't like the idea of exposing ourselves to some known hardware limitation just because we don't like the fix.  Already a lot of hours have been wasted on this issue.  Why drag it out further.

I've complained bitterly already and those nits have been acknowledged by hardware people in TI and ARM.

Regards,
Richard W.

Re: Spurious interrupt warning

[Russell King]

On Wed, Jan 07, 2009 at 10:13:30AM -0600, Woodruff, Richard wrote:
> > From: Tony Lindgren [mailto:tony@atomide.com]
> > Sent: Wednesday, January 07, 2009 2:26 AM
> > > It just happens to be the IRQ path has a very slow dissertation path when
> > you take into account timing domains that it shows up.
> > >
> > > Shrinking the race window with SO then using a read back to get the last one
> > or two is most conservative and safe.  Probably 97% for IO accesses are ok on
> > OMAP as device, but that is probably 99.9% with SO.  In our testing we may be
> > able to make that 97 a 99.  Still I'd rather not risk the 1% for production.
> >
> > Sure you can patch the SO back if you want to. However, it's a more
> > generic problem, and Russell has stated that he does not want to
> > merge it.
> 
> Yes Russell doesn't like it.  However, I didn't see any viable alternate
> solution offered.  If ARM or any hardware vendor does something badly in
> hardware punishing the current generation in hopes of changing the next
> is not so good. Especially if some work around exists.

I do hope you realise that precisely the same argument can be made for
the posting of writes to the timers.

And this is what's soo hard to grasp about your position - at one moment
you're talking about needing maximal performance (which involves the use
of device mappings.)  The next moment you're talking about wanting 100%
predictability by using strongly ordered mappings.

The fact of the matter is that from point of view of the ARM CPU, a
readback of the same address which was written to with a device mapping
can not bypass the write.  So a write to register X followed by an
immediate read of X and an execution dependency on that result _will_
cause the write to appear on the bus before execution can continue.
If anything else happens, the CPU is buggy because its allowing writes
to pass reads which is unpredictable from the system point of view.

If you can't get it to work with a readback and dependency, that suggests
that something else external to the CPU is misbehaving or broken.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

RE: Spurious interrupt warning

[Woodruff, Richard]

> > Yes Russell doesn't like it.  However, I didn't see any viable alternate
> > solution offered.  If ARM or any hardware vendor does something badly in
> > hardware punishing the current generation in hopes of changing the next
> > is not so good. Especially if some work around exists.
>
> I do hope you realise that precisely the same argument can be made for
> the posting of writes to the timers.
>
> And this is what's soo hard to grasp about your position - at one moment
> you're talking about needing maximal performance (which involves the use
> of device mappings.)  The next moment you're talking about wanting 100%
> predictability by using strongly ordered mappings.

The posting talked about for timers is posting in the device.  The device itself has very specific rules about waiting.

I expect to have to program carefully for this critical device.  I don't expect to have to program as carefully for every other address in the system.

The interconnect performance aspect of DEVICE vs. SO is not in the same order of magnitude as the effect with timer.

> The fact of the matter is that from point of view of the ARM CPU, a
> readback of the same address which was written to with a device mapping
> can not bypass the write.  So a write to register X followed by an
> immediate read of X and an execution dependency on that result _will_
> cause the write to appear on the bus before execution can continue.
> If anything else happens, the CPU is buggy because its allowing writes
> to pass reads which is unpredictable from the system point of view.
>
> If you can't get it to work with a readback and dependency, that suggests
> that something else external to the CPU is misbehaving or broken.

I don't mind using a read back dependency.  I don't like it but that may be necessary in some cases for a given device.

What I don't like is constantly hitting them and getting a broken system.  How many of these are out there?  If we can remove most of them in current software with SO then that isn't a bad tradeoff.

[x] Minimally having an option to enable or disable it would be good.  I just sent you such a patch.  If you dis-allow SO mapping to exist in the kernel I don't even get the option.

When interconnects and the like were really slow these effects were masked.

Regards,
Richard W.

Re: Spurious interrupt warning

[David Brownell]

On Wednesday 07 January 2009, Woodruff, Richard wrote:
> I don't mind using a read back dependency.  I don't like it but
> that may be necessary in some cases for a given device. 
> 
> What I don't like is constantly hitting them and getting a broken
> system. How many of these are out there?  If we can remove most
> of them in current software with SO then that isn't a bad tradeoff.  

The same issue comes up with PCI drivers though ... and there,
it's routinely accepted that critical paths in (some) drivers
need to issue readbacks to flush the relevant posted writes.
That's *especially* true in IRQ paths, and when DMA engines
update state in shared memory.

Which says to me that grumbling about strongly ordered regions
is more of a developer discipline/training problem than any
kind of real technical issue.  Not unlike how to do locking
right; or satisfy coding style guidelines; or follow correct
procedures to merge patches; or synchronize between several
execution contexts; or ... lots of other things that waste
time when folk don't bother doing them right at first.

Good drivers are not easy to do, and this is just one of the
many ways that shows up.  Fortunately *this* class of problem
tends to be easy to address by a careful code audit.

- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Spurious interrupt warning

[Woodruff, Richard]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]


> Which says to me that grumbling about strongly ordered regions
> is more of a developer discipline/training problem than any
> kind of real technical issue.  Not unlike how to do locking
> right; or satisfy coding style guidelines; or follow correct
> procedures to merge patches; or synchronize between several
> execution contexts; or ... lots of other things that waste
> time when folk don't bother doing them right at first.

Problem is of course much of the code base we written with out this in mind.

I've not checked for a long time but I seem to recall shutting off posting was a common thing for devices.  It is also something old-featured BIOS used to give as options.  This was a way for working around hardware and software issues.

Mind you, this change does not disable posting to the device in the way I interpret what a PC used to do (I could be wrong).  It just disables the ARM's initiator's ability to post into the interconnect (it may still post into a device).  DMA engines and others still can generated posted accesses on interconnect and device.

Much of the ARM-Linux main line and certainly the OMAP code have not been exposed to the issues.  Before MPCORE bug fixes all generic ARM used SO here.  And, only ARMv6 and above has this ability to generate this behavior.  ARMv5 and before behavior is SO.  And, until recently most interconnects used with ARM SOCs were slower and dumber about these things.

ARM-SO accesses are somewhat close to self-synchronizing. Provides a way to get more intuitive behavior for control accesses.  This is not true for DEVICE on OMAP. When you do a series of writes to an IO range with DEVICE.  Do you mind they will post up?  Also, it is possible the writes will be merged.  The effect is registers a, a+1, a+2, a+3 all get written in a burst on OMAP3 at some time latter.  The ARM may move right along given AXI2OCP mapping.  Given the way the IPs have been exercised for so long, its seems there may be a good number of subtle issues about.

Even with SO, you still may need some readbacks.  This happens for a couple of reasons:
        - omap device itself has explicit device side posting (like timer)
        - omap device needs to re-synchronize to slower functional clock
                - write-irq clear, but it requires some number of clocks to de-assert irq line.  Once the message gets to the ip it will ack.  Its not going to wait to all effects are done.

** Both the above are in addition to Interconnect posting.

SO for OMAP has an effect on the interconnect and the ARM pipeline.  It has less effect on end peripheral.

I'll attach the rough patch we added to one tree.  It gives an OPTION to disable posting. If I'm shipping tomorrow I want such an option.

Regards,
Richard W.


[-- Attachment #2: bus_posting.diff --]
[-- Type: application/octet-stream, Size: 14207 bytes --]

 arch/arm/configs/omap_3430ldp_defconfig |    3 ++-
 arch/arm/configs/omap_3430sdp_defconfig |    3 ++-
 arch/arm/mach-omap2/Kconfig             |   11 +++++++++++
 arch/arm/mach-omap2/io.c                |   16 ++++++++--------
 arch/arm/mach-omap2/pm_34xx.c           |    5 +++--
 arch/arm/mach-omap2/pm_34xx.h           |   21 +++++++++++++++------
 arch/arm/mach-omap2/pm_idle_34xx.c      |    6 +++---
 arch/arm/mach-omap2/pm_idle_34xx.h      |    2 +-
 arch/arm/mach-omap2/prcm_34xx.c         |   30 ++++++++++++++++++++++--------
 arch/arm/mm/mmu.c                       |    8 +++++++-
 arch/arm/plat-omap/gpio.c               |    4 ++++
 arch/arm/plat-omap/sram.c               |    5 +++--
 include/asm-arm/arch-omap/io.h          |   11 +++++++++++
 include/asm-arm/arch-omap/prcm_34xx.h   |    1 +
 include/asm-arm/mach/map.h              |    1 +
 15 files changed, 94 insertions(+), 33 deletions(-)

================================================================================

--- arch/arm/configs/omap_3430ldp_defconfig@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-12-03 15:30:49.000000000 -0600
+++ arch/arm/configs/omap_3430ldp_defconfig	2008-12-09 20:16:22.000000000 -0600
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.24.7-omap1-arm2
-# Fri Nov 28 22:36:13 2008
+# Tue Dec  9 20:53:43 2008
 #
 CONFIG_ARM=y
 CONFIG_SYS_SUPPORTS_APM_EMULATION=y
@@ -185,6 +185,7 @@ CONFIG_OMAP3430_ES2=y
 #
 # CONFIG_MACH_OMAP_3430SDP is not set
 CONFIG_MACH_OMAP_3430LABRADOR=y
+# CONFIG_INTERCONNECT_IO_POSTING is not set
 CONFIG_OMAP3_PM=y
 CONFIG_OMAP_VOLT_SR_BYPASS=y
 # CONFIG_OMAP_VOLT_SR is not set
--- arch/arm/configs/omap_3430sdp_defconfig@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-12-03 15:30:47.000000000 -0600
+++ arch/arm/configs/omap_3430sdp_defconfig	2008-12-09 20:16:26.000000000 -0600
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.24.7-omap1-arm2
-# Fri Nov 28 22:36:57 2008
+# Tue Dec  9 20:52:04 2008
 #
 CONFIG_ARM=y
 CONFIG_SYS_SUPPORTS_APM_EMULATION=y
@@ -198,6 +198,7 @@ CONFIG_OMAP3430_ES2=y
 #
 CONFIG_MACH_OMAP_3430SDP=y
 # CONFIG_MACH_OMAP_3430LABRADOR is not set
+# CONFIG_INTERCONNECT_IO_POSTING is not set
 CONFIG_OMAP3_PM=y
 CONFIG_OMAP_VOLT_SR_BYPASS=y
 # CONFIG_OMAP_VOLT_SR is not set
--- arch/arm/mach-omap2/Kconfig@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-11-06 21:38:43.000000000 -0600
+++ arch/arm/mach-omap2/Kconfig	2008-12-09 20:16:33.000000000 -0600
@@ -129,6 +129,17 @@ config MACH_OMAP_2430OSK
 	bool "OMAP 2430 OSK board"
 	depends on ARCH_OMAP2 && ARCH_OMAP24XX
 
+config INTERCONNECT_IO_POSTING
+	bool "Enable bus posting for PIO accesses"
+	depends on ARCH_OMAP34XX
+	default n
+	---help---
+	This option sets PIO access for internal OMAP3 registers to follow the
+	ARMv7 DEVICE attribute. For 3430 this will allow posted writes in the
+	interconnect. Software will need to synchronize writes to ensure
+	completion. When not set the attribute is Strongly Ordered which is
+	non-posted on the OMAP3 interconnect. 
+
 config OMAP3_PM
 	bool "Enable TI OMAP Power Management"
 	depends on ARCH_OMAP3
--- arch/arm/mach-omap2/io.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-06-13 20:01:15.000000000 -0500
+++ arch/arm/mach-omap2/io.c	2008-12-09 20:16:30.000000000 -0600
@@ -125,49 +125,49 @@ static struct map_desc omap34xx_io_desc[
 		.virtual	= L3_34XX_VIRT,
 		.pfn		= __phys_to_pfn(L3_34XX_PHYS),
 		.length		= L3_34XX_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= L4_34XX_VIRT,
 		.pfn		= __phys_to_pfn(L4_34XX_PHYS),
 		.length		= L4_34XX_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= L4_WK_34XX_VIRT,
 		.pfn		= __phys_to_pfn(L4_WK_34XX_PHYS),
 		.length		= L4_WK_34XX_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= OMAP34XX_GPMC_VIRT,
 		.pfn		= __phys_to_pfn(OMAP34XX_GPMC_PHYS),
 		.length		= OMAP34XX_GPMC_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= OMAP343X_SMS_VIRT,
 		.pfn		= __phys_to_pfn(OMAP343X_SMS_PHYS),
 		.length		= OMAP343X_SMS_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= OMAP343X_SDRC_VIRT,
 		.pfn		= __phys_to_pfn(OMAP343X_SDRC_PHYS),
 		.length		= OMAP343X_SDRC_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= L4_PER_34XX_VIRT,
 		.pfn		= __phys_to_pfn(L4_PER_34XX_PHYS),
 		.length		= L4_PER_34XX_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 	{
 		.virtual	= L4_EMU_34XX_VIRT,
 		.pfn		= __phys_to_pfn(L4_EMU_34XX_PHYS),
 		.length		= L4_EMU_34XX_SIZE,
-		.type		= MT_DEVICE
+		.type		= IO_MAP_TYPE
 	},
 };
 #endif
--- arch/arm/mach-omap2/pm_34xx.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-12-05 12:22:20.000000000 -0600
+++ arch/arm/mach-omap2/pm_34xx.c	2008-12-09 20:16:35.000000000 -0600
@@ -213,6 +213,7 @@ void omap3_save_per_context(void)
 void omap3_restore_per_context(void)
 {
 	omap_gpio_restore();
+	prcm_wait_for_clock(PRCM_UART3);
 	omap_uart_restore_ctx(2);
 }
 
@@ -272,7 +273,7 @@ void omap3_restore_core_settings(void)
 
 }
 
-void memory_logic_res_seting(void)
+void memory_logic_res_setting(void)
 {
 	res1_level = resource_get_level(memret1);
 	res2_level = resource_get_level(memret2);
@@ -379,7 +380,7 @@ static int omap3_pm_suspend(void)
 	target_state.core_state = PRCM_CORE_CSWR_MEMRET;
 #endif
 	if (target_state.core_state == PRCM_CORE_CSWR_MEMRET) {
-		memory_logic_res_seting();
+		memory_logic_res_setting();
 	}
 	if (core_off_notification != NULL)
 		core_off_notification(PRCM_TRUE);
--- arch/arm/mach-omap2/pm_34xx.h@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-11-06 21:38:44.000000000 -0600
+++ arch/arm/mach-omap2/pm_34xx.h	2008-12-09 20:16:38.000000000 -0600
@@ -31,12 +31,21 @@
 /* Context memory: w/oETM->u32[61], w/ETM->u32[213] */
 u32 context_mem[256];
 
-#define SCRATCHPAD_ROM  0x48002860
-#define SCRATCHPAD      0x48002910
-#define SCRATHPAD_ROM_OFFSET    0x19C
-#define TABLE_ADDRESS_OFFSET    0x31
-#define TABLE_VALUE_OFFSET      0x30
-#define CONTROL_REG_VALUE_OFFSET        0x32
+/*
+ * Read-Only Pad conf:
+ *   0x4800 2600 -> 4800 28A0
+ *          2600 ->      286C (padconf SAR)
+ * Scratch PAD
+ *  Early errata requires clear on cold reset
+ *   0x4800 2870 ->      29FC (scratchpad)
+ *   0x4800 2894 (first writeable scratchpad)
+ */
+#define SCRATCHPAD_ROM            0x480028A4
+#define SCRATCHPAD                0x48002910
+#define SCRATHPAD_ROM_OFFSET      0x158 /* size to clear to 29fc */
+#define TABLE_ADDRESS_OFFSET      0x31
+#define TABLE_VALUE_OFFSET        0x30
+#define CONTROL_REG_VALUE_OFFSET  0x32
 
 struct system_power_state target_state;
 
--- arch/arm/mach-omap2/pm_idle_34xx.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-11-05 13:31:31.000000000 -0600
+++ arch/arm/mach-omap2/pm_idle_34xx.c	2008-12-09 20:16:40.000000000 -0600
@@ -450,7 +450,7 @@ static void correct_target_state(void)
 	} else
 		target_state.per_state = PRCM_ON;
 		if (target_state.core_state == PRCM_CORE_CSWR_MEMRET)
-			memory_logic_res_seting();
+			memory_logic_res_setting();
 }
 
 static int omap3_enter_idle(struct cpuidle_device *dev,
@@ -741,12 +741,12 @@ restore:
 
 		if (target_state.core_state >= PRCM_CORE_OSWR_MEMRET) {
 #ifdef CONFIG_OMAP34XX_OFFMODE
-		context_restore_update(DOM_CORE1);
+			context_restore_update(DOM_CORE1);
 #endif
 			prcm_restore_registers(&target_state);
 			prcm_restore_core_context(target_state.core_state);
 			omap3_restore_core_settings();
-			}
+		}
 		/* Errata 1.4
 		* if the timer device gets idled which is when we
 		* are cutting the timer ICLK which is when we try
--- arch/arm/mach-omap2/pm_idle_34xx.h@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-09-10 10:56:03.000000000 -0500
+++ arch/arm/mach-omap2/pm_idle_34xx.h	2008-12-09 20:16:43.000000000 -0600
@@ -21,7 +21,7 @@ extern void omap3_restore_neon_context(v
 extern void omap3_save_per_context(void);
 extern void omap3_restore_per_context(void);
 extern void omap3_push_sram_functions(void);
-extern void memory_logic_res_seting(void);
+extern void memory_logic_res_setting(void);
 extern void omap3_restore_core_settings(void);
 extern void enable_smartreflex(int srid);
 extern void disable_smartreflex(int srid);
--- arch/arm/mach-omap2/prcm_34xx.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-12-05 13:49:26.000000000 -0600
+++ arch/arm/mach-omap2/prcm_34xx.c	2008-12-09 20:16:45.000000000 -0600
@@ -1355,6 +1355,19 @@ int prcm_is_device_accessible(u32 device
 }
 
 /*============================================================================*/
+/*======================== WAIT FOR CLOCK  ===================================*/
+/*============================================================================*/
+/*= Provide simple dead wait for access on clocks.                           =*/
+/*============================================================================*/
+void prcm_wait_for_clock(u32 deviceid)
+{
+	u8 ok;
+	do {
+		prcm_is_device_accessible(deviceid, &ok);
+	} while (ok != PRCM_TRUE);
+}
+
+/*============================================================================*/
 /*======================== INTERFACE CLOCK AUTOIDLE  =========================*/
 /*============================================================================*/
 /*= This command will either enable or disable the interface clock AutoIdle  =*/
@@ -4639,14 +4652,6 @@ int prcm_save_registers(struct system_po
 /* Restore registers */
 int prcm_restore_registers(struct system_power_state *target_state)
 {
-	PRCM_RESTORE(INTC_MIR_0);
-	PRCM_RESTORE(INTC_MIR_1);
-	PRCM_RESTORE(INTC_MIR_2);
-	PRCM_RESTORE(CONTROL_PADCONF_SYS_NIRQ);
-	PRCM_RESTORE(GPIO1_IRQENABLE1);
-	PRCM_RESTORE(GPIO1_WAKEUPENABLE);
-	PRCM_RESTORE(GPIO1_FALLINGDETECT);
-
 	PRCM_RESTORE(CM_CLKSEL2_PLL_IVA2);
 	PRCM_RESTORE(CM_SYSCONFIG);
 
@@ -4783,6 +4788,15 @@ int prcm_restore_registers(struct system
 	PRCM_RESTORE(PM_IVA2GRPSEL_USBHOST);
 #endif
 	PRCM_RESTORE(PM_WKEN_WKUP);
+
+	PRCM_RESTORE(INTC_MIR_0);
+	PRCM_RESTORE(INTC_MIR_1);
+	PRCM_RESTORE(INTC_MIR_2);
+	PRCM_RESTORE(CONTROL_PADCONF_SYS_NIRQ);
+	PRCM_RESTORE(GPIO1_IRQENABLE1);
+	PRCM_RESTORE(GPIO1_WAKEUPENABLE);
+	PRCM_RESTORE(GPIO1_FALLINGDETECT);
+
 	return PRCM_PASS;
 }
 
--- arch/arm/mm/mmu.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-05-07 14:13:00.000000000 -0500
+++ arch/arm/mm/mmu.c	2008-12-09 20:16:48.000000000 -0600
@@ -231,7 +231,13 @@ static struct mem_type mem_types[] = {
 		.domain    = DOMAIN_USER,
 	},
 	[MT_MEMORY_SO] = {
-		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_SECT_UNCACHED,
+		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE |
+				PMD_SECT_UNCACHED | PMD_SECT_XN,
+		.domain    = DOMAIN_KERNEL,
+	},
+	[MT_MEMORY_SO_EXE] = {
+		.prot_sect = PMD_TYPE_SECT | PMD_SECT_AP_WRITE |
+				PMD_SECT_UNCACHED,
 		.domain    = DOMAIN_KERNEL,
 	},
 	[MT_MEMORY] = {
--- arch/arm/plat-omap/gpio.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-12-05 12:22:50.000000000 -0600
+++ arch/arm/plat-omap/gpio.c	2008-12-09 20:16:51.000000000 -0600
@@ -1505,7 +1505,11 @@ static int __init _omap_gpio_init(void)
 #if defined(CONFIG_ARCH_OMAP24XX) || defined(CONFIG_ARCH_OMAP34XX)
 		if (bank->method == METHOD_GPIO_24XX) {
 			static const u32 non_wakeup_gpios[] = {
+#if defined(CONFIG_ARCH_OMAP24XX)
 				0xe203ffc0, 0x08700040
+#else
+				0
+#endif
 			};
 
 			__raw_writel(0x00000000, bank->base + OMAP24XX_GPIO_IRQENABLE1);
--- arch/arm/plat-omap/sram.c@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-10-02 05:20:19.000000000 -0500
+++ arch/arm/plat-omap/sram.c	2008-12-09 20:16:54.000000000 -0600
@@ -195,7 +195,7 @@ static struct map_desc omap_sram_io_desc
 	{	/* .length gets filled in at runtime */
 		.virtual	= OMAP1_SRAM_VA,
 		.pfn		= __phys_to_pfn(OMAP1_SRAM_PA),
-		.type		= MT_MEMORY_SO
+		.type		= MT_MEMORY_SO_EXE
 	}
 };
 
@@ -258,7 +258,8 @@ void * omap_sram_push(void * start, unsi
 	omap_sram_ceil -= size;
 	omap_sram_ceil = ROUND_DOWN(omap_sram_ceil, sizeof(void *));
 	memcpy((void *)omap_sram_ceil, start, size);
-	flush_icache_range((unsigned long)start, (unsigned long)(start + size));
+	flush_icache_range((unsigned long)omap_sram_ceil,
+			(unsigned long)(omap_sram_ceil + size));
 
 	return (void *)omap_sram_ceil;
 }
--- include/asm-arm/arch-omap/io.h@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-04-04 01:26:58.000000000 -0500
+++ include/asm-arm/arch-omap/io.h	2008-12-09 20:16:56.000000000 -0600
@@ -107,6 +107,17 @@
 
 #elif defined(CONFIG_ARCH_OMAP3)
 
+/* Select ARM view IO behavior */
+#ifdef CONFIG_INTERCONNECT_IO_POSTING
+/* ARM writes to devices are postable.  Further software 
+ * sychronization neeed ex: DSB or register read back 
+ */
+#define IO_MAP_TYPE	MT_DEVICE
+#else
+/* ARM writes to devices are sychronized */
+#define IO_MAP_TYPE	MT_MEMORY_SO
+#endif
+
 /* We map both L3 and L4 on OMAP3 */
 #define L3_34XX_PHYS		L3_34XX_BASE	/* 0x68000000 */
 #define L3_34XX_VIRT		0xf8000000
--- include/asm-arm/arch-omap/prcm_34xx.h@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-11-05 13:31:35.000000000 -0600
+++ include/asm-arm/arch-omap/prcm_34xx.h	2008-12-09 20:16:59.000000000 -0600
@@ -1273,6 +1273,7 @@ extern struct dvfs_config omap3_vdd2_con
 extern int prcm_clock_control(u32 deviceid, u8 clk_type, u8 control,
 			      u8 checkaccessibility);
 extern int prcm_is_device_accessible(u32 deviceid, u8 *result);
+extern void prcm_wait_for_clock(u32 deviceid);
 extern int prcm_interface_clock_autoidle(u32 deviceid, u8 control);
 extern int prcm_wakeup_event_control(u32 deviceid, u8 control);
 
--- include/asm-arm/mach/map.h@@/LINUX-GIT-2.6K_INT_FLOAT_4.0	2008-04-11 23:03:54.000000000 -0500
+++ include/asm-arm/mach/map.h	2008-12-09 20:17:02.000000000 -0600
@@ -26,6 +26,7 @@ struct map_desc {
 #define MT_MEMORY		8
 #define MT_ROM			9
 #define MT_MEMORY_SO		10
+#define MT_MEMORY_SO_EXE	11
 
 #define MT_NONSHARED_DEVICE	MT_DEVICE_NONSHARED
 #define MT_IXP2000_DEVICE	MT_DEVICE_IXP2000