omap3 cpuidle interrupt latency

omap3 cpuidle interrupt latency

[Premi, Sanjeev]

I have noticed large interrupt latency when the cpuidle is enabled.
e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
(I am at HEAD of the 'pm' branch)
 
The IRQs and FIQs are disabled at the beginning of the function
omap3_enter_idle() but WFI is executed much later in _omap_sram_idle().
In between, there is only one check for pending IRQs - omap_irq_pending()
 
If any interrupt occurs beyond this point is it considered by the WFI?
 
To reduce this latency, I am planning to do either/both of thse:
- Add more checks for pending IRQs
- Reduce the time for which the IRQs and FIQs are disabled
 
Benefits will depend upon the behavior of WFI.
 
Best regards,
Sanjeev

RE: omap3 cpuidle interrupt latency

[Dasgupta, Romit]

I think the original code had checks in 3 places but as you say it might have changed in linux-omap. I feel the right way to do this is to add a latency (using pm_qos) so that the CPU idle driver doesn't even attempt deeper idle states. Ideally we should use PM_QOS_NETWORK_LATENCY but the menu governor seems to check only PM_QOS_CPU_DMA_LATENCY class. I am not sure why menu governor does not check the other classes of latency. But for now I think if we add to PM_QOS_CPU_DMA_LATENCY when we detect some network activity and remove latency req after a while of network inactivity. 

Thanks,
-Romit

________________________________________
From: linux-omap-owner@vger.kernel.org [linux-omap-owner@vger.kernel.org] On Behalf Of Premi, Sanjeev [premi@ti.com]
Sent: Saturday, February 28, 2009 11:22 PM
To: linux-omap@vger.kernel.org
Subject: omap3 cpuidle interrupt latency

I have noticed large interrupt latency when the cpuidle is enabled.
e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
(I am at HEAD of the 'pm' branch)

The IRQs and FIQs are disabled at the beginning of the function
omap3_enter_idle() but WFI is executed much later in _omap_sram_idle().
In between, there is only one check for pending IRQs - omap_irq_pending()

If any interrupt occurs beyond this point is it considered by the WFI?

To reduce this latency, I am planning to do either/both of thse:
- Add more checks for pending IRQs
- Reduce the time for which the IRQs and FIQs are disabled

Benefits will depend upon the behavior of WFI.

Best regards,
Sanjeev

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: omap3 cpuidle interrupt latency

[Woodruff, Richard]

Part of it is a poor prediction.  A while back, I had hacked in a simple irq time stamp and used a kind of crude running average over all irq sources. Then I added this info into cpuidle for next state prediction. It did improve latency a quite a bit.  Simple idea is irqs might cluster and there is always a little activity after irq.

The effect of this is easily seen on musb driver. It uses some kind of streaming api where data is only sent up after a bunch of irqs have brought in data above some threshold.  CPUidle doesn't take this kind of thing into account well and will choose a poor cstate which causes effective throughput to drop.

I did post some of this code on linux pm.  I did get a response from one power person at Intel.  He indicated that cpuidle wasn't so good at in this aspect and that they were looking at a similar but different approach to the issue.  He didn't like the idea of extra overhead on each irq.  It was pretty minor to me.

It would be nice if this device could dynamically set constraints (say by using a short kernel timer during activity or set/remove pm_qos latency) or the prediction aspect of cpuidle could be improved using rules feed from things like interrupt source.

Regards
Richard W.

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
> owner@vger.kernel.org] On Behalf Of Dasgupta, Romit
> Sent: Saturday, February 28, 2009 9:25 PM
> To: Premi, Sanjeev; linux-omap@vger.kernel.org
> Subject: RE: omap3 cpuidle interrupt latency
>
> I think the original code had checks in 3 places but as you say it might have
> changed in linux-omap. I feel the right way to do this is to add a latency
> (using pm_qos) so that the CPU idle driver doesn't even attempt deeper idle
> states. Ideally we should use PM_QOS_NETWORK_LATENCY but the menu governor
> seems to check only PM_QOS_CPU_DMA_LATENCY class. I am not sure why menu
> governor does not check the other classes of latency. But for now I think if
> we add to PM_QOS_CPU_DMA_LATENCY when we detect some network activity and
> remove latency req after a while of network inactivity.
>
> Thanks,
> -Romit
>
> ________________________________________
> From: linux-omap-owner@vger.kernel.org [linux-omap-owner@vger.kernel.org] On
> Behalf Of Premi, Sanjeev [premi@ti.com]
> Sent: Saturday, February 28, 2009 11:22 PM
> To: linux-omap@vger.kernel.org
> Subject: omap3 cpuidle interrupt latency
>
> I have noticed large interrupt latency when the cpuidle is enabled.
> e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
> (I am at HEAD of the 'pm' branch)
>
> The IRQs and FIQs are disabled at the beginning of the function
> omap3_enter_idle() but WFI is executed much later in _omap_sram_idle().
> In between, there is only one check for pending IRQs - omap_irq_pending()
>
> If any interrupt occurs beyond this point is it considered by the WFI?
>
> To reduce this latency, I am planning to do either/both of thse:
> - Add more checks for pending IRQs
> - Reduce the time for which the IRQs and FIQs are disabled
>
> Benefits will depend upon the behavior of WFI.
>
> Best regards,
> Sanjeev
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: omap3 cpuidle interrupt latency

[Dasgupta, Romit]

It would be nice if this device could dynamically set constraints (say by using a short kernel timer during activity or set/remove pm_qos latency) or the prediction aspect of cpuidle could be improved using rules feed from things like interrupt source.

[Romit] I was thinking that in the irq handler we should add a requirement via the pm_qos or via the constraint framework (not sure if linux-omap has this) and set some kind of a timer (the idea is to keep arming the timer for clustered or rather closely spaced irq instances). If after a while the timer expires we release the latency req. This way perhaps we can handle the latency problem. Again, I am not sure why the menu governor only looks into the PM_QOS_CPU_DMA_LATENCY class only. I was thinking that it should look into other classes as well.

Thanks,
-Romit

RE: omap3 cpuidle interrupt latency

[Premi, Sanjeev]

> -----Original Message-----
> From: Dasgupta, Romit 
> Sent: Sunday, March 01, 2009 12:18 PM
> To: Woodruff, Richard; Premi, Sanjeev; linux-omap@vger.kernel.org
> Subject: RE: omap3 cpuidle interrupt latency
> 
> It would be nice if this device could dynamically set 
> constraints (say by using a short kernel timer during 
> activity or set/remove pm_qos latency) or the prediction 
> aspect of cpuidle could be improved using rules feed from 
> things like interrupt source.

Despite all constraints; the latency concern is still valid.

The constraints would either prevent idle state transition;
or in the idle processing we could still have repeat of current
situation - though less often.

I feel checking for pending interrupt before executing WFI
would help. Will try in the morning.

> [Romit] I was thinking that in the irq handler we should add 
> a requirement via the pm_qos or via the constraint framework 
> (not sure if linux-omap has this) and set some kind of a 
> timer (the idea is to keep arming the timer for clustered or 
> rather closely spaced irq instances). If after a while the 
> timer expires we release the latency req. This way perhaps we 
> can handle the latency problem. Again, I am not sure why the 
> menu governor only looks into the PM_QOS_CPU_DMA_LATENCY 
> class only. I was thinking that it should look into other 
> classes as well.
> 
> Thanks,
> -Romit

RE: omap3 cpuidle interrupt latency

[TK, Pratheesh Gangadhar]

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
> owner@vger.kernel.org] On Behalf Of Premi, Sanjeev
> Sent: Monday, March 02, 2009 1:31 AM
> To: Dasgupta, Romit; Woodruff, Richard; linux-omap@vger.kernel.org
> Subject: RE: omap3 cpuidle interrupt latency
> 
> > -----Original Message-----
> > From: Dasgupta, Romit
> > Sent: Sunday, March 01, 2009 12:18 PM
> > To: Woodruff, Richard; Premi, Sanjeev; linux-omap@vger.kernel.org
> > Subject: RE: omap3 cpuidle interrupt latency
> >
> > It would be nice if this device could dynamically set
> > constraints (say by using a short kernel timer during
> > activity or set/remove pm_qos latency) or the prediction
> > aspect of cpuidle could be improved using rules feed from
> > things like interrupt source.
> 
> Despite all constraints; the latency concern is still valid.
> 
> The constraints would either prevent idle state transition;
> or in the idle processing we could still have repeat of current
> situation - though less often.
> 
> I feel checking for pending interrupt before executing WFI
> would help. Will try in the morning.

Moving time keeping functions out of interrupt disabled context may also help. Something like this

return_sleep_time:
        local_irq_enable();
        local_fiq_enable();
        getnstimeofday(&ts_postidle);
        ts_idle = timespec_sub(ts_postidle, ts_preidle);

ARM will wakeup from WFI idle on nIRQ/nFIQ signal assertion. But interrupts will be processed only after they are enabled in CPSR AFAIK.

Regards,
Pratheesh

Re: omap3 cpuidle interrupt latency

[Kevin Hilman]

"Premi, Sanjeev" <premi@ti.com> writes:

> I have noticed large interrupt latency when the cpuidle is enabled.
> e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
> (I am at HEAD of the 'pm' branch)

Is it interrupt latency in general you are measuring?  or just the
interrupt latency for the smc network driver.  I think what you are
seeing is the result of the SMC IRQ not being configured as a wakeup
source, thus a network interrupt will not wake the system, but you end
up waiting for the next idle timer until the system wakes and handles
the network interrupt.

By default, I don't believe the GPIO interrupt used by the smc is
configured as a wakeup source.  Have you configured that GPIO as a
wakeup source?

Kevin


> The IRQs and FIQs are disabled at the beginning of the function
> omap3_enter_idle() but WFI is executed much later in _omap_sram_idle().
> In between, there is only one check for pending IRQs - omap_irq_pending()
>  
> If any interrupt occurs beyond this point is it considered by the WFI?
>  
> To reduce this latency, I am planning to do either/both of thse:
> - Add more checks for pending IRQs
> - Reduce the time for which the IRQs and FIQs are disabled
>  
> Benefits will depend upon the behavior of WFI.
>  
> Best regards,
> Sanjeev
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: omap3 cpuidle interrupt latency

[Premi, Sanjeev]

> -----Original Message-----
> From: Kevin Hilman [mailto:khilman@deeprootsystems.com] 
> Sent: Tuesday, March 03, 2009 1:16 AM
> To: Premi, Sanjeev
> Cc: linux-omap@vger.kernel.org
> Subject: Re: omap3 cpuidle interrupt latency
> 
> "Premi, Sanjeev" <premi@ti.com> writes:
> 
> > I have noticed large interrupt latency when the cpuidle is enabled.
> > e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
> > (I am at HEAD of the 'pm' branch)
> 
> Is it interrupt latency in general you are measuring?  or 
> just the interrupt latency for the smc network driver.  I 
> think what you are seeing is the result of the SMC IRQ not 
> being configured as a wakeup source, thus a network interrupt 
> will not wake the system, but you end up waiting for the next 
> idle timer until the system wakes and handles the network interrupt.

I used SMC as an example. The read over USB is also slow...

I feel there is generic latency. Initially, I was suspecting
the hrtimer functions taking long time. But that may not be
the case.

> 
> By default, I don't believe the GPIO interrupt used by the 
> smc is configured as a wakeup source.  Have you configured 
> that GPIO as a wakeup source?

My current measurements are based with all idle related flags
'sleep_while_idle', 'clocks_off_while_idle' and 'enable_off_mode'
set at 0.

Does WFI require a wakeup event? Even when system is not in 'off' stare?

~sanjeev

> 
> Kevin
> 
> 
> > The IRQs and FIQs are disabled at the beginning of the function
> > omap3_enter_idle() but WFI is executed much later in 
> _omap_sram_idle().
> > In between, there is only one check for pending IRQs - 
> > omap_irq_pending()
> >  
> > If any interrupt occurs beyond this point is it considered 
> by the WFI?
> >  
> > To reduce this latency, I am planning to do either/both of thse:
> > - Add more checks for pending IRQs
> > - Reduce the time for which the IRQs and FIQs are disabled
> >  
> > Benefits will depend upon the behavior of WFI.
> >  
> > Best regards,
> > Sanjeev
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" 
> > in the body of a message to majordomo@vger.kernel.org More 
> majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: omap3 cpuidle interrupt latency

[Premi, Sanjeev]

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of Premi, Sanjeev
> Sent: Tuesday, March 03, 2009 3:54 PM
> To: Kevin Hilman
> Cc: linux-omap@vger.kernel.org
> Subject: RE: omap3 cpuidle interrupt latency
> 
> > -----Original Message-----
> > From: Kevin Hilman [mailto:khilman@deeprootsystems.com]
> > Sent: Tuesday, March 03, 2009 1:16 AM
> > To: Premi, Sanjeev
> > Cc: linux-omap@vger.kernel.org
> > Subject: Re: omap3 cpuidle interrupt latency
> > 
> > "Premi, Sanjeev" <premi@ti.com> writes:
> > 
> > > I have noticed large interrupt latency when the cpuidle 
> is enabled.
> > > e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
> > > (I am at HEAD of the 'pm' branch)
> > 
> > Is it interrupt latency in general you are measuring?  or just the 
> > interrupt latency for the smc network driver.  I think what you are 
> > seeing is the result of the SMC IRQ not being configured as 
> a wakeup 
> > source, thus a network interrupt will not wake the system, 
> but you end 
> > up waiting for the next idle timer until the system wakes 
> and handles 
> > the network interrupt.
> 
> I used SMC as an example. The read over USB is also slow...
> 
> I feel there is generic latency. Initially, I was suspecting 
> the hrtimer functions taking long time. But that may not be the case.
> 
> > 
> > By default, I don't believe the GPIO interrupt used by the smc is 
> > configured as a wakeup source.  Have you configured that GPIO as a 
> > wakeup source?
> 
> My current measurements are based with all idle related flags 
> 'sleep_while_idle', 'clocks_off_while_idle' and 'enable_off_mode'
> set at 0.
> 
> Does WFI require a wakeup event? Even when system is not in 
> 'off' stare?

For simple debug, I replaced the _omap_sram_idle() processing with:
	__asm__ ("dmb");
	__asm__ ("dsb");
	__asm__ ("wfi");

Here, again behavior was same for ethernet.

For more simplification, I focused on keypad and touchscreen.
1) Let the system stay idle for about 10secs
2) Press the keypad & tap the touchscreen
3) cat /proc/interrupts

While the keypad interrupts (via SYS_NIRQ) are getting updated;
most of the TS interrupts (via GPIO) seem to be missing; With a
busy loop in the background OR _omap_sram_idle() commented - not
doing WFI - changes the behavior. Each tap on the TS is registered
in /proc/interrupts.

> 
> ~sanjeev
> 
> > 
> > Kevin
> > 
> > 
> > > The IRQs and FIQs are disabled at the beginning of the function
> > > omap3_enter_idle() but WFI is executed much later in
> > _omap_sram_idle().
> > > In between, there is only one check for pending IRQs -
> > > omap_irq_pending()
> > >  
> > > If any interrupt occurs beyond this point is it considered
> > by the WFI?
> > >  
> > > To reduce this latency, I am planning to do either/both of thse:
> > > - Add more checks for pending IRQs
> > > - Reduce the time for which the IRQs and FIQs are disabled
> > >  
> > > Benefits will depend upon the behavior of WFI.
> > >  
> > > Best regards,
> > > Sanjeev
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > linux-omap" 
> > > in the body of a message to majordomo@vger.kernel.org More
> > majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> 
> 

Re: omap3 cpuidle interrupt latency

[Kevin Hilman]

Premi, Sanjeev wrote:
>> -----Original Message-----
>> From: linux-omap-owner@vger.kernel.org 
>> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of Premi, Sanjeev
>> Sent: Tuesday, March 03, 2009 3:54 PM
>> To: Kevin Hilman
>> Cc: linux-omap@vger.kernel.org
>> Subject: RE: omap3 cpuidle interrupt latency
>>
>>> -----Original Message-----
>>> From: Kevin Hilman [mailto:khilman@deeprootsystems.com]
>>> Sent: Tuesday, March 03, 2009 1:16 AM
>>> To: Premi, Sanjeev
>>> Cc: linux-omap@vger.kernel.org
>>> Subject: Re: omap3 cpuidle interrupt latency
>>>
>>> "Premi, Sanjeev" <premi@ti.com> writes:
>>>
>>>> I have noticed large interrupt latency when the cpuidle 
>> is enabled.
>>>> e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
>>>> (I am at HEAD of the 'pm' branch)
>>> Is it interrupt latency in general you are measuring?  or just the 
>>> interrupt latency for the smc network driver.  I think what you are 
>>> seeing is the result of the SMC IRQ not being configured as 
>> a wakeup 
>>> source, thus a network interrupt will not wake the system, 
>> but you end 
>>> up waiting for the next idle timer until the system wakes 
>> and handles 
>>> the network interrupt.
>> I used SMC as an example. The read over USB is also slow...
>>
>> I feel there is generic latency. Initially, I was suspecting 
>> the hrtimer functions taking long time. But that may not be the case.
>>
>>> By default, I don't believe the GPIO interrupt used by the smc is 
>>> configured as a wakeup source.  Have you configured that GPIO as a 
>>> wakeup source?
>> My current measurements are based with all idle related flags 
>> 'sleep_while_idle', 'clocks_off_while_idle' and 'enable_off_mode'
>> set at 0.
>>
>> Does WFI require a wakeup event? Even when system is not in 
>> 'off' stare?
> 
> For simple debug, I replaced the _omap_sram_idle() processing with:
> 	__asm__ ("dmb");
> 	__asm__ ("dsb");
> 	__asm__ ("wfi");
> 
> Here, again behavior was same for ethernet.
> 
> For more simplification, I focused on keypad and touchscreen.
> 1) Let the system stay idle for about 10secs
> 2) Press the keypad & tap the touchscreen
> 3) cat /proc/interrupts
> 
> While the keypad interrupts (via SYS_NIRQ) are getting updated;
> most of the TS interrupts (via GPIO) seem to be missing; With a
> busy loop in the background OR _omap_sram_idle() commented - not
> doing WFI - changes the behavior. Each tap on the TS is registered
> in /proc/interrupts.
> 

IIUC, the keypad interrupts are making it all the way to the kernel, but 
the touchscreen ones are not, correct?

What happens if you enable IRQ wakeups for the TS interrupt?

Kevin

RE: omap3 cpuidle interrupt latency

[Premi, Sanjeev]

> -----Original Message-----
> From: Premi, Sanjeev 
> Sent: Wednesday, March 04, 2009 7:52 PM
> To: Premi, Sanjeev; Kevin Hilman
> Cc: linux-omap@vger.kernel.org
> Subject: RE: omap3 cpuidle interrupt latency
> 
> > -----Original Message-----
> > From: linux-omap-owner@vger.kernel.org 
> > [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of 
> Premi, Sanjeev
> > Sent: Tuesday, March 03, 2009 3:54 PM
> > To: Kevin Hilman
> > Cc: linux-omap@vger.kernel.org
> > Subject: RE: omap3 cpuidle interrupt latency
> > 
> > > -----Original Message-----
> > > From: Kevin Hilman [mailto:khilman@deeprootsystems.com]
> > > Sent: Tuesday, March 03, 2009 1:16 AM
> > > To: Premi, Sanjeev
> > > Cc: linux-omap@vger.kernel.org
> > > Subject: Re: omap3 cpuidle interrupt latency
> > > 
> > > "Premi, Sanjeev" <premi@ti.com> writes:
> > > 
> > > > I have noticed large interrupt latency when the cpuidle
> > is enabled.
> > > > e.g. response time for ping goes from avg 10-20ms to 800-1000ms.
> > > > (I am at HEAD of the 'pm' branch)
> > > 
> > > Is it interrupt latency in general you are measuring?  or 
> just the 
> > > interrupt latency for the smc network driver.  I think 
> what you are 
> > > seeing is the result of the SMC IRQ not being configured as
> > a wakeup
> > > source, thus a network interrupt will not wake the system,
> > but you end
> > > up waiting for the next idle timer until the system wakes
> > and handles
> > > the network interrupt.
> > 
> > I used SMC as an example. The read over USB is also slow...
> > 
> > I feel there is generic latency. Initially, I was suspecting the 
> > hrtimer functions taking long time. But that may not be the case.
> > 
> > > 
> > > By default, I don't believe the GPIO interrupt used by the smc is 
> > > configured as a wakeup source.  Have you configured that 
> GPIO as a 
> > > wakeup source?
> > 
> > My current measurements are based with all idle related flags 
> > 'sleep_while_idle', 'clocks_off_while_idle' and 'enable_off_mode'
> > set at 0.
> > 
> > Does WFI require a wakeup event? Even when system is not in 'off' 
> > stare?
> 
> For simple debug, I replaced the _omap_sram_idle() processing with:
> 	__asm__ ("dmb");
> 	__asm__ ("dsb");
> 	__asm__ ("wfi");
> 
> Here, again behavior was same for ethernet.
> 
> For more simplification, I focused on keypad and touchscreen.
> 1) Let the system stay idle for about 10secs
> 2) Press the keypad & tap the touchscreen
> 3) cat /proc/interrupts
> 
> While the keypad interrupts (via SYS_NIRQ) are getting 
> updated; most of the TS interrupts (via GPIO) seem to be 
> missing; With a busy loop in the background OR 
> _omap_sram_idle() commented - not doing WFI - changes the 
> behavior. Each tap on the TS is registered in /proc/interrupts.

On Kevin's suggestion, I tried to disable cpuidle and repeat the
experiment with omap_pm_idle () - with minor mods to ensure that
WFI is executed.

Results was same. While the UART is active (via 5 sec timeout)
all taps on the touchscreen were detected. If the system is kept
idle for about 10 secs, the interrupts are lost.

~sanjeev

> 
> > 
> > ~sanjeev
> > 
> > > 
> > > Kevin
> > > 
> > > 
> > > > The IRQs and FIQs are disabled at the beginning of the function
> > > > omap3_enter_idle() but WFI is executed much later in
> > > _omap_sram_idle().
> > > > In between, there is only one check for pending IRQs -
> > > > omap_irq_pending()
> > > >  
> > > > If any interrupt occurs beyond this point is it considered
> > > by the WFI?
> > > >  
> > > > To reduce this latency, I am planning to do either/both of thse:
> > > > - Add more checks for pending IRQs
> > > > - Reduce the time for which the IRQs and FIQs are disabled
> > > >  
> > > > Benefits will depend upon the behavior of WFI.
> > > >  
> > > > Best regards,
> > > > Sanjeev
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe
> > > linux-omap" 
> > > > in the body of a message to majordomo@vger.kernel.org More
> > > majordomo
> > > > info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > --
> > To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" 
> > in the body of a message to majordomo@vger.kernel.org More 
> majordomo 
> > info at http://vger.kernel.org/majordomo-info.html
> > 
> >