reboot not working on linux-2.6.37 for ARMv7

reboot not working on linux-2.6.37 for ARMv7

[shiraz hashim]

Hi,

I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP platform
with PL310 as Level 2 cache. We observe that on reboot the control
is not able to reach to the arch_reset.

The problem happens when L1 cache is disabled in
arm_machine_restart, through cpu_proc_fin() and L2 cache is flushed.
L2 cache flushing takes a spinlock and our platform doesnot have monitor
support at L2 thus failing strex instruction. The strex (and hence spin locks)
behave sanely when L1 is enabled as monitor associated with SCU takes
care of it.

-- 
regards
Shiraz Hashim

reboot not working on linux-2.6.37 for ARMv7

[Russell King - ARM Linux]

On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
> Hi,
> 
> I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP platform
> with PL310 as Level 2 cache. We observe that on reboot the control
> is not able to reach to the arch_reset.
> 
> The problem happens when L1 cache is disabled in
> arm_machine_restart, through cpu_proc_fin() and L2 cache is flushed.

Where's the L2 cache flush?  The sequence is:

        /* Clean and invalidate caches */
        flush_cache_all();
        /* Turn off caching */
        cpu_proc_fin();
        /* Push out any further dirty data, and ensure cache is empty */
        flush_cache_all();

and flush_cache_all() calls v7_flush_kern_cache_all() in
arch/arm/mm/cache-v7.S.

I do hope you're not modifying flush_cache_all() to also call the L2
cache functions because that's wrong.

reboot not working on linux-2.6.37 for ARMv7

[Santosh Shilimkar]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-
> arm-kernel-bounces at lists.infradead.org] On Behalf Of Russell King -
> ARM Linux
> Sent: Friday, February 11, 2011 5:33 PM
> To: shiraz hashim
> Cc: Armando VISCONTI; amit.goel at st.com; vipin.kumar at st.com; linux-
> arm-kernel at lists.infradead.org
> Subject: Re: reboot not working on linux-2.6.37 for ARMv7
>
> On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
> > Hi,
> >
> > I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP
> platform
> > with PL310 as Level 2 cache. We observe that on reboot the control
> > is not able to reach to the arch_reset.
> >
> > The problem happens when L1 cache is disabled in
> > arm_machine_restart, through cpu_proc_fin() and L2 cache is
> flushed.
>
> Where's the L2 cache flush?  The sequence is:
>
>         /* Clean and invalidate caches */
>         flush_cache_all();
>         /* Turn off caching */
>         cpu_proc_fin();
>         /* Push out any further dirty data, and ensure cache is
> empty */
>         flush_cache_all();
>
> and flush_cache_all() calls v7_flush_kern_cache_all() in
> arch/arm/mm/cache-v7.S.
>
> I do hope you're not modifying flush_cache_all() to also call the L2
> cache functions because that's wrong.
>
I don't want to hijack this thread. But don't we need L2 flush too in
reboot path ? If the arch_reset function fails because of the data has
not made it main memory and stuck in L2 which is used as part of this
function. With C bit disabled, there won't be any look up done in L2.

May be I am missing something here.

Regards,
Santosh

reboot not working on linux-2.6.37 for ARMv7

[Russell King - ARM Linux]

On Fri, Feb 11, 2011 at 05:40:46PM +0530, Santosh Shilimkar wrote:
> > -----Original Message-----
> > From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-
> > arm-kernel-bounces at lists.infradead.org] On Behalf Of Russell King -
> > ARM Linux
> > Sent: Friday, February 11, 2011 5:33 PM
> > To: shiraz hashim
> > Cc: Armando VISCONTI; amit.goel at st.com; vipin.kumar at st.com; linux-
> > arm-kernel at lists.infradead.org
> > Subject: Re: reboot not working on linux-2.6.37 for ARMv7
> >
> > On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
> > > Hi,
> > >
> > > I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP
> > platform
> > > with PL310 as Level 2 cache. We observe that on reboot the control
> > > is not able to reach to the arch_reset.
> > >
> > > The problem happens when L1 cache is disabled in
> > > arm_machine_restart, through cpu_proc_fin() and L2 cache is
> > flushed.
> >
> > Where's the L2 cache flush?  The sequence is:
> >
> >         /* Clean and invalidate caches */
> >         flush_cache_all();
> >         /* Turn off caching */
> >         cpu_proc_fin();
> >         /* Push out any further dirty data, and ensure cache is
> > empty */
> >         flush_cache_all();
> >
> > and flush_cache_all() calls v7_flush_kern_cache_all() in
> > arch/arm/mm/cache-v7.S.
> >
> > I do hope you're not modifying flush_cache_all() to also call the L2
> > cache functions because that's wrong.
> >
> I don't want to hijack this thread. But don't we need L2 flush too in
> reboot path ? If the arch_reset function fails because of the data has
> not made it main memory and stuck in L2 which is used as part of this
> function. With C bit disabled, there won't be any look up done in L2.

We only actually need the flushes if we're going to do a soft-reboot.
Soft-reboot isn't supported on ARMv6 and ARMv7 (as the proc-*.S reset
doesn't contain the necessary code).  ARMv5 and lower don't have L2
caches, so its not a concern for them.

Hard-reboot doesn't actually need the cache flushes.  Unfortunately,
we've no way of knowing whether a platform implements soft-reboot or
hard-reboot.

reboot not working on linux-2.6.37 for ARMv7

[Santosh Shilimkar]

> -----Original Message-----
> From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
> Sent: Friday, February 11, 2011 5:50 PM
> To: Santosh Shilimkar
> Cc: shiraz hashim; Armando VISCONTI; amit.goel at st.com;
> vipin.kumar at st.com; linux-arm-kernel at lists.infradead.org
> Subject: Re: reboot not working on linux-2.6.37 for ARMv7
>
> On Fri, Feb 11, 2011 at 05:40:46PM +0530, Santosh Shilimkar wrote:
> > > -----Original Message-----
> > > From: linux-arm-kernel-bounces at lists.infradead.org
> [mailto:linux-
> > > arm-kernel-bounces at lists.infradead.org] On Behalf Of Russell
> King -
> > > ARM Linux
> > > Sent: Friday, February 11, 2011 5:33 PM
> > > To: shiraz hashim
> > > Cc: Armando VISCONTI; amit.goel at st.com; vipin.kumar at st.com;
> linux-
> > > arm-kernel at lists.infradead.org
> > > Subject: Re: reboot not working on linux-2.6.37 for ARMv7
> > >
> > > On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
> > > > Hi,
> > > >
> > > > I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP
> > > platform
> > > > with PL310 as Level 2 cache. We observe that on reboot the
> control
> > > > is not able to reach to the arch_reset.
> > > >
> > > > The problem happens when L1 cache is disabled in
> > > > arm_machine_restart, through cpu_proc_fin() and L2 cache is
> > > flushed.
> > >
> > > Where's the L2 cache flush?  The sequence is:
> > >
> > >         /* Clean and invalidate caches */
> > >         flush_cache_all();
> > >         /* Turn off caching */
> > >         cpu_proc_fin();
> > >         /* Push out any further dirty data, and ensure cache is
> > > empty */
> > >         flush_cache_all();
> > >
> > > and flush_cache_all() calls v7_flush_kern_cache_all() in
> > > arch/arm/mm/cache-v7.S.
> > >
> > > I do hope you're not modifying flush_cache_all() to also call
> the L2
> > > cache functions because that's wrong.
> > >
> > I don't want to hijack this thread. But don't we need L2 flush too
> in
> > reboot path ? If the arch_reset function fails because of the data
> has
> > not made it main memory and stuck in L2 which is used as part of
> this
> > function. With C bit disabled, there won't be any look up done in
> L2.
>
> We only actually need the flushes if we're going to do a soft-
> reboot.
> Soft-reboot isn't supported on ARMv6 and ARMv7 (as the proc-*.S
> reset
> doesn't contain the necessary code).  ARMv5 and lower don't have L2
> caches, so its not a concern for them.
>
> Hard-reboot doesn't actually need the cache flushes.  Unfortunately,
> we've no way of knowing whether a platform implements soft-reboot or
> hard-reboot.

Thanks for clarification.

Regards,
Santosh

reboot not working on linux-2.6.37 for ARMv7

[viresh kumar]

On 2/11/11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
>> I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP platform
>> with PL310 as Level 2 cache. We observe that on reboot the control
>> is not able to reach to the arch_reset.
>>
>> The problem happens when L1 cache is disabled in
>> arm_machine_restart, through cpu_proc_fin() and L2 cache is flushed.
>
> Where's the L2 cache flush?  The sequence is:
>
>         /* Clean and invalidate caches */
>         flush_cache_all();
>         /* Turn off caching */
>         cpu_proc_fin();
>         /* Push out any further dirty data, and ensure cache is empty */
>         flush_cache_all();
>
> and flush_cache_all() calls v7_flush_kern_cache_all() in
> arch/arm/mm/cache-v7.S.
>
> I do hope you're not modifying flush_cache_all() to also call the L2
> cache functions because that's wrong.
>

Russell,

We are not modifying flush_cache_all. L2 cache is not flushed from there.
Actually as soon as L1 Cache is flushed and disabled, spin_lock is taken from
some other thread. This creates problem for our platform, as we do not have
monitors support at L2 thus failing strex instruction.

If we disable L2 then reboot works fine, as expected.

--
viresh

reboot not working on linux-2.6.37 for ARMv7

[Russell King - ARM Linux]

On Sat, Feb 12, 2011 at 09:45:45PM +0530, viresh kumar wrote:
> On 2/11/11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> > On Fri, Feb 11, 2011 at 05:01:06PM +0530, shiraz hashim wrote:
> >> I am using linux-2.6.37 on our ARM Cortex A9 (dual core) SMP platform
> >> with PL310 as Level 2 cache. We observe that on reboot the control
> >> is not able to reach to the arch_reset.
> >>
> >> The problem happens when L1 cache is disabled in
> >> arm_machine_restart, through cpu_proc_fin() and L2 cache is flushed.
> >
> > Where's the L2 cache flush?  The sequence is:
> >
> >         /* Clean and invalidate caches */
> >         flush_cache_all();
> >         /* Turn off caching */
> >         cpu_proc_fin();
> >         /* Push out any further dirty data, and ensure cache is empty */
> >         flush_cache_all();
> >
> > and flush_cache_all() calls v7_flush_kern_cache_all() in
> > arch/arm/mm/cache-v7.S.
> >
> > I do hope you're not modifying flush_cache_all() to also call the L2
> > cache functions because that's wrong.
> >
> 
> Russell,
> 
> We are not modifying flush_cache_all. L2 cache is not flushed from there.
> Actually as soon as L1 Cache is flushed and disabled, spin_lock is taken from
> some other thread. This creates problem for our platform, as we do not have
> monitors support at L2 thus failing strex instruction.
> 
> If we disable L2 then reboot works fine, as expected.

Again, this should not be happening:

void smp_send_stop(void)
{
        unsigned long timeout;

        if (num_online_cpus() > 1) {
                cpumask_t mask = cpu_online_map;
                cpu_clear(smp_processor_id(), mask);

                smp_cross_call(&mask, IPI_CPU_STOP);
        }

        /* Wait up to one second for other CPUs to stop */
        timeout = USEC_PER_SEC;
        while (num_online_cpus() > 1 && timeout--)
                udelay(1);

        if (num_online_cpus() > 1)
                pr_warning("SMP: failed to stop secondary CPUs\n");
}

void machine_shutdown(void)
{
#ifdef CONFIG_SMP
        smp_send_stop();
#endif
}

void machine_restart(char *cmd)
{
        machine_shutdown();
        arm_pm_restart(reboot_mode, cmd);
}

smp_send_stop() brings the other CPUs down to a quiescent state, waits for
up to one second for that to happen, and then starts flushing caches etc.
No other CPU should be active at the point where arm_pm_restart() is
called.  There should be no other threads active on either this or any
other CPU.

reboot not working on linux-2.6.37 for ARMv7

[shiraz hashim]

Hello Russell,

On Sat, Feb 12, 2011 at 9:51 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Sat, Feb 12, 2011 at 09:45:45PM +0530, viresh kumar wrote:

[...]

>> We are not modifying flush_cache_all. L2 cache is not flushed from there.
>> Actually as soon as L1 Cache is flushed and disabled, spin_lock is taken from
>> some other thread. This creates problem for our platform, as we do not have
>> monitors support at L2 thus failing strex instruction.
>>
>> If we disable L2 then reboot works fine, as expected.
>
> Again, this should not be happening:
>
> void smp_send_stop(void)
> {
> ? ? ? ?unsigned long timeout;
>
> ? ? ? ?if (num_online_cpus() > 1) {
> ? ? ? ? ? ? ? ?cpumask_t mask = cpu_online_map;
> ? ? ? ? ? ? ? ?cpu_clear(smp_processor_id(), mask);
>
> ? ? ? ? ? ? ? ?smp_cross_call(&mask, IPI_CPU_STOP);
> ? ? ? ?}
>
> ? ? ? ?/* Wait up to one second for other CPUs to stop */
> ? ? ? ?timeout = USEC_PER_SEC;
> ? ? ? ?while (num_online_cpus() > 1 && timeout--)
> ? ? ? ? ? ? ? ?udelay(1);
>
> ? ? ? ?if (num_online_cpus() > 1)
> ? ? ? ? ? ? ? ?pr_warning("SMP: failed to stop secondary CPUs\n");
> }
>
> void machine_shutdown(void)
> {
> #ifdef CONFIG_SMP
> ? ? ? ?smp_send_stop();
> #endif
> }
>
> void machine_restart(char *cmd)
> {
> ? ? ? ?machine_shutdown();
> ? ? ? ?arm_pm_restart(reboot_mode, cmd);
> }
>
> smp_send_stop() brings the other CPUs down to a quiescent state, waits for
> up to one second for that to happen, and then starts flushing caches etc.
> No other CPU should be active at the point where arm_pm_restart() is
> called. ?There should be no other threads active on either this or any
> other CPU.

Thanks, we got it. This was missing in linux-2.6.37 where the code was

 void smp_send_stop(void)
 {
         cpumask_t mask = cpu_online_map;
         cpu_clear(smp_processor_id(), mask);
         if (!cpus_empty(mask))
                 send_ipi_message(&mask, IPI_CPU_STOP);
 }

I would try this change and see if it works for me.

-- 
regards
Shiraz Hashim

reboot not working on linux-2.6.37 for ARMv7

[Russell King - ARM Linux]

On Sat, Feb 12, 2011 at 10:53:54PM +0530, shiraz hashim wrote:
> Thanks, we got it. This was missing in linux-2.6.37 where the code was
> 
>  void smp_send_stop(void)
>  {
>          cpumask_t mask = cpu_online_map;
>          cpu_clear(smp_processor_id(), mask);
>          if (!cpus_empty(mask))
>                  send_ipi_message(&mask, IPI_CPU_STOP);
>  }
> 
> I would try this change and see if it works for me.

Probably best to pull the patch out of latest git, which is:

28e18293cf0f8d23a0950d7b1d2212d11af494dc

reboot not working on linux-2.6.37 for ARMv7

[shiraz hashim]

On Sat, Feb 12, 2011 at 11:02 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Sat, Feb 12, 2011 at 10:53:54PM +0530, shiraz hashim wrote:
>> Thanks, we got it. This was missing in linux-2.6.37 where the code was
>>
>> ?void smp_send_stop(void)
>> ?{
>> ? ? ? ? ?cpumask_t mask = cpu_online_map;
>> ? ? ? ? ?cpu_clear(smp_processor_id(), mask);
>> ? ? ? ? ?if (!cpus_empty(mask))
>> ? ? ? ? ? ? ? ? ?send_ipi_message(&mask, IPI_CPU_STOP);
>> ?}
>>
>> I would try this change and see if it works for me.
>
> Probably best to pull the patch out of latest git, which is:
>
> 28e18293cf0f8d23a0950d7b1d2212d11af494dc

Sure, thanks again.

-- 
regards
Shiraz Hashim

reboot not working on linux-2.6.37 for ARMv7

[Rabin Vincent]

On Sat, Feb 12, 2011 at 21:45, viresh kumar <viresh.linux@gmail.com> wrote:
> We are not modifying flush_cache_all. L2 cache is not flushed from
> there.  Actually as soon as L1 Cache is flushed and disabled,
> spin_lock is taken from some other thread. This creates problem for
> our platform, as we do not have monitors support at L2 thus failing
> strex instruction.

Do you use writel() in your arch_reset() routine (as plat-spear appears
to be doing)?  There's a spinlock in there too (l2x0_cache_sync(), via
the barriers) which will also not work in your situation.

reboot not working on linux-2.6.37 for ARMv7

[shiraz hashim]

Hi Rabin,

On Sun, Feb 13, 2011 at 9:26 AM, Rabin Vincent <rabin@rab.in> wrote:
> On Sat, Feb 12, 2011 at 21:45, viresh kumar <viresh.linux@gmail.com> wrote:
>> We are not modifying flush_cache_all. L2 cache is not flushed from
>> there. ?Actually as soon as L1 Cache is flushed and disabled,
>> spin_lock is taken from some other thread. This creates problem for
>> our platform, as we do not have monitors support at L2 thus failing
>> strex instruction.
>
> Do you use writel() in your arch_reset() routine (as plat-spear appears
> to be doing)? ?There's a spinlock in there too (l2x0_cache_sync(), via
> the barriers) which will also not work in your situation.

You are right. Changed this to writel_relaxed and picked
28e18293cf0f8d23a0950d7b1d2212d11af494dc. Now it is working fine.

-- 
regards
Shiraz Hashim