Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Mats Liljegren]

Hi Frederic,

(Sorry for the re-post, I spelled the mailing list wrong...)

I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.

As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.

Are there any caveats I should be aware of?
Has anyone else started any work in this area that I could co-operate with?

Regards
Mats Liljegren

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Frederic Weisbecker]

2013/2/4 Mats Liljegren <mats.liljegren@enea.com>:
> Hi Frederic,
>
> (Sorry for the re-post, I spelled the mailing list wrong...)
>
> I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
>
> As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.

No that part is handled by the core code. But you need to implement
the context tracking arch backend.
Here is what Li Zhong did to port it on Powerpc for example:
https://lkml.org/lkml/2013/2/1/216

> Are there any caveats I should be aware of?

The generic cputime accounting only works on 64 bits archs. We'll need
to do some safe accessors to read the kernel_stat. In the meantime you
can do home tests by removing the CONFIG_64BIT dependency. This
shouldn't be dangerous

> Has anyone else started any work in this area that I could co-operate with?

Yeah two people have been working on this. But it was several month
ago and I haven't hear from them since then. I'm adding Geoff in Cc. I
can't retrieve the other guy who has worked on it too.

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Geoff Levand]

Hi,

On Fri, 2013-02-08 at 17:31 +0100, Frederic Weisbecker wrote:
> 2013/2/4 Mats Liljegren <mats.liljegren@enea.com>:
> > I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
> 
> Yeah two people have been working on this. But it was several month
> ago and I haven't hear from them since then. I'm adding Geoff in Cc. I
> can't retrieve the other guy who has worked on it too.

I haven't kept the patches up to date, but what I have is here:

  http://git.kernel.org/?p=linux/kernel/git/geoff/nohz.git

I have some Panda specific debugging patches in there too.  The
last version that worked OK with Panda was cpuset-v3-arm.  The
HEAD of the master branch had some problems at startup which I
never had a chance to fix.

-Geoff

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Mats Liljegren]

Frederic Weisbecker wrote:
> 2013/2/4 Mats Liljegren <mats.liljegren@enea.com>:
> > Hi Frederic,
> >
> > (Sorry for the re-post, I spelled the mailing list wrong...)
> >
> > I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
> >
> > As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
> 
> No that part is handled by the core code. But you need to implement
> the context tracking arch backend.
> Here is what Li Zhong did to port it on Powerpc for example:
> https://lkml.org/lkml/2013/2/1/216
> 
> > Are there any caveats I should be aware of?
> 
> The generic cputime accounting only works on 64 bits archs. We'll need
> to do some safe accessors to read the kernel_stat. In the meantime you
> can do home tests by removing the CONFIG_64BIT dependency. This
> shouldn't be dangerous
> 
> > Has anyone else started any work in this area that I could co-operate with?
> 
> Yeah two people have been working on this. But it was several month
> ago and I haven't hear from them since then. I'm adding Geoff in Cc. I
> can't retrieve the other guy who has worked on it too.

Thanks for the hints! They helped me along the way.

I have now managed to boot Linux without getting RCU complaints with RCU
checking enabled. I enabled virtual CPU accounting by ignoring 64BIT
dependency, as you suggested. I had to add an __aeabi_uldivmod function to
get it to compile though.

It boots without RCU complaints when CONFIG_PROVE_RCU is enabled.

Next step will be to figure out those boot parameters to enable full
dynticks mode, and see whether I forgot something... ;-) I'll do so on Monday.

Regards
Mats Liljegren

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Mats Liljegren]

Frederic Weisbecker wrote:
> 2013/2/4 Mats Liljegren <mats.liljegren@enea.com>:
> > Hi Frederic,
> >
> > (Sorry for the re-post, I spelled the mailing list wrong...)
> >
> > I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
> >
> > As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
> 
> No that part is handled by the core code. But you need to implement
> the context tracking arch backend.
> Here is what Li Zhong did to port it on Powerpc for example:
> https://lkml.org/lkml/2013/2/1/216
> 
> > Are there any caveats I should be aware of?
> 
> The generic cputime accounting only works on 64 bits archs. We'll need
> to do some safe accessors to read the kernel_stat. In the meantime you
> can do home tests by removing the CONFIG_64BIT dependency. This
> shouldn't be dangerous
> 
> > Has anyone else started any work in this area that I could co-operate with?
> 
> Yeah two people have been working on this. But it was several month
> ago and I haven't hear from them since then. I'm adding Geoff in Cc. I
> can't retrieve the other guy who has worked on it too.

Could the other guy be Kevin Hillman? The name has been mentioned to me but
I don't have his e-mail address.

I have stumbled across a problem, I get the following error messages during
boot:

---- Start snippet ----
[   41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.
[   41.126373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   41.134765] rcuo1           D c045c5d8     0    10      2 0x00000000
[   41.141662] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   41.150329] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   41.159423] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   41.168487] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   41.177124] no locks held by rcuo1/10.
[   41.181182] INFO: task rcuo1:12 blocked for more than 20 seconds.
[   41.187683] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   41.196044] rcuo1           D c045c5d8     0    12      2 0x00000000
[   41.202911] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   41.211578] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   41.220733] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   41.229705] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   41.238372] no locks held by rcuo1/12.
[   61.239868] INFO: task rcuo1:10 blocked for more than 20 seconds.
[   61.246337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   61.254730] rcuo1           D c045c5d8     0    10      2 0x00000000
[   61.261596] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   61.270294] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   61.279357] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   61.288391] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   61.297058] no locks held by rcuo1/10.
[   61.301116] INFO: task rcuo1:12 blocked for more than 20 seconds.
[   61.307617] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   61.315979] rcuo1           D c045c5d8     0    12      2 0x00000000
[   61.322845] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   61.331512] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   61.340667] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   61.349639] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   61.358306] no locks held by rcuo1/12.
[   81.359863] INFO: task swapper/0:1 blocked for more than 20 seconds.
[   81.366638] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   81.374999] swapper/0       D c045c5d8     0     1      0 0x00000000
[   81.381866] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   81.390563] [<c045c880>] (schedule+0x74/0x78) from [<c0459c0c>] (schedule_timeout+0x1e4/0x244)
[   81.399719] [<c0459c0c>] (schedule_timeout+0x1e4/0x244) from [<c0459c9c>] (schedule_timeout_uninterruptible+0x30/0x34)
[   81.411163] [<c0459c9c>] (schedule_timeout_uninterruptible+0x30/0x34) from [<c0057a98>] (msleep+0x24/0x30)
[   81.421508] [<c0057a98>] (msleep+0x24/0x30) from [<c0659d20>] (mount_root+0x74/0xfc)
[   81.429748] [<c0659d20>] (mount_root+0x74/0xfc) from [<c0659ed0>] (prepare_namespace+0x128/0x18c)
[   81.439270] [<c0659ed0>] (prepare_namespace+0x128/0x18c) from [<c044e35c>] (kernel_init+0x1cc/0x338)
[   81.449066] [<c044e35c>] (kernel_init+0x1cc/0x338) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   81.458282] no locks held by swapper/0/1.
[   81.462615] INFO: task rcuo1:10 blocked for more than 20 seconds.
[   81.469085] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   81.477447] rcuo1           D c045c5d8     0    10      2 0x00000000
[   81.484313] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   81.493011] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   81.502136] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   81.511169] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   81.519836] no locks held by rcuo1/10.
[   81.523834] INFO: task rcuo1:12 blocked for more than 20 seconds.
[   81.530364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   81.538696] rcuo1           D c045c5d8     0    12      2 0x00000000
[   81.545562] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[   81.554229] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[   81.563354] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[   81.572387] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[   81.581054] no locks held by rcuo1/12.
[  101.579895] INFO: task rcuo1:10 blocked for more than 20 seconds.
[  101.586364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  101.594757] rcuo1           D c045c5d8     0    10      2 0x00000000
[  101.601623] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[  101.610290] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[  101.619384] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[  101.628417] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[  101.637084] no locks held by rcuo1/10.
[  101.641143] INFO: task rcuo1:12 blocked for more than 20 seconds.
[  101.647613] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  101.656005] rcuo1           D c045c5d8     0    12      2 0x00000000
[  101.662872] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78)
[  101.671539] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224)
[  101.680664] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc)
[  101.689636] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20)
[  101.698303] no locks held by rcuo1/12.
[  112.400695] VFS: Unable to mount root fs via NFS, trying floppy.
[  112.407226] List of all partitions:
[  112.411041] No filesystem could mount root, tried:
[  112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[  112.425262] CPU1: stopping
[  112.428161] [<c001dcb8>] (unwind_backtrace+0x0/0x104) from [<c0453cb8>] (dump_stack+0x20/0x24)
[  112.437316] [<c0453cb8>] (dump_stack+0x20/0x24) from [<c001b16c>] (handle_IPI+0xf4/0x154)
[  112.445983] [<c001b16c>] (handle_IPI+0xf4/0x154) from [<c0008600>] (gic_handle_irq+0x68/0x84)
[  112.455047] [<c0008600>] (gic_handle_irq+0x68/0x84) from [<c045df84>] (__irq_svc+0x44/0x58)
[  112.463897] Exception stack(0xeda29f68 to 0xeda29fb0)
[  112.469238] 9f60:                   00000001 00000001 00000000 eda20a80 eda28000 eda28000
[  112.477905] 9f80: c073d9c8 c06c0574 c0468c60 eda28000 c06c4f90 eda29fdc eda29f80 eda29fb0
[  112.486572] 9fa0: c00ab854 c00156fc 20000113 ffffffff
[  112.491943] [<c045df84>] (__irq_svc+0x44/0x58) from [<c00156fc>] (cpu_idle+0xac/0x128)
[  112.500366] [<c00156fc>] (cpu_idle+0xac/0x128) from [<c0450a1c>] (secondary_start_kernel+0xec/0x104)
[  112.510040] [<c0450a1c>] (secondary_start_kernel+0xec/0x104) from [<8044ff88>] (0x8044ff88)
PANIC: VFS: Unable to mount root fs on unknown-block(2,0)

---- End snipped ----

Do you have any suggestions for the best way of tracking down the cause?

Regards
Mats Liljegren

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Kevin Hilman]

Mats Liljegren <mats.liljegren@enea.com> writes:

> Frederic Weisbecker wrote:
>> 2013/2/4 Mats Liljegren <mats.liljegren@enea.com>:
>> > Hi Frederic,
>> >
>> > (Sorry for the re-post, I spelled the mailing list wrong...)
>> >
>> > I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture,
>> > and try it out on the PandaBoard. Initially this is for evaluating
>> > it, but if it looks promising I'd like to upstream this.
>> >
>> > As far as I can see the first thing to do is to implement virtual
>> > time accounting. After an initial look at the patches done for
>> > ia64, it doesn't look too difficult.
>> 
>> No that part is handled by the core code. But you need to implement
>> the context tracking arch backend.
>> Here is what Li Zhong did to port it on Powerpc for example:
>> https://lkml.org/lkml/2013/2/1/216
>> 
>> > Are there any caveats I should be aware of?
>> 
>> The generic cputime accounting only works on 64 bits archs. We'll need
>> to do some safe accessors to read the kernel_stat. In the meantime you
>> can do home tests by removing the CONFIG_64BIT dependency. This
>> shouldn't be dangerous
>> 
>> > Has anyone else started any work in this area that I could co-operate with?
>> 
>> Yeah two people have been working on this. But it was several month
>> ago and I haven't hear from them since then. I'm adding Geoff in Cc. I
>> can't retrieve the other guy who has worked on it too.
>
> Could the other guy be Kevin Hillman? 

Yup, that's him (though my last name has only one 'l' ;)

> The name has been mentioned to me but I don't have his e-mail address.

Now you do. :)

After some time away, I'm back to working on getting adaptive nohz
working on ARM, but things have changed quite a bit since I last worked
on it.  Things actually look a bit simpler now with the new
context_tracking layer, but that needs a little work for ARM which I'm
now looking into.

However, just to get things compiling needs a little work too.  I've
setup a work-in-progress branch which you can start from to at least get
things compiling[1].

I noticed you already ran accross the 64-bit division problem.  I've
proposed a patch for that[2] which is included in my branch along with a
few other things.

Hope that helps,

Kevin

[1] git://git.kernel.org/pub/scm/linux/kernel/git/khilman/linux.git wip/3.8-rc6-nohz4-arm
[2] http://marc.info/?l=linux-kernel&m=136087139214252&w=2

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Frederic Weisbecker]

2013/2/11 Mats Liljegren <mats.liljegren@enea.com>:
> Could the other guy be Kevin Hillman? The name has been mentioned to me but
> I don't have his e-mail address.

Exactly :)

>
> I have stumbled across a problem, I get the following error messages during
> boot:
>
> ---- Start snippet ----
> [   41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.

This showed up upstream as well, it hshould have been fixed in linus tree now.

[...]
> [  112.400695] VFS: Unable to mount root fs via NFS, trying floppy.
> [  112.407226] List of all partitions:
> [  112.411041] No filesystem could mount root, tried:
> [  112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

Ah, that probably not due to my patches :)

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Mats Liljegren]

> 2013/2/11 Mats Liljegren <mats.liljegren@enea.com>:
>> Could the other guy be Kevin Hillman? The name has been mentioned to me but
>> I don't have his e-mail address.
> 
> Exactly :)
> 
>> 
>> I have stumbled across a problem, I get the following error messages during
>> boot:
>> 
>> ---- Start snippet ----
>> [   41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.
> 
> This showed up upstream as well, it hshould have been fixed in linus tree now.

Ok.

> 
> [...]
>> [  112.400695] VFS: Unable to mount root fs via NFS, trying floppy.
>> [  112.407226] List of all partitions:
>> [  112.411041] No filesystem could mount root, tried:
>> [  112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
> 
> Ah, that probably not due to my patches :)

By removing some kernel parameters I managed to get further. They were'nt important, just tried to disable watchdog and idle calls.
My current problem is that it refuses to enter full nohz mode due to too many runable threads. I need to check were those extra threads come from.

Could it be related to rcu problem above? If so I might need to switch to a newer version of Linus tree.

/Mats

Porting 3.8-rc4-nohz3 to ARM (PandaBoard)

[Frederic Weisbecker]

2013/2/20 Mats Liljegren <mats.liljegren@enea.com>:
>
>
>> 2013/2/11 Mats Liljegren <mats.liljegren@enea.com>:
>>> Could the other guy be Kevin Hillman? The name has been mentioned to me but
>>> I don't have his e-mail address.
>>
>> Exactly :)
>>
>>>
>>> I have stumbled across a problem, I get the following error messages during
>>> boot:
>>>
>>> ---- Start snippet ----
>>> [   41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.
>>
>> This showed up upstream as well, it hshould have been fixed in linus tree now.
>
> Ok.
>
>>
>> [...]
>>> [  112.400695] VFS: Unable to mount root fs via NFS, trying floppy.
>>> [  112.407226] List of all partitions:
>>> [  112.411041] No filesystem could mount root, tried:
>>> [  112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
>>
>> Ah, that probably not due to my patches :)
>
> By removing some kernel parameters I managed to get further. They were'nt important, just tried to disable watchdog and idle calls.
> My current problem is that it refuses to enter full nohz mode due to too many runable threads. I need to check were those extra threads come from.
>
> Could it be related to rcu problem above? If so I might need to switch to a newer version of Linus tree.

May be. If so you may need to migrate the rcuoX kthreads. You probably
need to use the sched switch tracepoint to check what's going on. And
migrate some threads affinity if needed.