Re: [Powertop] Powertop doesn't detects sleeping ARM cores

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On 9/20/2012 12:40 PM, Igor Zhbanov wrote:
> But is it correct to consider ARM core to be offline when it is just in WFI state?

nope

for all intents and purposes, the cpu is still there.

in an idle state, the cpu logically is there, and ready to resume execution if needed...
that's the definition of idle ;-)

offline is where the administrator takes a cpu out of circulation.

neither specifies any physical hardware state... in fact offline is usually implemented as an idle state, at
least on x86. the semantics for how to get back are different (system autonomous versus administrator)

now, quite possible some ARM platforms abuse this and don't implement something as "idle" but as "offline" instead.
that sounds not very smart to me.

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

Arjan van de Ven wrote:
> On 9/20/2012 12:40 PM, Igor Zhbanov wrote:
>> But is it correct to consider ARM core to be offline when it is just in WFI state?
> nope
>
> for all intents and purposes, the cpu is still there.
>
> in an idle state, the cpu logically is there, and ready to resume execution if needed...
> that's the definition of idle ;-)
>
> offline is where the administrator takes a cpu out of circulation.
>
> neither specifies any physical hardware state... in fact offline is usually implemented as an idle state, at
> least on x86. the semantics for how to get back are different (system autonomous versus administrator)
>
> now, quite possible some ARM platforms abuse this and don't implement something as "idle" but as "offline" instead.
> that sounds not very smart to me.

Yes, it seems that on some ARM platforms offline and idle are mixed.
On my test device it is even hard to bring the core online. If I write 1 to "online" file,
then I see other pseudo-files only for a less than a second, then the core
will be brought offline again. It seems that kernel powers up the core, then sees that there
no task for it, and powers it down again.

May be we should talk to ARM developers to do things more correctly. It seems that now impossible
to administratively enable/disable some cores. The CPU governor will always consider all available
cores to be possible to use (except of limit of nr_cpus boot parameter).

So I don't know what to do best.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 2359 bytes --]

Arjan van de Ven wrote:
> On 9/20/2012 10:24 AM, Igor Zhbanov wrote:
>
>> Now I'm looking for solution. But I see now only four ways:
>>
>> 1) Develop another kernel mechanism for providing information about all available
>>     CPUs and cores. Or ask the kernel authors not to remove offline cores from
>>     /proc/cpuinfo and their information files from /sys/.../cpu/.
>> 2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
>> 3) Ask the user to run PowerTOP on a heavy loaded system for the first time
>>     (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere
>>     in /var/cache/powertop and reuse it on later runs.
>> 4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear.
>>     (Perhaps there could be something like D-Bus or netlink notification when a core
>>      brought online -- I don't know about it).
>>
>> At your personal view, what way fits better in the PowerTOP project?
> it's messy. ARM in some ways is abusing CPU Offline for idle...
>
>> P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing
>> to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3.
>> Although default PC's CPU governor as I know doesn't turn off CPUs frequently.
> on x86, you don't win any power at all by offlining CPUs (in fact, it's very power inefficient to do so,
> idle tends to be THAT good)

Or it could be that kernel on ARM wrongly removes information from /proc/cpuinfo and /sys/
for offline cores. Or it wrongly considers them offline.

As I know many of the ARM CPUs has only two C-states. First is independent for each core:
WFI -- Wait For Interrupt. In this state the CPU just calls the halt instruction.

And the second state is ARM power down. This is the coupled state. As I understand,
the whole package could enter this state only when all cores (except current) are in WFI state.
So the last active core powers the CPU down.

But is it correct to consider ARM core to be offline when it is just in WFI state?

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

Sergey Senozhatsky wrote:

> On (09/20/12 13:29), Igor Zhbanov wrote:
>>> On (09/20/12 12:24), Igor Zhbanov wrote:
>>>> Chris Ferron wrote:
>>>>
>>>> Now I'm looking for solution. But I see now only four ways:
>>>>
>>>> 1) Develop another kernel mechanism for providing information about all available
>>>>     CPUs and cores. Or ask the kernel authors not to remove offline cores from
>>>>     /proc/cpuinfo and their information files from /sys/.../cpu/.
>>> procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for
>>> that purpose.
>>>
>>> https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html
>> Counting of the CPUs is the least problem. (By the way, how sysconf() figures
>> out the number? I suppose that it reads /proc/cpuinfo of scans /sys/.
>> Is there any other way?).
>>
>> The problem is that the PowerTOP needs to read information about cores
>> from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings
>> etc). And needed files in these directories are present only for cores
>> which are currently online.
> right, I was thinking about that some time ago: and from the top of my head kind of a solution
> seemed to be to track number of CPUs before measure, compare with the previously known one,
> and "re-init" powertop (at least rebuild CPUs part), but it's not that often that I offline/online
> CPUs, so I decided to drop it. with ARM we have a different story.
>
> on x86(_64) "dirty" CPU offline support looks "easy", just make get_msr() error non-fatal.
> online -- that's the problem, agree.

Perhaps PowerTOP could wake up every core in its calibration phase and story collected
information in platform config file.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]

On 9/20/2012 10:24 AM, Igor Zhbanov wrote:
> Chris Ferron wrote:
> 
>> On 09/19/2012 01:26 AM, Igor Zhbanov wrote:
>>> I mean that only online cores are shown in /proc/cpuinfo.
>>> So the PowerTOP doesn't count currently offline ones.
>>> And you must run the PowerTOP on a heavy loaded system
>>> to let the PowerTOP see all CPU cores.
>>> Therefore it is impossible to run the PowerTOP before starting
>>> of heavy application to get information about all CPU cores.
>>> Probably the PowerTOP could cache CPU information in some file,
>>> so it will be possible to register trace events for all cores.
>> Off hand I don't have a good solution to help your ARM story. Your notes
>> are interesting and I agree this is something that ARM is going to need.
>> Do you have a complete solution you would like to purpose? I assume you
>> have done some hacking to get around this, but if you have taken that
>> and tuned it i suggest purposing the change. If you find a solution that
>> doesn't digress the Architectures, and corrects the ARM story then
>> please submit patches. I would be interested in seeing what you have
>> come up with.
>> I will try and look for a general solution, are there any other ARM
>> advocates on this list who would like to chime in?
> 
> Now I'm looking for solution. But I see now only four ways:
> 
> 1) Develop another kernel mechanism for providing information about all available
>    CPUs and cores. Or ask the kernel authors not to remove offline cores from
>    /proc/cpuinfo and their information files from /sys/.../cpu/.
> 2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
> 3) Ask the user to run PowerTOP on a heavy loaded system for the first time
>    (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere
>    in /var/cache/powertop and reuse it on later runs.
> 4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear.
>    (Perhaps there could be something like D-Bus or netlink notification when a core
>     brought online -- I don't know about it).
> 
> At your personal view, what way fits better in the PowerTOP project?

it's messy. ARM in some ways is abusing CPU Offline for idle...

> 
> P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing
> to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3.
> Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

on x86, you don't win any power at all by offlining CPUs (in fact, it's very power inefficient to do so,
idle tends to be THAT good)
> 

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Sergey Senozhatsky]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

On (09/20/12 13:29), Igor Zhbanov wrote:
> >On (09/20/12 12:24), Igor Zhbanov wrote:
> >>Chris Ferron wrote:
> >>
> >>Now I'm looking for solution. But I see now only four ways:
> >>
> >>1) Develop another kernel mechanism for providing information about all available
> >>    CPUs and cores. Or ask the kernel authors not to remove offline cores from
> >>    /proc/cpuinfo and their information files from /sys/.../cpu/.
> >procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for
> >that purpose.
> >
> >https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html
> 
> Counting of the CPUs is the least problem. (By the way, how sysconf() figures
> out the number? I suppose that it reads /proc/cpuinfo of scans /sys/.
> Is there any other way?).
> 
> The problem is that the PowerTOP needs to read information about cores
> from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings
> etc). And needed files in these directories are present only for cores
> which are currently online.
>

right, I was thinking about that some time ago: and from the top of my head kind of a solution
seemed to be to track number of CPUs before measure, compare with the previously known one,
and "re-init" powertop (at least rebuild CPUs part), but it's not that often that I offline/online
CPUs, so I decided to drop it. with ARM we have a different story.

on x86(_64) "dirty" CPU offline support looks "easy", just make get_msr() error non-fatal.
online -- that's the problem, agree.
 
	-ss

> So the Powertop is unable
> 1) To count the number of available CPUs.
> 2) To read information about all CPUs.
> 

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

Sergey Senozhatsky wrote:

> On (09/20/12 12:24), Igor Zhbanov wrote:
>> Chris Ferron wrote:
>>
>> Now I'm looking for solution. But I see now only four ways:
>>
>> 1) Develop another kernel mechanism for providing information about all available
>>     CPUs and cores. Or ask the kernel authors not to remove offline cores from
>>     /proc/cpuinfo and their information files from /sys/.../cpu/.
> procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for
> that purpose.
>
> https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html

Counting of the CPUs is the least problem. (By the way, how sysconf() figures
out the number? I suppose that it reads /proc/cpuinfo of scans /sys/.
Is there any other way?).

The problem is that the PowerTOP needs to read information about cores
from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings
etc). And needed files in these directories are present only for cores
which are currently online.

So the Powertop is unable
1) To count the number of available CPUs.
2) To read information about all CPUs.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Sergey Senozhatsky]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

On (09/20/12 12:24), Igor Zhbanov wrote:
> Chris Ferron wrote:
> 
> Now I'm looking for solution. But I see now only four ways:
> 
> 1) Develop another kernel mechanism for providing information about all available
>    CPUs and cores. Or ask the kernel authors not to remove offline cores from
>    /proc/cpuinfo and their information files from /sys/.../cpu/.

procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for
that purpose.

https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html


	-ss

> 2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
> 3) Ask the user to run PowerTOP on a heavy loaded system for the first time
>    (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere
>    in /var/cache/powertop and reuse it on later runs.
> 4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear.
>    (Perhaps there could be something like D-Bus or netlink notification when a core
>     brought online -- I don't know about it).
> 
> At your personal view, what way fits better in the PowerTOP project?
> 
> P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing
> to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3.
> Although default PC's CPU governor as I know doesn't turn off CPUs frequently.
> 

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]

Chris Ferron wrote:

> On 09/19/2012 01:26 AM, Igor Zhbanov wrote:
>> I mean that only online cores are shown in /proc/cpuinfo.
>> So the PowerTOP doesn't count currently offline ones.
>> And you must run the PowerTOP on a heavy loaded system
>> to let the PowerTOP see all CPU cores.
>> Therefore it is impossible to run the PowerTOP before starting
>> of heavy application to get information about all CPU cores.
>> Probably the PowerTOP could cache CPU information in some file,
>> so it will be possible to register trace events for all cores.
> Off hand I don't have a good solution to help your ARM story. Your notes
> are interesting and I agree this is something that ARM is going to need.
> Do you have a complete solution you would like to purpose? I assume you
> have done some hacking to get around this, but if you have taken that
> and tuned it i suggest purposing the change. If you find a solution that
> doesn't digress the Architectures, and corrects the ARM story then
> please submit patches. I would be interested in seeing what you have
> come up with.
> I will try and look for a general solution, are there any other ARM
> advocates on this list who would like to chime in?

Now I'm looking for solution. But I see now only four ways:

1) Develop another kernel mechanism for providing information about all available
    CPUs and cores. Or ask the kernel authors not to remove offline cores from
    /proc/cpuinfo and their information files from /sys/.../cpu/.
2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
3) Ask the user to run PowerTOP on a heavy loaded system for the first time
    (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere
    in /var/cache/powertop and reuse it on later runs.
4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear.
    (Perhaps there could be something like D-Bus or netlink notification when a core
     brought online -- I don't know about it).

At your personal view, what way fits better in the PowerTOP project?

P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing
to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3.
Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Chris Ferron]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

On 09/19/2012 01:26 AM, Igor Zhbanov wrote:
> Hi,
>
> Here is some notes to my previous post.
>
> I mean that only online cores are shown in /proc/cpuinfo.
> So the PowerTOP doesn't count currently offline ones.
> And you must run the PowerTOP on a heavy loaded system
> to let the PowerTOP see all CPU cores.
>
> Therefore it is impossible to run the PowerTOP before starting
> of heavy application to get information about all CPU cores.
>
> Probably the PowerTOP could cache CPU information in some file,
> so it will be possible to register trace events for all cores.
>
Off hand I don't have a good solution to help your ARM story. Your notes 
are interesting and I agree this is something that ARM is going to need. 
Do you have a complete solution you would like to purpose? I assume you 
have done some hacking to get around this, but if you have taken that 
and tuned it i suggest purposing the change. If you find a solution that 
doesn't digress the Architectures, and corrects the ARM story then 
please submit patches. I would be interested in seeing what you have 
come up with.

I will try and look for a general solution, are there any other ARM 
advocates on this list who would like to chime in?

-C

Re: [Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

Hi,

Here is some notes to my previous post.

I mean that only online cores are shown in /proc/cpuinfo.
So the PowerTOP doesn't count currently offline ones.
And you must run the PowerTOP on a heavy loaded system
to let the PowerTOP see all CPU cores.

Therefore it is impossible to run the PowerTOP before starting
of heavy application to get information about all CPU cores.

Probably the PowerTOP could cache CPU information in some file,
so it will be possible to register trace events for all cores.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation

[Powertop] Powertop doesn't detects sleeping ARM cores

[Igor Zhbanov]

[-- Attachment #1: Type: text/plain, Size: 3294 bytes --]

Hello!

When you start the PowerTOP on a "cold" ARM phone with several cores and low CPU average
some cores could be powered down. It is very frequent situation on ARM platform to dynamically
turn cores on/off to save power.

And when the PowerTOP scans /proc/cpuinfo file in enumerate_cpus() function, it will see
only one working core, so PowerTOP calls handle_one_cpu(...) only once and sets number of CPUs to 1.
So the PowerTOP will not register trace events for other cores.

Later when the system load (during the lifetime of PowerTOP) increases, the Linux kernel powers-up
additional cores. But the PowerTOP will not catch events for them.

I thought about scanning the /sys/devices/system/cpu/ directory for finding out how many CPUs
the system has. But the problem is that the directories for CPUs in offline mode doesn't provide
needed subdirectories cpuidle/ and topology/ (also it misses some files in cpufreq/ directory).

The only way to enable these directories is to (temporarily?) turn the core on.
This could be done e.g. by writing 1 to cpu/cpuX/online.
But I'm not sure how long the core will stay online before the kernel turns it off again because of
low load. On my phone this happens in a less than a second.

Another possible way is to scan /sys/.../cpu/ for the number of CPUs in the system,
register trace events for all that CPUs (by the way, is it possible to register a trace event
for the core that is now off?), sleep for the needed amount of time as usual, and before
the processing of data, temporarily turn every core on to find out information about it.

Or the PowerTOP could remember which cores was turned on before the testing, turn it all on,
read needed info, turn some cores off to restore the state of the system and begin test.

Or it could scan /sys/.../cpu/ with some interval (1 second) instead of just sleeping,
and grab the information about new cores brought online.

By the way, the PowerTOP scans /proc/ directory twice -- once before the start and once after
to get information about processes (to provide more information from trace events).
But if some process starts after the PowerTOP begins to sleep, and dies before the sleep ends,
then the PowerTOP will be unable to get information about that process (e.g. command line).
So the PowerTOP will know only its PID and shortened version of the command line (task->comm[16]).

I have a patch adding scanning of /proc/ capabilities to the PowerTOP. I use this /proc/ scanner
in MALI supporting patch, because MALI provides only the PID of the process in their events,
and that is not enough to know about the process. It needs more discussion so I will send the patch
little later. But I want to notice, that it could be useful for the rest of the PowerTOP too
to catch that processes. (The best solution will be to catch the process exit, but that event
provides only short task->comm[16]).

Anyway, the PowerTOP could do more instead of just sleeping.

What do you think?

Thank you.

-- 
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation