* "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
@ 2016-04-02 9:32 Markus Trippelsdorf
2016-04-02 11:00 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Markus Trippelsdorf @ 2016-04-02 9:32 UTC (permalink / raw)
To: linux-kernel
Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
Current git kernel sometimes shows:
perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
...
when running e.g. "perf top" even when
/proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
--
Markus
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
2016-04-02 9:32 "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0 Markus Trippelsdorf
@ 2016-04-02 11:00 ` Peter Zijlstra
2016-04-02 11:17 ` Markus Trippelsdorf
2016-04-13 10:03 ` Markus Trippelsdorf
0 siblings, 2 replies; 6+ messages in thread
From: Peter Zijlstra @ 2016-04-02 11:00 UTC (permalink / raw)
To: Markus Trippelsdorf
Cc: linux-kernel, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> Current git kernel sometimes shows:
>
> perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> ...
>
> when running e.g. "perf top" even when
> /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
Ah, was 0 also meant to disable it?
Does the below help?
---
kernel/events/core.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 8c3b35f2a269..21ba024c9ed1 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -412,7 +412,8 @@ int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
if (ret || !write)
return ret;
- if (sysctl_perf_cpu_time_max_percent == 100) {
+ if (sysctl_perf_cpu_time_max_percent == 100 ||
+ sysctl_perf_cpu_time_max_percent == 0) {
printk(KERN_WARNING
"perf: Dynamic interrupt throttling disabled, can hang your system!\n");
WRITE_ONCE(perf_sample_allowed_ns, 0);
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
2016-04-02 11:00 ` Peter Zijlstra
@ 2016-04-02 11:17 ` Markus Trippelsdorf
2016-04-02 11:23 ` Markus Trippelsdorf
2016-04-02 16:33 ` Peter Zijlstra
2016-04-13 10:03 ` Markus Trippelsdorf
1 sibling, 2 replies; 6+ messages in thread
From: Markus Trippelsdorf @ 2016-04-02 11:17 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-kernel, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> > Current git kernel sometimes shows:
> >
> > perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> > ...
> >
> > when running e.g. "perf top" even when
> > /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
>
>
> Ah, was 0 also meant to disable it?
Yes. From Documentation/sysctl/kernel.txt:
perf_cpu_time_max_percent:
Hints to the kernel how much CPU time it should be allowed to
use to handle perf sampling events. If the perf subsystem
is informed that its samples are exceeding this limit, it
will drop its sampling frequency to attempt to reduce its CPU
usage.
Some perf sampling happens in NMIs. If these samples
unexpectedly take too long to execute, the NMIs can become
stacked up next to each other so much that nothing else is
allowed to execute.
0: disable the mechanism. Do not monitor or correct perf's
sampling rate no matter how CPU time it takes.
1-100: attempt to throttle perf's sample rate to this
percentage of CPU. Note: the kernel calculates an
"expected" length of each sample event. 100 here means
100% of that expected length. Even if this is set to
100, you may still see sample throttling if this
length is exceeded. Set to 0 if you truly do not care
how much CPU is consumed.
> Does the below help?
Thanks. I will test it later. But 91a612eea9a3 makes the assumption that only
sysctl_perf_cpu_time_max_percent==100 disables the feature also in
kernel/events/core.c.
--
Markus
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
2016-04-02 11:17 ` Markus Trippelsdorf
@ 2016-04-02 11:23 ` Markus Trippelsdorf
2016-04-02 16:33 ` Peter Zijlstra
1 sibling, 0 replies; 6+ messages in thread
From: Markus Trippelsdorf @ 2016-04-02 11:23 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-kernel, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
On 2016.04.02 at 13:17 +0200, Markus Trippelsdorf wrote:
> > Does the below help?
>
> Thanks. I will test it later. But 91a612eea9a3 makes the assumption that only
> sysctl_perf_cpu_time_max_percent==100 disables the feature also in
> kernel/events/core.c.
Please ignore the last sentence. It doesn't make sense.
--
Markus
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
2016-04-02 11:17 ` Markus Trippelsdorf
2016-04-02 11:23 ` Markus Trippelsdorf
@ 2016-04-02 16:33 ` Peter Zijlstra
1 sibling, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2016-04-02 16:33 UTC (permalink / raw)
To: Markus Trippelsdorf
Cc: linux-kernel, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
On Sat, Apr 02, 2016 at 01:17:26PM +0200, Markus Trippelsdorf wrote:
> On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> > Ah, was 0 also meant to disable it?
>
> Yes. From Documentation/sysctl/kernel.txt:
Oh wow, we have documentation on this! Learn something new every day :-)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0
2016-04-02 11:00 ` Peter Zijlstra
2016-04-02 11:17 ` Markus Trippelsdorf
@ 2016-04-13 10:03 ` Markus Trippelsdorf
1 sibling, 0 replies; 6+ messages in thread
From: Markus Trippelsdorf @ 2016-04-13 10:03 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-kernel, Ingo Molnar, Arnaldo Carvalho de Melo,
Alexander Shishkin
On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> > Current git kernel sometimes shows:
> >
> > perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> > perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> > ...
> >
> > when running e.g. "perf top" even when
> > /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
>
>
> Ah, was 0 also meant to disable it?
>
> Does the below help?
Yes, it obviously fixes the issue.
--
Markus
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-04-13 10:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-02 9:32 "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0 Markus Trippelsdorf
2016-04-02 11:00 ` Peter Zijlstra
2016-04-02 11:17 ` Markus Trippelsdorf
2016-04-02 11:23 ` Markus Trippelsdorf
2016-04-02 16:33 ` Peter Zijlstra
2016-04-13 10:03 ` Markus Trippelsdorf
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).