[BUG] perf and kmemcheck : fatal combination

[BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Hi guys

Just got a panic on a kmemcheck kernel, latest linux-2.6 tree.

I forgot I had kmemcheck enabled, and started "perf top" just because my
machine was damn slow... Oh well...

Crash in do_nmi -> nmi_enter() -> BUG_ON(in_nmi());

Thanks

Re: [BUG] perf and kmemcheck : fatal combination

[Peter Zijlstra]

On Mon, 2011-04-25 at 18:08 +0200, Eric Dumazet wrote:
> Hi guys
> 
> Just got a panic on a kmemcheck kernel, latest linux-2.6 tree.
> 
> I forgot I had kmemcheck enabled, and started "perf top" just because my
> machine was damn slow... Oh well...
> 
> Crash in do_nmi -> nmi_enter() -> BUG_ON(in_nmi());

Hmm,. I bet because kmemcheck triggers faults from nmi context because
it messes about with the page protection bits a lot to track things.

Can't really think of anything except not making perf available on
kmemcheck kernels.

---
 init/Kconfig |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 32745bf..94735b4 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1125,6 +1125,7 @@ config PERF_EVENTS
 	bool "Kernel performance events and counters"
 	default y if (PROFILING || PERF_COUNTERS)
 	depends on HAVE_PERF_EVENTS
+	depends on !KMEMCHECK
 	select ANON_INODES
 	select IRQ_WORK
 	help

Re: [BUG] perf and kmemcheck : fatal combination

[Pekka Enberg]

On Tue, Apr 26, 2011 at 10:38 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> On Mon, 2011-04-25 at 18:08 +0200, Eric Dumazet wrote:
>> Hi guys
>>
>> Just got a panic on a kmemcheck kernel, latest linux-2.6 tree.
>>
>> I forgot I had kmemcheck enabled, and started "perf top" just because my
>> machine was damn slow... Oh well...
>>
>> Crash in do_nmi -> nmi_enter() -> BUG_ON(in_nmi());
>
> Hmm,. I bet because kmemcheck triggers faults from nmi context because
> it messes about with the page protection bits a lot to track things.
>
> Can't really think of anything except not making perf available on
> kmemcheck kernels.
>
> ---
>  init/Kconfig |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 32745bf..94735b4 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1125,6 +1125,7 @@ config PERF_EVENTS
>        bool "Kernel performance events and counters"
>        default y if (PROFILING || PERF_COUNTERS)
>        depends on HAVE_PERF_EVENTS
> +       depends on !KMEMCHECK
>        select ANON_INODES
>        select IRQ_WORK
>        help

Acked-by: Pekka Enberg <penberg@kernel.org>

Re: [BUG] perf and kmemcheck : fatal combination

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Mon, 2011-04-25 at 18:08 +0200, Eric Dumazet wrote:
> > Hi guys
> > 
> > Just got a panic on a kmemcheck kernel, latest linux-2.6 tree.
> > 
> > I forgot I had kmemcheck enabled, and started "perf top" just because my
> > machine was damn slow... Oh well...
> > 
> > Crash in do_nmi -> nmi_enter() -> BUG_ON(in_nmi());
> 
> Hmm,. I bet because kmemcheck triggers faults from nmi context because
> it messes about with the page protection bits a lot to track things.
> 
> Can't really think of anything except not making perf available on
> kmemcheck kernels.
> 
> ---
>  init/Kconfig |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/init/Kconfig b/init/Kconfig
> index 32745bf..94735b4 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1125,6 +1125,7 @@ config PERF_EVENTS
>  	bool "Kernel performance events and counters"
>  	default y if (PROFILING || PERF_COUNTERS)
>  	depends on HAVE_PERF_EVENTS
> +	depends on !KMEMCHECK
>  	select ANON_INODES
>  	select IRQ_WORK
>  	help

Eric, does it manage to limp along if you remove the BUG_ON()?

That risks NMI recursion but maybe it allows you to see why things are slow, 
before it crashes ;-)

Thanks,

	Ingo

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

Le mardi 26 avril 2011 à 10:04 +0200, Ingo Molnar a écrit :

> Eric, does it manage to limp along if you remove the BUG_ON()?
> 
> That risks NMI recursion but maybe it allows you to see why things are slow, 
> before it crashes ;-)
> 

If I remove the BUG_ON from nmi_enter, it seems to crash very fast 



[-- Attachment #2: capture2.png --]
[-- Type: image/png, Size: 132197 bytes --]

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Le mardi 26 avril 2011 à 10:57 +0200, Eric Dumazet a écrit :
> Le mardi 26 avril 2011 à 10:04 +0200, Ingo Molnar a écrit :
> 
> > Eric, does it manage to limp along if you remove the BUG_ON()?
> > 
> > That risks NMI recursion but maybe it allows you to see why things are slow, 
> > before it crashes ;-)
> > 
> 
> If I remove the BUG_ON from nmi_enter, it seems to crash very fast 
> 
> 

Before you ask, some more complete netconsole traces :



[  306.657192] ------------[ cut here ]------------
[  306.657195] ------------[ cut here ]------------
[  306.657202] WARNING: at arch/x86/mm/kmemcheck/kmemcheck.c:634 kmemcheck_fault+0xa9/0xc0()
[  306.657204] Hardware name: ProLiant BL460c G6
[  306.657205] Modules linked in: nfsd lockd auth_rpcgss sunrpc tg3 libphy sg [last unloaded: x_tables]
[  306.657211] Pid: 3955, comm: perf Not tainted 2.6.39-rc4-00369-g23cf772-dirty #559
[  306.657212] Call Trace:
[  306.657214]  <NMI>  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[  306.657221]  [<ffffffff810427db>] warn_slowpath_common+0x8b/0xc0
[  306.657223]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
[  306.657226]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
[  306.657229]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
[  306.657234]  [<ffffffff811d0289>] ? put_dec+0x59/0x60
[  306.657237]  [<ffffffff811d0591>] ? number+0x301/0x330
[  306.657239]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
[  306.657245]  [<ffffffff8124dce5>] ? vt_console_print+0x85/0x360
[  306.657247]  [<ffffffff8124dcda>] ? vt_console_print+0x7a/0x360
[  306.657250]  [<ffffffff81043159>] __call_console_drivers+0x89/0xa0
[  306.657252]  [<ffffffff810431bb>] _call_console_drivers+0x4b/0x80
[  306.657254]  [<ffffffff810432d7>] console_unlock+0xe7/0x1e0
[  306.657257]  [<ffffffff8104388e>] vprintk+0x1ee/0x4a0
[  306.657260]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[  306.657262]  [<ffffffff81043ba7>] printk+0x67/0x70
[  306.657264]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[  306.657267]  [<ffffffff81042789>] warn_slowpath_common+0x39/0xc0
[  306.657269]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
[  306.657271]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
[  306.657273]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
[  306.657276]  [<ffffffff8101167b>] ? intel_pmu_drain_bts_buffer+0x2b/0x170
[  306.657279]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
[  306.657282]  [<ffffffff8100ef42>] ? x86_perf_event_update+0x12/0x70
[  306.657284]  [<ffffffff810104b1>] ? intel_pmu_save_and_restart+0x11/0x20
[  306.657287]  [<ffffffff81012e84>] intel_pmu_handle_irq+0x1d4/0x420
[  306.657290]  [<ffffffff8147b570>] perf_event_nmi_handler+0x50/0xc0
[  306.657292]  [<ffffffff8147cfa3>] notifier_call_chain+0x53/0x80
[  306.657294]  [<ffffffff8147d018>] __atomic_notifier_call_chain+0x48/0x70
[  306.657296]  [<ffffffff8147d051>] atomic_notifier_call_chain+0x11/0x20
[  306.657298]  [<ffffffff8147d08e>] notify_die+0x2e/0x30
[  306.657300]  [<ffffffff8147a8af>] do_nmi+0x4f/0x200
[  306.657302]  [<ffffffff8147a6ea>] nmi+0x1a/0x20
[  306.657304]  [<ffffffff8100fd4d>] ? intel_pmu_enable_all+0x9d/0x110
[  306.657305]  <<EOE>>  [<ffffffff810104da>] intel_pmu_nhm_enable_all+0x1a/0x120
[  306.657309]  [<ffffffff810131d4>] x86_pmu_enable+0x104/0x260
[  306.657313]  [<ffffffff810a84e9>] perf_pmu_enable+0x39/0x50
[  306.657314]  [<ffffffff8101236c>] x86_pmu_add+0xac/0x120
[  306.657317]  [<ffffffff810aae68>] ? perf_install_in_context+0x18/0xa0
[  306.657319]  [<ffffffff8102b001>] ? kmemcheck_pte_lookup+0x11/0x40
[  306.657322]  [<ffffffff8147a48f>] ? page_fault+0x1f/0x30
[  306.657325]  [<ffffffff810acf15>] event_sched_in+0x65/0x110
[  306.657327]  [<ffffffff810afb95>] __perf_install_in_context+0x125/0x140
[  306.657330]  [<ffffffff810ab100>] ? perf_remove_from_context+0xa0/0xa0
[  306.657332]  [<ffffffff810ab159>] remote_function+0x59/0x70
[  306.657335]  [<ffffffff81075d6e>] smp_call_function_single+0x8e/0x170
[  306.657338]  [<ffffffff810a86a4>] cpu_function_call+0x34/0x40
[  306.657340]  [<ffffffff810afa70>] ? perf_tp_event+0xf0/0xf0
[  306.657342]  [<ffffffff810aaedf>] perf_install_in_context+0x8f/0xa0
[  306.657345]  [<ffffffff810b0792>] sys_perf_event_open+0x592/0x7a0
[  306.657348]  [<ffffffff814819a9>] sysenter_dispatch+0x7/0x27
[  306.657350] ---[ end trace 7333dc2d81c31e96 ]---
[  306.699715] WARNING: at arch/x86/mm/kmemcheck/kmemcheck.c:634 kmemcheck_fault+0xa9/0xc0()
[  306.700659] Hardware name: ProLiant BL460c G6
[  306.701487] Modules linked in: nfsd lockd auth_rpcgss sunrpc tg3 libphy sg [last unloaded: x_tables]
[  306.704964] Pid: 3955, comm: perf Tainted: G        W   2.6.39-rc4-00369-g23cf772-dirty #559
[  306.705922] Call Trace:
[  306.706405]  <NMI>  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[  306.707439]  [<ffffffff810427db>] warn_slowpath_common+0x8b/0xc0
[  306.708173]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
[  306.708893]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
[  306.709597]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
[  306.710301]  [<ffffffff8101167b>] ? intel_pmu_drain_bts_buffer+0x2b/0x170
[  306.711091]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
[  306.711764]  [<ffffffff8100ef42>] ? x86_perf_event_update+0x12/0x70
[  306.712727]  [<ffffffff810104b1>] ? intel_pmu_save_and_restart+0x11/0x20
[  306.713509]  [<ffffffff81012e84>] intel_pmu_handle_irq+0x1d4/0x420
[  306.714254]  [<ffffffff8147b570>] perf_event_nmi_handler+0x50/0xc0
[  306.714999]  [<ffffffff8147cfa3>] notifier_call_chain+0x53/0x80
[  306.715728]  [<ffffffff8147d018>] __atomic_notifier_call_chain+0x48/0x70
[  306.716510]  [<ffffffff8147d051>] atomic_notifier_call_chain+0x11/0x20
[  306.717279]  [<ffffffff8147d08e>] notify_die+0x2e/0x30
[  306.717951]  [<ffffffff8147a8af>] do_nmi+0x4f/0x200
[  306.718605]  [<ffffffff8147a6ea>] nmi+0x1a/0x20
[  306.719237]  [<ffffffff8100fd4d>] ? intel_pmu_enable_all+0x9d/0x110
[  306.719988]  <<EOE>>  [<ffffffff810104da>] intel_pmu_nhm_enable_all+0x1a/0x120
[  306.721347]  [<ffffffff810131d4>] x86_pmu_enable+0x104/0x260
[  306.722056]  [<ffffffff810a84e9>] perf_pmu_enable+0x39/0x50
[  306.722760]  [<ffffffff8101236c>] x86_pmu_add+0xac/0x120
[  306.723445]  [<ffffffff810aae68>] ? perf_install_in_context+0x18/0xa0
[  306.724210]  [<ffffffff8102b001>] ? kmemcheck_pte_lookup+0x11/0x40
[  306.724955]  [<ffffffff8147a48f>] ? page_fault+0x1f/0x30
[  306.725640]  [<ffffffff810acf15>] event_sched_in+0x65/0x110
[  306.726345]  [<ffffffff810afb95>] __perf_install_in_context+0x125/0x140
[  306.727124]  [<ffffffff810ab100>] ? perf_remove_from_context+0xa0/0xa0
[  306.727893]  [<ffffffff810ab159>] remote_function+0x59/0x70
[  306.728597]  [<ffffffff81075d6e>] smp_call_function_single+0x8e/0x170
[  306.729363]  [<ffffffff810a86a4>] cpu_function_call+0x34/0x40
[  306.730079]  [<ffffffff810afa70>] ? perf_tp_event+0xf0/0xf0
[  306.730783]  [<ffffffff810aaedf>] perf_install_in_context+0x8f/0xa0
[  306.731535]  [<ffffffff810b0792>] sys_perf_event_open+0x592/0x7a0
[  306.732277]  [<ffffffff814819a9>] sysenter_dispatch+0x7/0x27
[  306.735272] ---[ end trace 7333dc2d81c31e97 ]---
[  306.736401] BUG: unable to handle kernel 

Re: [BUG] perf and kmemcheck : fatal combination

[Pekka Enberg]

On Tue, Apr 26, 2011 at 12:53 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le mardi 26 avril 2011 à 10:57 +0200, Eric Dumazet a écrit :
>> Le mardi 26 avril 2011 à 10:04 +0200, Ingo Molnar a écrit :
>>
>> > Eric, does it manage to limp along if you remove the BUG_ON()?
>> >
>> > That risks NMI recursion but maybe it allows you to see why things are slow,
>> > before it crashes ;-)
>> >
>>
>> If I remove the BUG_ON from nmi_enter, it seems to crash very fast
>
> Before you ask, some more complete netconsole traces :
>
> [  306.657192] ------------[ cut here ]------------
> [  306.657195] ------------[ cut here ]------------
> [  306.657202] WARNING: at arch/x86/mm/kmemcheck/kmemcheck.c:634 kmemcheck_fault+0xa9/0xc0()
> [  306.657204] Hardware name: ProLiant BL460c G6
> [  306.657205] Modules linked in: nfsd lockd auth_rpcgss sunrpc tg3 libphy sg [last unloaded: x_tables]
> [  306.657211] Pid: 3955, comm: perf Not tainted 2.6.39-rc4-00369-g23cf772-dirty #559
> [  306.657212] Call Trace:
> [  306.657214]  <NMI>  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
> [  306.657221]  [<ffffffff810427db>] warn_slowpath_common+0x8b/0xc0
> [  306.657223]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
> [  306.657226]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
> [  306.657229]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
> [  306.657234]  [<ffffffff811d0289>] ? put_dec+0x59/0x60
> [  306.657237]  [<ffffffff811d0591>] ? number+0x301/0x330
> [  306.657239]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
> [  306.657245]  [<ffffffff8124dce5>] ? vt_console_print+0x85/0x360
> [  306.657247]  [<ffffffff8124dcda>] ? vt_console_print+0x7a/0x360
> [  306.657250]  [<ffffffff81043159>] __call_console_drivers+0x89/0xa0
> [  306.657252]  [<ffffffff810431bb>] _call_console_drivers+0x4b/0x80
> [  306.657254]  [<ffffffff810432d7>] console_unlock+0xe7/0x1e0
> [  306.657257]  [<ffffffff8104388e>] vprintk+0x1ee/0x4a0
> [  306.657260]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
> [  306.657262]  [<ffffffff81043ba7>] printk+0x67/0x70
> [  306.657264]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
> [  306.657267]  [<ffffffff81042789>] warn_slowpath_common+0x39/0xc0
> [  306.657269]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
> [  306.657271]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
> [  306.657273]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
> [  306.657276]  [<ffffffff8101167b>] ? intel_pmu_drain_bts_buffer+0x2b/0x170
> [  306.657279]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
> [  306.657282]  [<ffffffff8100ef42>] ? x86_perf_event_update+0x12/0x70
> [  306.657284]  [<ffffffff810104b1>] ? intel_pmu_save_and_restart+0x11/0x20
> [  306.657287]  [<ffffffff81012e84>] intel_pmu_handle_irq+0x1d4/0x420
> [  306.657290]  [<ffffffff8147b570>] perf_event_nmi_handler+0x50/0xc0
> [  306.657292]  [<ffffffff8147cfa3>] notifier_call_chain+0x53/0x80
> [  306.657294]  [<ffffffff8147d018>] __atomic_notifier_call_chain+0x48/0x70
> [  306.657296]  [<ffffffff8147d051>] atomic_notifier_call_chain+0x11/0x20
> [  306.657298]  [<ffffffff8147d08e>] notify_die+0x2e/0x30
> [  306.657300]  [<ffffffff8147a8af>] do_nmi+0x4f/0x200
> [  306.657302]  [<ffffffff8147a6ea>] nmi+0x1a/0x20
> [  306.657304]  [<ffffffff8100fd4d>] ? intel_pmu_enable_all+0x9d/0x110
> [  306.657305]  <<EOE>>  [<ffffffff810104da>] intel_pmu_nhm_enable_all+0x1a/0x120
> [  306.657309]  [<ffffffff810131d4>] x86_pmu_enable+0x104/0x260
> [  306.657313]  [<ffffffff810a84e9>] perf_pmu_enable+0x39/0x50
> [  306.657314]  [<ffffffff8101236c>] x86_pmu_add+0xac/0x120
> [  306.657317]  [<ffffffff810aae68>] ? perf_install_in_context+0x18/0xa0
> [  306.657319]  [<ffffffff8102b001>] ? kmemcheck_pte_lookup+0x11/0x40
> [  306.657322]  [<ffffffff8147a48f>] ? page_fault+0x1f/0x30
> [  306.657325]  [<ffffffff810acf15>] event_sched_in+0x65/0x110
> [  306.657327]  [<ffffffff810afb95>] __perf_install_in_context+0x125/0x140
> [  306.657330]  [<ffffffff810ab100>] ? perf_remove_from_context+0xa0/0xa0
> [  306.657332]  [<ffffffff810ab159>] remote_function+0x59/0x70
> [  306.657335]  [<ffffffff81075d6e>] smp_call_function_single+0x8e/0x170
> [  306.657338]  [<ffffffff810a86a4>] cpu_function_call+0x34/0x40
> [  306.657340]  [<ffffffff810afa70>] ? perf_tp_event+0xf0/0xf0
> [  306.657342]  [<ffffffff810aaedf>] perf_install_in_context+0x8f/0xa0
> [  306.657345]  [<ffffffff810b0792>] sys_perf_event_open+0x592/0x7a0
> [  306.657348]  [<ffffffff814819a9>] sysenter_dispatch+0x7/0x27
> [  306.657350] ---[ end trace 7333dc2d81c31e96 ]---

That's just kmemcheck fault handler warning about in_nmi(). You could
try to make the relevant perf allocations use __GFP_NOTRACK and/or
SLAB_NOTRACK to avoid page faulting in the perf nmi handler.

                        Pekka

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Le mardi 26 avril 2011 à 13:08 +0300, Pekka Enberg a écrit :

> That's just kmemcheck fault handler warning about in_nmi(). You could
> try to make the relevant perf allocations use __GFP_NOTRACK and/or
> SLAB_NOTRACK to avoid page faulting in the perf nmi handler.

Yes, I am going to try that, thanks

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Le mardi 26 avril 2011 à 12:27 +0200, Eric Dumazet a écrit :
> Le mardi 26 avril 2011 à 13:08 +0300, Pekka Enberg a écrit :
> 
> > That's just kmemcheck fault handler warning about in_nmi(). You could
> > try to make the relevant perf allocations use __GFP_NOTRACK and/or
> > SLAB_NOTRACK to avoid page faulting in the perf nmi handler.
> 
> Yes, I am going to try that, thanks
> 

Thats far from trivial, maybe because we dont have NOTRACK api for
percpu allocations ?

I tried without success following patch

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 632e5dc..bea4949 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1632,7 +1632,7 @@ static int validate_event(struct perf_event *event)
 	struct event_constraint *c;
 	int ret = 0;
 
-	fake_cpuc = kmalloc(sizeof(*fake_cpuc), GFP_KERNEL | __GFP_ZERO);
+	fake_cpuc = kmalloc(sizeof(*fake_cpuc), GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK);
 	if (!fake_cpuc)
 		return -ENOMEM;
 
@@ -1667,7 +1667,7 @@ static int validate_group(struct perf_event *event)
 	int ret, n;
 
 	ret = -ENOMEM;
-	fake_cpuc = kmalloc(sizeof(*fake_cpuc), GFP_KERNEL | __GFP_ZERO);
+	fake_cpuc = kmalloc(sizeof(*fake_cpuc), GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK);
 	if (!fake_cpuc)
 		goto out;
 
diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 43fa20b..a659b61 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -1209,7 +1209,7 @@ static int intel_pmu_cpu_prepare(int cpu)
 		return NOTIFY_OK;
 
 	cpuc->per_core = kzalloc_node(sizeof(struct intel_percore),
-				      GFP_KERNEL, cpu_to_node(cpu));
+				      GFP_KERNEL | ___GFP_NOTRACK, cpu_to_node(cpu));
 	if (!cpuc->per_core)
 		return NOTIFY_BAD;
 
diff --git a/arch/x86/kernel/cpu/perf_event_intel_ds.c b/arch/x86/kernel/cpu/perf_event_intel_ds.c
index bab491b..e921a2f 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_ds.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_ds.c
@@ -84,7 +84,7 @@ static int alloc_pebs_buffer(int cpu)
 	if (!x86_pmu.pebs)
 		return 0;
 
-	buffer = kmalloc_node(PEBS_BUFFER_SIZE, GFP_KERNEL | __GFP_ZERO, node);
+	buffer = kmalloc_node(PEBS_BUFFER_SIZE, GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK, node);
 	if (unlikely(!buffer))
 		return -ENOMEM;
 
@@ -122,7 +122,7 @@ static int alloc_bts_buffer(int cpu)
 	if (!x86_pmu.bts)
 		return 0;
 
-	buffer = kmalloc_node(BTS_BUFFER_SIZE, GFP_KERNEL | __GFP_ZERO, node);
+	buffer = kmalloc_node(BTS_BUFFER_SIZE, GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK, node);
 	if (unlikely(!buffer))
 		return -ENOMEM;
 
@@ -155,7 +155,7 @@ static int alloc_ds_buffer(int cpu)
 	int node = cpu_to_node(cpu);
 	struct debug_store *ds;
 
-	ds = kmalloc_node(sizeof(*ds), GFP_KERNEL | __GFP_ZERO, node);
+	ds = kmalloc_node(sizeof(*ds), GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK, node);
 	if (unlikely(!ds))
 		return -ENOMEM;
 
diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h
index ba36217..8c2e3e6 100644
--- a/include/linux/hardirq.h
+++ b/include/linux/hardirq.h
@@ -211,7 +211,6 @@ extern void irq_exit(void);
 #define nmi_enter()						\
 	do {							\
 		ftrace_nmi_enter();				\
-		BUG_ON(in_nmi());				\
 		add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET);	\
 		lockdep_off();					\
 		rcu_nmi_enter();				\
diff --git a/kernel/perf_event.c b/kernel/perf_event.c
index 8e81a98..b09ba81 100644
--- a/kernel/perf_event.c
+++ b/kernel/perf_event.c
@@ -2589,14 +2589,14 @@ static int alloc_callchain_buffers(void)
 	 */
 	size = offsetof(struct callchain_cpus_entries, cpu_entries[nr_cpu_ids]);
 
-	entries = kzalloc(size, GFP_KERNEL);
+	entries = kzalloc(size, GFP_KERNEL | ___GFP_NOTRACK);
 	if (!entries)
 		return -ENOMEM;
 
 	size = sizeof(struct perf_callchain_entry) * PERF_NR_CONTEXTS;
 
 	for_each_possible_cpu(cpu) {
-		entries->cpu_entries[cpu] = kmalloc_node(size, GFP_KERNEL,
+		entries->cpu_entries[cpu] = kmalloc_node(size, GFP_KERNEL | ___GFP_NOTRACK,
 							 cpu_to_node(cpu));
 		if (!entries->cpu_entries[cpu])
 			goto fail;
@@ -2756,7 +2756,8 @@ alloc_perf_context(struct pmu *pmu, struct task_struct *task)
 {
 	struct perf_event_context *ctx;
 
-	ctx = kzalloc(sizeof(struct perf_event_context), GFP_KERNEL);
+	ctx = kzalloc(sizeof(struct perf_event_context),
+		      GFP_KERNEL | ___GFP_NOTRACK);
 	if (!ctx)
 		return NULL;
 
@@ -3451,7 +3452,7 @@ static void *perf_mmap_alloc_page(int cpu)
 	int node;
 
 	node = (cpu == -1) ? cpu : cpu_to_node(cpu);
-	page = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
+	page = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO | ___GFP_NOTRACK, 0);
 	if (!page)
 		return NULL;
 
@@ -3468,7 +3469,7 @@ perf_buffer_alloc(int nr_pages, long watermark, int cpu, int flags)
 	size = sizeof(struct perf_buffer);
 	size += nr_pages * sizeof(void *);
 
-	buffer = kzalloc(size, GFP_KERNEL);
+	buffer = kzalloc(size, GFP_KERNEL | ___GFP_NOTRACK);
 	if (!buffer)
 		goto fail;
 
@@ -3585,7 +3586,7 @@ perf_buffer_alloc(int nr_pages, long watermark, int cpu, int flags)
 	size = sizeof(struct perf_buffer);
 	size += sizeof(void *);
 
-	buffer = kzalloc(size, GFP_KERNEL);
+	buffer = kzalloc(size, GFP_KERNEL | ___GFP_NOTRACK);
 	if (!buffer)
 		goto fail;
 
@@ -4841,7 +4842,7 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event)
 		 * need to add enough zero bytes after the string to handle
 		 * the 64bit alignment we do later.
 		 */
-		buf = kzalloc(PATH_MAX + sizeof(u64), GFP_KERNEL);
+		buf = kzalloc(PATH_MAX + sizeof(u64), GFP_KERNEL | ___GFP_NOTRACK);
 		if (!buf) {
 			name = strncpy(tmp, "//enomem", sizeof(tmp));
 			goto got_name;
@@ -5385,7 +5386,7 @@ static int swevent_hlist_get_cpu(struct perf_event *event, int cpu)
 	if (!swevent_hlist_deref(swhash) && cpu_online(cpu)) {
 		struct swevent_hlist *hlist;
 
-		hlist = kzalloc(sizeof(*hlist), GFP_KERNEL);
+		hlist = kzalloc(sizeof(*hlist), GFP_KERNEL | ___GFP_NOTRACK);
 		if (!hlist) {
 			err = -ENOMEM;
 			goto exit;
@@ -5969,7 +5970,7 @@ static int pmu_dev_alloc(struct pmu *pmu)
 {
 	int ret = -ENOMEM;
 
-	pmu->dev = kzalloc(sizeof(struct device), GFP_KERNEL);
+	pmu->dev = kzalloc(sizeof(struct device), GFP_KERNEL | ___GFP_NOTRACK);
 	if (!pmu->dev)
 		goto out;
 
@@ -6170,7 +6171,7 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 			return ERR_PTR(-EINVAL);
 	}
 
-	event = kzalloc(sizeof(*event), GFP_KERNEL);
+	event = kzalloc(sizeof(*event), GFP_KERNEL | ___GFP_NOTRACK);
 	if (!event)
 		return ERR_PTR(-ENOMEM);
 
@@ -7222,7 +7223,8 @@ static void __cpuinit perf_event_init_cpu(int cpu)
 	if (swhash->hlist_refcount > 0) {
 		struct swevent_hlist *hlist;
 
-		hlist = kzalloc_node(sizeof(*hlist), GFP_KERNEL, cpu_to_node(cpu));
+		hlist = kzalloc_node(sizeof(*hlist), GFP_KERNEL | ___GFP_NOTRACK,
+				     cpu_to_node(cpu));
 		WARN_ON(!hlist);
 		rcu_assign_pointer(swhash->swevent_hlist, hlist);
 	}

Re: [BUG] perf and kmemcheck : fatal combination

[Peter Zijlstra]

On Tue, 2011-04-26 at 14:27 +0200, Eric Dumazet wrote:
> Thats far from trivial, maybe because we dont have NOTRACK api for
> percpu allocations ? 

We can't use per-cpu allocations from NMI context because of the same
problem, per-cpu uses vmalloc and vmalloc needs faults. Hence that
shouldn't be a problem.

It looks like you covered most of it though, the buffer and the
callchain stuff, aside from that it should only use some static data.

Pekka, what does kmemcheck do for .data and .bss things?

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Le mardi 26 avril 2011 à 14:33 +0200, Peter Zijlstra a écrit :
> On Tue, 2011-04-26 at 14:27 +0200, Eric Dumazet wrote:
> > Thats far from trivial, maybe because we dont have NOTRACK api for
> > percpu allocations ? 
> 
> We can't use per-cpu allocations from NMI context because of the same
> problem, per-cpu uses vmalloc and vmalloc needs faults. Hence that
> shouldn't be a problem.
> 
> It looks like you covered most of it though, the buffer and the
> callchain stuff, aside from that it should only use some static data.
> 
> Pekka, what does kmemcheck do for .data and .bss things?

Hmm, maybe I have a problem because of the WARN_ON in kmemcheck and my
boot had "log_buf_len=32M", so kmemcheck was called again


I am now trying to remve the line 634 from
arch/x86/mm/kmemcheck/kmemcheck.c

[ 4564.554310] WARNING: at arch/x86/mm/kmemcheck/kmemcheck.c:634 kmemcheck_fault+0xa9/0xc0()
[ 4564.554312] Hardware name: ProLiant BL460c G6
[ 4564.554313] Modules linked in: nfsd lockd auth_rpcgss sunrpc tg3 libphy sg [last unloaded: x_tables]
[ 4564.554319] Pid: 4276, comm: perf Not tainted 2.6.39-rc4-00369-g23cf772-dirty #561
[ 4564.554320] Call Trace:
[ 4564.554322]  <NMI>  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[ 4564.554329]  [<ffffffff810427db>] warn_slowpath_common+0x8b/0xc0
[ 4564.554331]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
[ 4564.554333]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
[ 4564.554337]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
[ 4564.554342]  [<ffffffff811d0289>] ? put_dec+0x59/0x60
[ 4564.554344]  [<ffffffff811d0591>] ? number+0x301/0x330
[ 4564.554347]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
[ 4564.554352]  [<ffffffff8124dce5>] ? vt_console_print+0x85/0x360
[ 4564.554355]  [<ffffffff8124dcda>] ? vt_console_print+0x7a/0x360
[ 4564.554358]  [<ffffffff81043159>] __call_console_drivers+0x89/0xa0
[ 4564.554360]  [<ffffffff810431bb>] _call_console_drivers+0x4b/0x80
[ 4564.554362]  [<ffffffff810432d7>] console_unlock+0xe7/0x1e0
[ 4564.554365]  [<ffffffff8104388e>] vprintk+0x1ee/0x4a0
[ 4564.554368]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[ 4564.554370]  [<ffffffff81043ba7>] printk+0x67/0x70
[ 4564.554372]  [<ffffffff8102ac39>] ? kmemcheck_fault+0xa9/0xc0
[ 4564.554375]  [<ffffffff81042789>] warn_slowpath_common+0x39/0xc0
[ 4564.554377]  [<ffffffff81042825>] warn_slowpath_null+0x15/0x20
[ 4564.554379]  [<ffffffff8102ac39>] kmemcheck_fault+0xa9/0xc0
[ 4564.554381]  [<ffffffff8147ca4b>] do_page_fault+0x1fb/0x560
[ 4564.554384]  [<ffffffff8101167b>] ? intel_pmu_drain_bts_buffer+0x2b/0x170
[ 4564.554387]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
[ 4564.554390]  [<ffffffff8100ef42>] ? x86_perf_event_update+0x12/0x70
[ 4564.554392]  [<ffffffff810104b1>] ? intel_pmu_save_and_restart+0x11/0x20
[ 4564.554394]  [<ffffffff81012e84>] intel_pmu_handle_irq+0x1d4/0x420
[ 4564.554397]  [<ffffffff8147b570>] perf_event_nmi_handler+0x50/0xc0
[ 4564.554399]  [<ffffffff8147cfa3>] notifier_call_chain+0x53/0x80
[ 4564.554401]  [<ffffffff8147d018>] __atomic_notifier_call_chain+0x48/0x70
[ 4564.554404]  [<ffffffff8147d051>] atomic_notifier_call_chain+0x11/0x20
[ 4564.554406]  [<ffffffff8147d08e>] notify_die+0x2e/0x30
[ 4564.554407]  [<ffffffff8147a8af>] do_nmi+0x4f/0x200
[ 4564.554409]  [<ffffffff8147a6ea>] nmi+0x1a/0x20
[ 4564.554411]  [<ffffffff8100fd4d>] ? intel_pmu_enable_all+0x9d/0x110

Re: [BUG] perf and kmemcheck : fatal combination

[Eric Dumazet]

Le mardi 26 avril 2011 à 14:56 +0200, Eric Dumazet a écrit :
> Le mardi 26 avril 2011 à 14:33 +0200, Peter Zijlstra a écrit :
> > On Tue, 2011-04-26 at 14:27 +0200, Eric Dumazet wrote:
> > > Thats far from trivial, maybe because we dont have NOTRACK api for
> > > percpu allocations ? 
> > 
> > We can't use per-cpu allocations from NMI context because of the same
> > problem, per-cpu uses vmalloc and vmalloc needs faults. Hence that
> > shouldn't be a problem.
> > 
> > It looks like you covered most of it though, the buffer and the
> > callchain stuff, aside from that it should only use some static data.
> > 
> > Pekka, what does kmemcheck do for .data and .bss things?
> 
> Hmm, maybe I have a problem because of the WARN_ON in kmemcheck and my
> boot had "log_buf_len=32M", so kmemcheck was called again
> 
> 
> I am now trying to remve the line 634 from
> arch/x86/mm/kmemcheck/kmemcheck.c

Yes, its making some progress

[  328.696312] BUG: unable to handle kernel 
[  328.697078] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[  328.698284] BUG: unable to handle kernel paging request at ffff88011fc07b58
[  328.699569] IP: [<ffff88011fc07968>] 0xffff88011fc07967
[  328.700488] PGD 1a94063 PUD 11f6f9067 PMD 11f7f8067 PTE 800000011fc07163
[  328.702327] Oops: 0011 [#1] PREEMPT SMP 
[  328.703693] last sysfs file: /sys/devices/system/cpu/online
[  328.704361] CPU 0 
[  328.704777] Modules linked in: nfsd lockd auth_rpcgss sunrpc tg3 libphy sg [last unloaded: x_tables]
[  328.708693] 
[  328.709075] Pid: 4019, comm: perf Not tainted 2.6.39-rc4-00369-g23cf772-dirty #562 HP ProLiant BL460c G6
[  328.710581] RIP: 3c4c:[<ffff88011fc07968>]  [<ffff88011fc07968>] 0xffff88011fc07967
[  328.711692] RSP: 0000:ffff88011b3cbfd8  EFLAGS: ffff88011fc0793c
[  328.712388] RAX: ffff88011b3ca000 RBX: ffff88011fc07ff8 RCX: ffff88011fc07158
[  328.713180] RDX: ffff88011fc03fc0 RSI: ffff88011fc00000 RDI: ffff88011fc07158
[  328.713959] RBP: ffff88011fc07888 R08: ffffffff81601680 R09: ffff880114278400
[  328.714738] R10: ffffffff81004ed3 R11: ffff88011fc078d8 R12: ffffffff8105eb99
[  328.715516] R13: ffff88011fc07888 R14: 0000000000000018 R15: ffff880112638238
[  328.716294] FS:  0000000000000000(0000) GS:ffff88011fc00000(0063) knlGS:00000000f77176c0

Re: [BUG] perf and kmemcheck : fatal combination

[Mathieu Desnoyers]

* Eric Dumazet (eric.dumazet@gmail.com) wrote:
> Le mardi 26 avril 2011 à 10:57 +0200, Eric Dumazet a écrit :
> > Le mardi 26 avril 2011 à 10:04 +0200, Ingo Molnar a écrit :
> > 
> > > Eric, does it manage to limp along if you remove the BUG_ON()?
> > > 
> > > That risks NMI recursion but maybe it allows you to see why things are slow, 
> > > before it crashes ;-)
> > > 
> > 
> > If I remove the BUG_ON from nmi_enter, it seems to crash very fast 
> > 
> > 
> 
> Before you ask, some more complete netconsole traces :
[...]
> [  306.657279]  [<ffffffff8147a48f>] page_fault+0x1f/0x30
> [  306.657282]  [<ffffffff8100ef42>] ? x86_perf_event_update+0x12/0x70
> [  306.657284]  [<ffffffff810104b1>] ? intel_pmu_save_and_restart+0x11/0x20
> [  306.657287]  [<ffffffff81012e84>] intel_pmu_handle_irq+0x1d4/0x420
> [  306.657290]  [<ffffffff8147b570>] perf_event_nmi_handler+0x50/0xc0
> [  306.657292]  [<ffffffff8147cfa3>] notifier_call_chain+0x53/0x80
> [  306.657294]  [<ffffffff8147d018>] __atomic_notifier_call_chain+0x48/0x70
> [  306.657296]  [<ffffffff8147d051>] atomic_notifier_call_chain+0x11/0x20
> [  306.657298]  [<ffffffff8147d08e>] notify_die+0x2e/0x30
> [  306.657300]  [<ffffffff8147a8af>] do_nmi+0x4f/0x200
> [  306.657302]  [<ffffffff8147a6ea>] nmi+0x1a/0x20
> [  306.657304]  [<ffffffff8100fd4d>] ? intel_pmu_enable_all+0x9d/0x110

just a thought: I've seen this kind of issue with LTTng before, and my
approach is to ensure this does not happen by issuing a
vmalloc_sync_all() call between all vmalloc/vmap calls and accesses to
those memory regions from the tracer code. So it boild down to :

1 - perform all memory allocation at trace session creation (from thread
    context). I do the page table in software (and allocate my buffer
    pages with alloc_pages()), so not page fault is generated by those
    accesses. However, I use kmalloc() to allocate my own
    software-page-table, which uses vmalloc if the allocation is larger
    than a certain threshold. Therefore, I need to issue
    vmalloc_sync_all() before NMI starts using the buffers.

2 - issue vmalloc_sync_all() from the tracer code, after buffer
    allocation, but before the trace session is added to the RCU list of
    active traces.

3 - issue vmalloc_sync_all() when each LTTng module is loaded, before
    they are registered to LTTng, so the memory used to keep their
    code and data is faulted in.

Until we find time and resources to finally implement the virtualized
NMI handling (which handles pages faults within NMIs) as discussed with
Linus last summer, I am staying with this work-around. It might be good
enough for perf too.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: [BUG] perf and kmemcheck : fatal combination

[Pekka Enberg]

On Tue, 2011-04-26 at 14:33 +0200, Peter Zijlstra wrote:
> On Tue, 2011-04-26 at 14:27 +0200, Eric Dumazet wrote:
> > Thats far from trivial, maybe because we dont have NOTRACK api for
> > percpu allocations ? 
> 
> We can't use per-cpu allocations from NMI context because of the same
> problem, per-cpu uses vmalloc and vmalloc needs faults. Hence that
> shouldn't be a problem.
> 
> It looks like you covered most of it though, the buffer and the
> callchain stuff, aside from that it should only use some static data.
> 
> Pekka, what does kmemcheck do for .data and .bss things?

No, kmemcheck is only active for memory allocated with slab and the page
allocator .