* [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions
@ 2016-05-25 17:47 Steven Rostedt
2016-08-08 20:09 ` Steven Rostedt
2016-09-02 14:46 ` Steven Rostedt
0 siblings, 2 replies; 4+ messages in thread
From: Steven Rostedt @ 2016-05-25 17:47 UTC (permalink / raw)
To: LKML
Cc: Jeremy Fitzhardinge, Chris Wright, Alok Kataria, Rusty Russell,
virtualization, Ingo Molnar, H. Peter Anvin, Thomas Gleixner,
Łukasz Daniluk
Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
after enabling function tracer. I asked him to bisect the functions within
available_filter_functions, which he did and it came down to three:
_paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
It was found that this is only an issue when noreplace-paravirt is added to
the kernel command line.
This means that those functions are most likely called within critical
sections of the funtion tracer, and must not be traced.
In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
longer an issue. But both _paravirt_ident_{32,64}() causes the following
splat when they are traced:
mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
Modules linked in: e1000e
CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
RIP: 0010:[<ffffffff81134148>] [<ffffffff81134148>] queued_spin_lock_slowpath+0x118/0x1a0
RSP: 0018:ffff8800d4aefb90 EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
FS: 00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
Stack:
ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
Call Trace:
[<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
[<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
[<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
[<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
[<ffffffff8122e322>] handle_mm_fault+0x312/0x670
[<ffffffff81231068>] ? find_vma+0x68/0x70
[<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
[<ffffffff810aba92>] do_page_fault+0x22/0x30
[<ffffffff81cc7f68>] page_fault+0x28/0x30
[<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
[<ffffffff8129dec5>] ? seq_read+0x305/0x370
[<ffffffff81279668>] __vfs_read+0x28/0xe0
[<ffffffff81279645>] ? __vfs_read+0x5/0xe0
[<ffffffff81279645>] ? __vfs_read+0x5/0xe0
[<ffffffff81279df6>] vfs_read+0x86/0x130
[<ffffffff8127b216>] SyS_read+0x46/0xa0
[<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
Reported-by: Łukasz Daniluk <lukasz.daniluk@intel.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index f08ac28b8136..f975d226be6e 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
".popsection");
/* identity function, which can be inlined */
-u32 _paravirt_ident_32(u32 x)
+u32 notrace _paravirt_ident_32(u32 x)
{
return x;
}
-u64 _paravirt_ident_64(u64 x)
+u64 notrace _paravirt_ident_64(u64 x)
{
return x;
}
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions
2016-05-25 17:47 [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions Steven Rostedt
@ 2016-08-08 20:09 ` Steven Rostedt
2016-09-02 14:46 ` Steven Rostedt
1 sibling, 0 replies; 4+ messages in thread
From: Steven Rostedt @ 2016-08-08 20:09 UTC (permalink / raw)
To: LKML
Cc: Jeremy Fitzhardinge, Chris Wright, Alok Kataria, Rusty Russell,
virtualization, Ingo Molnar, H. Peter Anvin, Thomas Gleixner,
Łukasz Daniluk, x86
Hmm, I'm guessing this patch got lost.
-- Steve
On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
>
> _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
>
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
>
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
>
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
>
> mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
> mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
> mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
> mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
> NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
> Modules linked in: e1000e
> CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
> Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
> task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
> RIP: 0010:[<ffffffff81134148>] [<ffffffff81134148>] queued_spin_lock_slowpath+0x118/0x1a0
> RSP: 0018:ffff8800d4aefb90 EFLAGS: 00000246
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
> RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
> RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
> R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
> R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
> FS: 00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
> Stack:
> ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
> ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
> 0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
> Call Trace:
> [<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
> [<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
> [<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
> [<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
> [<ffffffff8122e322>] handle_mm_fault+0x312/0x670
> [<ffffffff81231068>] ? find_vma+0x68/0x70
> [<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
> [<ffffffff810aba92>] do_page_fault+0x22/0x30
> [<ffffffff81cc7f68>] page_fault+0x28/0x30
> [<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
> [<ffffffff8129dec5>] ? seq_read+0x305/0x370
> [<ffffffff81279668>] __vfs_read+0x28/0xe0
> [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
> [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
> [<ffffffff81279df6>] vfs_read+0x86/0x130
> [<ffffffff8127b216>] SyS_read+0x46/0xa0
> [<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
> Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
> c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
> 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
>
> Reported-by: Łukasz Daniluk <lukasz.daniluk@intel.com>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
>
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
> ".popsection");
>
> /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
> {
> return x;
> }
>
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
> {
> return x;
> }
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions
2016-05-25 17:47 [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions Steven Rostedt
2016-08-08 20:09 ` Steven Rostedt
@ 2016-09-02 14:46 ` Steven Rostedt
2016-09-02 16:41 ` Linus Torvalds
1 sibling, 1 reply; 4+ messages in thread
From: Steven Rostedt @ 2016-09-02 14:46 UTC (permalink / raw)
To: LKML
Cc: Jeremy Fitzhardinge, Chris Wright, Alok Kataria, Rusty Russell,
virtualization, Ingo Molnar, H. Peter Anvin, Thomas Gleixner,
Łukasz Daniluk, Andrew Morton, Linus Torvalds
I just spent half a day bisecting function tracing because I tripped
over this again. I thought this was merged, but I guess it was missed
again.
Can someone please pull this in. And mark it for stable, it goes
probably as far back as 2.6.32.
Thanks!
-- Steve
On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:
> Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
>
> _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
>
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
>
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
>
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
>
> mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
> mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
> mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
> mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
> NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
> Modules linked in: e1000e
> CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
> Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 05/07/2012
> task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
> RIP: 0010:[<ffffffff81134148>] [<ffffffff81134148>] queued_spin_lock_slowpath+0x118/0x1a0
> RSP: 0018:ffff8800d4aefb90 EFLAGS: 00000246
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
> RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
> RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
> R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
> R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
> FS: 00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
> Stack:
> ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
> ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
> 0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
> Call Trace:
> [<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
> [<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
> [<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
> [<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
> [<ffffffff8122e322>] handle_mm_fault+0x312/0x670
> [<ffffffff81231068>] ? find_vma+0x68/0x70
> [<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
> [<ffffffff810aba92>] do_page_fault+0x22/0x30
> [<ffffffff81cc7f68>] page_fault+0x28/0x30
> [<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
> [<ffffffff8129dec5>] ? seq_read+0x305/0x370
> [<ffffffff81279668>] __vfs_read+0x28/0xe0
> [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
> [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
> [<ffffffff81279df6>] vfs_read+0x86/0x130
> [<ffffffff8127b216>] SyS_read+0x46/0xa0
> [<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
> Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
> c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
> 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
>
> Reported-by: Łukasz Daniluk <lukasz.daniluk@intel.com>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
>
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
> ".popsection");
>
> /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
> {
> return x;
> }
>
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
> {
> return x;
> }
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions
2016-09-02 14:46 ` Steven Rostedt
@ 2016-09-02 16:41 ` Linus Torvalds
0 siblings, 0 replies; 4+ messages in thread
From: Linus Torvalds @ 2016-09-02 16:41 UTC (permalink / raw)
To: Steven Rostedt
Cc: LKML, Jeremy Fitzhardinge, Chris Wright, Alok Kataria,
Rusty Russell, virtualization, Ingo Molnar, H. Peter Anvin,
Thomas Gleixner, Łukasz Daniluk, Andrew Morton
On Fri, Sep 2, 2016 at 7:46 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Can someone please pull this in. And mark it for stable, it goes
> probably as far back as 2.6.32.
Applied.
Linus
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-09-02 16:41 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-05-25 17:47 [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions Steven Rostedt
2016-08-08 20:09 ` Steven Rostedt
2016-09-02 14:46 ` Steven Rostedt
2016-09-02 16:41 ` Linus Torvalds
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).