From: Johannes Erdfelt <johannes@erdfelt.com>
To: Steven Rostedt <rostedt@goodmis.org>,
Ingo Molnar <mingo@redhat.com>,
live-patching@vger.kernel.org
Subject: ftrace regression with CONFIG_UNWINDER_FRAME_POINTER
Date: Mon, 15 Aug 2022 12:58:28 -0700 [thread overview]
Message-ID: <20220815195828.GE3948@sventech.com> (raw)
I recently ran across an oops with ftrace when using the frame pointer
unwinder. It does not affect Linus' tree (tested 6.0-rc1), but it does
affect 5.15.60 and 5.10.136 (at least, have not tested other kernels).
It appears to be related to the recently merged retbleed mitigations,
in particular the unret mitigation. I originally reproduced it with
a live patch (livepatch-sample.ko can reproduce it) but can also
reproduce by setting a function filter with ftrace.
Oops from loading live patch:
[ 0.672824] livepatch: 'livepatch_sample': patching complete
[ 0.673809] BUG: unable to handle page fault for address: ffffffffc14dd888
[ 0.674693] #PF: supervisor instruction fetch in kernel mode
[ 0.675441] #PF: error_code(0x0010) - not-present page
[ 0.676098] PGD 1ae10067 P4D 1ae10067 PUD 1ae12067 PMD 0
[ 0.676790] Oops: 0010 [#1] SMP PTI
[ 0.677239] CPU: 0 PID: 347 Comm: cat Tainted: G K 5.15.60 #1
[ 0.678127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ArchLinux 1.16.0-1 04/01/2014
[ 0.679318] RIP: 0010:0xffffffffc14dd888
[ 0.679820] Code: Unable to access opcode bytes at RIP 0xffffffffc14dd85e.
[ 0.680689] RSP: 0018:ffffac644075bc68 EFLAGS: 00010282
[ 0.681352] RAX: ffffffffa74d9cf0 RBX: 0000000000000000 RCX: 0000000000000001
[ 0.682282] RDX: 0000000000400cc0 RSI: 0000000000000001 RDI: ffff970c8245f000
[ 0.683184] RBP: ffffac644075bcc8 R08: 0000000000001000 R09: 0000000000000001
[ 0.684081] R10: 0000000000000000 R11: ffff970c81868200 R12: ffffac644075bd20
[ 0.684982] R13: ffffac644075bcf8 R14: 0000000000000001 R15: ffff970c8245f000
[ 0.685925] FS: 0000000000713198(0000) GS:ffff970c9f400000(0000) knlGS:0000000000000000
[ 0.686949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.687677] CR2: ffffffffc14dd85e CR3: 000000000105c003 CR4: 0000000000370ef0
[ 0.688579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.689513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.690411] Call Trace:
[ 0.690730] <TASK>
[ 0.691006] ? 0xffffffffc03c9000
[ 0.691425] ? seq_read_iter+0x10e/0x440
[ 0.691955] proc_reg_read_iter+0x52/0x90
[ 0.692487] generic_file_splice_read+0xf1/0x190
[ 0.693079] do_splice_to+0x7d/0xc0
[ 0.693528] splice_direct_to_actor+0xbd/0x220
[ 0.694100] ? generic_file_splice_read+0x190/0x190
[ 0.694723] do_splice_direct+0x8a/0xd0
[ 0.695214] do_sendfile+0x352/0x4b0
[ 0.695751] __x64_sys_sendfile64+0xa9/0xc0
[ 0.696285] ? __context_tracking_exit+0x57/0x90
[ 0.696868] do_syscall_64+0x37/0x90
[ 0.697324] entry_SYSCALL_64_after_hwframe+0x61/0xcb
[ 0.697964] RIP: 0033:0x4bf81e
[ 0.698361] Code: de e1 fe 50 be 69 19 12 28 b8 a9 00 00 00 0f 05 48 89 c7 e8 84 0a f4 ff 5a c3 49 89 ca 48 63 f6 48 63 ff b8 28 00 00 00 0f 05 <48> 89 c7 e9 6a 0a f4 ff 50 b8 74 00 00 00 0f 05 48 89 c7 e8 5a 0a
[ 0.700729] RSP: 002b:00007ffd027102b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
[ 0.701670] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004bf81e
[ 0.702594] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[ 0.703483] RBP: 00007ffd02710310 R08: 0000000000000000 R09: 0000000001000000
[ 0.704375] R10: 0000000001000000 R11: 0000000000000246 R12: 000000000000001b
[ 0.705285] R13: 0000000000000001 R14: 0000000001000000 R15: 0000000000000001
[ 0.706197] </TASK>
[ 0.706480] Modules linked in: livepatch_sample(K)
[ 0.707085] CR2: ffffffffc14dd888
[ 0.707508] ---[ end trace f6f689798d1c9538 ]---
[ 0.708094] RIP: 0010:0xffffffffc14dd888
[ 0.708610] Code: Unable to access opcode bytes at RIP 0xffffffffc14dd85e.
[ 0.709480] RSP: 0018:ffffac644075bc68 EFLAGS: 00010282
[ 0.710136] RAX: ffffffffa74d9cf0 RBX: 0000000000000000 RCX: 0000000000000001
[ 0.711026] RDX: 0000000000400cc0 RSI: 0000000000000001 RDI: ffff970c8245f000
[ 0.711915] RBP: ffffac644075bcc8 R08: 0000000000001000 R09: 0000000000000001
[ 0.712864] R10: 0000000000000000 R11: ffff970c81868200 R12: ffffac644075bd20
[ 0.713753] R13: ffffac644075bcf8 R14: 0000000000000001 R15: ffff970c8245f000
[ 0.714649] FS: 0000000000713198(0000) GS:ffff970c9f400000(0000) knlGS:0000000000000000
[ 0.715705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.716426] CR2: ffffffffc14dd85e CR3: 000000000105c003 CR4: 0000000000370ef0
[ 0.717317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.718208] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Killed
[ 0.720709] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[ 0.721740] Kernel Offset: 0x26000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 0.723133] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 ]---
Oops from setting ftrace filter:
[ 0.644436] BUG: unable to handle page fault for address: ffffffffc1172888
[ 0.645244] #PF: supervisor instruction fetch in kernel mode
[ 0.645902] #PF: error_code(0x0010) - not-present page
[ 0.646536] PGD 18e10067 P4D 18e10067 PUD 18e12067 PMD 0
[ 0.647166] Oops: 0010 [#1] SMP PTI
[ 0.647577] CPU: 0 PID: 1 Comm: init Not tainted 5.15.60 #1
[ 0.648227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ArchLinux 1.16.0-1 04/01/2014
[ 0.649284] BUG: unable to handle page fault for address: ffffffffc1172888
[ 0.650117] #PF: supervisor instruction fetch in kernel mode
[ 0.650772] #PF: error_code(0x0010) - not-present page
[ 0.651370] PGD 18e10067 P4D 18e10067 PUD 18e12067 PMD 0
[ 0.651998] Oops: 0010 [#2] SMP PTI
[ 0.652409] CPU: 0 PID: 1 Comm: init Not tainted 5.15.60 #1
[ 0.653059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ArchLinux 1.16.0-1 04/01/2014
Oddly this oops is incomplete but the portion that does get emitted
repeats a total of 4 times.
Requirements I've found to reproduce the oops:
- Recent LTS kernel (reproduced with 5.15.60 and 5.10.136 at least)
- I could not cause the oops with Linus' tree (6.0-rc1 tested)
- Have not tested other stable trees
- CONFIG_RETHUNK enabled
- CONFIG_UNWINDER_FRAME_POINTER enabled
- CONFIG_UNWINDER_ORC does not cause an oops
- Using the default retbleed mitigations
- Using retbleed=ibpb does not cause an oops on an AMD host
I've put together a quick reproducer using qemu. This tarball includes
a kernel and a minimal initrd that will reproduce the oops. Also
included is the .config used. I built this test kernel using gcc 12.1.0
(12.1.0-2) on an Arch Linux host but have also reproduced the oops using
gcc 9.3.0 (9.3.0-17ubuntu1~20.04) on an Ubuntu host.
http://johannes.erdfelt.com/ftrace-frame-pointer-regression.tar.gz
JE
next reply other threads:[~2022-08-15 23:28 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-15 19:58 Johannes Erdfelt [this message]
2022-08-15 20:06 ` ftrace regression with CONFIG_UNWINDER_FRAME_POINTER Steven Rostedt
2022-08-15 20:16 ` Borislav Petkov
2022-08-15 20:29 ` Johannes Erdfelt
2022-08-16 18:57 ` Paul Gortmaker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220815195828.GE3948@sventech.com \
--to=johannes@erdfelt.com \
--cc=live-patching@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.