From: Steven Rostedt <rostedt@goodmis.org>
To: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Christian Brauner <brauner@kernel.org>,
Naresh Kamboju <naresh.kamboju@linaro.org>,
open list <linux-kernel@vger.kernel.org>,
linux-stable <stable@vger.kernel.org>,
LTP List <ltp@lists.linux.it>,
lkft-triage@lists.linaro.org,
Stephen Rothwell <sfr@canb.auug.org.au>,
Al Viro <viro@zeniv.linux.org.uk>,
linux-fsdevel@vger.kernel.org, Arnd Bergmann <arnd@arndb.de>,
Dan Carpenter <dan.carpenter@linaro.org>,
Anders Roxell <anders.roxell@linaro.org>,
Linux Trace Kernel <linux-trace-kernel@vger.kernel.org>
Subject: Re: LTP: tracing: RIP: 0010:security_inode_permission+0x5/0x70
Date: Thu, 25 May 2023 00:47:47 -0400 [thread overview]
Message-ID: <20230525004747.26ab2b27@rorschach.local.home> (raw)
In-Reply-To: <20230525082148.af95df069c337fa570f24331@kernel.org>
On Thu, 25 May 2023 08:21:48 +0800
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> On Wed, 24 May 2023 12:39:40 +0200
> Christian Brauner <brauner@kernel.org> wrote:
>
> > On Wed, May 24, 2023 at 02:32:25PM +0530, Naresh Kamboju wrote:
> > > While running LTP tracing tests on qemu-x86_64 following kernel crash noticed
> > > with stable-rc 6.3.4-rc2.
> > >
> > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> > >
> > > FAILED COMMAND File:
> > > /lava-1/0/tests/0_ltp-tracing/automated/linux/ltp/output/LTP_tracing.failed
> > > TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-LTP_tracing.log.tconf
> > > Running tests.......
> >
> > No idea and seems very odd. Only thing I see is that there's ftrace
> > regression testing running while the crash happens.
>
> It sounds like there is a timing issue on int3 self-modifying code @ alternative.c
> ftrace@x86 uses it for arming/disarming mcount(fentry) call.
> I would like to know what is the actual command to run it. The above files are
> local file, and no actual commands recorded in the full log.
>
> Thank you,
>
> >
> > >
> > > <4>[ 57.932577] int3: 0000 [#1] PREEMPT SMP PTI
Was there other text before the above? I believe there should have been
more about what caused this bug.
> > > <4>[ 57.933090] CPU: 0 PID: 138 Comm: systemd-udevd Not tainted 6.3.4-rc2 #1
> > > <4>[ 57.933243] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> > > BIOS 1.14.0-2 04/01/2014
> > > <4>[ 57.933447] RIP: 0010:security_inode_permission+0x5/0x70
The +0x5 above is jumping to after the nop for fentry.
> > > <4>[ 57.934163] Code: c0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00
> > > 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
> > > 0f 1f 00 e8 <e7> 8d 64 18 f6 47 0d 02 75 50 55 48 89 e5 41 55 41 89 f5
Five bytes before the <e7> is 66 0f 1f 00 e8, which is e8 00 1f 0f 66,
which should be a call to fentry. Why this failed, we need more
information though (like perhaps there was more lines at the start
before what is posted here).
-- Steve
> > > 41 54 49
> > > <4>[ 57.934257] RSP: 0018:ffffa300c050bca0 EFLAGS: 00000246
> > > <4>[ 57.934363] RAX: 00000000000041ed RBX: ffff9bda012805c0 RCX:
> > > 0000000000000000
> > > <4>[ 57.934390] RDX: ffff9bda01f6df00 RSI: 0000000000000081 RDI:
> > > ffff9bda012805c0
> > > <4>[ 57.934415] RBP: ffffa300c050bcd0 R08: ffffa300c050bd80 R09:
> > > 00000000ffffff9c
> > > <4>[ 57.934440] R10: 0000000000000fe0 R11: ffc9d09b99d09993 R12:
> > > 0000000000000081
> > > <4>[ 57.934465] R13: 0000000000000000 R14: ffffffffa958f970 R15:
> > > 2f2f2f2f2f2f2f2f
> > > <4>[ 57.934544] FS: 00007fb89665f800(0000)
> > > GS:ffff9bda7bc00000(0000) knlGS:0000000000000000
> > > <4>[ 57.934578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > <4>[ 57.934604] CR2: 00007f60c9b0e000 CR3: 0000000104402000 CR4:
> > > 00000000000006f0
> > > <4>[ 57.934793] Call Trace:
> > > <4>[ 57.934958] <TASK>
> > > <4>[ 57.935082] ? inode_permission+0x70/0x1a0
> > > <4>[ 57.935235] link_path_walk.part.0.constprop.0+0xdd/0x3b0
> > > <4>[ 57.935360] path_lookupat+0x3e/0x190
> > > <4>[ 57.935426] filename_lookup+0xe8/0x1f0
> > > <4>[ 57.935638] user_path_at_empty+0x42/0x60
> > > <4>[ 57.935692] do_fchmodat+0x5f/0xc0
> > > <4>[ 57.935809] __x64_sys_chmod+0x1f/0x30
> > > <4>[ 57.935845] do_syscall_64+0x3e/0x90
> > > <4>[ 57.935886] entry_SYSCALL_64_after_hwframe+0x72/0xdc
> > > <4>[ 57.935996] RIP: 0033:0x7fb8964fbd6b
> > > <4>[ 57.936393] Code: ff ff ff ff eb e6 66 0f 1f 84 00 00 00 00 00
> > > f3 0f 1e fa b8 5f 00 00 00 0f 05 c3 0f 1f 40 00 f3 0f 1e fa b8 5a 00
> > > 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 40 0f 00 f7 d8 64
> > > 89 01 48
> > > <4>[ 57.936422] RSP: 002b:00007ffcf4c2aa38 EFLAGS: 00000202
> > > ORIG_RAX: 000000000000005a
> > > <4>[ 57.936471] RAX: ffffffffffffffda RBX: 0000000000000190 RCX:
> > > 00007fb8964fbd6b
> > > <4>[ 57.936490] RDX: 00007ffcf4c2aa4f RSI: 0000000000000190 RDI:
> > > 00007ffcf4c2aa40
> > > <4>[ 57.936504] RBP: 0000000000000190 R08: 0000000000000000 R09:
> > > 00007ffcf4c2a8e0
> > > <4>[ 57.936528] R10: 0000000000000000 R11: 0000000000000202 R12:
> > > 00007ffcf4c2aa40
> > > <4>[ 57.936542] R13: 0000000000000005 R14: 0000000000000000 R15:
> > > 0000000000000001
> > > <4>[ 57.936754] </TASK>
> > > <4>[ 57.936853] Modules linked in:
> > > <4>[ 57.962890] ---[ end trace 0000000000000000 ]---
> > > <4>[ 57.963006] RIP: 0010:security_inode_permission+0x5/0x70
> > > <4>[ 57.963080] Code: c0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00
> > > 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
> > > 0f 1f 00 e8 <e7> 8d 64 18 f6 47 0d 02 75 50 55 48 89 e5 41 55 41 89 f5
> > > 41 54 49
> > > <4>[ 57.963105] RSP: 0018:ffffa300c050bca0 EFLAGS: 00000246
> > > <4>[ 57.963145] RAX: 00000000000041ed RBX: ffff9bda012805c0 RCX:
> > > 0000000000000000
> > > <4>[ 57.963163] RDX: ffff9bda01f6df00 RSI: 0000000000000081 RDI:
> > > ffff9bda012805c0
> > > <4>[ 57.963180] RBP: ffffa300c050bcd0 R08: ffffa300c050bd80 R09:
> > > 00000000ffffff9c
> > > <4>[ 57.963195] R10: 0000000000000fe0 R11: ffc9d09b99d09993 R12:
> > > 0000000000000081
> > > <4>[ 57.963209] R13: 0000000000000000 R14: ffffffffa958f970 R15:
> > > 2f2f2f2f2f2f2f2f
> > > <4>[ 57.963226] FS: 00007fb89665f800(0000)
> > > GS:ffff9bda7bc00000(0000) knlGS:0000000000000000
> > > <4>[ 57.963246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > <4>[ 57.963261] CR2: 00007f60c9b0e000 CR3: 0000000104402000 CR4:
> > > 00000000000006f0
> > > <0>[ 57.963444] Kernel panic - not syncing: Fatal exception in interrupt
> > > <0>[ 57.964629] Kernel Offset: 0x26600000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> > >
> > > links,
> > > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.3.y/build/v6.3.3-364-ga37c304c022d/testrun/17168198/suite/log-parser-test/tests/
> > > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.3.y/build/v6.3.3-364-ga37c304c022d/testrun/17168198/suite/log-parser-test/test/check-kernel-panic/log
> > > - https://storage.tuxsuite.com/public/linaro/lkft/builds/2QCeudZ18KF3RXw3A5qfr5lPC2N/
> > > - https://storage.tuxsuite.com/public/linaro/lkft/builds/2QCeudZ18KF3RXw3A5qfr5lPC2N/config
> > >
> > > --
> > > Linaro LKFT
> > > https://lkft.linaro.org
>
>
WARNING: multiple messages have this Message-ID (diff)
From: Steven Rostedt <rostedt@goodmis.org>
To: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>,
Christian Brauner <brauner@kernel.org>,
Arnd Bergmann <arnd@arndb.de>,
open list <linux-kernel@vger.kernel.org>,
lkft-triage@lists.linaro.org,
Dan Carpenter <dan.carpenter@linaro.org>,
linux-stable <stable@vger.kernel.org>,
linux-fsdevel@vger.kernel.org,
Linux Trace Kernel <linux-trace-kernel@vger.kernel.org>,
LTP List <ltp@lists.linux.it>, Al Viro <viro@zeniv.linux.org.uk>
Subject: Re: [LTP] LTP: tracing: RIP: 0010:security_inode_permission+0x5/0x70
Date: Thu, 25 May 2023 00:47:47 -0400 [thread overview]
Message-ID: <20230525004747.26ab2b27@rorschach.local.home> (raw)
In-Reply-To: <20230525082148.af95df069c337fa570f24331@kernel.org>
On Thu, 25 May 2023 08:21:48 +0800
Masami Hiramatsu (Google) <mhiramat@kernel.org> wrote:
> On Wed, 24 May 2023 12:39:40 +0200
> Christian Brauner <brauner@kernel.org> wrote:
>
> > On Wed, May 24, 2023 at 02:32:25PM +0530, Naresh Kamboju wrote:
> > > While running LTP tracing tests on qemu-x86_64 following kernel crash noticed
> > > with stable-rc 6.3.4-rc2.
> > >
> > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> > >
> > > FAILED COMMAND File:
> > > /lava-1/0/tests/0_ltp-tracing/automated/linux/ltp/output/LTP_tracing.failed
> > > TCONF COMMAND File: /opt/ltp/output/LTP_RUN_ON-LTP_tracing.log.tconf
> > > Running tests.......
> >
> > No idea and seems very odd. Only thing I see is that there's ftrace
> > regression testing running while the crash happens.
>
> It sounds like there is a timing issue on int3 self-modifying code @ alternative.c
> ftrace@x86 uses it for arming/disarming mcount(fentry) call.
> I would like to know what is the actual command to run it. The above files are
> local file, and no actual commands recorded in the full log.
>
> Thank you,
>
> >
> > >
> > > <4>[ 57.932577] int3: 0000 [#1] PREEMPT SMP PTI
Was there other text before the above? I believe there should have been
more about what caused this bug.
> > > <4>[ 57.933090] CPU: 0 PID: 138 Comm: systemd-udevd Not tainted 6.3.4-rc2 #1
> > > <4>[ 57.933243] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> > > BIOS 1.14.0-2 04/01/2014
> > > <4>[ 57.933447] RIP: 0010:security_inode_permission+0x5/0x70
The +0x5 above is jumping to after the nop for fentry.
> > > <4>[ 57.934163] Code: c0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00
> > > 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
> > > 0f 1f 00 e8 <e7> 8d 64 18 f6 47 0d 02 75 50 55 48 89 e5 41 55 41 89 f5
Five bytes before the <e7> is 66 0f 1f 00 e8, which is e8 00 1f 0f 66,
which should be a call to fentry. Why this failed, we need more
information though (like perhaps there was more lines at the start
before what is posted here).
-- Steve
> > > 41 54 49
> > > <4>[ 57.934257] RSP: 0018:ffffa300c050bca0 EFLAGS: 00000246
> > > <4>[ 57.934363] RAX: 00000000000041ed RBX: ffff9bda012805c0 RCX:
> > > 0000000000000000
> > > <4>[ 57.934390] RDX: ffff9bda01f6df00 RSI: 0000000000000081 RDI:
> > > ffff9bda012805c0
> > > <4>[ 57.934415] RBP: ffffa300c050bcd0 R08: ffffa300c050bd80 R09:
> > > 00000000ffffff9c
> > > <4>[ 57.934440] R10: 0000000000000fe0 R11: ffc9d09b99d09993 R12:
> > > 0000000000000081
> > > <4>[ 57.934465] R13: 0000000000000000 R14: ffffffffa958f970 R15:
> > > 2f2f2f2f2f2f2f2f
> > > <4>[ 57.934544] FS: 00007fb89665f800(0000)
> > > GS:ffff9bda7bc00000(0000) knlGS:0000000000000000
> > > <4>[ 57.934578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > <4>[ 57.934604] CR2: 00007f60c9b0e000 CR3: 0000000104402000 CR4:
> > > 00000000000006f0
> > > <4>[ 57.934793] Call Trace:
> > > <4>[ 57.934958] <TASK>
> > > <4>[ 57.935082] ? inode_permission+0x70/0x1a0
> > > <4>[ 57.935235] link_path_walk.part.0.constprop.0+0xdd/0x3b0
> > > <4>[ 57.935360] path_lookupat+0x3e/0x190
> > > <4>[ 57.935426] filename_lookup+0xe8/0x1f0
> > > <4>[ 57.935638] user_path_at_empty+0x42/0x60
> > > <4>[ 57.935692] do_fchmodat+0x5f/0xc0
> > > <4>[ 57.935809] __x64_sys_chmod+0x1f/0x30
> > > <4>[ 57.935845] do_syscall_64+0x3e/0x90
> > > <4>[ 57.935886] entry_SYSCALL_64_after_hwframe+0x72/0xdc
> > > <4>[ 57.935996] RIP: 0033:0x7fb8964fbd6b
> > > <4>[ 57.936393] Code: ff ff ff ff eb e6 66 0f 1f 84 00 00 00 00 00
> > > f3 0f 1e fa b8 5f 00 00 00 0f 05 c3 0f 1f 40 00 f3 0f 1e fa b8 5a 00
> > > 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 40 0f 00 f7 d8 64
> > > 89 01 48
> > > <4>[ 57.936422] RSP: 002b:00007ffcf4c2aa38 EFLAGS: 00000202
> > > ORIG_RAX: 000000000000005a
> > > <4>[ 57.936471] RAX: ffffffffffffffda RBX: 0000000000000190 RCX:
> > > 00007fb8964fbd6b
> > > <4>[ 57.936490] RDX: 00007ffcf4c2aa4f RSI: 0000000000000190 RDI:
> > > 00007ffcf4c2aa40
> > > <4>[ 57.936504] RBP: 0000000000000190 R08: 0000000000000000 R09:
> > > 00007ffcf4c2a8e0
> > > <4>[ 57.936528] R10: 0000000000000000 R11: 0000000000000202 R12:
> > > 00007ffcf4c2aa40
> > > <4>[ 57.936542] R13: 0000000000000005 R14: 0000000000000000 R15:
> > > 0000000000000001
> > > <4>[ 57.936754] </TASK>
> > > <4>[ 57.936853] Modules linked in:
> > > <4>[ 57.962890] ---[ end trace 0000000000000000 ]---
> > > <4>[ 57.963006] RIP: 0010:security_inode_permission+0x5/0x70
> > > <4>[ 57.963080] Code: c0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00
> > > 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66
> > > 0f 1f 00 e8 <e7> 8d 64 18 f6 47 0d 02 75 50 55 48 89 e5 41 55 41 89 f5
> > > 41 54 49
> > > <4>[ 57.963105] RSP: 0018:ffffa300c050bca0 EFLAGS: 00000246
> > > <4>[ 57.963145] RAX: 00000000000041ed RBX: ffff9bda012805c0 RCX:
> > > 0000000000000000
> > > <4>[ 57.963163] RDX: ffff9bda01f6df00 RSI: 0000000000000081 RDI:
> > > ffff9bda012805c0
> > > <4>[ 57.963180] RBP: ffffa300c050bcd0 R08: ffffa300c050bd80 R09:
> > > 00000000ffffff9c
> > > <4>[ 57.963195] R10: 0000000000000fe0 R11: ffc9d09b99d09993 R12:
> > > 0000000000000081
> > > <4>[ 57.963209] R13: 0000000000000000 R14: ffffffffa958f970 R15:
> > > 2f2f2f2f2f2f2f2f
> > > <4>[ 57.963226] FS: 00007fb89665f800(0000)
> > > GS:ffff9bda7bc00000(0000) knlGS:0000000000000000
> > > <4>[ 57.963246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > <4>[ 57.963261] CR2: 00007f60c9b0e000 CR3: 0000000104402000 CR4:
> > > 00000000000006f0
> > > <0>[ 57.963444] Kernel panic - not syncing: Fatal exception in interrupt
> > > <0>[ 57.964629] Kernel Offset: 0x26600000 from 0xffffffff81000000
> > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> > >
> > >
> > > links,
> > > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.3.y/build/v6.3.3-364-ga37c304c022d/testrun/17168198/suite/log-parser-test/tests/
> > > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.3.y/build/v6.3.3-364-ga37c304c022d/testrun/17168198/suite/log-parser-test/test/check-kernel-panic/log
> > > - https://storage.tuxsuite.com/public/linaro/lkft/builds/2QCeudZ18KF3RXw3A5qfr5lPC2N/
> > > - https://storage.tuxsuite.com/public/linaro/lkft/builds/2QCeudZ18KF3RXw3A5qfr5lPC2N/config
> > >
> > > --
> > > Linaro LKFT
> > > https://lkft.linaro.org
>
>
--
Mailing list info: https://lists.linux.it/listinfo/ltp
next prev parent reply other threads:[~2023-05-25 4:48 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-24 9:02 [LTP] LTP: tracing: RIP: 0010:security_inode_permission+0x5/0x70 Naresh Kamboju
2023-05-24 9:02 ` Naresh Kamboju
2023-05-24 10:39 ` [LTP] " Christian Brauner
2023-05-24 10:39 ` Christian Brauner
2023-05-25 0:21 ` Masami Hiramatsu
2023-05-25 0:21 ` [LTP] " Masami Hiramatsu
2023-05-25 4:47 ` Steven Rostedt [this message]
2023-05-25 4:47 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230525004747.26ab2b27@rorschach.local.home \
--to=rostedt@goodmis.org \
--cc=anders.roxell@linaro.org \
--cc=arnd@arndb.de \
--cc=brauner@kernel.org \
--cc=dan.carpenter@linaro.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=lkft-triage@lists.linaro.org \
--cc=ltp@lists.linux.it \
--cc=mhiramat@kernel.org \
--cc=naresh.kamboju@linaro.org \
--cc=sfr@canb.auug.org.au \
--cc=stable@vger.kernel.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.