BUG: optimized kprobes illegal instructions in v4.19 stable kernels

BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Mathieu Desnoyers]

Hi,

I notice this commit as a possible culprit of the illegal instructions my lttng
users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
in a Yocto environment [1]. They were able to reproduce the issue with perf
as well.

commit e46daee53bb50bde38805f1823a182979724c229
Author: Kees Cook <keescook@chromium.org>
Date:   Tue Oct 30 22:12:56 2018 +0100

    ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE

I *think* the intent there was to do

-       memcpy(code, &optprobe_template_entry,
+       memcpy(code, (unsigned long *)&optprobe_template_entry,

But if you look at the commit, the "&" seems to have been stripped away,
which happens to change the behavior significantly.

Has this change ever been runtime-tested ?

It has been backported to:
- 4.19 stable as commit 3fe0c68aea21
- 4.14 stable as commit f9e0bc710347

Thanks,

Mathieu

[1] https://bugs.lttng.org/issues/1174

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Kees Cook]

On Mon, Feb 4, 2019 at 7:15 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> Hi,
>
> I notice this commit as a possible culprit of the illegal instructions my lttng
> users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
> in a Yocto environment [1]. They were able to reproduce the issue with perf
> as well.
>
> commit e46daee53bb50bde38805f1823a182979724c229
> Author: Kees Cook <keescook@chromium.org>
> Date:   Tue Oct 30 22:12:56 2018 +0100
>
>     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>
> I *think* the intent there was to do
>
> -       memcpy(code, &optprobe_template_entry,
> +       memcpy(code, (unsigned long *)&optprobe_template_entry,
>
> But if you look at the commit, the "&" seems to have been stripped away,
> which happens to change the behavior significantly.

Yeah, this was a typo on my part. :(

> Has this change ever been runtime-tested ?

I thought I had, given the details from the original bug report, but
clearly it didn't exercise it.

Thanks for fixing this!

-Kees

>
> It has been backported to:
> - 4.19 stable as commit 3fe0c68aea21
> - 4.14 stable as commit f9e0bc710347
>
> Thanks,
>
> Mathieu
>
> [1] https://bugs.lttng.org/issues/1174
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com



-- 
Kees Cook

Re: BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Masami Hiramatsu]

On Tue, 5 Feb 2019 15:06:10 +0000
Kees Cook <keescook@chromium.org> wrote:

> On Mon, Feb 4, 2019 at 7:15 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > Hi,
> >
> > I notice this commit as a possible culprit of the illegal instructions my lttng
> > users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
> > in a Yocto environment [1]. They were able to reproduce the issue with perf
> > as well.
> >
> > commit e46daee53bb50bde38805f1823a182979724c229
> > Author: Kees Cook <keescook@chromium.org>
> > Date:   Tue Oct 30 22:12:56 2018 +0100
> >
> >     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> >
> > I *think* the intent there was to do
> >
> > -       memcpy(code, &optprobe_template_entry,
> > +       memcpy(code, (unsigned long *)&optprobe_template_entry,
> >
> > But if you look at the commit, the "&" seems to have been stripped away,
> > which happens to change the behavior significantly.
> 
> Yeah, this was a typo on my part. :(

Ah, I thought it had been fixed as same as x86.
On x86, all optprobe_template_* are defined as kprobe_opcode_t [],
but on arm, it still be kprobe_opcode_t.

Hmm, but I think we should use kprobe_opcode_t [] or char[] as asm/sections.h does.
OK, I'll prepare for the change.

Thank you,


> 
> > Has this change ever been runtime-tested ?
> 
> I thought I had, given the details from the original bug report, but
> clearly it didn't exercise it.
> 
> Thanks for fixing this!
> 
> -Kees
> 
> >
> > It has been backported to:
> > - 4.19 stable as commit 3fe0c68aea21
> > - 4.14 stable as commit f9e0bc710347
> >
> > Thanks,
> >
> > Mathieu
> >
> > [1] https://bugs.lttng.org/issues/1174
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > http://www.efficios.com
> 
> 
> 
> -- 
> Kees Cook


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Greg KH]

On Wed, Feb 06, 2019 at 01:41:15PM +0900, Masami Hiramatsu wrote:
> On Tue, 5 Feb 2019 15:06:10 +0000
> Kees Cook <keescook@chromium.org> wrote:
> 
> > On Mon, Feb 4, 2019 at 7:15 PM Mathieu Desnoyers
> > <mathieu.desnoyers@efficios.com> wrote:
> > >
> > > Hi,
> > >
> > > I notice this commit as a possible culprit of the illegal instructions my lttng
> > > users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
> > > in a Yocto environment [1]. They were able to reproduce the issue with perf
> > > as well.
> > >
> > > commit e46daee53bb50bde38805f1823a182979724c229
> > > Author: Kees Cook <keescook@chromium.org>
> > > Date:   Tue Oct 30 22:12:56 2018 +0100
> > >
> > >     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> > >
> > > I *think* the intent there was to do
> > >
> > > -       memcpy(code, &optprobe_template_entry,
> > > +       memcpy(code, (unsigned long *)&optprobe_template_entry,
> > >
> > > But if you look at the commit, the "&" seems to have been stripped away,
> > > which happens to change the behavior significantly.
> > 
> > Yeah, this was a typo on my part. :(
> 
> Ah, I thought it had been fixed as same as x86.
> On x86, all optprobe_template_* are defined as kprobe_opcode_t [],
> but on arm, it still be kprobe_opcode_t.
> 
> Hmm, but I think we should use kprobe_opcode_t [] or char[] as asm/sections.h does.
> OK, I'll prepare for the change.

Did this ever get fixed in Linus's tree?  If so, what is the git commit
id, I can't seem to find anything...

thanks,

greg k-h

Re: BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Masami Hiramatsu]

On Mon, 18 Feb 2019 13:26:31 +0100
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Wed, Feb 06, 2019 at 01:41:15PM +0900, Masami Hiramatsu wrote:
> > On Tue, 5 Feb 2019 15:06:10 +0000
> > Kees Cook <keescook@chromium.org> wrote:
> > 
> > > On Mon, Feb 4, 2019 at 7:15 PM Mathieu Desnoyers
> > > <mathieu.desnoyers@efficios.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I notice this commit as a possible culprit of the illegal instructions my lttng
> > > > users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
> > > > in a Yocto environment [1]. They were able to reproduce the issue with perf
> > > > as well.
> > > >
> > > > commit e46daee53bb50bde38805f1823a182979724c229
> > > > Author: Kees Cook <keescook@chromium.org>
> > > > Date:   Tue Oct 30 22:12:56 2018 +0100
> > > >
> > > >     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> > > >
> > > > I *think* the intent there was to do
> > > >
> > > > -       memcpy(code, &optprobe_template_entry,
> > > > +       memcpy(code, (unsigned long *)&optprobe_template_entry,
> > > >
> > > > But if you look at the commit, the "&" seems to have been stripped away,
> > > > which happens to change the behavior significantly.
> > > 
> > > Yeah, this was a typo on my part. :(
> > 
> > Ah, I thought it had been fixed as same as x86.
> > On x86, all optprobe_template_* are defined as kprobe_opcode_t [],
> > but on arm, it still be kprobe_opcode_t.
> > 
> > Hmm, but I think we should use kprobe_opcode_t [] or char[] as asm/sections.h does.
> > OK, I'll prepare for the change.
> 
> Did this ever get fixed in Linus's tree?  If so, what is the git commit
> id, I can't seem to find anything...

I would like to wait for picking up Mathieu's patch which I acked, since
it is simpler and minimum fix. 

https://patchwork.kernel.org/patch/10797511/

After this, I will send my update to change the optprobe_template_*
definitions which will involves wider code, and need to be tested.

Thank you,


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: BUG: optimized kprobes illegal instructions in v4.19 stable kernels

[Mathieu Desnoyers]

----- On Feb 18, 2019, at 7:26 AM, Greg Kroah-Hartman gregkh@linuxfoundation.org wrote:

> On Wed, Feb 06, 2019 at 01:41:15PM +0900, Masami Hiramatsu wrote:
>> On Tue, 5 Feb 2019 15:06:10 +0000
>> Kees Cook <keescook@chromium.org> wrote:
>> 
>> > On Mon, Feb 4, 2019 at 7:15 PM Mathieu Desnoyers
>> > <mathieu.desnoyers@efficios.com> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I notice this commit as a possible culprit of the illegal instructions my lttng
>> > > users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
>> > > in a Yocto environment [1]. They were able to reproduce the issue with perf
>> > > as well.
>> > >
>> > > commit e46daee53bb50bde38805f1823a182979724c229
>> > > Author: Kees Cook <keescook@chromium.org>
>> > > Date:   Tue Oct 30 22:12:56 2018 +0100
>> > >
>> > >     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
>> > >
>> > > I *think* the intent there was to do
>> > >
>> > > -       memcpy(code, &optprobe_template_entry,
>> > > +       memcpy(code, (unsigned long *)&optprobe_template_entry,
>> > >
>> > > But if you look at the commit, the "&" seems to have been stripped away,
>> > > which happens to change the behavior significantly.
>> > 
>> > Yeah, this was a typo on my part. :(
>> 
>> Ah, I thought it had been fixed as same as x86.
>> On x86, all optprobe_template_* are defined as kprobe_opcode_t [],
>> but on arm, it still be kprobe_opcode_t.
>> 
>> Hmm, but I think we should use kprobe_opcode_t [] or char[] as asm/sections.h
>> does.
>> OK, I'll prepare for the change.
> 
> Did this ever get fixed in Linus's tree?  If so, what is the git commit
> id, I can't seem to find anything...

It seems to still be in the arm patch tracking system:

https://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=8834/1

If I understand its status correctly, it is applied to the arm tree, but
perhaps it has not been pulled by Linus yet ? The code is still broken
in Linus' master.

It would be important to get this arm kprobes fix upstream before 5.0
final.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

RE: optimized kprobes illegal instructions in v4.19 stable kernels

[David Laight]

From: Mathieu Desnoyers
> Sent: 04 February 2019 19:15
> I notice this commit as a possible culprit of the illegal instructions my lttng
> users are noticing on arm32 when using kprobes on a v4.19.13 Linux kernel
> in a Yocto environment [1]. They were able to reproduce the issue with perf
> as well.
> 
> commit e46daee53bb50bde38805f1823a182979724c229
> Author: Kees Cook <keescook@chromium.org>
> Date:   Tue Oct 30 22:12:56 2018 +0100
> 
>     ARM: 8806/1: kprobes: Fix false positive with FORTIFY_SOURCE
> 
> I *think* the intent there was to do
> 
> -       memcpy(code, &optprobe_template_entry,
> +       memcpy(code, (unsigned long *)&optprobe_template_entry,
> 
> But if you look at the commit, the "&" seems to have been stripped away,
> which happens to change the behavior significantly.

Which is why you shouldn't add casts to shut up compiler warnings ....

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)