Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Wed, 3 Apr 2024 19:00:07 -0700
Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Wed, Apr 3, 2024 at 5:58 PM Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > On Wed, 3 Apr 2024 09:58:12 -0700
> > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> > > On Wed, Apr 3, 2024 at 7:09 AM Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > > >
> > > > On Wed, 3 Apr 2024 11:47:41 +0200
> > > > Jiri Olsa <olsajiri@gmail.com> wrote:
> > > >
> > > > > On Wed, Apr 03, 2024 at 10:07:08AM +0900, Masami Hiramatsu wrote:
> > > > > > Hi Jiri,
> > > > > >
> > > > > > On Tue,  2 Apr 2024 11:33:00 +0200
> > > > > > Jiri Olsa <jolsa@kernel.org> wrote:
> > > > > >
> > > > > > > Adding uretprobe syscall instead of trap to speed up return probe.
> > > > > >
> > > > > > This is interesting approach. But I doubt we need to add additional
> > > > > > syscall just for this purpose. Can't we use another syscall or ioctl?
> > > > >
> > > > > so the plan is to optimize entry uprobe in a similar way and given
> > > > > the syscall is not a scarce resource I wanted to add another syscall
> > > > > for that one as well
> > > > >
> > > > > tbh I'm not sure sure which syscall or ioctl to reuse for this, it's
> > > > > possible to do that, the trampoline will just have to save one or
> > > > > more additional registers, but adding new syscall seems cleaner to me
> > > >
> > > > Hmm, I think a similar syscall is ptrace? prctl may also be a candidate.
> > >
> > > I think both ptrace and prctl are for completely different use cases
> > > and it would be an abuse of existing API to reuse them for uretprobe
> > > tracing. Also, keep in mind, that any extra argument that has to be
> > > passed into this syscall means that we need to complicate and slow
> > > generated assembly code that is injected into user process (to
> > > save/restore registers) and also kernel-side (again, to deal with all
> > > the extra registers that would be stored/restored on stack).
> > >
> > > Given syscalls are not some kind of scarce resources, what's the
> > > downside to have a dedicated and simple syscall?
> >
> > Syscalls are explicitly exposed to user space, thus, even if it is used
> > ONLY for a very specific situation, it is an official kernel interface,
> > and need to care about the compatibility. (If it causes SIGILL unless
> > a specific use case, I don't know there is a "compatibility".)
> 
> Check rt_sigreturn syscall (manpage at [0], for example).
> 
>        sigreturn() exists only to allow the implementation of signal
>        handlers.  It should never be called directly.  (Indeed, a simple
>        sigreturn() wrapper in the GNU C library simply returns -1, with
>        errno set to ENOSYS.)  Details of the arguments (if any) passed
>        to sigreturn() vary depending on the architecture.  (On some
>        architectures, such as x86-64, sigreturn() takes no arguments,
>        since all of the information that it requires is available in the
>        stack frame that was previously created by the kernel on the
>        user-space stack.)
> 
> This is a very similar use case. Also, check its source code in
> arch/x86/kernel/signal_64.c. It sends SIGSEGV to the calling process
> on any sign of something not being right. It's exactly the same with
> sys_uretprobe.
> 
>   [0] https://man7.org/linux/man-pages/man2/sigreturn.2.html

Thanks for a good example.
Hm, in the case of rt_sigreturn, it has no other way to do it so it
needs to use syscall. OTOH, sys_uretprobe is only for performance
optimization, and the performance may depend on the architecture.

> > And the number of syscalls are limited resource.
> 
> We have almost 500 of them, it didn't seems like adding 1-2 for good
> reasons would be a problem. Can you please point to where the limits
> on syscalls as a resource are described? I'm curious to learn.

Syscall table is compiled as a fixed array, so if we increase
the number, we need more tables. Of course this just increase 1 entry
and at least for x86 we already allocated bigger table, so it is OK.
But I'm just afraid if we can add more syscalls without any clear
rules, we may fill the tables with more specific syscalls.

Ah, we also should follow this document.

https://docs.kernel.org/process/adding-syscalls.html

Let me Cc linux-api@vger.kernel.org.

> >
> > I'm actually not sure how much we need to care of it, but adding a new
> > syscall is worth to be discussed carefully because all of them are
> > user-space compatibility.
> 
> Absolutely, it's a good discussion to have.

Thanks, if this is discussed enough and agreed from other maintainers,
I can safely pick this on my tree.

> 
> >
> > > > > > Also, we should run syzkaller on this syscall. And if uretprobe is
> > > > >
> > > > > right, I'll check on syzkaller
> > > > >
> > > > > > set in the user function, what happen if the user function directly
> > > > > > calls this syscall? (maybe it consumes shadow stack?)
> > > > >
> > > > > the process should receive SIGILL if there's no pending uretprobe for
> > > > > the current task, or it will trigger uretprobe if there's one pending
> > > >
> > > > No, that is too aggressive and not safe. Since the syscall is exposed to
> > > > user program, it should return appropriate error code instead of SIGILL.
> > > >
> > >
> > > This is the way it is today with uretprobes even through interrupt.
> >
> > I doubt that the interrupt (exception) and syscall should be handled
> > differently. Especially, this exception is injected by uprobes but
> > syscall will be caused by itself. But syscall can be called from user
> > program (of couse this works as sys_kill(self, SIGILL)).
> 
> Yep, I'd keep the behavior the same between uretprobes implemented
> through int3 and sys_uretprobe.

OK, so this syscall is something like coding int3 without debugger.

> >
> > > E.g., it could happen that user process is using fibers and is
> > > replacing stack pointer without kernel realizing this, which will
> > > trigger some defensive checks in uretprobe handling code and kernel
> > > will send SIGILL because it can't support such cases. This is
> > > happening today already, and it works fine in practice (except for
> > > applications that manually change stack pointer, too bad, you can't
> > > trace them with uretprobes, unfortunately).
> >
> > OK, we at least need to document it.
> 
> +1, yep

Can we make this syscall and uprobe behavior clearer? As you said, if
the application use sigreturn or longjump, it may skip returns and
shadow stack entries are left in the kernel. In such cases, can uretprobe
detect it properly, or just crash the process (or process runs wrongly)?

> 
> >
> > >
> > > So I think it's absolutely adequate to have this behavior if the user
> > > process is *intentionally* abusing this API.
> >
> > Of course user expected that it is abusing. So at least we need to
> > add a document that this syscall number is reserved to uprobes and
> > user program must not use it.
> >
> 
> Totally agree about documenting this.
> 
> > >
> > > > >
> > > > > but we could limit the syscall to be executed just from the trampoline,
> > > > > that should prevent all the user space use cases, I'll do that in next
> > > > > version and add more tests for that
> > > >
> > > > Why not limit? :) The uprobe_handle_trampoline() expects it is called
> > > > only from the trampoline, so it is natural to check the caller address.
> > > > (and uprobe should know where is the trampoline)
> > > >
> > > > Since the syscall is always exposed to the user program, it should
> > > > - Do nothing and return an error unless it is properly called.
> > > > - check the prerequisites for operation strictly.
> > > > I concern that new system calls introduce vulnerabilities.
> > > >
> > >
> > > As Oleg and Jiri mentioned, this syscall can't harm kernel or other
> > > processes, only the process that is abusing the API. So any extra
> > > checks that would slow down this approach is an unnecessary overhead
> > > and complication that will never be useful in practice.
> >
> > I think at least it should check the caller address to ensure the
> > address is in the trampoline.
> > But anyway, uprobes itself can break the target process, so no one
> > might care if this system call breaks the process now.
> 
> If we already have an expected range of addresses, then I think it's
> fine to do a quick unlikely() check. I'd be more concerned if we need
> to do another lookup or search to just validate this. I'm sure Jiri
> will figure it out.

Good.

> 
> >
> > >
> > > Also note that sys_uretprobe is a kind of internal and unstable API
> > > and it is explicitly called out that its contract can change at any
> > > time and user space shouldn't rely on it. It's purely for the kernel's
> > > own usage.
> >
> > Is that OK to use a syscall as "internal" and "unstable" API?
> 
> See above about rt_sigreturn. It seems like yes, for some highly
> specialized syscalls it is the case already.

OK, but as I said it is just for performance optimization, that is
a bit different from rt_sigreturn case.

Thank you,

> >
> > >
> > > So let's please keep it fast and simple.
> > >
> > >
> > > > Thank you,
> > > >
> > > >
> > > > >
> > > > > thanks,
> > > > > jirka
> > > > >
> > > > >
> > > > > >
> > >
> > > [...]
> >
> >
> > ([OT] If we can add syscall so casually, I would like to add sys_traceevent
> > for recording user space events :-) .)
> 
> Have you proposed this upstream? :) I have no clue and no opinion about it...
> 
> >
> > --
> > Masami Hiramatsu (Google) <mhiramat@kernel.org>


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Oleg Nesterov]

On 04/05, Masami Hiramatsu wrote:
>
> Can we make this syscall and uprobe behavior clearer? As you said, if
> the application use sigreturn or longjump, it may skip returns and
> shadow stack entries are left in the kernel. In such cases, can uretprobe
> detect it properly, or just crash the process (or process runs wrongly)?

Please see the comment in handle_trampoline(), it tries to detect this case.
This patch should not make any difference.

Oleg.

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Thu, 4 Apr 2024 18:11:09 +0200
Oleg Nesterov <oleg@redhat.com> wrote:

> On 04/05, Masami Hiramatsu wrote:
> >
> > Can we make this syscall and uprobe behavior clearer? As you said, if
> > the application use sigreturn or longjump, it may skip returns and
> > shadow stack entries are left in the kernel. In such cases, can uretprobe
> > detect it properly, or just crash the process (or process runs wrongly)?
> 
> Please see the comment in handle_trampoline(), it tries to detect this case.
> This patch should not make any difference.

I think you mean this loop will skip and discard the stacked return_instance
to find the valid one.

----
        do {
                /*
                 * We should throw out the frames invalidated by longjmp().
                 * If this chain is valid, then the next one should be alive
                 * or NULL; the latter case means that nobody but ri->func
                 * could hit this trampoline on return. TODO: sigaltstack().
                 */
                next = find_next_ret_chain(ri);
                valid = !next || arch_uretprobe_is_alive(next, RP_CHECK_RET, regs);

                instruction_pointer_set(regs, ri->orig_ret_vaddr);
                do {
                        if (valid)
                                handle_uretprobe_chain(ri, regs);
                        ri = free_ret_instance(ri);
                        utask->depth--;
                } while (ri != next);
        } while (!valid);
----

I think this expects setjmp/longjmp as below

foo() { <- retprobe1
	setjmp()
	bar() { <- retprobe2
		longjmp()
	}
} <- return to trampoline

In this case, we need to skip retprobe2's instance.
My concern is, if we can not find appropriate return instance, what happen?
e.g.

foo() { <-- retprobe1
   bar() { # sp is decremented
       sys_uretprobe() <-- ??
    }
}

It seems sys_uretprobe() will handle retprobe1 at that point instead of
SIGILL.

Can we avoid this with below strict check?

if (ri->stack != regs->sp + expected_offset)
	goto sigill;

expected_offset should be 16 (push * 3 - ret) on x64 if we ri->stack is the
regs->sp right after call.

Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Jiri Olsa]

On Fri, Apr 05, 2024 at 10:22:03AM +0900, Masami Hiramatsu wrote:
> On Thu, 4 Apr 2024 18:11:09 +0200
> Oleg Nesterov <oleg@redhat.com> wrote:
> 
> > On 04/05, Masami Hiramatsu wrote:
> > >
> > > Can we make this syscall and uprobe behavior clearer? As you said, if
> > > the application use sigreturn or longjump, it may skip returns and
> > > shadow stack entries are left in the kernel. In such cases, can uretprobe
> > > detect it properly, or just crash the process (or process runs wrongly)?
> > 
> > Please see the comment in handle_trampoline(), it tries to detect this case.
> > This patch should not make any difference.
> 
> I think you mean this loop will skip and discard the stacked return_instance
> to find the valid one.
> 
> ----
>         do {
>                 /*
>                  * We should throw out the frames invalidated by longjmp().
>                  * If this chain is valid, then the next one should be alive
>                  * or NULL; the latter case means that nobody but ri->func
>                  * could hit this trampoline on return. TODO: sigaltstack().
>                  */
>                 next = find_next_ret_chain(ri);
>                 valid = !next || arch_uretprobe_is_alive(next, RP_CHECK_RET, regs);
> 
>                 instruction_pointer_set(regs, ri->orig_ret_vaddr);
>                 do {
>                         if (valid)
>                                 handle_uretprobe_chain(ri, regs);
>                         ri = free_ret_instance(ri);
>                         utask->depth--;
>                 } while (ri != next);
>         } while (!valid);
> ----
> 
> I think this expects setjmp/longjmp as below
> 
> foo() { <- retprobe1
> 	setjmp()
> 	bar() { <- retprobe2
> 		longjmp()
> 	}
> } <- return to trampoline
> 
> In this case, we need to skip retprobe2's instance.
> My concern is, if we can not find appropriate return instance, what happen?
> e.g.
> 
> foo() { <-- retprobe1
>    bar() { # sp is decremented
>        sys_uretprobe() <-- ??
>     }
> }
> 
> It seems sys_uretprobe() will handle retprobe1 at that point instead of
> SIGILL.

yes, and I think it's fine, you get the consumer called in wrong place,
but it's your fault and kernel won't crash

this can be fixed by checking the syscall is called from the trampoline
and prevent handle_trampoline call if it's not

> 
> Can we avoid this with below strict check?
> 
> if (ri->stack != regs->sp + expected_offset)
> 	goto sigill;

hm the current uprobe 'alive' check makes sure the return_instance is above
or at the same stack address, not sure we can match it exactly, need to think
about that more

> 
> expected_offset should be 16 (push * 3 - ret) on x64 if we ri->stack is the
> regs->sp right after call.

the syscall trampoline already updates the regs->sp before calling
handle_trampoline

        regs->sp += sizeof(r11_cx_ax);

jirka

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Oleg Nesterov]

On 04/05, Jiri Olsa wrote:
>
> On Fri, Apr 05, 2024 at 10:22:03AM +0900, Masami Hiramatsu wrote:
> >
> > I think this expects setjmp/longjmp as below
> >
> > foo() { <- retprobe1
> > 	setjmp()
> > 	bar() { <- retprobe2
> > 		longjmp()
> > 	}
> > } <- return to trampoline
> >
> > In this case, we need to skip retprobe2's instance.

Yes,

> > My concern is, if we can not find appropriate return instance, what happen?
> > e.g.
> >
> > foo() { <-- retprobe1
> >    bar() { # sp is decremented
> >        sys_uretprobe() <-- ??
> >     }
> > }
> >
> > It seems sys_uretprobe() will handle retprobe1 at that point instead of
> > SIGILL.
>
> yes, and I think it's fine, you get the consumer called in wrong place,
> but it's your fault and kernel won't crash

Agreed.

With or without this patch userpace can also do

	foo() { <-- retprobe1
		bar() {
			jump to xol_area
		}
	}

handle_trampoline() will handle retprobe1.

> this can be fixed by checking the syscall is called from the trampoline
> and prevent handle_trampoline call if it's not

Yes, but I still do not think this makes a lot of sense. But I won't argue.

And what should sys_uretprobe() do if it is not called from the trampoline?
I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.

I agree very much with Andrii,

       sigreturn()  exists only to allow the implementation of signal handlers.  It should never be
       called directly.  Details of the arguments (if any) passed to sigreturn() vary depending  on
       the architecture.

this is how sys_uretprobe() should be treated/documented.

sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
and return -EINVAL if this addr is not valid. But why?

Oleg.

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Fri, 5 Apr 2024 13:02:30 +0200
Oleg Nesterov <oleg@redhat.com> wrote:

> On 04/05, Jiri Olsa wrote:
> >
> > On Fri, Apr 05, 2024 at 10:22:03AM +0900, Masami Hiramatsu wrote:
> > >
> > > I think this expects setjmp/longjmp as below
> > >
> > > foo() { <- retprobe1
> > > 	setjmp()
> > > 	bar() { <- retprobe2
> > > 		longjmp()
> > > 	}
> > > } <- return to trampoline
> > >
> > > In this case, we need to skip retprobe2's instance.
> 
> Yes,
> 
> > > My concern is, if we can not find appropriate return instance, what happen?
> > > e.g.
> > >
> > > foo() { <-- retprobe1
> > >    bar() { # sp is decremented
> > >        sys_uretprobe() <-- ??
> > >     }
> > > }
> > >
> > > It seems sys_uretprobe() will handle retprobe1 at that point instead of
> > > SIGILL.
> >
> > yes, and I think it's fine, you get the consumer called in wrong place,
> > but it's your fault and kernel won't crash
> 
> Agreed.
> 
> With or without this patch userpace can also do
> 
> 	foo() { <-- retprobe1
> 		bar() {
> 			jump to xol_area
> 		}
> 	}
> 
> handle_trampoline() will handle retprobe1.

This is OK because the execution path has been changed to trampoline, but
the above will continue running bar() after sys_uretprobe().

> 
> > this can be fixed by checking the syscall is called from the trampoline
> > and prevent handle_trampoline call if it's not
> 
> Yes, but I still do not think this makes a lot of sense. But I won't argue.
> 
> And what should sys_uretprobe() do if it is not called from the trampoline?
> I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.
> 
> I agree very much with Andrii,
> 
>        sigreturn()  exists only to allow the implementation of signal handlers.  It should never be
>        called directly.  Details of the arguments (if any) passed to sigreturn() vary depending  on
>        the architecture.
> 
> this is how sys_uretprobe() should be treated/documented.

OK.

> 
> sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
> and return -EINVAL if this addr is not valid. But why?

Because sigreturn() never returns, but sys_uretprobe() will return.
If it changes the regs->ip and directly returns to the original return address,
I think it is natural to send SIGILL.


Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Oleg Nesterov]

On 04/06, Masami Hiramatsu wrote:
>
> On Fri, 5 Apr 2024 13:02:30 +0200
> Oleg Nesterov <oleg@redhat.com> wrote:
>
> > With or without this patch userpace can also do
> >
> > 	foo() { <-- retprobe1
> > 		bar() {
> > 			jump to xol_area
> > 		}
> > 	}
> >
> > handle_trampoline() will handle retprobe1.
>
> This is OK because the execution path has been changed to trampoline,

Agreed, in this case the misuse is more clear. But please see below.

> but the above will continue running bar() after sys_uretprobe().

... and most probably crash

> > sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
> > and return -EINVAL if this addr is not valid. But why?
>
> Because sigreturn() never returns, but sys_uretprobe() will return.

You mean, sys_uretprobe() returns to the next insn after syscall.

Almost certainly yes, but this is not necessarily true. If one of consumers
changes regs->sp sys_uretprobe() "returns" to another location, just like
sys_rt_sigreturn().

That said.

Masami, it is not that I am trying to prove that you are "wrong" ;) No.

I see your points even if I am biased, I understand that my objections are
not 100% "fair".

I am just trying to explain why, rightly or not, I care much less about the
abuse of sys_uretprobe().

Thanks!

Oleg.

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Sat, 6 Apr 2024 19:55:59 +0200
Oleg Nesterov <oleg@redhat.com> wrote:

> On 04/06, Masami Hiramatsu wrote:
> >
> > On Fri, 5 Apr 2024 13:02:30 +0200
> > Oleg Nesterov <oleg@redhat.com> wrote:
> >
> > > With or without this patch userpace can also do
> > >
> > > 	foo() { <-- retprobe1
> > > 		bar() {
> > > 			jump to xol_area
> > > 		}
> > > 	}
> > >
> > > handle_trampoline() will handle retprobe1.
> >
> > This is OK because the execution path has been changed to trampoline,
> 
> Agreed, in this case the misuse is more clear. But please see below.
> 
> > but the above will continue running bar() after sys_uretprobe().
> 
> .. and most probably crash

Yes, unless it returns with longjmp(). (but this is rare case and
maybe malicious program.)

> 
> > > sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
> > > and return -EINVAL if this addr is not valid. But why?
> >
> > Because sigreturn() never returns, but sys_uretprobe() will return.
> 
> You mean, sys_uretprobe() returns to the next insn after syscall.
> 
> Almost certainly yes, but this is not necessarily true. If one of consumers
> changes regs->sp sys_uretprobe() "returns" to another location, just like
> sys_rt_sigreturn().
> 
> That said.
> 
> Masami, it is not that I am trying to prove that you are "wrong" ;) No.
> 
> I see your points even if I am biased, I understand that my objections are
> not 100% "fair".
> 
> I am just trying to explain why, rightly or not, I care much less about the
> abuse of sys_uretprobe().

I would like to clear that the abuse of this syscall will not possible to harm
the normal programs, and even if it is used by malicious code (e.g. injected by
stack overflow) it doesn't cause a problem. At least thsese points are cleared,
and documented. it is easier to push it as new Linux API.

Thank you,

> 
> Thanks!
> 
> Oleg.
> 
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Jiri Olsa]

On Fri, Apr 05, 2024 at 01:02:30PM +0200, Oleg Nesterov wrote:
> On 04/05, Jiri Olsa wrote:
> >
> > On Fri, Apr 05, 2024 at 10:22:03AM +0900, Masami Hiramatsu wrote:
> > >
> > > I think this expects setjmp/longjmp as below
> > >
> > > foo() { <- retprobe1
> > > 	setjmp()
> > > 	bar() { <- retprobe2
> > > 		longjmp()
> > > 	}
> > > } <- return to trampoline
> > >
> > > In this case, we need to skip retprobe2's instance.
> 
> Yes,
> 
> > > My concern is, if we can not find appropriate return instance, what happen?
> > > e.g.
> > >
> > > foo() { <-- retprobe1
> > >    bar() { # sp is decremented
> > >        sys_uretprobe() <-- ??
> > >     }
> > > }
> > >
> > > It seems sys_uretprobe() will handle retprobe1 at that point instead of
> > > SIGILL.
> >
> > yes, and I think it's fine, you get the consumer called in wrong place,
> > but it's your fault and kernel won't crash
> 
> Agreed.
> 
> With or without this patch userpace can also do
> 
> 	foo() { <-- retprobe1
> 		bar() {
> 			jump to xol_area
> 		}
> 	}
> 
> handle_trampoline() will handle retprobe1.
> 
> > this can be fixed by checking the syscall is called from the trampoline
> > and prevent handle_trampoline call if it's not
> 
> Yes, but I still do not think this makes a lot of sense. But I won't argue.
> 
> And what should sys_uretprobe() do if it is not called from the trampoline?
> I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.

so the similar behaviour with int3 ends up with immediate SIGTRAP
and not invoking pending uretprobe consumers, like:

  - setup uretprobe for foo
  - foo() {
      executes int 3 -> sends SIGTRAP
    }

because the int3 handler checks if it got executed from the uretprobe's
trampoline.. if not it treats that int3 as regular trap

while for uretprobe syscall we have at the moment following behaviour:

  - setup uretprobe for foo
  - foo() {
     uretprobe_syscall -> executes foo's uretprobe consumers
    }
  - at some point we get to the 'ret' instruction that jump into uretprobe
    trampoline and the uretprobe_syscall won't find pending uretprobe and
    will send SIGILL


so I think we should mimic int3 behaviour and:

  - setup uretprobe for foo
  - foo() {
     uretprobe_syscall -> check if we got executed from uretprobe's
     trampoline and send SIGILL if that's not the case

I think it's better to have the offending process killed right away,
rather than having more undefined behaviour, waiting for final 'ret'
instruction that jumps to uretprobe trampoline and causes SIGILL

> 
> I agree very much with Andrii,
> 
>        sigreturn()  exists only to allow the implementation of signal handlers.  It should never be
>        called directly.  Details of the arguments (if any) passed to sigreturn() vary depending  on
>        the architecture.
> 
> this is how sys_uretprobe() should be treated/documented.

yes, will include man page patch in new version

jirka

> 
> sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
> and return -EINVAL if this addr is not valid. But why?
> 
> Oleg.
> 

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Oleg Nesterov]

On 04/08, Jiri Olsa wrote:
>
> On Fri, Apr 05, 2024 at 01:02:30PM +0200, Oleg Nesterov wrote:
> >
> > And what should sys_uretprobe() do if it is not called from the trampoline?
> > I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.
>
> so the similar behaviour with int3 ends up with immediate SIGTRAP
> and not invoking pending uretprobe consumers, like:
>
>   - setup uretprobe for foo
>   - foo() {
>       executes int 3 -> sends SIGTRAP
>     }
>
> because the int3 handler checks if it got executed from the uretprobe's
> trampoline.

... or the task has uprobe at this address

> if not it treats that int3 as regular trap

Yes this mimics the "default" behaviour without uprobes/uretprobes

> so I think we should mimic int3 behaviour and:
>
>   - setup uretprobe for foo
>   - foo() {
>      uretprobe_syscall -> check if we got executed from uretprobe's
>      trampoline and send SIGILL if that's not the case

Agreed,

> I think it's better to have the offending process killed right away,
> rather than having more undefined behaviour, waiting for final 'ret'
> instruction that jumps to uretprobe trampoline and causes SIGILL

Agreed. In fact I think it should be also killed if copy_to/from_user()
fails by the same reason.

Oleg.

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Jiri Olsa]

On Mon, Apr 08, 2024 at 06:22:59PM +0200, Oleg Nesterov wrote:
> On 04/08, Jiri Olsa wrote:
> >
> > On Fri, Apr 05, 2024 at 01:02:30PM +0200, Oleg Nesterov wrote:
> > >
> > > And what should sys_uretprobe() do if it is not called from the trampoline?
> > > I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.
> >
> > so the similar behaviour with int3 ends up with immediate SIGTRAP
> > and not invoking pending uretprobe consumers, like:
> >
> >   - setup uretprobe for foo
> >   - foo() {
> >       executes int 3 -> sends SIGTRAP
> >     }
> >
> > because the int3 handler checks if it got executed from the uretprobe's
> > trampoline.
> 
> ... or the task has uprobe at this address
> 
> > if not it treats that int3 as regular trap
> 
> Yes this mimics the "default" behaviour without uprobes/uretprobes
> 
> > so I think we should mimic int3 behaviour and:
> >
> >   - setup uretprobe for foo
> >   - foo() {
> >      uretprobe_syscall -> check if we got executed from uretprobe's
> >      trampoline and send SIGILL if that's not the case
> 
> Agreed,
> 
> > I think it's better to have the offending process killed right away,
> > rather than having more undefined behaviour, waiting for final 'ret'
> > instruction that jumps to uretprobe trampoline and causes SIGILL
> 
> Agreed. In fact I think it should be also killed if copy_to/from_user()
> fails by the same reason.

+1 makes sense

jirka

> 
> Oleg.
> 

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Mon, 8 Apr 2024 18:02:13 +0200
Jiri Olsa <olsajiri@gmail.com> wrote:

> On Fri, Apr 05, 2024 at 01:02:30PM +0200, Oleg Nesterov wrote:
> > On 04/05, Jiri Olsa wrote:
> > >
> > > On Fri, Apr 05, 2024 at 10:22:03AM +0900, Masami Hiramatsu wrote:
> > > >
> > > > I think this expects setjmp/longjmp as below
> > > >
> > > > foo() { <- retprobe1
> > > > 	setjmp()
> > > > 	bar() { <- retprobe2
> > > > 		longjmp()
> > > > 	}
> > > > } <- return to trampoline
> > > >
> > > > In this case, we need to skip retprobe2's instance.
> > 
> > Yes,
> > 
> > > > My concern is, if we can not find appropriate return instance, what happen?
> > > > e.g.
> > > >
> > > > foo() { <-- retprobe1
> > > >    bar() { # sp is decremented
> > > >        sys_uretprobe() <-- ??
> > > >     }
> > > > }
> > > >
> > > > It seems sys_uretprobe() will handle retprobe1 at that point instead of
> > > > SIGILL.
> > >
> > > yes, and I think it's fine, you get the consumer called in wrong place,
> > > but it's your fault and kernel won't crash
> > 
> > Agreed.
> > 
> > With or without this patch userpace can also do
> > 
> > 	foo() { <-- retprobe1
> > 		bar() {
> > 			jump to xol_area
> > 		}
> > 	}
> > 
> > handle_trampoline() will handle retprobe1.
> > 
> > > this can be fixed by checking the syscall is called from the trampoline
> > > and prevent handle_trampoline call if it's not
> > 
> > Yes, but I still do not think this makes a lot of sense. But I won't argue.
> > 
> > And what should sys_uretprobe() do if it is not called from the trampoline?
> > I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.
> 
> so the similar behaviour with int3 ends up with immediate SIGTRAP
> and not invoking pending uretprobe consumers, like:
> 
>   - setup uretprobe for foo
>   - foo() {
>       executes int 3 -> sends SIGTRAP
>     }
> 
> because the int3 handler checks if it got executed from the uretprobe's
> trampoline.. if not it treats that int3 as regular trap

Yeah, that is consistent behavior. Sounds good to me.

> 
> while for uretprobe syscall we have at the moment following behaviour:
> 
>   - setup uretprobe for foo
>   - foo() {
>      uretprobe_syscall -> executes foo's uretprobe consumers
>     }
>   - at some point we get to the 'ret' instruction that jump into uretprobe
>     trampoline and the uretprobe_syscall won't find pending uretprobe and
>     will send SIGILL
> 
> 
> so I think we should mimic int3 behaviour and:
> 
>   - setup uretprobe for foo
>   - foo() {
>      uretprobe_syscall -> check if we got executed from uretprobe's
>      trampoline and send SIGILL if that's not the case

OK, this looks good to me.

> 
> I think it's better to have the offending process killed right away,
> rather than having more undefined behaviour, waiting for final 'ret'
> instruction that jumps to uretprobe trampoline and causes SIGILL
> 
> > 
> > I agree very much with Andrii,
> > 
> >        sigreturn()  exists only to allow the implementation of signal handlers.  It should never be
> >        called directly.  Details of the arguments (if any) passed to sigreturn() vary depending  on
> >        the architecture.
> > 
> > this is how sys_uretprobe() should be treated/documented.
> 
> yes, will include man page patch in new version

And please follow Documentation/process/adding-syscalls.rst in new version,
then we can avoid repeating the same discussion :-)

Thank you!

> 
> jirka
> 
> > 
> > sigreturn() can be "improved" too. Say, it could validate sigcontext->ip
> > and return -EINVAL if this addr is not valid. But why?
> > 
> > Oleg.
> > 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Jiri Olsa]

On Tue, Apr 09, 2024 at 09:34:39AM +0900, Masami Hiramatsu wrote:

SNIP

> > > 
> > > > this can be fixed by checking the syscall is called from the trampoline
> > > > and prevent handle_trampoline call if it's not
> > > 
> > > Yes, but I still do not think this makes a lot of sense. But I won't argue.
> > > 
> > > And what should sys_uretprobe() do if it is not called from the trampoline?
> > > I'd prefer force_sig(SIGILL) to punish the abuser ;) OK, OK, EINVAL.
> > 
> > so the similar behaviour with int3 ends up with immediate SIGTRAP
> > and not invoking pending uretprobe consumers, like:
> > 
> >   - setup uretprobe for foo
> >   - foo() {
> >       executes int 3 -> sends SIGTRAP
> >     }
> > 
> > because the int3 handler checks if it got executed from the uretprobe's
> > trampoline.. if not it treats that int3 as regular trap
> 
> Yeah, that is consistent behavior. Sounds good to me.
> 
> > 
> > while for uretprobe syscall we have at the moment following behaviour:
> > 
> >   - setup uretprobe for foo
> >   - foo() {
> >      uretprobe_syscall -> executes foo's uretprobe consumers
> >     }
> >   - at some point we get to the 'ret' instruction that jump into uretprobe
> >     trampoline and the uretprobe_syscall won't find pending uretprobe and
> >     will send SIGILL
> > 
> > 
> > so I think we should mimic int3 behaviour and:
> > 
> >   - setup uretprobe for foo
> >   - foo() {
> >      uretprobe_syscall -> check if we got executed from uretprobe's
> >      trampoline and send SIGILL if that's not the case
> 
> OK, this looks good to me.
> 
> > 
> > I think it's better to have the offending process killed right away,
> > rather than having more undefined behaviour, waiting for final 'ret'
> > instruction that jumps to uretprobe trampoline and causes SIGILL
> > 
> > > 
> > > I agree very much with Andrii,
> > > 
> > >        sigreturn()  exists only to allow the implementation of signal handlers.  It should never be
> > >        called directly.  Details of the arguments (if any) passed to sigreturn() vary depending  on
> > >        the architecture.
> > > 
> > > this is how sys_uretprobe() should be treated/documented.
> > 
> > yes, will include man page patch in new version
> 
> And please follow Documentation/process/adding-syscalls.rst in new version,
> then we can avoid repeating the same discussion :-)

yep, will do

thanks,
jirka

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Fri, 5 Apr 2024 10:56:15 +0200
Jiri Olsa <olsajiri@gmail.com> wrote:

> > 
> > Can we avoid this with below strict check?
> > 
> > if (ri->stack != regs->sp + expected_offset)
> > 	goto sigill;
> 
> hm the current uprobe 'alive' check makes sure the return_instance is above
> or at the same stack address, not sure we can match it exactly, need to think
> about that more
> 
> > 
> > expected_offset should be 16 (push * 3 - ret) on x64 if we ri->stack is the
> > regs->sp right after call.
> 
> the syscall trampoline already updates the regs->sp before calling
> handle_trampoline
> 
>         regs->sp += sizeof(r11_cx_ax);

Yes, that is "push * 3" part. And "- ret"  is that the stack entry is consumed
by the "ret", which is stored by call.

1: |--------| <- sp at function entry == ri->stack
0: |ret-addr| <- call pushed it

0: |ret-addr| <- sp at return trampoline

3: |r11     | <- regs->sp at syscall
2: |rcx     |
1: |rax     | <- ri->stack
0: |ret-addr|

(Note: The lower the line, the larger the address.)

Thus, we can check the stack address by (regs->sp + 16 == ri->stack).

Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PATCHv2 1/3] uprobe: Add uretprobe syscall to speed up return probe

[Masami Hiramatsu]

On Thu, 4 Apr 2024 13:58:43 +0200
Jiri Olsa <olsajiri@gmail.com> wrote:

> On Wed, Apr 03, 2024 at 07:00:07PM -0700, Andrii Nakryiko wrote:
> 
> SNIP
> 
> > Check rt_sigreturn syscall (manpage at [0], for example).
> > 
> >        sigreturn() exists only to allow the implementation of signal
> >        handlers.  It should never be called directly.  (Indeed, a simple
> >        sigreturn() wrapper in the GNU C library simply returns -1, with
> >        errno set to ENOSYS.)  Details of the arguments (if any) passed
> >        to sigreturn() vary depending on the architecture.  (On some
> >        architectures, such as x86-64, sigreturn() takes no arguments,
> >        since all of the information that it requires is available in the
> >        stack frame that was previously created by the kernel on the
> >        user-space stack.)
> > 
> > This is a very similar use case. Also, check its source code in
> > arch/x86/kernel/signal_64.c. It sends SIGSEGV to the calling process
> > on any sign of something not being right. It's exactly the same with
> > sys_uretprobe.
> > 
> >   [0] https://man7.org/linux/man-pages/man2/sigreturn.2.html
> > 
> > > And the number of syscalls are limited resource.
> > 
> > We have almost 500 of them, it didn't seems like adding 1-2 for good
> > reasons would be a problem. Can you please point to where the limits
> > on syscalls as a resource are described? I'm curious to learn.
> > 
> > >
> > > I'm actually not sure how much we need to care of it, but adding a new
> > > syscall is worth to be discussed carefully because all of them are
> > > user-space compatibility.
> > 
> > Absolutely, it's a good discussion to have.
> > 
> > >
> > > > > > > Also, we should run syzkaller on this syscall. And if uretprobe is
> > > > > >
> > > > > > right, I'll check on syzkaller
> > > > > >
> > > > > > > set in the user function, what happen if the user function directly
> > > > > > > calls this syscall? (maybe it consumes shadow stack?)
> > > > > >
> > > > > > the process should receive SIGILL if there's no pending uretprobe for
> > > > > > the current task, or it will trigger uretprobe if there's one pending
> > > > >
> > > > > No, that is too aggressive and not safe. Since the syscall is exposed to
> > > > > user program, it should return appropriate error code instead of SIGILL.
> > > > >
> > > >
> > > > This is the way it is today with uretprobes even through interrupt.
> > >
> > > I doubt that the interrupt (exception) and syscall should be handled
> > > differently. Especially, this exception is injected by uprobes but
> > > syscall will be caused by itself. But syscall can be called from user
> > > program (of couse this works as sys_kill(self, SIGILL)).
> > 
> > Yep, I'd keep the behavior the same between uretprobes implemented
> > through int3 and sys_uretprobe.
> 
> +1 
> 
> > 
> > >
> > > > E.g., it could happen that user process is using fibers and is
> > > > replacing stack pointer without kernel realizing this, which will
> > > > trigger some defensive checks in uretprobe handling code and kernel
> > > > will send SIGILL because it can't support such cases. This is
> > > > happening today already, and it works fine in practice (except for
> > > > applications that manually change stack pointer, too bad, you can't
> > > > trace them with uretprobes, unfortunately).
> > >
> > > OK, we at least need to document it.
> > 
> > +1, yep
> > 
> > >
> > > >
> > > > So I think it's absolutely adequate to have this behavior if the user
> > > > process is *intentionally* abusing this API.
> > >
> > > Of course user expected that it is abusing. So at least we need to
> > > add a document that this syscall number is reserved to uprobes and
> > > user program must not use it.
> > >
> > 
> > Totally agree about documenting this.
> 
> ok there's map page on sigreturn.. do you think we should add man page
> for uretprobe or you can think of some other place to document it?

I think it is better to have a man-page. Anyway, to discuss and explain
this syscall, the man-page is a good format to describe it.

> 
> > 
> > > >
> > > > > >
> > > > > > but we could limit the syscall to be executed just from the trampoline,
> > > > > > that should prevent all the user space use cases, I'll do that in next
> > > > > > version and add more tests for that
> > > > >
> > > > > Why not limit? :) The uprobe_handle_trampoline() expects it is called
> > > > > only from the trampoline, so it is natural to check the caller address.
> > > > > (and uprobe should know where is the trampoline)
> > > > >
> > > > > Since the syscall is always exposed to the user program, it should
> > > > > - Do nothing and return an error unless it is properly called.
> > > > > - check the prerequisites for operation strictly.
> > > > > I concern that new system calls introduce vulnerabilities.
> > > > >
> > > >
> > > > As Oleg and Jiri mentioned, this syscall can't harm kernel or other
> > > > processes, only the process that is abusing the API. So any extra
> > > > checks that would slow down this approach is an unnecessary overhead
> > > > and complication that will never be useful in practice.
> > >
> > > I think at least it should check the caller address to ensure the
> > > address is in the trampoline.
> > > But anyway, uprobes itself can break the target process, so no one
> > > might care if this system call breaks the process now.
> > 
> > If we already have an expected range of addresses, then I think it's
> > fine to do a quick unlikely() check. I'd be more concerned if we need
> > to do another lookup or search to just validate this. I'm sure Jiri
> > will figure it out.
> 
> Oleg mentioned the trampoline address check could race with another
> thread's mremap call, however trap is using that check as well, it
> still seems like good idea to do it also in the uretprobe syscall

Yeah, and also, can we add a stack pointer check if the trampoline is
shared with other probe points?

Thank you,

> 
> jirka


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>