Re: ftrace_kill() leads to kmalloc-512 UAF

Re: ftrace_kill() leads to kmalloc-512 UAF

[Steven Rostedt]

[ Adding live kernel patching folks ]

On Wed, 20 Jul 2022 18:22:25 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:

> I'm debugging crash of our product which does live kernel patching together
> with ISV security scanner which uses ftrace kprobes to do whatever it does.
> 
> What happens is that is ftrace ever detects .text change, refuses to patch
> and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> to cause UAF on kmalloc-512 cache by trying to register kprobe with
> perf_event_open() and then unregistering it by exiting the process.
> 
> 1) live kernel patching happens, first instruction of some function changes

But live kernel patching uses ftrace to do this, so the question remains,
why doesn't ftrace know about this change?

> 
> 2) kprobe on that function is registered with perf_event_open()
> 
> 	WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> 	 [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> 	 [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> 	 [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> 	 [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> 	 [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> 	 [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> 	 [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> 	 [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> 	 [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> 	 [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> 	 [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> 	 [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> 	 [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> 	 [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> 	 [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> 	 [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> 	 [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> 	 [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> 	 [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> 	 [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> 	ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> 	 actual: e9:4b:50:2e:3f
> 

Again, why did the above happen. This is a kernel bug that needs to be
fixed. Everything else after this is unimportant, because it's just fall
out to the above bug.

-- Steve



> 3) FTRACE_WARN_ON_ONCE() calls ftrace_kill() which sets ftrace_disabled:
> 
> 	ftrace_disabled = 1;
> 
> 4) process exits, all kprobes are unregistered but "ftrace_disabled" is 1 now
>    so disarming fails with -ENODEV:
> 
> 	[  253.042821] WARNING: CPU: 2 PID: 1682 at kernel/kprobes.c:1006 disarm_kprobe+0x102/0x180
> 	[  253.044424] Failed to disarm kprobe-ftrace at do_exit+0x0/0xa30 (-19)
> 	[  253.086658]  [<ffffffff811655a0>] ? ftrace_set_filter_ip+0x60/0x70
> 	[  253.089599]  [<ffffffff810a1810>] ? mm_update_next_owner+0x230/0x230
> 	[  253.090878]  [<ffffffff81796492>] disarm_kprobe+0x102/0x180
> 	[  253.092084]  [<ffffffff817965dd>] __disable_kprobe+0xcd/0xf0
> 	[  253.093242]  [<ffffffff81796c13>] disable_kprobe+0x23/0x40
> 	[  253.094390]  [<ffffffff81187d14>] disable_trace_kprobe+0xc4/0x100
> 	[  253.095751]  [<ffffffff81797e53>] kprobe_register+0x23/0x60
> 	[  253.097420]  [<ffffffff8118327c>] perf_trace_event_unreg.isra.1+0x3c/0xa0
> 	[  253.098866]  [<ffffffff81183730>] perf_kprobe_destroy+0x30/0x40
> 	[  253.100085]  [<ffffffff811af50a>] _free_event+0xfa/0x2f0
> 	[  253.101147]  [<ffffffff811af769>] put_event+0x19/0x20
> 	[  253.102154]  [<ffffffff811af993>] perf_event_release_kernel+0x223/0x300
> 	[  253.103457]  [<ffffffff811afa80>] perf_release+0x10/0x20
> 	[  253.106188]  [<ffffffff8125063c>] __fput+0xec/0x230
> 
> and more importantly underlying kprobe is not removed from "kprobe_table" hashtable:
> 
> 	unregister_kprobe
> 	unregister_kprobes
> 	__unregister_kprobe_top
> 	__disable_kprobe
> 		ret = disarm_kprobe(orig_p, true);
> 			disarm_kprobe_ftrace
> 			__disarm_kprobe_ftrace
> 			ftrace_set_filter_ip
> 			ftrace_set_addr
> 			ftrace_set_hash
> 			        if (unlikely(ftrace_disabled))
> 			                return -ENODEV;
> 
> 				// BOOM, function does nothing!!!
> 
> 		if (ret) {
> 			p->flags &= ~KPROBE_FLAG_DISABLED;
> 			return ERR_PTR(ret);
> 		}	
> 
> 
> perf_kprobe_destroy() will free the containing kprobe with inner kprobe
> still in hashtable manifesting the bug as regular oopses, mystical oopses
> in unrelated processes and doublefaults even.
> 
> 	void perf_kprobe_destroy(struct perf_event *p_event)
> 	{
> 	        perf_trace_event_close(p_event);
> 
> 		// does more or less nothing
> 	        perf_trace_event_unreg(p_event);
> 		// does kfree
> 	        destroy_local_trace_kprobe(p_event->tp_event);
> 	}
> 
> 
> crash> p kprobe_table  
> kprobe_table = $1 =
>  {{
>   }, {
>     first = 0xffff880135e9ddd8
>   }, {
> 	...
> 
> crash> struct kprobe 0xffff880135e9ddd8  
> struct kprobe {
>   hlist = {
>     next = 0x6b6b6b6b6b6b6b6b,
>     pprev = 0x6b6b6b6b6b6b6b6b
>   },
>   list = {
>     next = 0x6b6b6b6b6b6b6b6b,
>     prev = 0x6b6b6b6b6b6b6b6b
>   },
>   nmissed = 7740398493674204011,
>   addr = 0x6b6b6b6b6b6b6b6b <Address 0x6b6b6b6b6b6b6b6b out of bounds>,
> 		....
> 
> This is much easier to reproduce by adding the following BUG_ON:
> 
> 	--- a/kernel/trace/trace_kprobe.c.orig  2022-07-20 14:35:06.760511285 +0300
> 	+++ b/kernel/trace/trace_kprobe.c       2022-07-20 14:41:14.699877744 +0300
> 	@@ -512,8 +512,10 @@ static void __unregister_trace_kprobe(st
> 	        if (trace_probe_is_registered(&tk->tp)) {
> 	                if (trace_kprobe_is_return(tk))
> 	                        unregister_kretprobe(&tk->rp);
> 	-               else
> 	+               else {
> 	                        unregister_kprobe(&tk->rp.kp);
> 	+                       BUG_ON(!hlist_unhashed(&tk->rp.kp.hlist));
> 	+               }
> 	                tk->tp.flags &= ~TP_FLAG_REGISTERED;
> 	                /* Cleanup kprobe for reuse */
> 	                if (tk->rp.kp.symbol_name)
> 
> Basically, if ftrace_kill() is ever called ever there is a ticking UAFbomb.
> 
> 	Alexey (CloudLinux)

Re: ftrace_kill() leads to kmalloc-512 UAF

[Alexey Dobriyan]

On Wed, Jul 20, 2022 at 12:11:02PM -0400, Steven Rostedt wrote:
> 
> [ Adding live kernel patching folks ]
> 
> On Wed, 20 Jul 2022 18:22:25 +0300
> Alexey Dobriyan <adobriyan@gmail.com> wrote:
> 
> > I'm debugging crash of our product which does live kernel patching together
> > with ISV security scanner which uses ftrace kprobes to do whatever it does.
> > 
> > What happens is that is ftrace ever detects .text change, refuses to patch
> > and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> > to cause UAF on kmalloc-512 cache by trying to register kprobe with
> > perf_event_open() and then unregistering it by exiting the process.
> > 
> > 1) live kernel patching happens, first instruction of some function changes
> 
> But live kernel patching uses ftrace to do this, so the question remains,
> why doesn't ftrace know about this change?

That a line from our product which changes first instruction of the function.

> > 2) kprobe on that function is registered with perf_event_open()
> > 
> > 	WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> > 	 [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> > 	 [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> > 	 [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> > 	 [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> > 	 [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> > 	 [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> > 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > 	 [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> > 	 [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> > 	 [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> > 	 [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> > 	 [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> > 	 [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> > 	 [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> > 	 [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> > 	 [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> > 	 [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> > 	 [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> > 	 [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> > 	 [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> > 	 [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> > 	ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> > 	 actual: e9:4b:50:2e:3f
> > 
> 
> Again, why did the above happen. This is a kernel bug that needs to be
> fixed. Everything else after this is unimportant, because it's just fall
> out to the above bug.

I'll double check what we (not KLP) do and maybe even what KLP does!

Re: ftrace_kill() leads to kmalloc-512 UAF

[Steven Rostedt]

On Wed, 20 Jul 2022 19:46:02 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:

> On Wed, Jul 20, 2022 at 12:11:02PM -0400, Steven Rostedt wrote:
> > 
> > [ Adding live kernel patching folks ]
> > 
> > On Wed, 20 Jul 2022 18:22:25 +0300
> > Alexey Dobriyan <adobriyan@gmail.com> wrote:
> >   
> > > I'm debugging crash of our product which does live kernel patching together
> > > with ISV security scanner which uses ftrace kprobes to do whatever it does.
> > > 
> > > What happens is that is ftrace ever detects .text change, refuses to patch
> > > and prints a warning with FTRACE_WARN_ON_ONCE() then there is reliable way
> > > to cause UAF on kmalloc-512 cache by trying to register kprobe with
> > > perf_event_open() and then unregistering it by exiting the process.
> > > 
> > > 1) live kernel patching happens, first instruction of some function changes  
> > 
> > But live kernel patching uses ftrace to do this, so the question remains,
> > why doesn't ftrace know about this change?  
> 
> That a line from our product which changes first instruction of the function.
> 

So this is an out-of-tree change?

> > > 2) kprobe on that function is registered with perf_event_open()
> > > 
> > > 	WARNING: CPU: 5 PID: 2109 at kernel/trace/ftrace.c:1853 ftrace_bug+0x25d/0x270
> > > 	 [<ffffffff811638ed>] ftrace_bug+0x25d/0x270
> > > 	 [<ffffffff81065571>] ftrace_replace_code+0x2b1/0x420
> > > 	 [<ffffffff81163f9a>] ftrace_modify_all_code+0x6a/0xb0
> > > 	 [<ffffffff810656f0>] arch_ftrace_update_code+0x10/0x20
> > > 	 [<ffffffff81164077>] ftrace_run_update_code+0x17/0x70
> > > 	 [<ffffffff81165512>] ftrace_set_hash+0x1c2/0x1f0
> > > 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > > 	 [<ffffffff8126fee0>] ? SyS_dup2+0x60/0x60
> > > 	 [<ffffffff811655a0>] ftrace_set_filter_ip+0x60/0x70
> > > 	 [<ffffffff8179624c>] arm_kprobe+0x9c/0x140
> > > 	 [<ffffffff81796368>] enable_kprobe+0x78/0xa0
> > > 	 [<ffffffff81187bab>] enable_trace_kprobe+0x7b/0x120
> > > 	 [<ffffffff81797e5f>] kprobe_register+0x2f/0x60
> > > 	 [<ffffffff8118348a>] perf_trace_event_init+0x1aa/0x230
> > > 	 [<ffffffff811836b7>] perf_kprobe_init+0xa7/0xf0
> > > 	 [<ffffffff811a8919>] perf_kprobe_event_init+0x49/0x70
> > > 	 [<ffffffff811aa569>] perf_try_init_event+0x99/0xc0
> > > 	 [<ffffffff811b29f2>] perf_init_event+0x92/0x150
> > > 	 [<ffffffff811b2fa1>] perf_event_alloc+0x4f1/0x910
> > > 	 [<ffffffff811b3789>] SYSC_perf_event_open+0x3c9/0xe50
> > > 	 [<ffffffff811b4679>] SyS_perf_event_open+0x9/0x10
> > > 	 [<ffffffff81799f92>] system_call_fastpath+0x25/0x2a
> > > 	ftrace failed to modify [<ffffffff8126fee0>] SyS_dup+0x0/0x120
> > > 	 actual: e9:4b:50:2e:3f
> > >   
> > 
> > Again, why did the above happen. This is a kernel bug that needs to be
> > fixed. Everything else after this is unimportant, because it's just fall
> > out to the above bug.  
> 
> I'll double check what we (not KLP) do and maybe even what KLP does!

I would guess the fix is to have your stuff not monkey around with ftrace
locations or to modify your own kernel to have ftrace know about this.

-- Steve