Re: perf: fuzzer causes lockup in x86_pmu_event_init()

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Feb 23, 2015 at 10:56:10PM -0500, Vince Weaver wrote:
> On Tue, 17 Feb 2015, Vince Weaver wrote:
> 
> > This is on a Haswell machine, current git as of this past Friday.
> > 
> > I let the perf_fuzzer run and it took 4 days to find this.
> > Sadly it doesn't seem to be reproducible so I am not sure
> > how it exactly got into this state.
> 
> I have hit this on another machine, my core2 machine (after 10 days of 
> fuzzing).  So this seems to be a real issue although hard to hit.
> 
> The problem seems to map to 
> 	arch/x86/kernel/cpu/perf_event.c:824
> 
> It is stuck forever in this loop in collect_events()
> 
>         list_for_each_entry(event, &leader->sibling_list, group_entry) {
>                 if (!is_x86_event(event) ||
>                     event->state <= PERF_EVENT_STATE_OFF)
>                         continue;
> 
>                 if (n >= max_count)
>                         return -EINVAL;
> 
>                 cpuc->event_list[n] = event;
>                 n++;
>         }
> 

> [884044.228001] RIP: 0010:[<ffffffff810138a8>]  [<ffffffff810138a8>] x86_pmu_event_init+0x138/0x31d

> [884044.228001] Call Trace:
> [884044.228001]  [<ffffffff810cec1b>] perf_try_init_event+0x25/0x47
> [884044.228001]  [<ffffffff810d488d>] perf_init_event+0x93/0xca
> [884044.228001]  [<ffffffff810d4b5f>] perf_event_alloc+0x29b/0x32d
> [884044.228001]  [<ffffffff810d5008>] SYSC_perf_event_open+0x417/0x89c
> [884044.228001]  [<ffffffff810d57fe>] SyS_perf_event_open+0x9/0xb

That smells like a corrupted sibling_list, I see no other way for that
loop to not end.

It occurs to me that that list iteration is entirely unserialized, we
should be holding a ctx lock or mutex, but we do not.

Now IIRC the perf fuzzer is single threaded, so it would not actually
trigger the most horrible cases here; but this does smell bad.

Does something like the below make sense and/or help? Jolsa?

---
 kernel/events/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index af924bc38121..763e7c02e796 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -7049,12 +7049,23 @@ EXPORT_SYMBOL_GPL(perf_pmu_unregister);
 
 static int perf_try_init_event(struct pmu *pmu, struct perf_event *event)
 {
+	struct perf_event_context *ctx = NULL;
 	int ret;
 
 	if (!try_module_get(pmu->module))
 		return -ENODEV;
+
+	if (event->group_leader != event) {
+		ctx = perf_event_ctx_lock(event->group_leader);
+		BUG_ON(!ctx);
+	}
+
 	event->pmu = pmu;
 	ret = pmu->event_init(event);
+
+	if (ctx)
+		perf_event_ctx_unlock(event->group_leader, ctx);
+
 	if (ret)
 		module_put(pmu->module);