[RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Wang Nan]

This patch set has been posted multiple times (with and without
corresponding 'perf tool' patches), and doesn't receive further
comment. I think it should be okay to merge them into mainline.
There are many perf's improvement depend on it. However, Peter
is not responsive after I fixed some problems he pointed out.

Introduces 'write_backward' into perf_event_attr, allows kernel
writing the ring buffer from the end of it. This feature allows
extracting data from overwritable ring buffer.

Wang Nan (5):
  perf core: Introduce new ioctl options to pause and resume ring buffer
  perf core: Set event's default overflow_handler
  perf core: Prepare writing into ring buffer from end
  perf core: Add backward attribute to perf event
  perf core: Reduce perf event output overhead by new overflow handler

 include/linux/perf_event.h      | 22 +++++++++++--
 include/uapi/linux/perf_event.h |  4 ++-
 kernel/events/core.c            | 73 +++++++++++++++++++++++++++++++++++------
 kernel/events/internal.h        | 11 +++++++
 kernel/events/ring_buffer.c     | 63 +++++++++++++++++++++++++++++++----
 5 files changed, 153 insertions(+), 20 deletions(-)

Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com

-- 
1.8.3.4

[RESEND PATCH 1/5] perf core: Introduce new ioctl options to pause and resume ring buffer

[Wang Nan]

Add new ioctl() to pause/resume ring-buffer output.

In some situations we want to read from ring buffer only when we
ensure nothing can write to the ring buffer during reading. Without
this patch we have to turn off all events attached to this ring buffer
to achieve this.

This patch is for supporting overwrite ring buffer. Following
commits will introduce new methods support reading from overwrite ring
buffer. Before reading caller must ensure the ring buffer is frozen, or
the reading is unreliable.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 include/uapi/linux/perf_event.h |  1 +
 kernel/events/core.c            | 13 +++++++++++++
 kernel/events/internal.h        | 11 +++++++++++
 kernel/events/ring_buffer.c     |  7 ++++++-
 4 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index 1afe962..a3c1903 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -401,6 +401,7 @@ struct perf_event_attr {
 #define PERF_EVENT_IOC_SET_FILTER	_IOW('$', 6, char *)
 #define PERF_EVENT_IOC_ID		_IOR('$', 7, __u64 *)
 #define PERF_EVENT_IOC_SET_BPF		_IOW('$', 8, __u32)
+#define PERF_EVENT_IOC_PAUSE_OUTPUT	_IOW('$', 9, __u32)
 
 enum perf_event_ioc_flags {
 	PERF_IOC_FLAG_GROUP		= 1U << 0,
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 5dcc0bd..088c7fe 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -4306,6 +4306,19 @@ static long _perf_ioctl(struct perf_event *event, unsigned int cmd, unsigned lon
 	case PERF_EVENT_IOC_SET_BPF:
 		return perf_event_set_bpf_prog(event, arg);
 
+	case PERF_EVENT_IOC_PAUSE_OUTPUT: {
+		struct ring_buffer *rb;
+
+		rcu_read_lock();
+		rb = rcu_dereference(event->rb);
+		if (!event->rb) {
+			rcu_read_unlock();
+			return -EINVAL;
+		}
+		rb_toggle_paused(rb, !!arg);
+		rcu_read_unlock();
+		return 0;
+	}
 	default:
 		return -ENOTTY;
 	}
diff --git a/kernel/events/internal.h b/kernel/events/internal.h
index 2bbad9c..6a93d1b 100644
--- a/kernel/events/internal.h
+++ b/kernel/events/internal.h
@@ -18,6 +18,7 @@ struct ring_buffer {
 #endif
 	int				nr_pages;	/* nr of data pages  */
 	int				overwrite;	/* can overwrite itself */
+	int				paused;		/* can write into ring buffer */
 
 	atomic_t			poll;		/* POLL_ for wakeups */
 
@@ -65,6 +66,16 @@ static inline void rb_free_rcu(struct rcu_head *rcu_head)
 	rb_free(rb);
 }
 
+static inline void
+rb_toggle_paused(struct ring_buffer *rb,
+		 bool pause)
+{
+	if (!pause && rb->nr_pages)
+		rb->paused = 0;
+	else
+		rb->paused = 1;
+}
+
 extern struct ring_buffer *
 rb_alloc(int nr_pages, long watermark, int cpu, int flags);
 extern void perf_event_wakeup(struct perf_event *event);
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 1faad2c..22e1a47 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -125,8 +125,11 @@ int perf_output_begin(struct perf_output_handle *handle,
 	if (unlikely(!rb))
 		goto out;
 
-	if (unlikely(!rb->nr_pages))
+	if (unlikely(rb->paused)) {
+		if (rb->nr_pages)
+			local_inc(&rb->lost);
 		goto out;
+	}
 
 	handle->rb    = rb;
 	handle->event = event;
@@ -244,6 +247,8 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
 	INIT_LIST_HEAD(&rb->event_list);
 	spin_lock_init(&rb->event_lock);
 	init_irq_work(&rb->irq_work, rb_irq_work);
+
+	rb->paused = rb->nr_pages ? 0 : 1;
 }
 
 static void ring_buffer_put_async(struct ring_buffer *rb)
-- 
1.8.3.4

[RESEND PATCH 2/5] perf core: Set event's default overflow_handler

[Wang Nan]

Set a default event->overflow_handler in perf_event_alloc() so don't
need checking event->overflow_handler in __perf_event_overflow().
Following commits can give a different default overflow_handler.

No extra performance introduced into hot path because in the original
code we still need reading this handler from memory. A conditional branch
is avoided so actually we remove some instructions.

Initial idea comes from Peter at [1].

[1] http://lkml.kernel.org/r/20130708121557.GA17211@twins.programming.kicks-ass.net

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 kernel/events/core.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 088c7fe..ed94c91 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -6467,10 +6467,7 @@ static int __perf_event_overflow(struct perf_event *event,
 		irq_work_queue(&event->pending);
 	}
 
-	if (event->overflow_handler)
-		event->overflow_handler(event, data, regs);
-	else
-		perf_event_output(event, data, regs);
+	event->overflow_handler(event, data, regs);
 
 	if (*perf_event_fasync(event) && event->pending_kill) {
 		event->pending_wakeup = 1;
@@ -7963,8 +7960,13 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 		context = parent_event->overflow_handler_context;
 	}
 
-	event->overflow_handler	= overflow_handler;
-	event->overflow_handler_context = context;
+	if (overflow_handler) {
+		event->overflow_handler	= overflow_handler;
+		event->overflow_handler_context = context;
+	} else {
+		event->overflow_handler = perf_event_output;
+		event->overflow_handler_context = NULL;
+	}
 
 	perf_event__state_init(event);
 
-- 
1.8.3.4

[RESEND PATCH 3/5] perf core: Prepare writing into ring buffer from end

[Wang Nan]

Convert perf_output_begin to __perf_output_begin and make the later
function able to write records from the end of the ring buffer.
Following commits will utilize the 'backward' flag.

This patch doesn't introduce any extra performance overhead since we
use always_inline.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 kernel/events/ring_buffer.c | 42 ++++++++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 22e1a47..37c11c6 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -102,8 +102,21 @@ out:
 	preempt_enable();
 }
 
-int perf_output_begin(struct perf_output_handle *handle,
-		      struct perf_event *event, unsigned int size)
+static bool __always_inline
+ring_buffer_has_space(unsigned long head, unsigned long tail,
+		      unsigned long data_size, unsigned int size,
+		      bool backward)
+{
+	if (!backward)
+		return CIRC_SPACE(head, tail, data_size) >= size;
+	else
+		return CIRC_SPACE(tail, head, data_size) >= size;
+}
+
+static int __always_inline
+__perf_output_begin(struct perf_output_handle *handle,
+		    struct perf_event *event, unsigned int size,
+		    bool backward)
 {
 	struct ring_buffer *rb;
 	unsigned long tail, offset, head;
@@ -146,9 +159,12 @@ int perf_output_begin(struct perf_output_handle *handle,
 	do {
 		tail = READ_ONCE(rb->user_page->data_tail);
 		offset = head = local_read(&rb->head);
-		if (!rb->overwrite &&
-		    unlikely(CIRC_SPACE(head, tail, perf_data_size(rb)) < size))
-			goto fail;
+		if (!rb->overwrite) {
+			if (unlikely(!ring_buffer_has_space(head, tail,
+							    perf_data_size(rb),
+							    size, backward)))
+				goto fail;
+		}
 
 		/*
 		 * The above forms a control dependency barrier separating the
@@ -162,9 +178,17 @@ int perf_output_begin(struct perf_output_handle *handle,
 		 * See perf_output_put_handle().
 		 */
 
-		head += size;
+		if (!backward)
+			head += size;
+		else
+			head -= size;
 	} while (local_cmpxchg(&rb->head, offset, head) != offset);
 
+	if (backward) {
+		offset = head;
+		head = (u64)(-head);
+	}
+
 	/*
 	 * We rely on the implied barrier() by local_cmpxchg() to ensure
 	 * none of the data stores below can be lifted up by the compiler.
@@ -206,6 +230,12 @@ out:
 	return -ENOSPC;
 }
 
+int perf_output_begin(struct perf_output_handle *handle,
+		      struct perf_event *event, unsigned int size)
+{
+	return __perf_output_begin(handle, event, size, false);
+}
+
 unsigned int perf_output_copy(struct perf_output_handle *handle,
 		      const void *buf, unsigned int len)
 {
-- 
1.8.3.4

[RESEND PATCH 4/5] perf core: Add backward attribute to perf event

[Wang Nan]

In perf_event_attr a new bit 'write_backward' is appended to indicate
this event should write ring buffer from its end to beginning.

In perf_output_begin(), prepare ring buffer according this bit.

This patch introduces small overhead into perf_output_begin():
an extra memory read and a conditional branch. Further patch can remove
this overhead by using custom output handler.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 include/linux/perf_event.h      | 5 +++++
 include/uapi/linux/perf_event.h | 3 ++-
 kernel/events/core.c            | 7 +++++++
 kernel/events/ring_buffer.c     | 2 ++
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index a9d8cab..88e2b47 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1031,6 +1031,11 @@ static inline bool has_aux(struct perf_event *event)
 	return event->pmu->setup_aux;
 }
 
+static inline bool is_write_backward(struct perf_event *event)
+{
+	return !!event->attr.write_backward;
+}
+
 extern int perf_output_begin(struct perf_output_handle *handle,
 			     struct perf_event *event, unsigned int size);
 extern void perf_output_end(struct perf_output_handle *handle);
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index a3c1903..43fc8d2 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -340,7 +340,8 @@ struct perf_event_attr {
 				comm_exec      :  1, /* flag comm events that are due to an exec */
 				use_clockid    :  1, /* use @clockid for time fields */
 				context_switch :  1, /* context switch data */
-				__reserved_1   : 37;
+				write_backward :  1, /* Write ring buffer from end to beginning */
+				__reserved_1   : 36;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index ed94c91..513b556 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8196,6 +8196,13 @@ perf_event_set_output(struct perf_event *event, struct perf_event *output_event)
 		goto out;
 
 	/*
+	 * Either writing ring buffer from beginning or from end.
+	 * Mixing is not allowed.
+	 */
+	if (is_write_backward(output_event) != is_write_backward(event))
+		goto out;
+
+	/*
 	 * If both events generate aux data, they must be on the same PMU
 	 */
 	if (has_aux(event) && has_aux(output_event) &&
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 37c11c6..80b1fa7 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -233,6 +233,8 @@ out:
 int perf_output_begin(struct perf_output_handle *handle,
 		      struct perf_event *event, unsigned int size)
 {
+	if (unlikely(is_write_backward(event)))
+		return __perf_output_begin(handle, event, size, true);
 	return __perf_output_begin(handle, event, size, false);
 }
 
-- 
1.8.3.4

[RESEND PATCH 5/5] perf core: Reduce perf event output overhead by new overflow handler

[Wang Nan]

By creating onward and backward specific overflow handlers and setting
them according to event's backward setting, normal sampling events
don't need checking backward setting of an event any more.

This is the last patch of backward writing patchset. After this patch,
there's no extra overhead introduced to the fast path of sampling
output.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 include/linux/perf_event.h  | 17 +++++++++++++++--
 kernel/events/core.c        | 41 ++++++++++++++++++++++++++++++++++++-----
 kernel/events/ring_buffer.c | 12 ++++++++++++
 3 files changed, 63 insertions(+), 7 deletions(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 88e2b47..bbe913f 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -829,9 +829,15 @@ extern int perf_event_overflow(struct perf_event *event,
 				 struct perf_sample_data *data,
 				 struct pt_regs *regs);
 
+extern void perf_event_output_onward(struct perf_event *event,
+				     struct perf_sample_data *data,
+				     struct pt_regs *regs);
+extern void perf_event_output_backward(struct perf_event *event,
+				       struct perf_sample_data *data,
+				       struct pt_regs *regs);
 extern void perf_event_output(struct perf_event *event,
-				struct perf_sample_data *data,
-				struct pt_regs *regs);
+			      struct perf_sample_data *data,
+			      struct pt_regs *regs);
 
 extern void
 perf_event_header__init_id(struct perf_event_header *header,
@@ -1038,6 +1044,13 @@ static inline bool is_write_backward(struct perf_event *event)
 
 extern int perf_output_begin(struct perf_output_handle *handle,
 			     struct perf_event *event, unsigned int size);
+extern int perf_output_begin_onward(struct perf_output_handle *handle,
+				    struct perf_event *event,
+				    unsigned int size);
+extern int perf_output_begin_backward(struct perf_output_handle *handle,
+				      struct perf_event *event,
+				      unsigned int size);
+
 extern void perf_output_end(struct perf_output_handle *handle);
 extern unsigned int perf_output_copy(struct perf_output_handle *handle,
 			     const void *buf, unsigned int len);
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 513b556..969c43e 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -5606,9 +5606,13 @@ void perf_prepare_sample(struct perf_event_header *header,
 	}
 }
 
-void perf_event_output(struct perf_event *event,
-			struct perf_sample_data *data,
-			struct pt_regs *regs)
+static void __always_inline
+__perf_event_output(struct perf_event *event,
+		    struct perf_sample_data *data,
+		    struct pt_regs *regs,
+		    int (*output_begin)(struct perf_output_handle *,
+					struct perf_event *,
+					unsigned int))
 {
 	struct perf_output_handle handle;
 	struct perf_event_header header;
@@ -5618,7 +5622,7 @@ void perf_event_output(struct perf_event *event,
 
 	perf_prepare_sample(&header, data, event, regs);
 
-	if (perf_output_begin(&handle, event, header.size))
+	if (output_begin(&handle, event, header.size))
 		goto exit;
 
 	perf_output_sample(&handle, &header, data, event);
@@ -5629,6 +5633,30 @@ exit:
 	rcu_read_unlock();
 }
 
+void
+perf_event_output_onward(struct perf_event *event,
+			 struct perf_sample_data *data,
+			 struct pt_regs *regs)
+{
+	__perf_event_output(event, data, regs, perf_output_begin_onward);
+}
+
+void
+perf_event_output_backward(struct perf_event *event,
+			   struct perf_sample_data *data,
+			   struct pt_regs *regs)
+{
+	__perf_event_output(event, data, regs, perf_output_begin_backward);
+}
+
+void
+perf_event_output(struct perf_event *event,
+		  struct perf_sample_data *data,
+		  struct pt_regs *regs)
+{
+	__perf_event_output(event, data, regs, perf_output_begin);
+}
+
 /*
  * read event_id
  */
@@ -7963,8 +7991,11 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 	if (overflow_handler) {
 		event->overflow_handler	= overflow_handler;
 		event->overflow_handler_context = context;
+	} else if (is_write_backward(event)){
+		event->overflow_handler = perf_event_output_backward;
+		event->overflow_handler_context = NULL;
 	} else {
-		event->overflow_handler = perf_event_output;
+		event->overflow_handler = perf_event_output_onward;
 		event->overflow_handler_context = NULL;
 	}
 
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 80b1fa7..7e30e012 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -230,6 +230,18 @@ out:
 	return -ENOSPC;
 }
 
+int perf_output_begin_onward(struct perf_output_handle *handle,
+			     struct perf_event *event, unsigned int size)
+{
+	return __perf_output_begin(handle, event, size, false);
+}
+
+int perf_output_begin_backward(struct perf_output_handle *handle,
+			       struct perf_event *event, unsigned int size)
+{
+	return __perf_output_begin(handle, event, size, true);
+}
+
 int perf_output_begin(struct perf_output_handle *handle,
 		      struct perf_event *event, unsigned int size)
 {
-- 
1.8.3.4

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> This patch set has been posted multiple times (with and without
> corresponding 'perf tool' patches), and doesn't receive further
> comment. I think it should be okay to merge them into mainline.
> There are many perf's improvement depend on it. However, Peter
> is not responsive after I fixed some problems he pointed out.
> 
> Introduces 'write_backward' into perf_event_attr, allows kernel
> writing the ring buffer from the end of it. This feature allows
> extracting data from overwritable ring buffer.
> 
> Wang Nan (5):
>   perf core: Introduce new ioctl options to pause and resume ring buffer
>   perf core: Set event's default overflow_handler
>   perf core: Prepare writing into ring buffer from end
>   perf core: Add backward attribute to perf event
>   perf core: Reduce perf event output overhead by new overflow handler

perf kernel features are currently on hold until I can manage to run a
fuzzer for more than a few minutes without my machine having a seizure.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > This patch set has been posted multiple times (with and without
> > corresponding 'perf tool' patches), and doesn't receive further
> > comment. I think it should be okay to merge them into mainline.
> > There are many perf's improvement depend on it. However, Peter
> > is not responsive after I fixed some problems he pointed out.
> > 
> > Introduces 'write_backward' into perf_event_attr, allows kernel
> > writing the ring buffer from the end of it. This feature allows
> > extracting data from overwritable ring buffer.
> > 
> > Wang Nan (5):
> >   perf core: Introduce new ioctl options to pause and resume ring buffer
> >   perf core: Set event's default overflow_handler
> >   perf core: Prepare writing into ring buffer from end
> >   perf core: Add backward attribute to perf event
> >   perf core: Reduce perf event output overhead by new overflow handler
> 
> perf kernel features are currently on hold until I can manage to run a
> fuzzer for more than a few minutes without my machine having a seizure.

Btw., could you describe exactly what commands you are running, with what 
configuration options (if that matters), so that people who'd like our feature 
freeze to be lifted can help out?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 02:49:01PM +0100, Ingo Molnar wrote:
> 
> * Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > > This patch set has been posted multiple times (with and without
> > > corresponding 'perf tool' patches), and doesn't receive further
> > > comment. I think it should be okay to merge them into mainline.
> > > There are many perf's improvement depend on it. However, Peter
> > > is not responsive after I fixed some problems he pointed out.
> > > 
> > > Introduces 'write_backward' into perf_event_attr, allows kernel
> > > writing the ring buffer from the end of it. This feature allows
> > > extracting data from overwritable ring buffer.
> > > 
> > > Wang Nan (5):
> > >   perf core: Introduce new ioctl options to pause and resume ring buffer
> > >   perf core: Set event's default overflow_handler
> > >   perf core: Prepare writing into ring buffer from end
> > >   perf core: Add backward attribute to perf event
> > >   perf core: Reduce perf event output overhead by new overflow handler
> > 
> > perf kernel features are currently on hold until I can manage to run a
> > fuzzer for more than a few minutes without my machine having a seizure.
> 
> Btw., could you describe exactly what commands you are running, with what 
> configuration options (if that matters), so that people who'd like our feature 
> freeze to be lifted can help out?

Mostly syz-kaller, but also Vince's perf-fuzzer and your perf-stress
script, which I'm not sure is publicly available.

perf_fuzzer lives at:

  https://github.com/deater/perf_event_tests.git

Here's a thread on syz-kaller:

  lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com

If things have shifted again I'm sure Dmitry is willing to help.

I run the thing natively on actual real hardware, which ensure I get to
test the PMU drivers too.

# cat go-fuzz.sh
#!/bin/bash

echo 1 > /proc/sys/kernel/traceoff_on_warning
echo 1 > /debug/tracing/options/stacktrace
echo 1 > /debug/tracing/events/sched/enable
cd gopath/src/github.com/google/syzkaller/
./bin/syz-manager -config perf.cfg

# cat gopath/src/github.com/google/syzkaller/perf.cfg

{
        "http": "localhost:50000",
        "workdir": "/root/gopath/src/github.com/google/syzkaller/workdir",
        "syzkaller": "/root/gopath/src/github.com/google/syzkaller",
        "vmlinux": "-",
        "type": "local",
        "count": 1,
        "procs": 160,
        "cover": false,
        "dropprivs": false,
        "enable_syscalls": [
                "getpid",
                "gettid",
                "perf_event_open",
                "ioctl$PERF*",
                "prctl$void",
                "bpf$*",
                "sched_yield"
        ]
}

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Peter Zijlstra <peterz@infradead.org> wrote:

> Here's a thread on syz-kaller:
> 
>   lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> 
> If things have shifted again I'm sure Dmitry is willing to help.

So I tried to install 'go' but it's a _really_ unintuitive tool I have to say.

I installed golang-go on Ubuntu, which gave me a 'go' command:

  triton:~> go version
  go version go1.5.1 linux/amd64

that was the only step that worked. It's a dead end from that point on:

 triton:~> go get github.com/google/syzkaller
 package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath

ok, so lets set GOPATH to the distro src directory:

 triton:~> export GOPATH=/usr/lib/go/src/
 triton:~> go get github.com/google/syzkaller
 package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied

btw., this is what the directory contains:

 triton:~> ls /usr/lib/go/src/
 all.bash          bootstrap.bash  bytes       compress   debug     flag  html      io            make.bat   nacltest.bash  race.bash  run.bash  strconv  testing  unsafe
 all.bat           bufio           clean.bash  container  encoding  fmt   image     iostest.bash  Make.dist  net            race.bat   run.bat   strings  text
 androidtest.bash  buildall.bash   clean.bat   crypto     errors    go    index     log           math       os             reflect    runtime   sync     time
 archive           builtin         cmd         database   expvar    hash  internal  make.bash     mime       path           regexp     sort      syscall  unicode

so, according to the error message it wants a writable directory. Lets try it that 
way:

 triton:~> mkdir go
 triton:~> 
 triton:~> export GOPATH=/home/mingo/go/
 triton:~> go get github.com/google/syzkaller
 can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller


looks like someone wants 'Go' to be used as little as possible! ;-)

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 4:29 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Peter Zijlstra <peterz@infradead.org> wrote:
>
>> Here's a thread on syz-kaller:
>>
>>   lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>>
>> If things have shifted again I'm sure Dmitry is willing to help.
>
> So I tried to install 'go' but it's a _really_ unintuitive tool I have to say.
>
> I installed golang-go on Ubuntu, which gave me a 'go' command:
>
>   triton:~> go version
>   go version go1.5.1 linux/amd64
>
> that was the only step that worked. It's a dead end from that point on:
>
>  triton:~> go get github.com/google/syzkaller
>  package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath
>
> ok, so lets set GOPATH to the distro src directory:
>
>  triton:~> export GOPATH=/usr/lib/go/src/
>  triton:~> go get github.com/google/syzkaller
>  package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied
>
> btw., this is what the directory contains:
>
>  triton:~> ls /usr/lib/go/src/
>  all.bash          bootstrap.bash  bytes       compress   debug     flag  html      io            make.bat   nacltest.bash  race.bash  run.bash  strconv  testing  unsafe
>  all.bat           bufio           clean.bash  container  encoding  fmt   image     iostest.bash  Make.dist  net            race.bat   run.bat   strings  text
>  androidtest.bash  buildall.bash   clean.bat   crypto     errors    go    index     log           math       os             reflect    runtime   sync     time
>  archive           builtin         cmd         database   expvar    hash  internal  make.bash     mime       path           regexp     sort      syscall  unicode
>
> so, according to the error message it wants a writable directory. Lets try it that
> way:
>
>  triton:~> mkdir go
>  triton:~>
>  triton:~> export GOPATH=/home/mingo/go/
>  triton:~> go get github.com/google/syzkaller
>  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller

Yes, GOPATH needs to be set a writable dir.
You can ignore "can't load package" error. The goal of that step is
checkout syzkaller with all dependencies into correct dirs under
GOPATH. That's already done by now.
Or you can do (/... at the end):
$ go get github.com/google/syzkaller/...
That will checkout and build.
Either way you can continue with the make step.



> looks like someone wants 'Go' to be used as little as possible! ;-)

You probably did not observe recently a noobie trying to build a C
project with sufficiently-complicate-build-system and a bunch of
dependencies that needs to be on specific, unknown revisions, and that
still does not compile with you compiler and does not link with your
linker, and then of course target machine has a wrong glibc version :)

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> > so, according to the error message it wants a writable directory. Lets try it that
> > way:
> >
> >  triton:~> mkdir go
> >  triton:~>
> >  triton:~> export GOPATH=/home/mingo/go/
> >  triton:~> go get github.com/google/syzkaller
> >  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> 
> Yes, GOPATH needs to be set a writable dir.
> You can ignore "can't load package" error. The goal of that step is
> checkout syzkaller with all dependencies into correct dirs under
> GOPATH. That's already done by now.
> Or you can do (/... at the end):
> $ go get github.com/google/syzkaller/...
> That will checkout and build.
> Either way you can continue with the make step.

Cool, the '/...' trick works.


> > looks like someone wants 'Go' to be used as little as possible! ;-)
> 
> You probably did not observe recently a noobie trying to build a C
> project with sufficiently-complicate-build-system and a bunch of
> dependencies that needs to be on specific, unknown revisions, and that
> still does not compile with you compiler and does not link with your
> linker, and then of course target machine has a wrong glibc version :)

Nah, old Linux tools very much suck, we know that and we suffer from it.

But new tools should not suck! :-)

So, going from the description at:

http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com

I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg 
with:


triton:~/go/src/github.com/google/syzkaller> cat perf.cfg 
{
        "http": "localhost:50000",
        "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
        "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
        "vmlinux": "-",
        "type": "local",
        "count": 1,
        "procs": 16,
        "nocover": true,
        "nodropprivs": true,
        "enable_syscalls": [
                "getpid",
                "perf_event_open",
                "ioctl$PERF*",
                "prctl$void",
                "bpf$*",
                "sched_yield"
        ]
}

Then I tried to do:

   $ bin/syz-manager -config perf.cfg

in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no 
'bin' directory:

  triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
  bash: bin/syz-manager: No such file or directory

So that should really read something like:

  cd ~/go
  bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg

next roadblock:

fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer

so how do I proceed from here?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> > so, according to the error message it wants a writable directory. Lets try it that
>> > way:
>> >
>> >  triton:~> mkdir go
>> >  triton:~>
>> >  triton:~> export GOPATH=/home/mingo/go/
>> >  triton:~> go get github.com/google/syzkaller
>> >  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>>
>> Yes, GOPATH needs to be set a writable dir.
>> You can ignore "can't load package" error. The goal of that step is
>> checkout syzkaller with all dependencies into correct dirs under
>> GOPATH. That's already done by now.
>> Or you can do (/... at the end):
>> $ go get github.com/google/syzkaller/...
>> That will checkout and build.
>> Either way you can continue with the make step.
>
> Cool, the '/...' trick works.
>
>
>> > looks like someone wants 'Go' to be used as little as possible! ;-)
>>
>> You probably did not observe recently a noobie trying to build a C
>> project with sufficiently-complicate-build-system and a bunch of
>> dependencies that needs to be on specific, unknown revisions, and that
>> still does not compile with you compiler and does not link with your
>> linker, and then of course target machine has a wrong glibc version :)
>
> Nah, old Linux tools very much suck, we know that and we suffer from it.
>
> But new tools should not suck! :-)
>
> So, going from the description at:
>
> http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>
> I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> with:
>
>
> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> {
>         "http": "localhost:50000",
>         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>         "vmlinux": "-",
>         "type": "local",
>         "count": 1,
>         "procs": 16,
>         "nocover": true,
>         "nodropprivs": true,
>         "enable_syscalls": [
>                 "getpid",
>                 "perf_event_open",
>                 "ioctl$PERF*",
>                 "prctl$void",
>                 "bpf$*",
>                 "sched_yield"
>         ]
> }
>
> Then I tried to do:
>
>    $ bin/syz-manager -config perf.cfg
>
> in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> 'bin' directory:
>
>   triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
>   bash: bin/syz-manager: No such file or directory
>
> So that should really read something like:
>
>   cd ~/go
>   bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>
> next roadblock:
>
> fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>
> so how do I proceed from here?


You need to call make in syzkaller dir, it will create
syzkaller/bin/syz-manager. I.e.

$ cd /home/mingo/go/src/github.com/google/syzkaller
$ make
$ bin/syz-manager -config perf.cfg

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> > way:
> >> >
> >> >  triton:~> mkdir go
> >> >  triton:~>
> >> >  triton:~> export GOPATH=/home/mingo/go/
> >> >  triton:~> go get github.com/google/syzkaller
> >> >  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >>
> >> Yes, GOPATH needs to be set a writable dir.
> >> You can ignore "can't load package" error. The goal of that step is
> >> checkout syzkaller with all dependencies into correct dirs under
> >> GOPATH. That's already done by now.
> >> Or you can do (/... at the end):
> >> $ go get github.com/google/syzkaller/...
> >> That will checkout and build.
> >> Either way you can continue with the make step.
> >
> > Cool, the '/...' trick works.
> >
> >
> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >>
> >> You probably did not observe recently a noobie trying to build a C
> >> project with sufficiently-complicate-build-system and a bunch of
> >> dependencies that needs to be on specific, unknown revisions, and that
> >> still does not compile with you compiler and does not link with your
> >> linker, and then of course target machine has a wrong glibc version :)
> >
> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >
> > But new tools should not suck! :-)
> >
> > So, going from the description at:
> >
> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >
> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> > with:
> >
> >
> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> > {
> >         "http": "localhost:50000",
> >         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> >         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> >         "vmlinux": "-",
> >         "type": "local",
> >         "count": 1,
> >         "procs": 16,
> >         "nocover": true,
> >         "nodropprivs": true,
> >         "enable_syscalls": [
> >                 "getpid",
> >                 "perf_event_open",
> >                 "ioctl$PERF*",
> >                 "prctl$void",
> >                 "bpf$*",
> >                 "sched_yield"
> >         ]
> > }
> >
> > Then I tried to do:
> >
> >    $ bin/syz-manager -config perf.cfg
> >
> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> > 'bin' directory:
> >
> >   triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> >   bash: bin/syz-manager: No such file or directory
> >
> > So that should really read something like:
> >
> >   cd ~/go
> >   bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >
> > next roadblock:
> >
> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >
> > so how do I proceed from here?
> 
> 
> You need to call make in syzkaller dir, it will create
> syzkaller/bin/syz-manager. I.e.
> 
> $ cd /home/mingo/go/src/github.com/google/syzkaller
> $ make

Ok, cool, this got me further.

> $ bin/syz-manager -config perf.cfg

I now get periodic output of:

fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:24:19 loading corpus...
2016/03/08 17:24:19 loaded 0 programs
2016/03/08 17:24:19 serving http on http://localhost:50000
2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
2016/03/08 17:24:34 local-0: lost connection: exit status 1
2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
2016/03/08 17:24:49 local-0: lost connection: exit status 1
2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
2016/03/08 17:25:04 local-0: lost connection: exit status 1
2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
2016/03/08 17:25:20 local-0: lost connection: exit status 1
2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
2016/03/08 17:25:35 local-0: lost connection: exit status 1
2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
2016/03/08 17:25:50 local-0: lost connection: exit status 1
2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049

is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can 
see.

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> > * Dmitry Vyukov <dvyukov@google.com> wrote:
>> >
>> >> > so, according to the error message it wants a writable directory. Lets try it that
>> >> > way:
>> >> >
>> >> >  triton:~> mkdir go
>> >> >  triton:~>
>> >> >  triton:~> export GOPATH=/home/mingo/go/
>> >> >  triton:~> go get github.com/google/syzkaller
>> >> >  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>> >>
>> >> Yes, GOPATH needs to be set a writable dir.
>> >> You can ignore "can't load package" error. The goal of that step is
>> >> checkout syzkaller with all dependencies into correct dirs under
>> >> GOPATH. That's already done by now.
>> >> Or you can do (/... at the end):
>> >> $ go get github.com/google/syzkaller/...
>> >> That will checkout and build.
>> >> Either way you can continue with the make step.
>> >
>> > Cool, the '/...' trick works.
>> >
>> >
>> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
>> >>
>> >> You probably did not observe recently a noobie trying to build a C
>> >> project with sufficiently-complicate-build-system and a bunch of
>> >> dependencies that needs to be on specific, unknown revisions, and that
>> >> still does not compile with you compiler and does not link with your
>> >> linker, and then of course target machine has a wrong glibc version :)
>> >
>> > Nah, old Linux tools very much suck, we know that and we suffer from it.
>> >
>> > But new tools should not suck! :-)
>> >
>> > So, going from the description at:
>> >
>> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>> >
>> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
>> > with:
>> >
>> >
>> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
>> > {
>> >         "http": "localhost:50000",
>> >         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>> >         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>> >         "vmlinux": "-",
>> >         "type": "local",
>> >         "count": 1,
>> >         "procs": 16,
>> >         "nocover": true,
>> >         "nodropprivs": true,
>> >         "enable_syscalls": [
>> >                 "getpid",
>> >                 "perf_event_open",
>> >                 "ioctl$PERF*",
>> >                 "prctl$void",
>> >                 "bpf$*",
>> >                 "sched_yield"
>> >         ]
>> > }
>> >
>> > Then I tried to do:
>> >
>> >    $ bin/syz-manager -config perf.cfg
>> >
>> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
>> > 'bin' directory:
>> >
>> >   triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
>> >   bash: bin/syz-manager: No such file or directory
>> >
>> > So that should really read something like:
>> >
>> >   cd ~/go
>> >   bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> >
>> > next roadblock:
>> >
>> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>> >
>> > so how do I proceed from here?
>>
>>
>> You need to call make in syzkaller dir, it will create
>> syzkaller/bin/syz-manager. I.e.
>>
>> $ cd /home/mingo/go/src/github.com/google/syzkaller
>> $ make
>
> Ok, cool, this got me further.
>
>> $ bin/syz-manager -config perf.cfg
>
> I now get periodic output of:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> 2016/03/08 17:24:19 loading corpus...
> 2016/03/08 17:24:19 loaded 0 programs
> 2016/03/08 17:24:19 serving http on http://localhost:50000
> 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
>
> is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> see.

Change:
"nocover": true,
to:
"cover": false,
in the config file.

KCOV will increase efficiency of the fuzzer, but it is not necessary.
As far as I understand Peter tested without KCOV.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 05:29:50PM +0100, Dmitry Vyukov wrote:
> KCOV will increase efficiency of the fuzzer, but it is not necessary.
> As far as I understand Peter tested without KCOV.

Correct, I never quite got around to enabling it. Still on the TODO list
somewhere.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >> >
> >> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >> >
> >> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> >> > way:
> >> >> >
> >> >> >  triton:~> mkdir go
> >> >> >  triton:~>
> >> >> >  triton:~> export GOPATH=/home/mingo/go/
> >> >> >  triton:~> go get github.com/google/syzkaller
> >> >> >  can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >> >>
> >> >> Yes, GOPATH needs to be set a writable dir.
> >> >> You can ignore "can't load package" error. The goal of that step is
> >> >> checkout syzkaller with all dependencies into correct dirs under
> >> >> GOPATH. That's already done by now.
> >> >> Or you can do (/... at the end):
> >> >> $ go get github.com/google/syzkaller/...
> >> >> That will checkout and build.
> >> >> Either way you can continue with the make step.
> >> >
> >> > Cool, the '/...' trick works.
> >> >
> >> >
> >> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >> >>
> >> >> You probably did not observe recently a noobie trying to build a C
> >> >> project with sufficiently-complicate-build-system and a bunch of
> >> >> dependencies that needs to be on specific, unknown revisions, and that
> >> >> still does not compile with you compiler and does not link with your
> >> >> linker, and then of course target machine has a wrong glibc version :)
> >> >
> >> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >> >
> >> > But new tools should not suck! :-)
> >> >
> >> > So, going from the description at:
> >> >
> >> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >> >
> >> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> >> > with:
> >> >
> >> >
> >> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> >> > {
> >> >         "http": "localhost:50000",
> >> >         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> >> >         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> >> >         "vmlinux": "-",
> >> >         "type": "local",
> >> >         "count": 1,
> >> >         "procs": 16,
> >> >         "nocover": true,
> >> >         "nodropprivs": true,
> >> >         "enable_syscalls": [
> >> >                 "getpid",
> >> >                 "perf_event_open",
> >> >                 "ioctl$PERF*",
> >> >                 "prctl$void",
> >> >                 "bpf$*",
> >> >                 "sched_yield"
> >> >         ]
> >> > }
> >> >
> >> > Then I tried to do:
> >> >
> >> >    $ bin/syz-manager -config perf.cfg
> >> >
> >> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> >> > 'bin' directory:
> >> >
> >> >   triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> >> >   bash: bin/syz-manager: No such file or directory
> >> >
> >> > So that should really read something like:
> >> >
> >> >   cd ~/go
> >> >   bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> >
> >> > next roadblock:
> >> >
> >> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >> >
> >> > so how do I proceed from here?
> >>
> >>
> >> You need to call make in syzkaller dir, it will create
> >> syzkaller/bin/syz-manager. I.e.
> >>
> >> $ cd /home/mingo/go/src/github.com/google/syzkaller
> >> $ make
> >
> > Ok, cool, this got me further.
> >
> >> $ bin/syz-manager -config perf.cfg
> >
> > I now get periodic output of:
> >
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> > 2016/03/08 17:24:19 loading corpus...
> > 2016/03/08 17:24:19 loaded 0 programs
> > 2016/03/08 17:24:19 serving http on http://localhost:50000
> > 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> > 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> > 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> > 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> > 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> > 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> > 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> > 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> > 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> > 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> > 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> > 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> > 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
> >
> > is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> > see.
> 
> Change:
> "nocover": true,
> to:
> "cover": false,
> in the config file.
> 
> KCOV will increase efficiency of the fuzzer, but it is not necessary.
> As far as I understand Peter tested without KCOV.

Ok, so now I get:

fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:39:25 loading corpus...
2016/03/08 17:39:25 loaded 0 programs
2016/03/08 17:39:25 serving http on http://localhost:50000
2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809

           |            |-sshd(49552)---sshd(49555)---bash(49562)---syz-manager(49652)-+-syz-fuzzer(49661)-+-syz-executor(49935)
           |            |                                                              |                   |-syz-executor(49936)
           |            |                                                              |                   |-syz-executor(49937)
           |            |                                                              |                   |-syz-executor(49938)
           |            |                                                              |                   |-syz-executor(49939)
           |            |                                                              |                   |-syz-executor(49940)
           |            |                                                              |                   |-syz-executor(49941)
           |            |                                                              |                   |-syz-executor(49942)
           |            |                                                              |                   |-syz-executor(49943)
           |            |                                                              |                   |-syz-executor(49944)
           |            |                                                              |                   |-syz-executor(49945)
           |            |                                                              |                   |-syz-executor(49946)
           |            |                                                              |                   |-syz-executor(49947)
           |            |                                                              |                   |-syz-executor(49948)
           |            |                                                              |                   |-syz-executor(49949)
           |            |                                                              |                   |-syz-executor(49950)
           |            |                                                              |                   |-{syz-fuzzer}(49662)
           |            |                                                              |                   |-{syz-fuzzer}(49663)
           |            |                                                              |                   |-{syz-fuzzer}(49664)
           |            |                                                              |                   |-{syz-fuzzer}(49665)
           |            |                                                              |                   |-{syz-fuzzer}(49666)
           |            |                                                              |                   |-{syz-fuzzer}(49688)
           |            |                                                              |                   |-{syz-fuzzer}(49689)
           |            |                                                              |                   |-{syz-fuzzer}(49690)
           |            |                                                              |                   |-{syz-fuzzer}(49691)
           |            |                                                              |                   |-{syz-fuzzer}(49692)
           |            |                                                              |                   |-{syz-fuzzer}(49693)
           |            |                                                              |                   |-{syz-fuzzer}(49694)
           |            |                                                              |                   |-{syz-fuzzer}(49695)
           |            |                                                              |                   |-{syz-fuzzer}(49696)
           |            |                                                              |                   |-{syz-fuzzer}(49697)
           |            |                                                              |                   |-{syz-fuzzer}(49698)
           |            |                                                              |                   |-{syz-fuzzer}(49699)
           |            |                                                              |                   |-{syz-fuzzer}(49700)
           |            |                                                              |                   |-{syz-fuzzer}(49701)
           |            |                                                              |                   |-{syz-fuzzer}(49704)
           |            |                                                              |                   |-{syz-fuzzer}(49705)
           |            |                                                              |                   |-{syz-fuzzer}(49706)
           |            |                                                              |                   |-{syz-fuzzer}(49710)
           |            |                                                              |                   |-{syz-fuzzer}(49711)
           |            |                                                              |                   |-{syz-fuzzer}(49737)
           |            |                                                              |                   |-{syz-fuzzer}(49739)
           |            |                                                              |                   |-{syz-fuzzer}(49762)
           |            |                                                              |                   |-{syz-fuzzer}(49764)
           |            |                                                              |                   |-{syz-fuzzer}(49787)
           |            |                                                              |                   |-{syz-fuzzer}(49789)
           |            |                                                              |                   |-{syz-fuzzer}(49802)
           |            |                                                              |                   |-{syz-fuzzer}(49803)
           |            |                                                              |                   |-{syz-fuzzer}(49804)
           |            |                                                              |                   |-{syz-fuzzer}(49818)
           |            |                                                              |                   |-{syz-fuzzer}(49822)
           |            |                                                              |                   |-{syz-fuzzer}(49846)
           |            |                                                              |                   |-{syz-fuzzer}(49847)
           |            |                                                              |                   |-{syz-fuzzer}(49848)
           |            |                                                              |                   |-{syz-fuzzer}(49849)
           |            |                                                              |                   |-{syz-fuzzer}(49850)
           |            |                                                              |                   |-{syz-fuzzer}(49851)
           |            |                                                              |                   |-{syz-fuzzer}(49852)
           |            |                                                              |                   |-{syz-fuzzer}(49893)
           |            |                                                              |                   |-{syz-fuzzer}(49894)
           |            |                                                              |                   |-{syz-fuzzer}(49895)
           |            |                                                              |                   |-{syz-fuzzer}(49896)
           |            |                                                              |                   |-{syz-fuzzer}(49933)
           |            |                                                              |                   `-{syz-fuzzer}(49934)
           |            |                                                              |-{syz-manager}(49653)
           |            |                                                              |-{syz-manager}(49654)
           |            |                                                              |-{syz-manager}(49655)
           |            |                                                              |-{syz-manager}(49656)
           |            |                                                              |-{syz-manager}(49657)
           |            |                                                              |-{syz-manager}(49658)
           |            |                                                              |-{syz-manager}(49659)
           |            |                                                              |-{syz-manager}(49667)
           |            |                                                              |-{syz-manager}(49668)
           |            |                                                              |-{syz-manager}(49669)
           |            |                                                              |-{syz-manager}(49670)
           |            |                                                              |-{syz-manager}(49671)
           |            |                                                              |-{syz-manager}(49672)
           |            |                                                              |-{syz-manager}(49673)
           |            |                                                              |-{syz-manager}(49674)
           |            |                                                              |-{syz-manager}(49675)
           |            |                                                              |-{syz-manager}(49676)
           |            |                                                              |-{syz-manager}(49677)
           |            |                                                              |-{syz-manager}(49678)
           |            |                                                              |-{syz-manager}(49679)
           |            |                                                              |-{syz-manager}(49680)
           |            |                                                              |-{syz-manager}(49681)
           |            |                                                              |-{syz-manager}(49682)
           |            |                                                              |-{syz-manager}(49683)
           |            |                                                              |-{syz-manager}(49684)
           |            |                                                              |-{syz-manager}(49685)
           |            |                                                              |-{syz-manager}(49686)
           |            |                                                              |-{syz-manager}(49687)
           |            |                                                              |-{syz-manager}(49913)
           |            |                                                              `-{syz-manager}(49914)


I'm seeing some zombies:

 49935 pts/2    Z+     0:00 [syz-executor] <defunct>
 49936 pts/2    Z+     0:00 [syz-executor] <defunct>
 49937 pts/2    Z+     0:00 [syz-executor] <defunct>
 49938 pts/2    Z+     0:00 [syz-executor] <defunct>
 49939 pts/2    Z+     0:00 [syz-executor] <defunct>
 49940 pts/2    Z+     0:00 [syz-executor] <defunct>
 49941 pts/2    Z+     0:00 [syz-executor] <defunct>
 49942 pts/2    Z+     0:00 [syz-executor] <defunct>
 49943 pts/2    Z+     0:00 [syz-executor] <defunct>
 49944 pts/2    Z+     0:00 [syz-executor] <defunct>
 49945 pts/2    Z+     0:00 [syz-executor] <defunct>
 49946 pts/2    Z+     0:00 [syz-executor] <defunct>
 49947 pts/2    Z+     0:00 [syz-executor] <defunct>
 49948 pts/2    Z+     0:00 [syz-executor] <defunct>
 49949 pts/2    Z+     0:00 [syz-executor] <defunct>
 49950 pts/2    Z+     0:00 [syz-executor] <defun

does it mean it found a kernel bug already?

It only had a couple of seconds of runtime:

 49652 mingo     20   0 1434276  52144  11344 S   0.0  0.0   0:00.54 syz-manager                                                                                                      
 49661 mingo     20   0 2196672  43948  10448 S   0.0  0.0   0:05.59 syz-fuzzer                 

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> It only had a couple of seconds of runtime:
> 
>  49652 mingo     20   0 1434276  52144  11344 S   0.0  0.0   0:00.54 syz-manager                                                                                                      
>  49661 mingo     20   0 2196672  43948  10448 S   0.0  0.0   0:05.59 syz-fuzzer                 

Ah, so it appears to making some progress:

 49652 mingo     20   0 1581740  47600  11344 S   0.0  0.0   0:00.58 syz-manager                                                                                                      
 49661 mingo     20   0 2204868  43720  10448 S   0.0  0.0   0:07.49 syz-fuzzer                             

 49652 mingo     20   0 1598132  31512  11344 S   0.0  0.0   0:00.61 syz-manager                                                                                                      
 49661 mingo     20   0 2204868  44252  10448 S   0.0  0.0   0:09.09 syz-fuzzer               

but only about +1 second runtime added every minute or so. Is that expected?

There's no progress mark anywhere suggesting that the tool thinks it is going 
fine. You might want to emit periodic (once a minute or so) 'I am still OK!' 
messages or so.

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Ingo Molnar <mingo@kernel.org> wrote:
>
>> It only had a couple of seconds of runtime:
>>
>>  49652 mingo     20   0 1434276  52144  11344 S   0.0  0.0   0:00.54 syz-manager
>>  49661 mingo     20   0 2196672  43948  10448 S   0.0  0.0   0:05.59 syz-fuzzer
>
> Ah, so it appears to making some progress:
>
>  49652 mingo     20   0 1581740  47600  11344 S   0.0  0.0   0:00.58 syz-manager
>  49661 mingo     20   0 2204868  43720  10448 S   0.0  0.0   0:07.49 syz-fuzzer
>
>  49652 mingo     20   0 1598132  31512  11344 S   0.0  0.0   0:00.61 syz-manager
>  49661 mingo     20   0 2204868  44252  10448 S   0.0  0.0   0:09.09 syz-fuzzer
>
> but only about +1 second runtime added every minute or so. Is that expected?

The main work is done by child syz-executor processes.
syz-manager/syz-fuzzer only guide the process.
You can set "procs" param in config to higher value to increase CPU
utilization. To get more bugs you want to saturate all CPUs to trigger
more unusual thread interleavings.


> There's no progress mark anywhere suggesting that the tool thinks it is going
> fine. You might want to emit periodic (once a minute or so) 'I am still OK!'
> messages or so.

Will do.


Regarding the zombie processes, it may or may not be OK. If they hang
for minutes and you can't kill them, then it is a kernel bug. If they
hang for minutes and you can kill them, then it is either kernel bug
of my bug. If they are recycled eventually, then it is OK. The first
thing I do in such cases is:

$ cat /proc/$PID/task/**/stack

If there is a second unfinished thread hanging on a kernel spinlock or
mutex, then it's definitely bad.
It also helps to enable CONFIG_RCU_STALL_COMMON=y,
CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
debugging. These can detect various stalls.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Ingo Molnar <mingo@kernel.org> wrote:
> >
> >> It only had a couple of seconds of runtime:
> >>
> >>  49652 mingo     20   0 1434276  52144  11344 S   0.0  0.0   0:00.54 syz-manager
> >>  49661 mingo     20   0 2196672  43948  10448 S   0.0  0.0   0:05.59 syz-fuzzer
> >
> > Ah, so it appears to making some progress:
> >
> >  49652 mingo     20   0 1581740  47600  11344 S   0.0  0.0   0:00.58 syz-manager
> >  49661 mingo     20   0 2204868  43720  10448 S   0.0  0.0   0:07.49 syz-fuzzer
> >
> >  49652 mingo     20   0 1598132  31512  11344 S   0.0  0.0   0:00.61 syz-manager
> >  49661 mingo     20   0 2204868  44252  10448 S   0.0  0.0   0:09.09 syz-fuzzer
> >
> > but only about +1 second runtime added every minute or so. Is that expected?
> 
> The main work is done by child syz-executor processes.

Hm, they don't seem to be doing anything:

fomalhaut:~> ps aux | grep syz-executor
mingo     41506  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41509  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41513  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41515  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41523  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41601  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41608  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41662  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41764  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     41966  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42029  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42084  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42145  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42149  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42166  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     42175  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
mingo     57627  2.2  0.0 1860540 44884 pts/2   Sl+  18:16   0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0

... because they are recycling:

fomalhaut:~> ps aux | grep syz-executor
mingo     57627  1.6  0.0 1942468 44624 pts/2   Sl+  18:16   0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
mingo     98448  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98454  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98468  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98472  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98476  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98522  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98525  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98548  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98568  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98596  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98618  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98644  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98695  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98708  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98737  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
mingo     98756  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>

I'm not seeing anything happen in 'top' - only a mostly idle system.

> syz-manager/syz-fuzzer only guide the process.
> You can set "procs" param in config to higher value to increase CPU
> utilization. To get more bugs you want to saturate all CPUs to trigger
> more unusual thread interleavings.

So right now it doesn't seem to saturate 16 CPUs - not even close to it.

> If there is a second unfinished thread hanging on a kernel spinlock or
> mutex, then it's definitely bad.
> It also helps to enable CONFIG_RCU_STALL_COMMON=y,
> CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
> debugging. These can detect various stalls.

I can just Ctrl-C the manager and it shuts down within a few seconds:

2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
2016/03/08 17:51:45 local-0: lost connection: exit status 2
2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570
2016/03/08 18:04:04 local-0: lost connection: exit status 2
2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131
2016/03/08 18:16:24 local-0: lost connection: exit status 2
2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124
^C2016/03/08 18:22:53 shutting down...

with nothing hanging around:

fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz

and with no kernel messages in dmesg - and with a fully functional system.

I'm running the 16-task load on a 120 CPU system - should I increase it to 120? 
Does the code expect to saturate the system?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 6:24 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> > * Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> >> It only had a couple of seconds of runtime:
>> >>
>> >>  49652 mingo     20   0 1434276  52144  11344 S   0.0  0.0   0:00.54 syz-manager
>> >>  49661 mingo     20   0 2196672  43948  10448 S   0.0  0.0   0:05.59 syz-fuzzer
>> >
>> > Ah, so it appears to making some progress:
>> >
>> >  49652 mingo     20   0 1581740  47600  11344 S   0.0  0.0   0:00.58 syz-manager
>> >  49661 mingo     20   0 2204868  43720  10448 S   0.0  0.0   0:07.49 syz-fuzzer
>> >
>> >  49652 mingo     20   0 1598132  31512  11344 S   0.0  0.0   0:00.61 syz-manager
>> >  49661 mingo     20   0 2204868  44252  10448 S   0.0  0.0   0:09.09 syz-fuzzer
>> >
>> > but only about +1 second runtime added every minute or so. Is that expected?
>>
>> The main work is done by child syz-executor processes.
>
> Hm, they don't seem to be doing anything:
>
> fomalhaut:~> ps aux | grep syz-executor
> mingo     41506  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41509  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41513  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41515  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41523  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41601  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41608  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41662  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41764  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     41966  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42029  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42084  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42145  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42149  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42166  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     42175  0.0  0.0      0     0 pts/2    Z+   18:19   0:00 [syz-executor] <defunct>
> mingo     57627  2.2  0.0 1860540 44884 pts/2   Sl+  18:16   0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
>
> ... because they are recycling:
>
> fomalhaut:~> ps aux | grep syz-executor
> mingo     57627  1.6  0.0 1942468 44624 pts/2   Sl+  18:16   0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
> mingo     98448  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98454  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98468  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98472  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98476  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98522  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98525  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98548  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98568  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98596  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98618  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98644  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98695  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98708  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98737  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
> mingo     98756  0.0  0.0      0     0 pts/2    Z+   18:21   0:00 [syz-executor] <defunct>
>
> I'm not seeing anything happen in 'top' - only a mostly idle system.
>
>> syz-manager/syz-fuzzer only guide the process.
>> You can set "procs" param in config to higher value to increase CPU
>> utilization. To get more bugs you want to saturate all CPUs to trigger
>> more unusual thread interleavings.
>
> So right now it doesn't seem to saturate 16 CPUs - not even close to it.
>
>> If there is a second unfinished thread hanging on a kernel spinlock or
>> mutex, then it's definitely bad.
>> It also helps to enable CONFIG_RCU_STALL_COMMON=y,
>> CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
>> debugging. These can detect various stalls.
>
> I can just Ctrl-C the manager and it shuts down within a few seconds:
>
> 2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
> 2016/03/08 17:51:45 local-0: lost connection: exit status 2
> 2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570
> 2016/03/08 18:04:04 local-0: lost connection: exit status 2
> 2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131
> 2016/03/08 18:16:24 local-0: lost connection: exit status 2
> 2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124
> ^C2016/03/08 18:22:53 shutting down...
>
> with nothing hanging around:

OK, this all looks good.

> fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz
>
> and with no kernel messages in dmesg - and with a fully functional system.
>
> I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> Does the code expect to saturate the system?

No, it does not expect to saturate the system. Set "procs" to 480, or
something like that.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> > mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz
> >
> > and with no kernel messages in dmesg - and with a fully functional system.
> >
> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> > Does the code expect to saturate the system?
> 
> No, it does not expect to saturate the system. Set "procs" to 480, or
> something like that.

Does not seem to help much:

fomalhaut:~> vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

 1  0      0 257465904 219940 4736092    0    0     0   102 16022 4396  0  1 99  0  0
 2  0      0 257452144 220496 4755052    0    0     2  3649 14286 4627  0  1 99  0  0
 2  0      0 257473408 221188 4770824    0    0    15  1898 17175 4474  0  1 99  0  0

Only around 1% system utilization. Should I go for 1,000 or more? :)

Peter, do you experience with running syz-kaller on larger CPU count Intel 
systems?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
>> > mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz
>> >
>> > and with no kernel messages in dmesg - and with a fully functional system.
>> >
>> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
>> > Does the code expect to saturate the system?
>>
>> No, it does not expect to saturate the system. Set "procs" to 480, or
>> something like that.
>
> Does not seem to help much:
>
> fomalhaut:~> vmstat 10
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>
>  1  0      0 257465904 219940 4736092    0    0     0   102 16022 4396  0  1 99  0  0
>  2  0      0 257452144 220496 4755052    0    0     2  3649 14286 4627  0  1 99  0  0
>  2  0      0 257473408 221188 4770824    0    0    15  1898 17175 4474  0  1 99  0  0
>
> Only around 1% system utilization. Should I go for 1,000 or more? :)
>
> Peter, do you experience with running syz-kaller on larger CPU count Intel
> systems?


Try to set "dropprivs": false in config.
I've noticed that creation/destruction of namespaces is very slow and
globally serialized. So sometimes it takes tens of seconds for each
worker processes to startup. For perf-related syscalls it should be
"safe" to just run as root. And perf subsystem operation is also
unaffected by namespaces as far as I know, so it should not affect
behavior as well.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Dmitry Vyukov <dvyukov@google.com> wrote:

> On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > * Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> >> > mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz
> >> >
> >> > and with no kernel messages in dmesg - and with a fully functional system.
> >> >
> >> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> >> > Does the code expect to saturate the system?
> >>
> >> No, it does not expect to saturate the system. Set "procs" to 480, or
> >> something like that.
> >
> > Does not seem to help much:
> >
> > fomalhaut:~> vmstat 10
> > procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
> >
> >  1  0      0 257465904 219940 4736092    0    0     0   102 16022 4396  0  1 99  0  0
> >  2  0      0 257452144 220496 4755052    0    0     2  3649 14286 4627  0  1 99  0  0
> >  2  0      0 257473408 221188 4770824    0    0    15  1898 17175 4474  0  1 99  0  0
> >
> > Only around 1% system utilization. Should I go for 1,000 or more? :)
> >
> > Peter, do you experience with running syz-kaller on larger CPU count Intel
> > systems?
> 
> 
> Try to set "dropprivs": false in config.

Things got a lot more lively after that!

But most of the overhead seems to come from systemd trying to dump core or 
something like that:

 85872 mingo     20   0   34712   3016   2656 S   4.6  0.0   0:00.14 systemd-coredum                                                                                                  
 85440 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
 85751 mingo     20   0   34712   3076   2716 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
 85840 mingo     20   0   34712   2988   2624 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
 85861 mingo     20   0   34712   3080   2720 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
 85954 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum          

and I have:

 fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
 0

weird ... Has any of you seen such behavior?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Tue, Mar 8, 2016 at 6:48 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> > * Dmitry Vyukov <dvyukov@google.com> wrote:
>> >
>> >> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
>> >> > mingo      1374  0.0  0.0 118476  2376 pts/2    S+   18:23   0:00 grep --color=auto -i syz
>> >> >
>> >> > and with no kernel messages in dmesg - and with a fully functional system.
>> >> >
>> >> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
>> >> > Does the code expect to saturate the system?
>> >>
>> >> No, it does not expect to saturate the system. Set "procs" to 480, or
>> >> something like that.
>> >
>> > Does not seem to help much:
>> >
>> > fomalhaut:~> vmstat 10
>> > procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>> >
>> >  1  0      0 257465904 219940 4736092    0    0     0   102 16022 4396  0  1 99  0  0
>> >  2  0      0 257452144 220496 4755052    0    0     2  3649 14286 4627  0  1 99  0  0
>> >  2  0      0 257473408 221188 4770824    0    0    15  1898 17175 4474  0  1 99  0  0
>> >
>> > Only around 1% system utilization. Should I go for 1,000 or more? :)
>> >
>> > Peter, do you experience with running syz-kaller on larger CPU count Intel
>> > systems?
>>
>>
>> Try to set "dropprivs": false in config.
>
> Things got a lot more lively after that!
>
> But most of the overhead seems to come from systemd trying to dump core or
> something like that:
>
>  85872 mingo     20   0   34712   3016   2656 S   4.6  0.0   0:00.14 systemd-coredum
>  85440 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum
>  85751 mingo     20   0   34712   3076   2716 S   4.2  0.0   0:00.13 systemd-coredum
>  85840 mingo     20   0   34712   2988   2624 S   4.2  0.0   0:00.13 systemd-coredum
>  85861 mingo     20   0   34712   3080   2720 S   4.2  0.0   0:00.13 systemd-coredum
>  85954 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum
>
> and I have:
>
>  fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
>  0
>
> weird ... Has any of you seen such behavior?


I have not seen it.
Probably I need to directly disable core dumps within the syz-executor process.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 06:48:56PM +0100, Ingo Molnar wrote:
> weird ... Has any of you seen such behavior?

No systemd on any of my machines ;-)

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> Things got a lot more lively after that!
> 
> But most of the overhead seems to come from systemd trying to dump core or 
> something like that:
> 
>  85872 mingo     20   0   34712   3016   2656 S   4.6  0.0   0:00.14 systemd-coredum                                                                                                  
>  85440 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
>  85751 mingo     20   0   34712   3076   2716 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
>  85840 mingo     20   0   34712   2988   2624 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
>  85861 mingo     20   0   34712   3080   2720 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
>  85954 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum          
> 
> and I have:
> 
>  fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
>  0
> 
> weird ... Has any of you seen such behavior?

So the workaround for that is to disable systemd trying to log every core dump to 
the system journal (!), via:

  echo > /proc/sys/kernel/core_pattern 

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> 
> * Ingo Molnar <mingo@kernel.org> wrote:
> 
> > Things got a lot more lively after that!
> > 
> > But most of the overhead seems to come from systemd trying to dump core or 
> > something like that:
> > 
> >  85872 mingo     20   0   34712   3016   2656 S   4.6  0.0   0:00.14 systemd-coredum                                                                                                  
> >  85440 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
> >  85751 mingo     20   0   34712   3076   2716 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
> >  85840 mingo     20   0   34712   2988   2624 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
> >  85861 mingo     20   0   34712   3080   2720 S   4.2  0.0   0:00.13 systemd-coredum                                                                                                  
> >  85954 mingo     20   0   34712   3028   2664 S   4.2  0.0   0:00.13 systemd-coredum          
> > 
> > and I have:
> > 
> >  fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
> >  0
> > 
> > weird ... Has any of you seen such behavior?
> 
> So the workaround for that is to disable systemd trying to log every core dump to 
> the system journal (!), via:
> 
>   echo > /proc/sys/kernel/core_pattern 

So with that fixed, I finally started fuzzing for real.

With nproc set to 120 it seems to be chugging along at about 25% system 
utilization:

Tasks: 1271 total,   1 running, 1270 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.5 us, 35.3 sy,  0.0 ni, 55.2 id,  0.5 wa,  4.5 hi,  0.0 si,  0.0 st
KiB Mem : 26401230+total, 25401017+free,  1624640 used,  8377496 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 26143329+avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
 87593 mingo     20   0 7850436 162284  11672 S  2941  0.1  33:01.83 syz-fuzzer                                                                                                       
   923 root      20   0   84772  44344  43840 S  22.8  0.0   1:23.86 systemd-journal                                                                                                  
  1369 root      16  -4  114636   3256   2832 S  15.7  0.0   0:29.03 auditd                                                                                                           
  1379 root      12  -8   80236   1764   1432 S   8.3  0.0   0:15.79 audispd                                                                                                          
   878 root      20   0       0      0      0 S   6.9  0.0   0:16.55 jbd2/sda1-8                                                                                                      
  1381 root      16  -4   52216   3232   2892 S   3.8  0.0   0:07.08 sedispatch              

Even that one is not ideal - obviously there's way too much systemd-journal 
overhead, but I'm unable to turn the darn thing off ...

with nproc=480 it does not seem to be working very well - it quickly generates:

2016/03/08 18:59:38 local-0: lost connection: exit status 2
2016/03/08 18:59:38 local-0: saving crash 'lost connection' to crash-local-0-1457459978295423413

and then after some time starts to ramp up again. It's mostly idling around.

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> With nproc set to 120 it seems to be chugging along at about 25% system 
> utilization:
> 
> Tasks: 1271 total,   1 running, 1270 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  4.5 us, 35.3 sy,  0.0 ni, 55.2 id,  0.5 wa,  4.5 hi,  0.0 si,  0.0 st
> KiB Mem : 26401230+total, 25401017+free,  1624640 used,  8377496 buff/cache
> KiB Swap:        0 total,        0 free,        0 used. 26143329+avail Mem 
> 
>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
>  87593 mingo     20   0 7850436 162284  11672 S  2941  0.1  33:01.83 syz-fuzzer                                                                                                       
>    923 root      20   0   84772  44344  43840 S  22.8  0.0   1:23.86 systemd-journal                                                                                                  
>   1369 root      16  -4  114636   3256   2832 S  15.7  0.0   0:29.03 auditd                                                                                                           
>   1379 root      12  -8   80236   1764   1432 S   8.3  0.0   0:15.79 audispd                                                                                                          
>    878 root      20   0       0      0      0 S   6.9  0.0   0:16.55 jbd2/sda1-8                                                                                                      
>   1381 root      16  -4   52216   3232   2892 S   3.8  0.0   0:07.08 sedispatch              
> 
> Even that one is not ideal - obviously there's way too much systemd-journal 
> overhead, but I'm unable to turn the darn thing off ...

The journal.conf man page suggests that putting 'Storage=none' into 
/etc/systemd/journal.conf disables journalling - but that's not true.

It's apparently impossible to disable systemd logging via any normal means ...

So the workaround for all that is a brutal:

   mv /var/log/journal /var/log/journal.dontuse

after that there does not seem to be systemd logging anymore.

... but there is tons of auditd logging now! ;-)

Fortunately that's easily stopped via:

  service stop auditd

with that there's no log IO anymore during fuzzing (yay!).

Except that systemd journald rears its ugly head back:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                          
111464 mingo     20   0 7249024 143348  11476 S  2741  0.1   7:41.31 syz-fuzzer                                                                                                       
122632 root      20   0   29200   2768   2524 S   9.5  0.0   0:28.74 systemd-journal                                                                                                  
111463 root      20   0  165596   5772   3840 R   4.8  0.0   0:00.80 top                                                                                                              
112078 mingo     20   0   20928    704    644 S   2.9  0.0   0:00.17 syz-executor                                                                                                     
112516 mingo     20   0   20928    708    644 S   2.9  0.0   0:00.16 syz-executor                                                                                                     
   878 root      20   0       0      0      0 S   1.9  0.0   0:53.83 jbd2/sda1-8                                                                                                      
111711 mingo     20   0   20928    704    644 S   1.9  0.0   0:00.16 syz-executor                                                                                                     
111901 mingo     20   0   20928    704    644 S   1.9  0.0   0:00.15 syz-executor                                                                                                     
     8 root      20   0       0      0      0 S   1.0  0.0   1:00.55 rcu_sched                       

so it's eating about 10% of system overhead despite doing nothing (!).

The workaround for that systemd bug is a brutal:

 [root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
 [root@fomalhaut ~]# 

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Ingo Molnar]

* Ingo Molnar <mingo@kernel.org> wrote:

> so it's eating about 10% of system overhead despite doing nothing (!).
> 
> The workaround for that systemd bug is a brutal:
> 
>  [root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
>  [root@fomalhaut ~]# 

Except that if I do that, systemd stops working altogether - for example:

 [root@fomalhaut ~]# systemctl disable audit
 Failed to execute operation: Connection timed out

only reinstating the binary and re-starting journald fixes that.

systemd is the most passive-agressive utility I've ever seen.

So it's not possible to disable journald. Does anyone know any solution for that, 
which does not involve reinstalling a whole distro?

Thanks,

	Ingo

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 06:37:09PM +0100, Ingo Molnar wrote:
> Peter, do you experience with running syz-kaller on larger CPU count Intel 
> systems?

I run it on a 20 core (40 cpu) system mostly, as that tends to not take
ages to come back up again once I wrecked it.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 05:27:03PM +0100, Ingo Molnar wrote:
> > > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> > > {
> > >         "http": "localhost:50000",
> > >         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> > >         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> > >         "vmlinux": "-",
> > >         "type": "local",
> > >         "count": 1,
> > >         "procs": 16,
> > >         "nocover": true,

this^^^

> I now get periodic output of:
> 
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> 2016/03/08 17:24:19 loading corpus...
> 2016/03/08 17:24:19 loaded 0 programs
> 2016/03/08 17:24:19 serving http on http://localhost:50000
> 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> 2016/03/08 17:24:34 local-0: lost connection: exit status 1

> is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can 
> see.

Should have disabled that.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Borislav Petkov]

On Tue, Mar 08, 2016 at 04:54:23PM +0100, Ingo Molnar wrote:
> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg 
> {
>         "http": "localhost:50000",
>         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>         "vmlinux": "-",
>         "type": "local",
>         "count": 1,
>         "procs": 16,
>         "nocover": true,
>         "nodropprivs": true,
>         "enable_syscalls": [
>                 "getpid",
>                 "perf_event_open",

Btw, is there a way to specify range of arguments to feed into
perf_event_open? Like, limit @attr_uptr to single or multiple event IDs
or so, for example?

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Dmitry Vyukov]

On Wed, Mar 9, 2016 at 11:53 AM, Borislav Petkov <bp@alien8.de> wrote:
> On Tue, Mar 08, 2016 at 04:54:23PM +0100, Ingo Molnar wrote:
>> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
>> {
>>         "http": "localhost:50000",
>>         "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>>         "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>>         "vmlinux": "-",
>>         "type": "local",
>>         "count": 1,
>>         "procs": 16,
>>         "nocover": true,
>>         "nodropprivs": true,
>>         "enable_syscalls": [
>>                 "getpid",
>>                 "perf_event_open",
>
> Btw, is there a way to specify range of arguments to feed into
> perf_event_open? Like, limit @attr_uptr to single or multiple event IDs
> or so, for example?


No, there is no _that_ level of granularity in the config.

If you really-really want that, then you can alter description of the
perf_event_open syscall in sys/perf.txt file:

https://github.com/google/syzkaller/blob/master/sys/perf.txt

to be more restrictive. For example, you can limit set of values
passed in a particular argument, or even set some args/fields to const
values.
Then you will need to do:

$ make generate LINUX=/path/to/fresh/linux/checkout
$ make

But I would suggest to not do that. That perf config already limits
set of syscalls to a very small set. The fuzzer should be able to
examine all interesting combinations of arguments for these syscalls
in a reasonable time (provided that you use CONFIG_KCOV). And in the
end you don't know where the bugs. They are usually where you don't
expect them. So I would suggest the opposite: describe and more
perf-related syscalls, describe arguments in greater detail, enable
other syscalls that can have effect on perf subsystem. And then just
run it for longer using more machines.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Jiri Olsa]

On Tue, Mar 08, 2016 at 02:57:59PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 08, 2016 at 02:49:01PM +0100, Ingo Molnar wrote:
> > 
> > * Peter Zijlstra <peterz@infradead.org> wrote:
> > 
> > > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > > > This patch set has been posted multiple times (with and without
> > > > corresponding 'perf tool' patches), and doesn't receive further
> > > > comment. I think it should be okay to merge them into mainline.
> > > > There are many perf's improvement depend on it. However, Peter
> > > > is not responsive after I fixed some problems he pointed out.
> > > > 
> > > > Introduces 'write_backward' into perf_event_attr, allows kernel
> > > > writing the ring buffer from the end of it. This feature allows
> > > > extracting data from overwritable ring buffer.
> > > > 
> > > > Wang Nan (5):
> > > >   perf core: Introduce new ioctl options to pause and resume ring buffer
> > > >   perf core: Set event's default overflow_handler
> > > >   perf core: Prepare writing into ring buffer from end
> > > >   perf core: Add backward attribute to perf event
> > > >   perf core: Reduce perf event output overhead by new overflow handler
> > > 
> > > perf kernel features are currently on hold until I can manage to run a
> > > fuzzer for more than a few minutes without my machine having a seizure.
> > 
> > Btw., could you describe exactly what commands you are running, with what 
> > configuration options (if that matters), so that people who'd like our feature 
> > freeze to be lifted can help out?
> 
> Mostly syz-kaller, but also Vince's perf-fuzzer and your perf-stress
> script, which I'm not sure is publicly available.
> 
> perf_fuzzer lives at:
> 
>   https://github.com/deater/perf_event_tests.git
> 
> Here's a thread on syz-kaller:
> 
>   lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> 
> If things have shifted again I'm sure Dmitry is willing to help.
> 
> I run the thing natively on actual real hardware, which ensure I get to
> test the PMU drivers too.
> 
> # cat go-fuzz.sh
> #!/bin/bash
> 
> echo 1 > /proc/sys/kernel/traceoff_on_warning
> echo 1 > /debug/tracing/options/stacktrace
> echo 1 > /debug/tracing/events/sched/enable

are you running this under root?

jirka

> cd gopath/src/github.com/google/syzkaller/
> ./bin/syz-manager -config perf.cfg
> 
> # cat gopath/src/github.com/google/syzkaller/perf.cfg
> 
> {
>         "http": "localhost:50000",
>         "workdir": "/root/gopath/src/github.com/google/syzkaller/workdir",
>         "syzkaller": "/root/gopath/src/github.com/google/syzkaller",
>         "vmlinux": "-",
>         "type": "local",
>         "count": 1,
>         "procs": 160,
>         "cover": false,
>         "dropprivs": false,
>         "enable_syscalls": [
>                 "getpid",
>                 "gettid",
>                 "perf_event_open",
>                 "ioctl$PERF*",
>                 "prctl$void",
>                 "bpf$*",
>                 "sched_yield"
>         ]
> }

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > # cat go-fuzz.sh
> > #!/bin/bash
> > 
> > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > echo 1 > /debug/tracing/options/stacktrace
> > echo 1 > /debug/tracing/events/sched/enable
> 
> are you running this under root?

Of course ;-) I'm not even sure these machines have a user account.

Also, what's the point of fuzzing if half your 'fun' options return
-EPERM.

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Jiri Olsa]

On Tue, Mar 08, 2016 at 09:07:46PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > > # cat go-fuzz.sh
> > > #!/bin/bash
> > > 
> > > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > > echo 1 > /debug/tracing/options/stacktrace
> > > echo 1 > /debug/tracing/events/sched/enable
> > 
> > are you running this under root?
> 
> Of course ;-) I'm not even sure these machines have a user account.
> 
> Also, what's the point of fuzzing if half your 'fun' options return
> -EPERM.

IIRC the perf fuzzer (one from Vince) could screw your machine having
it run under root

jirka

Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer

[Peter Zijlstra]

On Tue, Mar 08, 2016 at 09:44:58PM +0100, Jiri Olsa wrote:
> On Tue, Mar 08, 2016 at 09:07:46PM +0100, Peter Zijlstra wrote:
> > On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > > > # cat go-fuzz.sh
> > > > #!/bin/bash
> > > > 
> > > > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > > > echo 1 > /debug/tracing/options/stacktrace
> > > > echo 1 > /debug/tracing/events/sched/enable
> > > 
> > > are you running this under root?
> > 
> > Of course ;-) I'm not even sure these machines have a user account.
> > 
> > Also, what's the point of fuzzing if half your 'fun' options return
> > -EPERM.
> 
> IIRC the perf fuzzer (one from Vince) could screw your machine having
> it run under root

Never noticed :-) Guess how I usually run it..