From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757722AbbIVD2C (ORCPT ); Mon, 21 Sep 2015 23:28:02 -0400 Received: from mail-pa0-f50.google.com ([209.85.220.50]:34621 "EHLO mail-pa0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756923AbbIVD2A (ORCPT ); Mon, 21 Sep 2015 23:28:00 -0400 Subject: Re: [RFC resend] Perf: Trigger and dump sample info to perf.data from user space ring buffer To: Yunlong Song , a.p.zijlstra@chello.nl, paulus@samba.org, mingo@redhat.com, acme@kernel.org, rostedt@goodmis.org, ast@kernel.org, jolsa@kernel.org, Namhyung Kim , masami.hiramatsu.pt@hitachi.com, adrian.hunter@intel.com, bp@alien8.de, rric@kernel.org References: <5600C7F7.70702@huawei.com> Cc: linux-kernel@vger.kernel.org, wangnan0@huawei.com From: David Ahern Message-ID: <5600CAC6.7050504@gmail.com> Date: Mon, 21 Sep 2015 21:28:06 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <5600C7F7.70702@huawei.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 9/21/15 9:16 PM, Yunlong Song wrote: > [Problem Background] > > We want to run perf in daemon mode and collect the traces when the exception > (e.g., machine crashes, app performance goes down) appears. Perf may run for a > long time (from days to weeks or even months), since we do not know when the > exception will appear at all, however it will appear at some time (especially > for a beta product). If we simply use “perf record” as usual, here come two > problems as time goes by: 1 there will be large amounts of IOs created for writing > perf.data which may affects the performance a lot; 2 the size of perf.data will > be larger and larger as well. Although we can use eBPF to reduce the traces in > normal case, but in our case, the perf runs in daemon mode for a long time and > that will accumulate the traces as time goes by. This is a perf-based scheduling daemon I wrote a few years ago: https://github.com/dsahern/linux/blob/perf/full-monty/tools/perf/schedmon.c It solves a similar problem by holding the last N-seconds or M-bytes of events in memory. When something of significance happens it is notified to dump events to a file. The events are scheduling tracepoints but could easily be anything else. David