From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932110AbcBXS1b (ORCPT ); Wed, 24 Feb 2016 13:27:31 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:48521 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758572AbcBXS13 (ORCPT ); Wed, 24 Feb 2016 13:27:29 -0500 Subject: Re: [PATCH] trace-cmd: use nonblocking reads for streaming To: Steven Rostedt References: <1450371712-12218-1-git-send-email-jbacik@fb.com> <20160223181751.31dafea1@gandalf.local.home> CC: , From: Josef Bacik Message-ID: <56CDF609.7010506@fb.com> Date: Wed, 24 Feb 2016 13:27:21 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <20160223181751.31dafea1@gandalf.local.home> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.54.13] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-02-24_09:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/23/2016 06:17 PM, Steven Rostedt wrote: > On Thu, 17 Dec 2015 12:01:52 -0500 > Josef Bacik wrote: > >> I noticed while using the streaming infrastructure in trace-cmd that I was >> seemingly missing events. Using other tracing methods I got these events and >> record->missed_events was never being set. This is because the streaming >> infrastructure uses blocking reads on the per cpu trace pipe's, which means >> we'll wait for an entire pages worth of data to be ready before passing it along >> to the recorder. This makes it impossible to do long term tracing that requires >> coupling two different events that could occur on different CPU's, and I imagine >> it has been what is screwing up my trace-cmd profile runs on our giant 40 cpu >> boxes. Fix trace-cmd instead to use a nonblocking read with select to wait for >> data on the pipe so we don't burn CPU unnecessarily. With this patch I'm no >> longer seeing missed events in my app. Thanks, > > I just want to make sure I understand what is happening here. > > This wasn't trace-cmd's default code right? This was your own app. And > I'm guessing you were matching events perhaps. That is, after seeing > some event, you looked for the other event. But if that event happened > on a CPU that isn't very active, it would wait forever, as the read > was waiting for a full page? > > Or is there something else. > > I don't have a problem with the patch. I just want to understand the > issue. Yup I had an app that was watching block request issue and completion events, and occasionally a completion event would happen on some mostly idle cpu, so I wouldn't get the completion request until several hours later (the app runs all the time) when we finally had a full page to read from that cpu's buffer. Thanks, Josef