From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f174.google.com (mail-pd0-f174.google.com [209.85.192.174]) by kanga.kvack.org (Postfix) with ESMTP id 16E146B0031 for ; Thu, 3 Apr 2014 17:44:04 -0400 (EDT) Received: by mail-pd0-f174.google.com with SMTP id y13so2387846pdi.33 for ; Thu, 03 Apr 2014 14:44:03 -0700 (PDT) Received: from g2t2353.austin.hp.com (g2t2353.austin.hp.com. [15.217.128.52]) by mx.google.com with ESMTPS id yx10si3640286pab.343.2014.04.03.14.44.02 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 03 Apr 2014 14:44:02 -0700 (PDT) Message-ID: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> Subject: [RFC] mm,tracing: improve current situation From: Davidlohr Bueso Date: Thu, 03 Apr 2014 14:44:00 -0700 Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton Hi All, During LSFMM Dave Jones discussed the current situation around testing/trinity in the mm. One of the conclusions was that basically we lack tools to gather the necessary information to make debugging a less painful process, making it pretty much a black box for a lot of cases. One of the suggested ways to do so was to improve our tracing. Currently we have events for kmem, vmscan and oom (which really just traces the tunable updates) -- In addition Dave Hansen also also been trying to add tracing for TLB range flushing, hopefully that can make it in some time soon. However, this lacks the more general data that governs all of the core VM, such as vmas and of course the mm_struct. To this end, I've started adding events to trace the vma lifecycle, including: creating, removing, splitting, merging, copying and adjusting. Currently it only prints out the start and end virtual addresses, such as: bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] Now, on a more general scenario, I basically would like to know, 1) is this actually useful... I'm hoping that, if in fact something like this gets merged, it won't just sit there. 2) What other general data would be useful for debugging purposes? I'm happy to collect feedback and send out something we can all benefit from. Thanks, Davidlohr -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f44.google.com (mail-pb0-f44.google.com [209.85.160.44]) by kanga.kvack.org (Postfix) with ESMTP id 8A0036B0038 for ; Thu, 3 Apr 2014 18:31:24 -0400 (EDT) Received: by mail-pb0-f44.google.com with SMTP id rp16so2506014pbb.17 for ; Thu, 03 Apr 2014 15:31:24 -0700 (PDT) Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTP id xt7si3677540pab.471.2014.04.03.15.31.23 for ; Thu, 03 Apr 2014 15:31:23 -0700 (PDT) Message-ID: <533DE12C.9030203@intel.com> Date: Thu, 03 Apr 2014 15:31:08 -0700 From: Dave Hansen MIME-Version: 1.0 Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton On 04/03/2014 02:44 PM, Davidlohr Bueso wrote: > Now, on a more general scenario, I basically would like to know, 1) is > this actually useful... I'm hoping that, if in fact something like this > gets merged, it won't just sit there. 2) What other general data would > be useful for debugging purposes? I'm happy to collect feedback and send > out something we can all benefit from. One thing that would be nice, specifically for the VM, would be to turn all of the things that touch the /proc/vmstat counters (count_vm_event(), etc...) in to tracepoints. I started on it once, but ran in to some header dependency hell and gave up before I got anything useful. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-f175.google.com (mail-yk0-f175.google.com [209.85.160.175]) by kanga.kvack.org (Postfix) with ESMTP id C6F426B0044 for ; Thu, 3 Apr 2014 19:25:00 -0400 (EDT) Received: by mail-yk0-f175.google.com with SMTP id 131so2238814ykp.6 for ; Thu, 03 Apr 2014 16:25:00 -0700 (PDT) Received: from aserp1040.oracle.com (aserp1040.oracle.com. [141.146.126.69]) by mx.google.com with ESMTPS id l5si7723296yhg.44.2014.04.03.16.25.00 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 03 Apr 2014 16:25:00 -0700 (PDT) Message-ID: <533DEDC5.5070500@oracle.com> Date: Thu, 03 Apr 2014 19:24:53 -0400 From: Sasha Levin MIME-Version: 1.0 Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Davidlohr Bueso , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dave Jones , Andrew Morton On 04/03/2014 05:44 PM, Davidlohr Bueso wrote: > Hi All, > > During LSFMM Dave Jones discussed the current situation around > testing/trinity in the mm. One of the conclusions was that basically we > lack tools to gather the necessary information to make debugging a less > painful process, making it pretty much a black box for a lot of cases. > > One of the suggested ways to do so was to improve our tracing. Currently > we have events for kmem, vmscan and oom (which really just traces the > tunable updates) -- In addition Dave Hansen also also been trying to add > tracing for TLB range flushing, hopefully that can make it in some time > soon. However, this lacks the more general data that governs all of the > core VM, such as vmas and of course the mm_struct. > > To this end, I've started adding events to trace the vma lifecycle, > including: creating, removing, splitting, merging, copying and > adjusting. Currently it only prints out the start and end virtual > addresses, such as: > > bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] > > Now, on a more general scenario, I basically would like to know, 1) is > this actually useful... I'm hoping that, if in fact something like this > gets merged, it won't just sit there. 2) What other general data would > be useful for debugging purposes? I'm happy to collect feedback and send > out something we can all benefit from. There's another thing we have to think about, which is the bottleneck of getting that debug info out. Turning on any sort of tracing/logging in mm/ would trigger huge amounts of data flowing out. Any attempt to store that data anywhere would result either in too much interference to the tests so that issues stop reproducing, or way too much data to even be able to get through the guest <-> host pipe. I was working on a similar idea, which is similar to what lockdep does now: when you get a lockdep spew you see a nice output which also shows call traces of relevant locks. What if, for example, we could make dump_page() also dump the traces of where each of it's flags was set or cleared? Thanks, Sasha -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f51.google.com (mail-oa0-f51.google.com [209.85.219.51]) by kanga.kvack.org (Postfix) with ESMTP id 5A5726B0055 for ; Thu, 3 Apr 2014 19:34:22 -0400 (EDT) Received: by mail-oa0-f51.google.com with SMTP id i4so2765029oah.24 for ; Thu, 03 Apr 2014 16:34:22 -0700 (PDT) Received: from g4t3427.houston.hp.com (g4t3427.houston.hp.com. [15.201.208.55]) by mx.google.com with ESMTPS id wu5si5532858oeb.121.2014.04.03.16.34.21 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 03 Apr 2014 16:34:21 -0700 (PDT) Message-ID: <1396568057.4661.38.camel@buesod1.americas.hpqcorp.net> Subject: Re: [RFC] mm,tracing: improve current situation From: Davidlohr Bueso Date: Thu, 03 Apr 2014 16:34:17 -0700 In-Reply-To: <533DE12C.9030203@intel.com> References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> <533DE12C.9030203@intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Dave Hansen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton On Thu, 2014-04-03 at 15:31 -0700, Dave Hansen wrote: > On 04/03/2014 02:44 PM, Davidlohr Bueso wrote: > > Now, on a more general scenario, I basically would like to know, 1) is > > this actually useful... I'm hoping that, if in fact something like this > > gets merged, it won't just sit there. 2) What other general data would > > be useful for debugging purposes? I'm happy to collect feedback and send > > out something we can all benefit from. > > One thing that would be nice, specifically for the VM, would be to turn > all of the things that touch the /proc/vmstat counters > (count_vm_event(), etc...) in to tracepoints. Hmm what would be the difference? what's the issue with /proc/vmstat? I guess for one, some of the stats depend on build rules (ie TLB flushing statistics). -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ee0-f47.google.com (mail-ee0-f47.google.com [74.125.83.47]) by kanga.kvack.org (Postfix) with ESMTP id C40C26B0035 for ; Tue, 8 Apr 2014 11:44:07 -0400 (EDT) Received: by mail-ee0-f47.google.com with SMTP id b15so837813eek.34 for ; Tue, 08 Apr 2014 08:44:05 -0700 (PDT) Received: from mx2.suse.de (cantor2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id u49si3300523eef.52.2014.04.08.08.44.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 08 Apr 2014 08:44:05 -0700 (PDT) Message-ID: <53441943.1080102@suse.cz> Date: Tue, 08 Apr 2014 17:44:03 +0200 From: Vlastimil Babka MIME-Version: 1.0 Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> <533DEDC5.5070500@oracle.com> In-Reply-To: <533DEDC5.5070500@oracle.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Sasha Levin , Davidlohr Bueso , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dave Jones , Andrew Morton On 04/04/2014 01:24 AM, Sasha Levin wrote: > On 04/03/2014 05:44 PM, Davidlohr Bueso wrote: >> Hi All, >> >> During LSFMM Dave Jones discussed the current situation around >> testing/trinity in the mm. One of the conclusions was that basically we >> lack tools to gather the necessary information to make debugging a less >> painful process, making it pretty much a black box for a lot of cases. >> >> One of the suggested ways to do so was to improve our tracing. Currently >> we have events for kmem, vmscan and oom (which really just traces the >> tunable updates) -- In addition Dave Hansen also also been trying to add >> tracing for TLB range flushing, hopefully that can make it in some time >> soon. However, this lacks the more general data that governs all of the >> core VM, such as vmas and of course the mm_struct. >> >> To this end, I've started adding events to trace the vma lifecycle, >> including: creating, removing, splitting, merging, copying and >> adjusting. Currently it only prints out the start and end virtual >> addresses, such as: >> >> bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] >> >> Now, on a more general scenario, I basically would like to know, 1) is >> this actually useful... I'm hoping that, if in fact something like this >> gets merged, it won't just sit there. 2) What other general data would >> be useful for debugging purposes? I'm happy to collect feedback and send >> out something we can all benefit from. I think that adding more tracepoints might be more useful for debugging performance-related problems (e.g. compaction) that don't manifest as panic, and that VM_BUG_ON is more suited for this kind of debugging. But I might be wrong. > There's another thing we have to think about, which is the bottleneck of > getting that debug info out. > > Turning on any sort of tracing/logging in mm/ would trigger huge amounts > of data flowing out. Any attempt to store that data anywhere would result > either in too much interference to the tests so that issues stop reproducing, > or way too much data to even be able to get through the guest <-> host pipe. > > I was working on a similar idea, which is similar to what lockdep does now: > when you get a lockdep spew you see a nice output which also shows call > traces of relevant locks. What if, for example, we could make dump_page() > also dump the traces of where each of it's flags was set or cleared? Hm doesn't the oops printing already print accumulated trace buffers? Wouldn't it be easier to post-process that instead of trying to do some smart "unwinding" during oops? Is it possible to enable tracing without actually consuming the data, just for this purpose? Vlastimil > Thanks, > Sasha > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753648AbaDCVob (ORCPT ); Thu, 3 Apr 2014 17:44:31 -0400 Received: from g2t2353.austin.hp.com ([15.217.128.52]:3503 "EHLO g2t2353.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752711AbaDCVoC (ORCPT ); Thu, 3 Apr 2014 17:44:02 -0400 Message-ID: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> Subject: [RFC] mm,tracing: improve current situation From: Davidlohr Bueso To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton Date: Thu, 03 Apr 2014 14:44:00 -0700 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi All, During LSFMM Dave Jones discussed the current situation around testing/trinity in the mm. One of the conclusions was that basically we lack tools to gather the necessary information to make debugging a less painful process, making it pretty much a black box for a lot of cases. One of the suggested ways to do so was to improve our tracing. Currently we have events for kmem, vmscan and oom (which really just traces the tunable updates) -- In addition Dave Hansen also also been trying to add tracing for TLB range flushing, hopefully that can make it in some time soon. However, this lacks the more general data that governs all of the core VM, such as vmas and of course the mm_struct. To this end, I've started adding events to trace the vma lifecycle, including: creating, removing, splitting, merging, copying and adjusting. Currently it only prints out the start and end virtual addresses, such as: bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] Now, on a more general scenario, I basically would like to know, 1) is this actually useful... I'm hoping that, if in fact something like this gets merged, it won't just sit there. 2) What other general data would be useful for debugging purposes? I'm happy to collect feedback and send out something we can all benefit from. Thanks, Davidlohr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753972AbaDCWbS (ORCPT ); Thu, 3 Apr 2014 18:31:18 -0400 Received: from mga02.intel.com ([134.134.136.20]:61515 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753330AbaDCWbR (ORCPT ); Thu, 3 Apr 2014 18:31:17 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.97,791,1389772800"; d="scan'208";a="514417090" Message-ID: <533DE12C.9030203@intel.com> Date: Thu, 03 Apr 2014 15:31:08 -0700 From: Dave Hansen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Davidlohr Bueso , linux-mm@kvack.org CC: linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/03/2014 02:44 PM, Davidlohr Bueso wrote: > Now, on a more general scenario, I basically would like to know, 1) is > this actually useful... I'm hoping that, if in fact something like this > gets merged, it won't just sit there. 2) What other general data would > be useful for debugging purposes? I'm happy to collect feedback and send > out something we can all benefit from. One thing that would be nice, specifically for the VM, would be to turn all of the things that touch the /proc/vmstat counters (count_vm_event(), etc...) in to tracepoints. I started on it once, but ran in to some header dependency hell and gave up before I got anything useful. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753941AbaDCXZH (ORCPT ); Thu, 3 Apr 2014 19:25:07 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:50557 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753483AbaDCXZC (ORCPT ); Thu, 3 Apr 2014 19:25:02 -0400 Message-ID: <533DEDC5.5070500@oracle.com> Date: Thu, 03 Apr 2014 19:24:53 -0400 From: Sasha Levin User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Davidlohr Bueso , linux-mm@kvack.org CC: linux-kernel@vger.kernel.org, Dave Jones , Andrew Morton Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> In-Reply-To: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/03/2014 05:44 PM, Davidlohr Bueso wrote: > Hi All, > > During LSFMM Dave Jones discussed the current situation around > testing/trinity in the mm. One of the conclusions was that basically we > lack tools to gather the necessary information to make debugging a less > painful process, making it pretty much a black box for a lot of cases. > > One of the suggested ways to do so was to improve our tracing. Currently > we have events for kmem, vmscan and oom (which really just traces the > tunable updates) -- In addition Dave Hansen also also been trying to add > tracing for TLB range flushing, hopefully that can make it in some time > soon. However, this lacks the more general data that governs all of the > core VM, such as vmas and of course the mm_struct. > > To this end, I've started adding events to trace the vma lifecycle, > including: creating, removing, splitting, merging, copying and > adjusting. Currently it only prints out the start and end virtual > addresses, such as: > > bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] > > Now, on a more general scenario, I basically would like to know, 1) is > this actually useful... I'm hoping that, if in fact something like this > gets merged, it won't just sit there. 2) What other general data would > be useful for debugging purposes? I'm happy to collect feedback and send > out something we can all benefit from. There's another thing we have to think about, which is the bottleneck of getting that debug info out. Turning on any sort of tracing/logging in mm/ would trigger huge amounts of data flowing out. Any attempt to store that data anywhere would result either in too much interference to the tests so that issues stop reproducing, or way too much data to even be able to get through the guest <-> host pipe. I was working on a similar idea, which is similar to what lockdep does now: when you get a lockdep spew you see a nice output which also shows call traces of relevant locks. What if, for example, we could make dump_page() also dump the traces of where each of it's flags was set or cleared? Thanks, Sasha From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754071AbaDCXeX (ORCPT ); Thu, 3 Apr 2014 19:34:23 -0400 Received: from g4t3427.houston.hp.com ([15.201.208.55]:52243 "EHLO g4t3427.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753886AbaDCXeV (ORCPT ); Thu, 3 Apr 2014 19:34:21 -0400 Message-ID: <1396568057.4661.38.camel@buesod1.americas.hpqcorp.net> Subject: Re: [RFC] mm,tracing: improve current situation From: Davidlohr Bueso To: Dave Hansen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dave Jones , Sasha Levin , Andrew Morton Date: Thu, 03 Apr 2014 16:34:17 -0700 In-Reply-To: <533DE12C.9030203@intel.com> References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> <533DE12C.9030203@intel.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 2014-04-03 at 15:31 -0700, Dave Hansen wrote: > On 04/03/2014 02:44 PM, Davidlohr Bueso wrote: > > Now, on a more general scenario, I basically would like to know, 1) is > > this actually useful... I'm hoping that, if in fact something like this > > gets merged, it won't just sit there. 2) What other general data would > > be useful for debugging purposes? I'm happy to collect feedback and send > > out something we can all benefit from. > > One thing that would be nice, specifically for the VM, would be to turn > all of the things that touch the /proc/vmstat counters > (count_vm_event(), etc...) in to tracepoints. Hmm what would be the difference? what's the issue with /proc/vmstat? I guess for one, some of the stats depend on build rules (ie TLB flushing statistics). From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757512AbaDHPoI (ORCPT ); Tue, 8 Apr 2014 11:44:08 -0400 Received: from cantor2.suse.de ([195.135.220.15]:36735 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756232AbaDHPoG (ORCPT ); Tue, 8 Apr 2014 11:44:06 -0400 Message-ID: <53441943.1080102@suse.cz> Date: Tue, 08 Apr 2014 17:44:03 +0200 From: Vlastimil Babka User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Sasha Levin , Davidlohr Bueso , linux-mm@kvack.org CC: linux-kernel@vger.kernel.org, Dave Jones , Andrew Morton Subject: Re: [RFC] mm,tracing: improve current situation References: <1396561440.4661.33.camel@buesod1.americas.hpqcorp.net> <533DEDC5.5070500@oracle.com> In-Reply-To: <533DEDC5.5070500@oracle.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/04/2014 01:24 AM, Sasha Levin wrote: > On 04/03/2014 05:44 PM, Davidlohr Bueso wrote: >> Hi All, >> >> During LSFMM Dave Jones discussed the current situation around >> testing/trinity in the mm. One of the conclusions was that basically we >> lack tools to gather the necessary information to make debugging a less >> painful process, making it pretty much a black box for a lot of cases. >> >> One of the suggested ways to do so was to improve our tracing. Currently >> we have events for kmem, vmscan and oom (which really just traces the >> tunable updates) -- In addition Dave Hansen also also been trying to add >> tracing for TLB range flushing, hopefully that can make it in some time >> soon. However, this lacks the more general data that governs all of the >> core VM, such as vmas and of course the mm_struct. >> >> To this end, I've started adding events to trace the vma lifecycle, >> including: creating, removing, splitting, merging, copying and >> adjusting. Currently it only prints out the start and end virtual >> addresses, such as: >> >> bash-3661 [000] .... 222.964847: split_vma: [8a8000-9a6000] => new: [9a6000-9b6000] >> >> Now, on a more general scenario, I basically would like to know, 1) is >> this actually useful... I'm hoping that, if in fact something like this >> gets merged, it won't just sit there. 2) What other general data would >> be useful for debugging purposes? I'm happy to collect feedback and send >> out something we can all benefit from. I think that adding more tracepoints might be more useful for debugging performance-related problems (e.g. compaction) that don't manifest as panic, and that VM_BUG_ON is more suited for this kind of debugging. But I might be wrong. > There's another thing we have to think about, which is the bottleneck of > getting that debug info out. > > Turning on any sort of tracing/logging in mm/ would trigger huge amounts > of data flowing out. Any attempt to store that data anywhere would result > either in too much interference to the tests so that issues stop reproducing, > or way too much data to even be able to get through the guest <-> host pipe. > > I was working on a similar idea, which is similar to what lockdep does now: > when you get a lockdep spew you see a nice output which also shows call > traces of relevant locks. What if, for example, we could make dump_page() > also dump the traces of where each of it's flags was set or cleared? Hm doesn't the oops printing already print accumulated trace buffers? Wouldn't it be easier to post-process that instead of trying to do some smart "unwinding" during oops? Is it possible to enable tracing without actually consuming the data, just for this purpose? Vlastimil > Thanks, > Sasha > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org >