From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Gleixner Subject: [patch V2 21/29] tracing: Use percpu stack trace buffer more intelligently Date: Thu, 18 Apr 2019 10:41:40 +0200 Message-ID: <20190418084254.999521114@linutronix.de> References: <20190418084119.056416939@linutronix.de> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: LKML Cc: Mike Snitzer , David Airlie , Catalin Marinas , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, dm-devel@redhat.com, Alexander Potapenko , Christoph Lameter , Christoph Hellwig , Alasdair Kergon , Marek Szyprowski , linux-arch@vger.kernel.org, x86@kernel.org, kasan-dev@googlegroups.com, Johannes Thumshirn , Andrey Ryabinin , Alexey Dobriyan , intel-gfx@lists.freedesktop.org, David Rientjes , Akinobu Mita , Steven Rostedt , Josef Bacik , Mike Rapoport , Andy Lutomirski , Josh Poimboeuf , David Sterba List-Id: iommu@lists.linux-foundation.org VGhlIHBlciBjcHUgc3RhY2sgdHJhY2UgYnVmZmVyIHVzYWdlIHBhdHRlcm4gaXMgb2RkIGF0IGJl c3QuIFRoZSBidWZmZXIgaGFzCnBsYWNlIGZvciA1MTIgc3RhY2sgdHJhY2UgZW50cmllcyBvbiA2 NC1iaXQgYW5kIDEwMjQgb24gMzItYml0LiBXaGVuCmludGVycnVwdHMgb3IgZXhjZXB0aW9ucyBu ZXN0IGFmdGVyIHRoZSBwZXIgY3B1IGJ1ZmZlciB3YXMgYWNxdWlyZWQgdGhlCnN0YWNrdHJhY2Ug bGVuZ3RoIGlzIGhhcmRjb2RlZCB0byA4IGVudHJpZXMuIDUxMi8xMDI0IHN0YWNrIHRyYWNlIGVu dHJpZXMKaW4ga2VybmVsIHN0YWNrcyBhcmUgdW5yZWFsaXN0aWMgc28gdGhlIGJ1ZmZlciBpcyBh IGNvbXBsZXRlIHdhc3RlLgoKU3BsaXQgdGhlIGJ1ZmZlciBpbnRvIGNodW5rcyBvZiA2NCBzdGFj ayBlbnRyaWVzIHdoaWNoIGlzIHBsZW50eS4gVGhpcwphbGxvd3MgbmVzdGluZyBjb250ZXh0cyAo aW50ZXJydXB0cywgZXhjZXB0aW9ucykgdG8gdXRpbGl6ZSB0aGUgY3B1IGJ1ZmZlcgpmb3Igc3Rh Y2sgcmV0cmlldmFsIGFuZCBhdm9pZHMgdGhlIGZpeGVkIGxlbmd0aCBhbGxvY2F0aW9uIGFsb25n IHdpdGggdGhlCmNvbmRpdGlvbmFsIGV4ZWN1dGlvbiBwYXRoZXMuCgpTaWduZWQtb2ZmLWJ5OiBU aG9tYXMgR2xlaXhuZXIgPHRnbHhAbGludXRyb25peC5kZT4KQ2M6IFN0ZXZlbiBSb3N0ZWR0IDxy b3N0ZWR0QGdvb2RtaXMub3JnPgotLS0KIGtlcm5lbC90cmFjZS90cmFjZS5jIHwgICA3NyArKysr KysrKysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIDEgZmlsZSBj aGFuZ2VkLCAzOSBpbnNlcnRpb25zKCspLCAzOCBkZWxldGlvbnMoLSkKCi0tLSBhL2tlcm5lbC90 cmFjZS90cmFjZS5jCisrKyBiL2tlcm5lbC90cmFjZS90cmFjZS5jCkBAIC0yNzQ5LDEyICsyNzQ5 LDIxIEBAIHRyYWNlX2Z1bmN0aW9uKHN0cnVjdCB0cmFjZV9hcnJheSAqdHIsCiAKICNpZmRlZiBD T05GSUdfU1RBQ0tUUkFDRQogCi0jZGVmaW5lIEZUUkFDRV9TVEFDS19NQVhfRU5UUklFUyAoUEFH RV9TSVpFIC8gc2l6ZW9mKHVuc2lnbmVkIGxvbmcpKQorLyogNjQgZW50cmllcyBmb3Iga2VybmVs IHN0YWNrcyBhcmUgcGxlbnR5ICovCisjZGVmaW5lIEZUUkFDRV9LU1RBQ0tfRU5UUklFUwk2NAor CiBzdHJ1Y3QgZnRyYWNlX3N0YWNrIHsKLQl1bnNpZ25lZCBsb25nCQljYWxsc1tGVFJBQ0VfU1RB Q0tfTUFYX0VOVFJJRVNdOworCXVuc2lnbmVkIGxvbmcJCWNhbGxzW0ZUUkFDRV9LU1RBQ0tfRU5U UklFU107CiB9OwogCi1zdGF0aWMgREVGSU5FX1BFUl9DUFUoc3RydWN0IGZ0cmFjZV9zdGFjaywg ZnRyYWNlX3N0YWNrKTsKKy8qIFRoaXMgYWxsb3dzIDggbGV2ZWwgbmVzdGluZyB3aGljaCBpcyBw bGVudHkgKi8KKyNkZWZpbmUgRlRSQUNFX0tTVEFDS19ORVNUSU5HCShQQUdFX1NJWkUgLyBzaXpl b2Yoc3RydWN0IGZ0cmFjZV9zdGFjaykpCisKK3N0cnVjdCBmdHJhY2Vfc3RhY2tzIHsKKwlzdHJ1 Y3QgZnRyYWNlX3N0YWNrCXN0YWNrc1tGVFJBQ0VfS1NUQUNLX05FU1RJTkddOworfTsKKworc3Rh dGljIERFRklORV9QRVJfQ1BVKHN0cnVjdCBmdHJhY2Vfc3RhY2tzLCBmdHJhY2Vfc3RhY2tzKTsK IHN0YXRpYyBERUZJTkVfUEVSX0NQVShpbnQsIGZ0cmFjZV9zdGFja19yZXNlcnZlKTsKIAogc3Rh dGljIHZvaWQgX19mdHJhY2VfdHJhY2Vfc3RhY2soc3RydWN0IHJpbmdfYnVmZmVyICpidWZmZXIs CkBAIC0yNzYzLDEwICsyNzcyLDExIEBAIHN0YXRpYyB2b2lkIF9fZnRyYWNlX3RyYWNlX3N0YWNr KHN0cnVjdAogewogCXN0cnVjdCB0cmFjZV9ldmVudF9jYWxsICpjYWxsID0gJmV2ZW50X2tlcm5l bF9zdGFjazsKIAlzdHJ1Y3QgcmluZ19idWZmZXJfZXZlbnQgKmV2ZW50OworCXN0cnVjdCBmdHJh Y2Vfc3RhY2sgKmZzdGFjazsKIAlzdHJ1Y3Qgc3RhY2tfZW50cnkgKmVudHJ5OwogCXN0cnVjdCBz dGFja190cmFjZSB0cmFjZTsKLQlpbnQgdXNlX3N0YWNrOwotCWludCBzaXplID0gRlRSQUNFX1NU QUNLX0VOVFJJRVM7CisJaW50IHNpemUgPSBGVFJBQ0VfS1NUQUNLX0VOVFJJRVM7CisJaW50IHN0 YWNraWR4OwogCiAJdHJhY2UubnJfZW50cmllcwk9IDA7CiAJdHJhY2Uuc2tpcAkJPSBza2lwOwpA QCAtMjc4OCwyOSArMjc5OCwzMiBAQCBzdGF0aWMgdm9pZCBfX2Z0cmFjZV90cmFjZV9zdGFjayhz dHJ1Y3QKIAkgKi8KIAlwcmVlbXB0X2Rpc2FibGVfbm90cmFjZSgpOwogCi0JdXNlX3N0YWNrID0g X190aGlzX2NwdV9pbmNfcmV0dXJuKGZ0cmFjZV9zdGFja19yZXNlcnZlKTsKKwlzdGFja2lkeCA9 IF9fdGhpc19jcHVfaW5jX3JldHVybihmdHJhY2Vfc3RhY2tfcmVzZXJ2ZSk7CisKKwkvKiBUaGlz IHNob3VsZCBuZXZlciBoYXBwZW4uIElmIGl0IGRvZXMsIHllbGwgb25jZSBhbmQgc2tpcCAqLwor CWlmIChXQVJOX09OX09OQ0Uoc3RhY2tpZHggPj0gRlRSQUNFX0tTVEFDS19ORVNUSU5HKSkKKwkJ Z290byBvdXQ7CisKIAkvKgotCSAqIFdlIGRvbid0IG5lZWQgYW55IGF0b21pYyB2YXJpYWJsZXMs IGp1c3QgYSBiYXJyaWVyLgotCSAqIElmIGFuIGludGVycnVwdCBjb21lcyBpbiwgd2UgZG9uJ3Qg Y2FyZSwgYmVjYXVzZSBpdCB3b3VsZAotCSAqIGhhdmUgZXhpdGVkIGFuZCBwdXQgdGhlIGNvdW50 ZXIgYmFjayB0byB3aGF0IHdlIHdhbnQuCi0JICogV2UganVzdCBuZWVkIGEgYmFycmllciB0byBr ZWVwIGdjYyBmcm9tIG1vdmluZyB0aGluZ3MKLQkgKiBhcm91bmQuCisJICogVGhlIGFib3ZlIF9f dGhpc19jcHVfaW5jX3JldHVybigpIGlzICdhdG9taWMnIGNwdSBsb2NhbC4gQW4KKwkgKiBpbnRl cnJ1cHQgd2lsbCBlaXRoZXIgc2VlIHRoZSB2YWx1ZSBwcmUgaW5jcmVtZW50IG9yIHBvc3QKKwkg KiBpbmNyZW1lbnQuIElmIHRoZSBpbnRlcnJ1cHQgaGFwcGVucyBwcmUgaW5jcmVtZW50IGl0IHdp bGwgaGF2ZQorCSAqIHJlc3RvcmVkIHRoZSBjb3VudGVyIHdoZW4gaXQgcmV0dXJucy4gIFdlIGp1 c3QgbmVlZCBhIGJhcnJpZXIgdG8KKwkgKiBrZWVwIGdjYyBmcm9tIG1vdmluZyB0aGluZ3MgYXJv dW5kLgogCSAqLwogCWJhcnJpZXIoKTsKLQlpZiAodXNlX3N0YWNrID09IDEpIHsKLQkJdHJhY2Uu ZW50cmllcwkJPSB0aGlzX2NwdV9wdHIoZnRyYWNlX3N0YWNrLmNhbGxzKTsKLQkJdHJhY2UubWF4 X2VudHJpZXMJPSBGVFJBQ0VfU1RBQ0tfTUFYX0VOVFJJRVM7Ci0KLQkJaWYgKHJlZ3MpCi0JCQlz YXZlX3N0YWNrX3RyYWNlX3JlZ3MocmVncywgJnRyYWNlKTsKLQkJZWxzZQotCQkJc2F2ZV9zdGFj a190cmFjZSgmdHJhY2UpOwotCi0JCWlmICh0cmFjZS5ucl9lbnRyaWVzID4gc2l6ZSkKLQkJCXNp emUgPSB0cmFjZS5ucl9lbnRyaWVzOwotCX0gZWxzZQotCQkvKiBGcm9tIG5vdyBvbiwgdXNlX3N0 YWNrIGlzIGEgYm9vbGVhbiAqLwotCQl1c2Vfc3RhY2sgPSAwOworCisJZnN0YWNrID0gdGhpc19j cHVfcHRyKGZ0cmFjZV9zdGFja3Muc3RhY2tzKSArIChzdGFja2lkeCAtIDEpOworCXRyYWNlLmVu dHJpZXMJCT0gZnN0YWNrLT5jYWxsczsKKwl0cmFjZS5tYXhfZW50cmllcwk9IEZUUkFDRV9LU1RB Q0tfRU5UUklFUzsKKworCWlmIChyZWdzKQorCQlzYXZlX3N0YWNrX3RyYWNlX3JlZ3MocmVncywg JnRyYWNlKTsKKwllbHNlCisJCXNhdmVfc3RhY2tfdHJhY2UoJnRyYWNlKTsKKworCWlmICh0cmFj ZS5ucl9lbnRyaWVzID4gc2l6ZSkKKwkJc2l6ZSA9IHRyYWNlLm5yX2VudHJpZXM7CiAKIAlzaXpl ICo9IHNpemVvZih1bnNpZ25lZCBsb25nKTsKIApAQCAtMjgyMCwxOSArMjgzMyw3IEBAIHN0YXRp YyB2b2lkIF9fZnRyYWNlX3RyYWNlX3N0YWNrKHN0cnVjdAogCQlnb3RvIG91dDsKIAllbnRyeSA9 IHJpbmdfYnVmZmVyX2V2ZW50X2RhdGEoZXZlbnQpOwogCi0JbWVtc2V0KCZlbnRyeS0+Y2FsbGVy LCAwLCBzaXplKTsKLQotCWlmICh1c2Vfc3RhY2spCi0JCW1lbWNweSgmZW50cnktPmNhbGxlciwg dHJhY2UuZW50cmllcywKLQkJICAgICAgIHRyYWNlLm5yX2VudHJpZXMgKiBzaXplb2YodW5zaWdu ZWQgbG9uZykpOwotCWVsc2UgewotCQl0cmFjZS5tYXhfZW50cmllcwk9IEZUUkFDRV9TVEFDS19F TlRSSUVTOwotCQl0cmFjZS5lbnRyaWVzCQk9IGVudHJ5LT5jYWxsZXI7Ci0JCWlmIChyZWdzKQot CQkJc2F2ZV9zdGFja190cmFjZV9yZWdzKHJlZ3MsICZ0cmFjZSk7Ci0JCWVsc2UKLQkJCXNhdmVf c3RhY2tfdHJhY2UoJnRyYWNlKTsKLQl9CisJbWVtY3B5KCZlbnRyeS0+Y2FsbGVyLCB0cmFjZS5l bnRyaWVzLCBzaXplKTsKIAogCWVudHJ5LT5zaXplID0gdHJhY2UubnJfZW50cmllczsKIAoKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkludGVsLWdmeCBt YWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwczovL2xpc3Rz LmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ludGVsLWdmeA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2855C10F0E for ; Thu, 18 Apr 2019 09:34:50 +0000 (UTC) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBC65214DA for ; Thu, 18 Apr 2019 09:34:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBC65214DA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 6F65211B0; Thu, 18 Apr 2019 09:34:44 +0000 (UTC) Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id A547D11B0 for ; Thu, 18 Apr 2019 09:34:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from Galois.linutronix.de (Galois.linutronix.de [146.0.238.70]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 19EFB828 for ; Thu, 18 Apr 2019 09:34:42 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=nanos.tec.linutronix.de) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1hH30G-0001uJ-9j; Thu, 18 Apr 2019 11:06:44 +0200 Message-Id: <20190418084254.999521114@linutronix.de> User-Agent: quilt/0.65 Date: Thu, 18 Apr 2019 10:41:40 +0200 From: Thomas Gleixner To: LKML Subject: [patch V2 21/29] tracing: Use percpu stack trace buffer more intelligently References: <20190418084119.056416939@linutronix.de> MIME-Version: 1.0 Cc: Mike Snitzer , David Airlie , Catalin Marinas , Joonas Lahtinen , dri-devel@lists.freedesktop.org, linux-mm@kvack.org, dm-devel@redhat.com, Alexander Potapenko , Christoph Lameter , Christoph Hellwig , Alasdair Kergon , linux-arch@vger.kernel.org, x86@kernel.org, kasan-dev@googlegroups.com, Johannes Thumshirn , Andrey Ryabinin , Alexey Dobriyan , intel-gfx@lists.freedesktop.org, David Rientjes , Maarten Lankhorst , Akinobu Mita , Steven Rostedt , Josef Bacik , Rodrigo Vivi , Mike Rapoport , Jani Nikula , Andy Lutomirski , Josh Poimboeuf , David Sterba , Dmitry Vyukov , Chris Mason , Pekka Enberg , iommu@lists.linux-foundation.org, Daniel Vetter , Andrew Morton , Robin Murphy , linux-btrfs@vger.kernel.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Sender: iommu-bounces@lists.linux-foundation.org Errors-To: iommu-bounces@lists.linux-foundation.org Message-ID: <20190418084140.8EtFh4yMWWalTXOgDrmTXkF7OVoRpQXlJzRYyMLxM-M@z> The per cpu stack trace buffer usage pattern is odd at best. The buffer has place for 512 stack trace entries on 64-bit and 1024 on 32-bit. When interrupts or exceptions nest after the per cpu buffer was acquired the stacktrace length is hardcoded to 8 entries. 512/1024 stack trace entries in kernel stacks are unrealistic so the buffer is a complete waste. Split the buffer into chunks of 64 stack entries which is plenty. This allows nesting contexts (interrupts, exceptions) to utilize the cpu buffer for stack retrieval and avoids the fixed length allocation along with the conditional execution pathes. Signed-off-by: Thomas Gleixner Cc: Steven Rostedt --- kernel/trace/trace.c | 77 +++++++++++++++++++++++++-------------------------- 1 file changed, 39 insertions(+), 38 deletions(-) --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -2749,12 +2749,21 @@ trace_function(struct trace_array *tr, #ifdef CONFIG_STACKTRACE -#define FTRACE_STACK_MAX_ENTRIES (PAGE_SIZE / sizeof(unsigned long)) +/* 64 entries for kernel stacks are plenty */ +#define FTRACE_KSTACK_ENTRIES 64 + struct ftrace_stack { - unsigned long calls[FTRACE_STACK_MAX_ENTRIES]; + unsigned long calls[FTRACE_KSTACK_ENTRIES]; }; -static DEFINE_PER_CPU(struct ftrace_stack, ftrace_stack); +/* This allows 8 level nesting which is plenty */ +#define FTRACE_KSTACK_NESTING (PAGE_SIZE / sizeof(struct ftrace_stack)) + +struct ftrace_stacks { + struct ftrace_stack stacks[FTRACE_KSTACK_NESTING]; +}; + +static DEFINE_PER_CPU(struct ftrace_stacks, ftrace_stacks); static DEFINE_PER_CPU(int, ftrace_stack_reserve); static void __ftrace_trace_stack(struct ring_buffer *buffer, @@ -2763,10 +2772,11 @@ static void __ftrace_trace_stack(struct { struct trace_event_call *call = &event_kernel_stack; struct ring_buffer_event *event; + struct ftrace_stack *fstack; struct stack_entry *entry; struct stack_trace trace; - int use_stack; - int size = FTRACE_STACK_ENTRIES; + int size = FTRACE_KSTACK_ENTRIES; + int stackidx; trace.nr_entries = 0; trace.skip = skip; @@ -2788,29 +2798,32 @@ static void __ftrace_trace_stack(struct */ preempt_disable_notrace(); - use_stack = __this_cpu_inc_return(ftrace_stack_reserve); + stackidx = __this_cpu_inc_return(ftrace_stack_reserve); + + /* This should never happen. If it does, yell once and skip */ + if (WARN_ON_ONCE(stackidx >= FTRACE_KSTACK_NESTING)) + goto out; + /* - * We don't need any atomic variables, just a barrier. - * If an interrupt comes in, we don't care, because it would - * have exited and put the counter back to what we want. - * We just need a barrier to keep gcc from moving things - * around. + * The above __this_cpu_inc_return() is 'atomic' cpu local. An + * interrupt will either see the value pre increment or post + * increment. If the interrupt happens pre increment it will have + * restored the counter when it returns. We just need a barrier to + * keep gcc from moving things around. */ barrier(); - if (use_stack == 1) { - trace.entries = this_cpu_ptr(ftrace_stack.calls); - trace.max_entries = FTRACE_STACK_MAX_ENTRIES; - - if (regs) - save_stack_trace_regs(regs, &trace); - else - save_stack_trace(&trace); - - if (trace.nr_entries > size) - size = trace.nr_entries; - } else - /* From now on, use_stack is a boolean */ - use_stack = 0; + + fstack = this_cpu_ptr(ftrace_stacks.stacks) + (stackidx - 1); + trace.entries = fstack->calls; + trace.max_entries = FTRACE_KSTACK_ENTRIES; + + if (regs) + save_stack_trace_regs(regs, &trace); + else + save_stack_trace(&trace); + + if (trace.nr_entries > size) + size = trace.nr_entries; size *= sizeof(unsigned long); @@ -2820,19 +2833,7 @@ static void __ftrace_trace_stack(struct goto out; entry = ring_buffer_event_data(event); - memset(&entry->caller, 0, size); - - if (use_stack) - memcpy(&entry->caller, trace.entries, - trace.nr_entries * sizeof(unsigned long)); - else { - trace.max_entries = FTRACE_STACK_ENTRIES; - trace.entries = entry->caller; - if (regs) - save_stack_trace_regs(regs, &trace); - else - save_stack_trace(&trace); - } + memcpy(&entry->caller, trace.entries, size); entry->size = trace.nr_entries; _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu