From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FSL_HELO_FAKE, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40313C43613 for ; Sun, 23 Jun 2019 05:28:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B1B5D20679 for ; Sun, 23 Jun 2019 05:28:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qB454TB1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726387AbfFWF0z (ORCPT ); Sun, 23 Jun 2019 01:26:55 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:45794 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725268AbfFWF0z (ORCPT ); Sun, 23 Jun 2019 01:26:55 -0400 Received: by mail-pg1-f195.google.com with SMTP id z19so2374033pgl.12; Sat, 22 Jun 2019 22:26:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=N8JnL+BRd9Cg+THTYIKpLrM8fsEULQaLnQoJGKhjpxE=; b=qB454TB1PMSa7V3u1jHaP9pS37vTWYdB360vJ4qU6OwXc8AMM/zoqQ4TjsmbcIAOo8 V6A0laaDuK89R0SSamGqigVXHffBdnLm+1YeCLEPtL6jxTpOemzSPtdH6ooIw9L3gFLx uk0+Dce1NwY68lLRQvarvlF6JCHDstbq5wLYyXybYwk4jzYwsFAyCHUBHz/VNCMlM81B 1gWWiv9vF6dnMvxLSbFjdyvZIRD+nbgPlSWu8AOzE0MJcruviEIiJri3qCjo8Rk9AHYt qf3G7YsKfzgD2falb/fl5F2BYqUP4qSxqIMC10wRwLq/F5p0h6/uF0n7AlJpE+YHGjJh +A4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=N8JnL+BRd9Cg+THTYIKpLrM8fsEULQaLnQoJGKhjpxE=; b=umYaQWD89w5SJ/3O93p6JoXdod0ORhLANZuXQl/eTGD8qkIbOyjmRvQt9TC/XIb0S4 et1P4jVg0ExamhLPu55FXr3uhAvwQnV6hj8BfRoNi7DhUjVrbrC15FDiu+gL+Dv3pHZ1 tRpaF7Y4zYeSlBoXfOXCbdvfLoUNjluX8GUyY7jCVLu9NebShkEW0vpYLYRPUy6SxPjm 0l1S5D3YuULHE0z2tX7s6+uj4exmTmaT+02nLlAbAybiET9eUFffy/5VfXJ70sfmsCRp jQODhbhWAEIfg+eJOEDwszZzld32f/Y5cSH5J3rSqiFtTqLxE7qRhZuwd79uWWiKboaN liGA== X-Gm-Message-State: APjAAAVrqVNQK02qL5pq+XbDCXy+ojqyDZzrEU9Wx3CqcALysyFVD3gc qfptFdXQDHBEs2fkwUGdMLk= X-Google-Smtp-Source: APXvYqwY4EVo5/YP1HAB7Z9z9YWIcjURphl747zcCBIff460prHrFQP3ugo/yePWl2Jw2858cGPQ3Q== X-Received: by 2002:a65:478a:: with SMTP id e10mr26040994pgs.245.1561267611836; Sat, 22 Jun 2019 22:26:51 -0700 (PDT) Received: from gmail.com (c-73-140-212-29.hsd1.wa.comcast.net. [73.140.212.29]) by smtp.gmail.com with ESMTPSA id s9sm6265873pjp.7.2019.06.22.22.26.49 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 22 Jun 2019 22:26:51 -0700 (PDT) Date: Sat, 22 Jun 2019 22:26:48 -0700 From: Andrei Vagin To: Thomas Gleixner Cc: Dmitry Safonov , linux-kernel@vger.kernel.org, Adrian Reber , Andrei Vagin , Andy Lutomirski , Arnd Bergmann , Christian Brauner , Cyrill Gorcunov , Dmitry Safonov <0x7f454c46@gmail.com>, "Eric W. Biederman" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Jeff Dike , Oleg Nesterov , Pavel Emelyanov , Shuah Khan , Vincenzo Frascino , containers@lists.linux-foundation.org, criu@openvz.org, linux-api@vger.kernel.org, x86@kernel.org Subject: Re: [PATCHv4 26/28] x86/vdso: Align VDSO functions by CPU L1 cache line Message-ID: <20190623052647.GA9838@gmail.com> References: <20190612192628.23797-1-dima@arista.com> <20190612192628.23797-27-dima@arista.com> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: User-Agent: Mutt/1.11.3 (2019-02-01) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 14, 2019 at 04:13:31PM +0200, Thomas Gleixner wrote: > On Wed, 12 Jun 2019, Dmitry Safonov wrote: >=20 > > From: Andrei Vagin > >=20 > > After performance testing VDSO patches a noticeable 20% regression was > > found on gettime_perf selftest with a cold cache. > > As it turns to be, before time namespaces introduction, VDSO functions > > were quite aligned to cache lines, but adding a new code to adjust > > timens offset inside namespace created a small shift and vdso functions > > become unaligned on cache lines. > >=20 > > Add align to vdso functions with gcc option to fix performance drop. > >=20 > > Coping the resulting numbers from cover letter: > >=20 > > Hot CPU cache (more gettime_perf.c cycles - the better): > > | before | CONFIG_TIME_NS=3Dn | host | inside timens > > --------|------------|------------------|-------------|------------- > > cycles | 139887013 | 139453003 | 139899785 | 128792458 > > diff (%)| 100 | 99.7 | 100 | 92 >=20 > Why is CONFIG_TIME_NS=3Dn behaving worse than current mainline and > worse than 'host' mode? We had to specify a precision of these numbers, it is more than this 0.3%, so at that time I decided that here is nothing to worry about. I did these measurments a few mounth ago for the second version of this series. I repeated measurments for this set of patches: | before | CONFIG_TIME_NS=3Dn | host | inside timens -------------------------------------------------------------- | 144645498 | 142916801 | 140364862 | 132378440 | 143440633 | 141545739 | 140540053 | 132714190 | 144876395 | 144650599 | 140026814 | 131843318 | 143984551 | 144595770 | 140359260 | 131683544 | 144875682 | 143799788 | 140692618 | 131300332 -------------------------------------------------------------- avg | 144364551 | 143501739 | 140396721 | 131983964 diff % | 100 | 99.4 | 97.2 | 91.4 ------------------------------------------------------------- stdev % | 0.4 | 0.9 | 0.1 | 0.4 >=20 > > Cold cache (lesser tsc per gettime_perf_cold.c cycle - the better): > > | before | CONFIG_TIME_NS=3Dn | host | inside timens > > --------|------------|------------------|-------------|------------- > > tsc | 6748 | 6718 | 6862 | 12682 > > diff (%)| 100 | 99.6 | 101.7 | 188 >=20 > Weird, now CONFIG_TIME_NS=3Dn is better than current mainline and 'host' = mode > drops. The precision of these numbers is much smaller than of the previous set. These numbers are for the second version of this series, so I decided to repeat measurements for this version. When I run the test, I found that there is some degradation in compare with v5.0. I bisected and found that the problem is in 2b539aefe9e4 ("mm/resource: Let walk_system_ram_range() search child resources"). At this point, I realized that my test isn't quite right. On each iteration, the test starts a new process, then do start=3Drdtsc();clock_gettime();end=3Drdtsc() and prints (end-start). The problem here is that when clock_gettime() is called the first time, vdso pages are not mapped into a process address space, so the test measures how fast vdso pages are mapped into the process address space. I modified this test, now it uses the clflush instruction to drop cpu caches. Here are the results: | before | CONFIG_TIME_NS=3Dn | host | inside timens -------------------------------------------------------------- tsc | 434 | 433 | 437 | 477 stdev(tsc) | 5 | 5 | 5 | 3 diff (%) | 1 | 1 | 100.1 | 109 Here is the source code for the modified test: https://github.com/avagin/linux-task-diag/blob/wip/timens-rfc-v4/tools/test= ing/selftests/timens/gettime_perf_cold.c This test does 10K iterations. At the first glance, the numbers look noisy, so I sort them and take only 8K numbers in the middle: $ ./gettime_perf_cold > raw $ cat raw | sort -n | tail -n 9000 | head -n 8000 > results >=20 > Either I'm misreading the numbers or missing something or I'm just confus= ed > as usual :) >=20 > Thanks, > = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = > tglx