From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0D1FC433DF for ; Wed, 19 Aug 2020 10:43:12 +0000 (UTC) Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 63DEA206DA for ; Wed, 19 Aug 2020 10:43:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kroah.com header.i=@kroah.com header.b="AQNYrbOw"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="TKWgKTRt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 63DEA206DA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kroah.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kernelnewbies-bounces@kernelnewbies.org Received: from localhost ([::1] helo=shelob.surriel.com) by shelob.surriel.com with esmtp (Exim 4.94) (envelope-from ) id 1k8LYE-0007rD-Ee; Wed, 19 Aug 2020 06:42:38 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]) by shelob.surriel.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1k8LYB-0007r7-MN for kernelnewbies@kernelnewbies.org; Wed, 19 Aug 2020 06:42:36 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id DA20F5C008F; Wed, 19 Aug 2020 06:42:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 19 Aug 2020 06:42:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kroah.com; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:content-transfer-encoding:in-reply-to; s=fm1; bh=G rhlYBHahFUnfdeextaZfRwniN2kk8VF9Pt6+h/0TuI=; b=AQNYrbOw0aMP2QXnQ MxiJwCO0NtYEIuSS7CTcvGrEQGtpaXYYB/V4qn1314029uZLK3DN8WxJtFYNtIHv vhiS4qVv2Hk9hVrudXu2FQ2lgkW8XrMwhvlBBOsai4BcuLyzycL+lqUhjygZni8K /E9yi3Xqbt3TV327d8VrHxaLskSMRCcaNfYuKtbkOSop8H0WMrBYhLbDIxf5aCmI dj6PgfujAxbYK3QSg85ylTxsAwPcpLDQk+sp9pC9eYQBETsuLXQ1kxS6WTBZAYx/ x6UJpD42oQ1coGOlYdhDcwv9soS47MwivXfLFqyiXYr5jUDCrdmO2h3K7Oy0OI80 kt9aA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=GrhlYBHahFUnfdeextaZfRwniN2kk8VF9Pt6+h/0T uI=; b=TKWgKTRt1bliMpQJSGf1O8FjBHHZBZ9H2JPyejT+pUzpdPpgEbpEwRxat k4GxdGGT8kauIcaEyOMAqG0QDbPB5GEyBN13iFNT98j/sRWuEoLfTaRkWflx3VaX 7ImPEZFL+5rJmDBdJPFNRwPBgy0L6ax1fV+s2KI8Y3Ln4/bmN6CxGJnRlX1vVBgA U+5qdPLxRF3Q5cA18BOS3bx2fEDRpMh/JxhkdxT/YqYU9vV7VxAVq7CTHg/azOFk Jjym2bwM/xukMynCiaKwZZ4unIWFJrvC7uVRqX7UuNSjR8+X0m1B+WQHy5ce8ChM Qg0JfkZjHx/N1iVm+eUwGvEEQYwsg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedruddtkedgvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtugfgjgesthekredttddtjeenucfhrhhomhepifhrvghg ucfmjfcuoehgrhgvgheskhhrohgrhhdrtghomheqnecuggftrfgrthhtvghrnhepueehke ehlefffeeiudetfeekjeffvdeuheejjeffheeludfgteekvdelkeduuddvnecukfhppeek fedrkeeirdekledruddtjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehgrhgvgheskhhrohgrhhdrtghomh X-ME-Proxy: Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) by mail.messagingengine.com (Postfix) with ESMTPA id 440153280059; Wed, 19 Aug 2020 06:42:31 -0400 (EDT) Date: Wed, 19 Aug 2020 12:42:54 +0200 From: Greg KH To: Muni Sekhar Subject: Re: Scheduler benchmarks Message-ID: <20200819104254.GA41946@kroah.com> References: <20200818143633.GA628293@kroah.com> <20200818171457.GA736234@kroah.com> <20200818173656.GA748290@kroah.com> <84362b8b-971f-fb89-115d-41d2457c24fd@sony.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, peter enderborg , kernelnewbies X-BeenThere: kernelnewbies@kernelnewbies.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Learn about the Linux kernel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: kernelnewbies-bounces@kernelnewbies.org T24gV2VkLCBBdWcgMTksIDIwMjAgYXQgMDM6NDY6MDZQTSArMDUzMCwgTXVuaSBTZWtoYXIgd3Jv dGU6Cj4gT24gVHVlLCBBdWcgMTgsIDIwMjAgYXQgMTE6NDUgUE0gcGV0ZXIgZW5kZXJib3JnCj4g PHBldGVyLmVuZGVyYm9yZ0Bzb255LmNvbT4gd3JvdGU6Cj4gPgo+ID4gT24gOC8xOC8yMCA3OjUz IFBNLCBNdW5pIFNla2hhciB3cm90ZToKPiA+ID4gT24gVHVlLCBBdWcgMTgsIDIwMjAgYXQgMTE6 MDYgUE0gR3JlZyBLSCA8Z3JlZ0Brcm9haC5jb20+IHdyb3RlOgo+ID4gPj4gT24gVHVlLCBBdWcg MTgsIDIwMjAgYXQgMTE6MDE6MzVQTSArMDUzMCwgTXVuaSBTZWtoYXIgd3JvdGU6Cj4gPiA+Pj4g T24gVHVlLCBBdWcgMTgsIDIwMjAgYXQgMTA6NDQgUE0gR3JlZyBLSCA8Z3JlZ0Brcm9haC5jb20+ IHdyb3RlOgo+ID4gPj4+PiBPbiBUdWUsIEF1ZyAxOCwgMjAyMCBhdCAxMDoyNDoxM1BNICswNTMw LCBNdW5pIFNla2hhciB3cm90ZToKPiA+ID4+Pj4+IE9uIFR1ZSwgQXVnIDE4LCAyMDIwIGF0IDg6 MDYgUE0gR3JlZyBLSCA8Z3JlZ0Brcm9haC5jb20+IHdyb3RlOgo+ID4gPj4+Pj4+IE9uIFR1ZSwg QXVnIDE4LCAyMDIwIGF0IDA4OjAwOjExUE0gKzA1MzAsIE11bmkgU2VraGFyIHdyb3RlOgo+ID4g Pj4+Pj4+PiBIaSBhbGwsCj4gPiA+Pj4+Pj4+Cj4gPiA+Pj4+Pj4+IEnigJl2ZSB0d28gaWRlbnRp Y2FsIExpbnV4IHN5c3RlbXMgd2l0aCBvbmx5IGtlcm5lbCBkaWZmZXJlbmNlcy4KPiA+ID4+Pj4+ PiBXaGF0IGFyZSB0aGUgZGlmZmVyZW5jZXMgaW4gdGhlIGtlcm5lbHM/Cj4gPiA+Pj4+IFlvdSBk aWRuJ3QgYW5zd2VyIHRoaXMgcXVlc3Rpb24sIGlzIHRoaXMgdGhlIHNhbWUga2VybmVsIHNvdXJj ZSBiZWluZwo+ID4gPj4+PiBjb21wYXJlZCBoZXJlPyAgU2FtZSB2ZXJzaW9uPyAgU2FtZSBjb21w aWxlcj8gIEV2ZXJ5dGhpbmcgaWRlbnRpY2FsPwo+ID4gPj4+IEJvdGggc3lzdGVtcyBhcmUgaGF2 aW5nIGV4YWN0bHkgdGhlIHNhbWUgaGFyZHdhcmUgY29uZmlndXJhdGlvbi4KPiA+ID4+PiBDb21w aWxlciBhbmQga2VybmVsIHZlcnNpb25zIGFyZSBkaWZmZXJlbnQuIE9uZSBzeXN0ZW0gaGFzIFVi dW50dQo+ID4gPj4+IDE2LjA0LjQgTFRTKDQuNC4wLTY2LWdlbmVyaWMga2VybmVsIHdpdGggZ2Nj IHZlcnNpb24gNS40LjApIGtlcm5lbCBhbmQKPiA+ID4+PiB0aGUgb3RoZXIgb25lIGhhcyBVYnVu dHUgMTguMDQuNCBMVFMoNC4xNS4wLTkxLWdlbmVyaWMga2VybmVsIHdpdGggZ2NjCj4gPiA+Pj4g dmVyc2lvbiA3LjUuMCkuCj4gPiA+PiBUaG9zZSBhcmUgX3ZlcnlfIGRpZmZlcmVudCBrZXJuZWwg dmVyc2lvbnMsIHdpdGggbWFueSB5ZWFycyBhbmQgdGVucyBvZgo+ID4gPj4gdGhvdXNhbmRzIG9m IGRpZmZlcmVudCBjaGFuZ2VzIGJldHdlZW4gdGhlbS4KPiA+ID4+Cj4gPiA+PiBIb3BlZnVsbHkg dGhlIG5ld2VyIGtlcm5lbCBpcyBmYXN0ZXIsIHNvIGp1c3Qgc3RpY2sgd2l0aCB0aGF0IDopCj4g PiA+IEJ1dCB1bmZvcnR1bmF0ZWx5IHRoZSBuZXdlciBrZXJuZWwgaXMgdmVyeSBzbG93LCB0aGF0 IGlzIHRoZSByZWFzb24KPiA+ID4gZm9yIHN0YXJ0aW5nIHRoaXMgaW52ZXN0aWdhdGlvbiA6KQo+ ID4gPiBBbnkgdHlwZSBvZiBoZWxwLCAgYW5kIGd1aWRlbGluZXMgdG8gZGl2ZSBkZWVwZXIgd2ls bCBiZSBoaWdobHkgYXBwcmVjaWF0ZWQuCj4gPgo+ID4gT24gdGhlIDQuNCBrZXJuZWwgeW91IGRv bnQgaGF2ZQo+ID4KPiA+ICtDT05GSUdfUkVUUE9MSU5FPXkKPiA+ICtDT05GSUdfSU5URUxfUkRU PXkKPiBUaGFua3MhIFRoYXQgaXMgaGVscGZ1bC4gWWVzLCBJIHNlZSA0LjQga2VybmVsIGRvbid0 IGhhdmUgdGhlIGFib3ZlCj4gdHdvIGNvbmZpZyBvcHRpb25zLgo+IFdoYXQgYW5hbHlzaXMgY2Fu IGJlIGRvbmUgdG8gbmFycm93IGRvd24gdGhlIHJvb3QgY2F1c2U/Cgoicm9vdCBjYXVzZSIgaXMg eW91IGFyZSBjb21wYXJpbmcgYSBrZXJuZWwgd2l0aG91dCBhbnkgc3BlY3RyZS9tZWx0ZG93bgpt aXRpZ2F0aW9ucywgd2l0aCBvbmUgdGhhdCBoYXMgaXQuCgpUaG9zZSBtaXRpZ2F0aW9ucyBhcmUg a25vd24gdG8gaGF2ZSBhIGh1Z2Ugc2xvd2Rvd24gb24gc29tZQpiZW5jaG1hcmtzL3dvcmtsb2Fk cywgYXMgeW91IGhhdmUgc2hvd24gaGVyZS4KCklmIHlvdSBkb24ndCBsaWtlIHRoaXMsIHlvdSBj YW4gZGlzYWJsZSB0aGVzZSBjaGFuZ2VzIG9uIG5ld2VyIGtlcm5lbHMsCmlmIHlvdSByZWFsbHkg cmVhbGx5IGtub3cgd2hhdCB5b3UgYXJlIGRvaW5nLCBidXQgSSBzdHJvbmdseSBkbyBub3QKcmVj b21tZW5kIGl0LgoKQWxzbyB5b3UgY2FuIGNvbXBsYWluIHRvIHlvdXIgaGFyZHdhcmUgdmVuZG9y cyBmb3Igc2VsbGluZyB5b3UgYnJva2VuCmhhcmR3YXJlIDopCgo+IEFueSBleGFtcGxlIG9mIHJl ZmVyZW5jZSBjb3VsZCBiZSBoZWxwZnVsIHRvIHVuZGVyc3RhbmQuCgpMb29rIHVwIFNwZWN0cmUg YW5kIE1lbHRkb3duIGZvciBtYW55IG1hbnkgZXhhbXBsZXMgb2Ygd2hhdCBoYXBwZW5lZCBhbmQK d2hhdCB3ZW50IHdyb25nIHdpdGggY2hpcCBkZXNpZ25zIGFuZCBob3cgd2UgaGFkIHRvIGZpeCB0 aGVzZSB0aGluZ3MgaW4KdGhlIGtlcm5lbCBhIGZldyB5ZWFycyBhZ28uCgpncmVnIGstaAoKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KS2VybmVsbmV3Ymll cyBtYWlsaW5nIGxpc3QKS2VybmVsbmV3Ymllc0BrZXJuZWxuZXdiaWVzLm9yZwpodHRwczovL2xp c3RzLmtlcm5lbG5ld2JpZXMub3JnL21haWxtYW4vbGlzdGluZm8va2VybmVsbmV3Ymllcwo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: Greg KH Subject: Re: Scheduler benchmarks Date: Wed, 19 Aug 2020 12:42:54 +0200 Message-ID: <20200819104254.GA41946@kroah.com> References: <20200818143633.GA628293@kroah.com> <20200818171457.GA736234@kroah.com> <20200818173656.GA748290@kroah.com> <84362b8b-971f-fb89-115d-41d2457c24fd@sony.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Muni Sekhar Cc: peter enderborg , kernelnewbies , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org List-Id: linux-perf-users.vger.kernel.org On Wed, Aug 19, 2020 at 03:46:06PM +0530, Muni Sekhar wrote: > On Tue, Aug 18, 2020 at 11:45 PM peter enderborg > wrote: > > > > On 8/18/20 7:53 PM, Muni Sekhar wrote: > > > On Tue, Aug 18, 2020 at 11:06 PM Greg KH wrote: > > >> On Tue, Aug 18, 2020 at 11:01:35PM +0530, Muni Sekhar wrote: > > >>> On Tue, Aug 18, 2020 at 10:44 PM Greg KH wrote: > > >>>> On Tue, Aug 18, 2020 at 10:24:13PM +0530, Muni Sekhar wrote: > > >>>>> On Tue, Aug 18, 2020 at 8:06 PM Greg KH wrote: > > >>>>>> On Tue, Aug 18, 2020 at 08:00:11PM +0530, Muni Sekhar wrote: > > >>>>>>> Hi all, > > >>>>>>> > > >>>>>>> I’ve two identical Linux systems with only kernel differences. > > >>>>>> What are the differences in the kernels? > > >>>> You didn't answer this question, is this the same kernel source being > > >>>> compared here? Same version? Same compiler? Everything identical? > > >>> Both systems are having exactly the same hardware configuration. > > >>> Compiler and kernel versions are different. One system has Ubuntu > > >>> 16.04.4 LTS(4.4.0-66-generic kernel with gcc version 5.4.0) kernel and > > >>> the other one has Ubuntu 18.04.4 LTS(4.15.0-91-generic kernel with gcc > > >>> version 7.5.0). > > >> Those are _very_ different kernel versions, with many years and tens of > > >> thousands of different changes between them. > > >> > > >> Hopefully the newer kernel is faster, so just stick with that :) > > > But unfortunately the newer kernel is very slow, that is the reason > > > for starting this investigation :) > > > Any type of help, and guidelines to dive deeper will be highly appreciated. > > > > On the 4.4 kernel you dont have > > > > +CONFIG_RETPOLINE=y > > +CONFIG_INTEL_RDT=y > Thanks! That is helpful. Yes, I see 4.4 kernel don't have the above > two config options. > What analysis can be done to narrow down the root cause? "root cause" is you are comparing a kernel without any spectre/meltdown mitigations, with one that has it. Those mitigations are known to have a huge slowdown on some benchmarks/workloads, as you have shown here. If you don't like this, you can disable these changes on newer kernels, if you really really know what you are doing, but I strongly do not recommend it. Also you can complain to your hardware vendors for selling you broken hardware :) > Any example of reference could be helpful to understand. Look up Spectre and Meltdown for many many examples of what happened and what went wrong with chip designs and how we had to fix these things in the kernel a few years ago. greg k-h