From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753667AbcCRIhL (ORCPT ); Fri, 18 Mar 2016 04:37:11 -0400 Received: from www.linutronix.de ([62.245.132.108]:35662 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752363AbcCRIhD (ORCPT ); Fri, 18 Mar 2016 04:37:03 -0400 Date: Fri, 18 Mar 2016 09:35:21 +0100 (CET) From: Thomas Gleixner To: "Huang, Ying" cc: Ingo Molnar , Peter Zijlstra , Chris Mason , Peter Zijlstra , lkp@01.org, Sebastian Andrzej Siewior , Davidlohr Bueso , Hugh Dickins , LKML , Linus Torvalds , Mel Gorman , Mel Gorman , Darren Hart , Wu Fengguang Subject: Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops In-Reply-To: <87d1qsgvpw.fsf@yhuang-dev.intel.com> Message-ID: References: <871t7vzze1.fsf@yhuang-dev.intel.com> <20160229093708.GA21037@gmail.com> <87d1qsgvpw.fsf@yhuang-dev.intel.com> User-Agent: Alpine 2.11 (DEB 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=UTF-8 X-Linutronix-Spam-Score: -1.0 X-Linutronix-Spam-Level: - X-Linutronix-Spam-Status: No , -1.0 points, 5.0 required, ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 18 Mar 2016, Huang, Ying wrote: > Usually we will put most important change we think in the subject of the > mail, for this email, it is, > > +25.6% will-it-scale.per_process_ops That is confusing on it's own, because the reader does not know at all whether this is an improvement or a regression. So something like this might be useful: Subject: subsystem: 12digitsha1: 25% performance improvement or in some other case Subject: subsystem: 12digitsha1: 25% performance regression So in the latter case I will look into that mail immediately. The improvement one can wait until I have cared about urgent stuff. In the subject line it is pretty much irrelevant which foo-bla-ops test has produced that result. It really does not matter. If it's a regression, it's urgent. If it's an improvement it's informal and it can wait to be read. So in that case it would be: futex: 65d8fc777f6d: 25% performance improvement You can grab the subsystem prefix from the commit. > and, we try to put most important changes at the top of the comparison > result below. That is the will-it-scale.xxx below. > > We are thinking about how to improve this. You input is valuable for > us. We are thinking change the "below changes" line to something like > below. > > FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on > ... > > Does this looks better? A bit, but it still does not tell me much. It's completely non obvious what 'will-it-scale.per_process_ops' means. Let me give you an example how a useful and easy to understand summary of the change could look like: FYI, we noticed 25.6% performance improvement due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. So those few lines tell precisely what this is about. It's something I already expected, so I really can skip the rest of the mail unless I'm interested in reproducing the result. Now lets look at a performance regression. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_process_ops test. will-it-scale.per_process_ops tests the futex operations for process shared futexes (Or whatever that test really does). The commit has no significant impact on any other test in the test suite. In that case I will certainly be interested how to reproduce that test. So I need the following information: Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM Config file: http://wherever.you.store/your/results/test-nr/config Test: http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2 That tarball should contain: README test_script.sh test_binary README should tell: will-it-scale.per_process_ops Short explanation of the test Preliminaries: - perf - whatever So that allows me to reproduce that test more or less with no effort. And that's the really important part. You can provide nice charts and full comparison tables for all tests on a web site for those who are interested in large stats and pretty charts. Full results: http://wherever.you.store/your/results/test-nr/results So now lets look at a scenario where that commit results in a performance improvement on will-it-scale.per_process_ops and a regression on will-it-scale.per_task. Subject: futex: 65d8fc777f6d: 25% performance regression FYI, we noticed a 25.2% performance regression due to commit 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()" in the will-it-scale.per_task test. will-it-scale.per_tasks tests the futex operations for process private futexes (Or whatever that test really does). The commit has significant impact on the following tests in the test suite: will-it-scale.per_process_ops: 25% improvement The 25% improvement is really not interesting. What's interesting is the regression. That's what people need to look at. You still can provide the information about all the tests including the 25% improvement data on Full results: http://wherever.you.store/your/results/test-nr/results Now, if you have commits which have mixed impact on various tests, then it's important to point out the most relevant issue clearly in the summary. I.e. you need to filter the tests for the maximum regression/improvement and use that as the main information. You still can provide the information about the impact on other tests in a very condensed form. The commit has significant impact on the following tests in the test suite: test1: 0.5% Regression test2: 2.0% Improvement .... testN: 0.2% Regression That's useful information, but it might be completely irrelevant if the primary impact is, e.g. a 10% Regression. Then it's nice to know, that it gave a 2% improvement on test2, but that's about it. The important information is the 10% regression, which needs to be addressed urgently. Hope that helps. Thanks, tglx