From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx155.postini.com [74.125.245.155]) by kanga.kvack.org (Postfix) with SMTP id 2566D6B00A5 for ; Fri, 12 Oct 2012 10:51:18 -0400 (EDT) Date: Fri, 12 Oct 2012 15:51:14 +0100 From: Mel Gorman Subject: MMTests 0.06 Message-ID: <20121012145114.GZ29125@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: Linux-MM Cc: LKML MMTests 0.06 is a configurable test suite that runs a number of common workloads of interest to MM developers. There are multiple additions all but in many respects the most useful will be automatic package installation. The package names are based on openSUSE but it's easy to create mappings in bin/install-depends where the package names differ. The very basics of monitoring NUMA efficiency is there as well and the autonuma benchmark has a test. The stats it reports for NUMA need significant improvement but for the most part that should be straight forward. Changelog since v0.05 o Automatically install packages (need name mappings for other distros) o Add benchmark for autonumabench o Add support for benchmarking NAS with MPI o Add pgbench for autonumabench (may need a bit more work) o Upgrade postgres version to 9.2.1 o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains o Alter mailserver config to finish in a reasonable time o Add monitor for perf sched o Add moinitor that gathers ftrace information with trace-cmd o Add preliminary monitors for NUMA stats (very basic) o Specify ftrace events to monitor from config file o Remove the bulk of whats left of VMRegress o Convert shellpacks to a template format to auto-generate boilerplate code o Collect lock_stat information if enabled o Run multiple iterations of aim9 o Add basic regression tests for Cross Memory Attach o Copy with preempt being enabled in highalloc stres tests o Have largedd cope with a missing large file to work with o Add a monitor-only mode to just capture logs o Report receive-side throughput in netperf for results At LSF/MM at some point a request was made that a series of tests be identified that were of interest to MM developers and that could be used for testing the Linux memory management subsystem. There is renewed interest in some sort of general testing framework during discussions for Kernel Summit 2012 so here is what I use. http://www.csn.ul.ie/~mel/projects/mmtests/ http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz There are a number of stock configurations stored in configs/. For example config-global-dhp__pagealloc-performance runs a number of tests that may be able to identify performance regressions or gains in the page allocator. Similarly there network and scheduler configs. There are also more complex options. config-global-dhp__parallelio-memcachetest will run memcachetest in the foreground while doing IO of different sizes in the background to measure how much unrelated IO affects the throughput of an in-memory database. This release is also a little rough and the extraction scripts could have been tidier but they were mostly written in an airport and for the most part they work as advertised. I'll fix bugs as according as they are brought to my attention. The stats reporting still needs work because while some tests know how to make a better estimate of mean by filtering outliers it is not being handled consistently and the methodology needs work. I know filtering statistics like this is a major flaw in the methodology but the decision was made in this case in the interest of the benchmarks with unstable results completing in a reasonable time. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx130.postini.com [74.125.245.130]) by kanga.kvack.org (Postfix) with SMTP id 00FBE6B0062 for ; Wed, 24 Oct 2012 16:14:31 -0400 (EDT) Received: by mail-ie0-f169.google.com with SMTP id 10so1633713ied.14 for ; Wed, 24 Oct 2012 13:14:31 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20121012145114.GZ29125@suse.de> References: <20121012145114.GZ29125@suse.de> Date: Wed, 24 Oct 2012 17:14:31 -0300 Message-ID: Subject: Re: MMTests 0.06 From: Ezequiel Garcia Content-Type: multipart/mixed; boundary=f46d044787d302cf6d04ccd3bc51 Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linux-MM , LKML --f46d044787d302cf6d04ccd3bc51 Content-Type: text/plain; charset=ISO-8859-1 Hi, On Fri, Oct 12, 2012 at 11:51 AM, Mel Gorman wrote: > MMTests 0.06 is a configurable test suite that runs a number of common > workloads of interest to MM developers. There are multiple additions > all but in many respects the most useful will be automatic package > installation. The package names are based on openSUSE but it's easy to > create mappings in bin/install-depends where the package names differ. The > very basics of monitoring NUMA efficiency is there as well and the autonuma > benchmark has a test. The stats it reports for NUMA need significant > improvement but for the most part that should be straight forward. > > Changelog since v0.05 > o Automatically install packages (need name mappings for other distros) > o Add benchmark for autonumabench > o Add support for benchmarking NAS with MPI > o Add pgbench for autonumabench (may need a bit more work) > o Upgrade postgres version to 9.2.1 > o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains > o Alter mailserver config to finish in a reasonable time > o Add monitor for perf sched > o Add moinitor that gathers ftrace information with trace-cmd > o Add preliminary monitors for NUMA stats (very basic) > o Specify ftrace events to monitor from config file > o Remove the bulk of whats left of VMRegress > o Convert shellpacks to a template format to auto-generate boilerplate code > o Collect lock_stat information if enabled > o Run multiple iterations of aim9 > o Add basic regression tests for Cross Memory Attach > o Copy with preempt being enabled in highalloc stres tests > o Have largedd cope with a missing large file to work with > o Add a monitor-only mode to just capture logs > o Report receive-side throughput in netperf for results > > At LSF/MM at some point a request was made that a series of tests > be identified that were of interest to MM developers and that could be > used for testing the Linux memory management subsystem. There is renewed > interest in some sort of general testing framework during discussions for > Kernel Summit 2012 so here is what I use. > > http://www.csn.ul.ie/~mel/projects/mmtests/ > http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz > > There are a number of stock configurations stored in configs/. For example > config-global-dhp__pagealloc-performance runs a number of tests that > may be able to identify performance regressions or gains in the page > allocator. Similarly there network and scheduler configs. There are also > more complex options. config-global-dhp__parallelio-memcachetest will run > memcachetest in the foreground while doing IO of different sizes in the > background to measure how much unrelated IO affects the throughput of an > in-memory database. > > This release is also a little rough and the extraction scripts could > have been tidier but they were mostly written in an airport and for the > most part they work as advertised. I'll fix bugs as according as they are > brought to my attention. > > The stats reporting still needs work because while some tests know how > to make a better estimate of mean by filtering outliers it is not being > handled consistently and the methodology needs work. I know filtering > statistics like this is a major flaw in the methodology but the decision > was made in this case in the interest of the benchmarks with unstable > results completing in a reasonable time. > FWIW, I found a minor problem with sudo and yum incantation when trying this. I'm attaching a patch. Hope it helps, Ezequiel --f46d044787d302cf6d04ccd3bc51 Content-Type: application/octet-stream; name="yum-fix-install-option.patch" Content-Disposition: attachment; filename="yum-fix-install-option.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h8ovsfl60 ZGlmZiAtTmF1ciBtbXRlc3RzLTAuMDYtbW10ZXN0cy0wLjAxL2Jpbi9pbnN0YWxsLWRlcGVuZHMg bW10ZXN0cy0wLjA2LW1tdGVzdHMtMC4wMS1taW5lL2Jpbi9pbnN0YWxsLWRlcGVuZHMKLS0tIG1t dGVzdHMtMC4wNi1tbXRlc3RzLTAuMDEvYmluL2luc3RhbGwtZGVwZW5kcwkyMDEyLTEwLTEyIDEx OjI4OjQyLjAwMDAwMDAwMCAtMDMwMAorKysgbW10ZXN0cy0wLjA2LW1tdGVzdHMtMC4wMS1taW5l L2Jpbi9pbnN0YWxsLWRlcGVuZHMJMjAxMi0xMC0yNCAxNjo1OTo0MS4wMjk0MTcxOTMgLTAzMDAK QEAgLTExLDcgKzExLDcgQEAKIAlteSAkYmluYXJ5ID0gJF9bMF07CiAKIAlvcGVuKEJPREdFLCAi d2hpY2ggJGJpbmFyeSAyPi9kZXYvbnVsbCB8IikgfHwgZGllKCJGYWlsZWQgdG8gb3BlbiBwaXBl IHRvIHdoaWNoIik7Ci0JbXkgJGxpbmUgPSA8Qk9ER0U+OworCWNob21wKG15ICRsaW5lID0gPEJP REdFPik7CiAJY2xvc2UoQk9ER0UpOwogCXJldHVybiAkbGluZTsKIH0KQEAgLTI2LDcgKzI2LDcg QEAKIG15ICVwYWNrYWdlX2JpbiA9ICgKIAkic3VzZSIgICA9PiAienlwcGVyIGluc3RhbGwgLXki LAogCSJkZWJpYW4iID0+ICJhcHQtZ2V0IGluc3RhbGwgLXkgLS1mb3JjZS15ZXMiLAotCSJyZWRo YXQiID0+ICJ5dW0gaW5zdGFsbCAtaSIsCisJInJlZGhhdCIgPT4gInl1bSBpbnN0YWxsIC15IiwK ICk7CiAKIGlmICgkRU5WeyJBU1NVTUVfUEFDS0FHRV9JTlNUQUxMRUQifSBlcSAieWVzIikgewo= --f46d044787d302cf6d04ccd3bc51-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx204.postini.com [74.125.245.204]) by kanga.kvack.org (Postfix) with SMTP id B03CD6B0070 for ; Thu, 25 Oct 2012 06:27:58 -0400 (EDT) Date: Thu, 25 Oct 2012 11:20:28 +0100 From: Mel Gorman Subject: Re: MMTests 0.06 Message-ID: <20121025102028.GB2558@suse.de> References: <20121012145114.GZ29125@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Ezequiel Garcia Cc: Linux-MM , LKML On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote: > > The stats reporting still needs work because while some tests know how > > to make a better estimate of mean by filtering outliers it is not being > > handled consistently and the methodology needs work. I know filtering > > statistics like this is a major flaw in the methodology but the decision > > was made in this case in the interest of the benchmarks with unstable > > results completing in a reasonable time. > > > > FWIW, I found a minor problem with sudo and yum incantation when trying this. > > I'm attaching a patch. > Thanks very much. I've picked it up and it'll be in MMTests 0.07. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx136.postini.com [74.125.245.136]) by kanga.kvack.org (Postfix) with SMTP id 55CEC6B0062 for ; Thu, 25 Oct 2012 06:47:02 -0400 (EDT) Received: by mail-ia0-f169.google.com with SMTP id h37so1516630iak.14 for ; Thu, 25 Oct 2012 03:47:01 -0700 (PDT) Message-ID: <5089189D.4050401@gmail.com> Date: Thu, 25 Oct 2012 18:46:53 +0800 From: Ni zhan Chen MIME-Version: 1.0 Subject: Re: MMTests 0.06 References: <20121012145114.GZ29125@suse.de> <20121025102028.GB2558@suse.de> In-Reply-To: <20121025102028.GB2558@suse.de> Content-Type: multipart/alternative; boundary="------------010706030301070401060909" Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Ezequiel Garcia , Linux-MM , LKML This is a multi-part message in MIME format. --------------010706030301070401060909 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit On 10/25/2012 06:20 PM, Mel Gorman wrote: > On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote: >>> The stats reporting still needs work because while some tests know how >>> to make a better estimate of mean by filtering outliers it is not being >>> handled consistently and the methodology needs work. I know filtering >>> statistics like this is a major flaw in the methodology but the decision >>> was made in this case in the interest of the benchmarks with unstable >>> results completing in a reasonable time. >>> >> FWIW, I found a minor problem with sudo and yum incantation when trying this. >> >> I'm attaching a patch. >> > Thanks very much. I've picked it up and it'll be in MMTests 0.07. Hi Gorman, Could MMTests 0.07 auto download related packages for different distributions? Regards, Chen > --------------010706030301070401060909 Content-Type: text/html; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit
On 10/25/2012 06:20 PM, Mel Gorman wrote:
On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote:

The stats reporting still needs work because while some tests know how
to make a better estimate of mean by filtering outliers it is not being
handled consistently and the methodology needs work. I know filtering
statistics like this is a major flaw in the methodology but the decision
was made in this case in the interest of the benchmarks with unstable
results completing in a reasonable time.


FWIW, I found a minor problem with sudo and yum incantation when trying this.

I'm attaching a patch.


Thanks very much. I've picked it up and it'll be in MMTests 0.07.

Hi Gorman,

Could MMTests 0.07 auto download related packages for different distributions?

Regards,
Chen

--------------010706030301070401060909-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx175.postini.com [74.125.245.175]) by kanga.kvack.org (Postfix) with SMTP id F158B6B0071 for ; Fri, 26 Oct 2012 03:48:53 -0400 (EDT) Received: by mail-ob0-f169.google.com with SMTP id va7so2955221obc.14 for ; Fri, 26 Oct 2012 00:48:53 -0700 (PDT) Message-ID: <508A4060.4060800@gmail.com> Date: Fri, 26 Oct 2012 15:48:48 +0800 From: Ni zhan Chen MIME-Version: 1.0 Subject: Re: MMTests 0.06 References: <20121012145114.GZ29125@suse.de> In-Reply-To: <20121012145114.GZ29125@suse.de> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linux-MM , LKML On 10/12/2012 10:51 PM, Mel Gorman wrote: > MMTests 0.06 is a configurable test suite that runs a number of common > workloads of interest to MM developers. There are multiple additions > all but in many respects the most useful will be automatic package > installation. The package names are based on openSUSE but it's easy to > create mappings in bin/install-depends where the package names differ. The > very basics of monitoring NUMA efficiency is there as well and the autonuma > benchmark has a test. The stats it reports for NUMA need significant > improvement but for the most part that should be straight forward. > > Changelog since v0.05 > o Automatically install packages (need name mappings for other distros) > o Add benchmark for autonumabench > o Add support for benchmarking NAS with MPI > o Add pgbench for autonumabench (may need a bit more work) > o Upgrade postgres version to 9.2.1 > o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains > o Alter mailserver config to finish in a reasonable time > o Add monitor for perf sched > o Add moinitor that gathers ftrace information with trace-cmd > o Add preliminary monitors for NUMA stats (very basic) > o Specify ftrace events to monitor from config file > o Remove the bulk of whats left of VMRegress > o Convert shellpacks to a template format to auto-generate boilerplate code > o Collect lock_stat information if enabled > o Run multiple iterations of aim9 > o Add basic regression tests for Cross Memory Attach > o Copy with preempt being enabled in highalloc stres tests > o Have largedd cope with a missing large file to work with > o Add a monitor-only mode to just capture logs > o Report receive-side throughput in netperf for results > > At LSF/MM at some point a request was made that a series of tests > be identified that were of interest to MM developers and that could be > used for testing the Linux memory management subsystem. There is renewed > interest in some sort of general testing framework during discussions for > Kernel Summit 2012 so here is what I use. > > http://www.csn.ul.ie/~mel/projects/mmtests/ > http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz > > There are a number of stock configurations stored in configs/. For example > config-global-dhp__pagealloc-performance runs a number of tests that > may be able to identify performance regressions or gains in the page > allocator. Similarly there network and scheduler configs. There are also > more complex options. config-global-dhp__parallelio-memcachetest will run > memcachetest in the foreground while doing IO of different sizes in the > background to measure how much unrelated IO affects the throughput of an > in-memory database. > > This release is also a little rough and the extraction scripts could > have been tidier but they were mostly written in an airport and for the > most part they work as advertised. I'll fix bugs as according as they are > brought to my attention. > > The stats reporting still needs work because while some tests know how > to make a better estimate of mean by filtering outliers it is not being > handled consistently and the methodology needs work. I know filtering > statistics like this is a major flaw in the methodology but the decision > was made in this case in the interest of the benchmarks with unstable > results completing in a reasonable time. Hi Gorman, Could MMTests 0.07 auto download related packages for different distributions? Regards, Chen -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx201.postini.com [74.125.245.201]) by kanga.kvack.org (Postfix) with SMTP id 375DE6B0073 for ; Fri, 26 Oct 2012 04:00:10 -0400 (EDT) Date: Fri, 26 Oct 2012 09:00:05 +0100 From: Mel Gorman Subject: Re: MMTests 0.06 Message-ID: <20121026080004.GA15591@suse.de> References: <20121012145114.GZ29125@suse.de> <508A4060.4060800@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <508A4060.4060800@gmail.com> Sender: owner-linux-mm@kvack.org List-ID: To: Ni zhan Chen Cc: Linux-MM , LKML On Fri, Oct 26, 2012 at 03:48:48PM +0800, Ni zhan Chen wrote: > On 10/12/2012 10:51 PM, Mel Gorman wrote: > >MMTests 0.06 is a configurable test suite that runs a number of common > >workloads of interest to MM developers. There are multiple additions > >all but in many respects the most useful will be automatic package > >installation. The package names are based on openSUSE but it's easy to > >create mappings in bin/install-depends where the package names differ. The > >very basics of monitoring NUMA efficiency is there as well and the autonuma > >benchmark has a test. The stats it reports for NUMA need significant > >improvement but for the most part that should be straight forward. > > > >Changelog since v0.05 > >o Automatically install packages (need name mappings for other distros) > >o Add benchmark for autonumabench > >o Add support for benchmarking NAS with MPI > >o Add pgbench for autonumabench (may need a bit more work) > >o Upgrade postgres version to 9.2.1 > >o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains > >o Alter mailserver config to finish in a reasonable time > >o Add monitor for perf sched > >o Add moinitor that gathers ftrace information with trace-cmd > >o Add preliminary monitors for NUMA stats (very basic) > >o Specify ftrace events to monitor from config file > >o Remove the bulk of whats left of VMRegress > >o Convert shellpacks to a template format to auto-generate boilerplate code > >o Collect lock_stat information if enabled > >o Run multiple iterations of aim9 > >o Add basic regression tests for Cross Memory Attach > >o Copy with preempt being enabled in highalloc stres tests > >o Have largedd cope with a missing large file to work with > >o Add a monitor-only mode to just capture logs > >o Report receive-side throughput in netperf for results > > > >At LSF/MM at some point a request was made that a series of tests > >be identified that were of interest to MM developers and that could be > >used for testing the Linux memory management subsystem. There is renewed > >interest in some sort of general testing framework during discussions for > >Kernel Summit 2012 so here is what I use. > > > >http://www.csn.ul.ie/~mel/projects/mmtests/ > >http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz > > > >There are a number of stock configurations stored in configs/. For example > >config-global-dhp__pagealloc-performance runs a number of tests that > >may be able to identify performance regressions or gains in the page > >allocator. Similarly there network and scheduler configs. There are also > >more complex options. config-global-dhp__parallelio-memcachetest will run > >memcachetest in the foreground while doing IO of different sizes in the > >background to measure how much unrelated IO affects the throughput of an > >in-memory database. > > > >This release is also a little rough and the extraction scripts could > >have been tidier but they were mostly written in an airport and for the > >most part they work as advertised. I'll fix bugs as according as they are > >brought to my attention. > > > >The stats reporting still needs work because while some tests know how > >to make a better estimate of mean by filtering outliers it is not being > >handled consistently and the methodology needs work. I know filtering > >statistics like this is a major flaw in the methodology but the decision > >was made in this case in the interest of the benchmarks with unstable > >results completing in a reasonable time. > > Hi Gorman, > > Could MMTests 0.07 auto download related packages for different > distributions? > Sure. Look at bin/install-depends and you'll see near the top of the file this declaration. my %package_map = ( "debian::zlib-devel" => "zlib1g-dev", "debian::gcc-fortran" => "gfortran", "debian::gcc-c++" => "g++", "debian::diffutils" => "diff", ); This says for example that the zlib-devel package in openSUSE is called zlib1g-dev in Debian. Send me the mappings of the different packages for your distribution or edit this file yourself, send me the patch and I'll add them. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933052Ab2JLOvS (ORCPT ); Fri, 12 Oct 2012 10:51:18 -0400 Received: from cantor2.suse.de ([195.135.220.15]:48679 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759051Ab2JLOvR (ORCPT ); Fri, 12 Oct 2012 10:51:17 -0400 Date: Fri, 12 Oct 2012 15:51:14 +0100 From: Mel Gorman To: Linux-MM Cc: LKML Subject: MMTests 0.06 Message-ID: <20121012145114.GZ29125@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org MMTests 0.06 is a configurable test suite that runs a number of common workloads of interest to MM developers. There are multiple additions all but in many respects the most useful will be automatic package installation. The package names are based on openSUSE but it's easy to create mappings in bin/install-depends where the package names differ. The very basics of monitoring NUMA efficiency is there as well and the autonuma benchmark has a test. The stats it reports for NUMA need significant improvement but for the most part that should be straight forward. Changelog since v0.05 o Automatically install packages (need name mappings for other distros) o Add benchmark for autonumabench o Add support for benchmarking NAS with MPI o Add pgbench for autonumabench (may need a bit more work) o Upgrade postgres version to 9.2.1 o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains o Alter mailserver config to finish in a reasonable time o Add monitor for perf sched o Add moinitor that gathers ftrace information with trace-cmd o Add preliminary monitors for NUMA stats (very basic) o Specify ftrace events to monitor from config file o Remove the bulk of whats left of VMRegress o Convert shellpacks to a template format to auto-generate boilerplate code o Collect lock_stat information if enabled o Run multiple iterations of aim9 o Add basic regression tests for Cross Memory Attach o Copy with preempt being enabled in highalloc stres tests o Have largedd cope with a missing large file to work with o Add a monitor-only mode to just capture logs o Report receive-side throughput in netperf for results At LSF/MM at some point a request was made that a series of tests be identified that were of interest to MM developers and that could be used for testing the Linux memory management subsystem. There is renewed interest in some sort of general testing framework during discussions for Kernel Summit 2012 so here is what I use. http://www.csn.ul.ie/~mel/projects/mmtests/ http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz There are a number of stock configurations stored in configs/. For example config-global-dhp__pagealloc-performance runs a number of tests that may be able to identify performance regressions or gains in the page allocator. Similarly there network and scheduler configs. There are also more complex options. config-global-dhp__parallelio-memcachetest will run memcachetest in the foreground while doing IO of different sizes in the background to measure how much unrelated IO affects the throughput of an in-memory database. This release is also a little rough and the extraction scripts could have been tidier but they were mostly written in an airport and for the most part they work as advertised. I'll fix bugs as according as they are brought to my attention. The stats reporting still needs work because while some tests know how to make a better estimate of mean by filtering outliers it is not being handled consistently and the methodology needs work. I know filtering statistics like this is a major flaw in the methodology but the decision was made in this case in the interest of the benchmarks with unstable results completing in a reasonable time. -- Mel Gorman SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759364Ab2JYK2A (ORCPT ); Thu, 25 Oct 2012 06:28:00 -0400 Received: from cantor2.suse.de ([195.135.220.15]:36436 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759295Ab2JYK16 (ORCPT ); Thu, 25 Oct 2012 06:27:58 -0400 Date: Thu, 25 Oct 2012 11:20:28 +0100 From: Mel Gorman To: Ezequiel Garcia Cc: Linux-MM , LKML Subject: Re: MMTests 0.06 Message-ID: <20121025102028.GB2558@suse.de> References: <20121012145114.GZ29125@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote: > > The stats reporting still needs work because while some tests know how > > to make a better estimate of mean by filtering outliers it is not being > > handled consistently and the methodology needs work. I know filtering > > statistics like this is a major flaw in the methodology but the decision > > was made in this case in the interest of the benchmarks with unstable > > results completing in a reasonable time. > > > > FWIW, I found a minor problem with sudo and yum incantation when trying this. > > I'm attaching a patch. > Thanks very much. I've picked it up and it'll be in MMTests 0.07. -- Mel Gorman SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757006Ab2JZHsz (ORCPT ); Fri, 26 Oct 2012 03:48:55 -0400 Received: from mail-oa0-f46.google.com ([209.85.219.46]:46778 "EHLO mail-oa0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756882Ab2JZHsx (ORCPT ); Fri, 26 Oct 2012 03:48:53 -0400 Message-ID: <508A4060.4060800@gmail.com> Date: Fri, 26 Oct 2012 15:48:48 +0800 From: Ni zhan Chen User-Agent: Mozilla/5.0 (X11; Linux i686; rv:16.0) Gecko/20121011 Thunderbird/16.0.1 MIME-Version: 1.0 To: Mel Gorman CC: Linux-MM , LKML Subject: Re: MMTests 0.06 References: <20121012145114.GZ29125@suse.de> In-Reply-To: <20121012145114.GZ29125@suse.de> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/12/2012 10:51 PM, Mel Gorman wrote: > MMTests 0.06 is a configurable test suite that runs a number of common > workloads of interest to MM developers. There are multiple additions > all but in many respects the most useful will be automatic package > installation. The package names are based on openSUSE but it's easy to > create mappings in bin/install-depends where the package names differ. The > very basics of monitoring NUMA efficiency is there as well and the autonuma > benchmark has a test. The stats it reports for NUMA need significant > improvement but for the most part that should be straight forward. > > Changelog since v0.05 > o Automatically install packages (need name mappings for other distros) > o Add benchmark for autonumabench > o Add support for benchmarking NAS with MPI > o Add pgbench for autonumabench (may need a bit more work) > o Upgrade postgres version to 9.2.1 > o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains > o Alter mailserver config to finish in a reasonable time > o Add monitor for perf sched > o Add moinitor that gathers ftrace information with trace-cmd > o Add preliminary monitors for NUMA stats (very basic) > o Specify ftrace events to monitor from config file > o Remove the bulk of whats left of VMRegress > o Convert shellpacks to a template format to auto-generate boilerplate code > o Collect lock_stat information if enabled > o Run multiple iterations of aim9 > o Add basic regression tests for Cross Memory Attach > o Copy with preempt being enabled in highalloc stres tests > o Have largedd cope with a missing large file to work with > o Add a monitor-only mode to just capture logs > o Report receive-side throughput in netperf for results > > At LSF/MM at some point a request was made that a series of tests > be identified that were of interest to MM developers and that could be > used for testing the Linux memory management subsystem. There is renewed > interest in some sort of general testing framework during discussions for > Kernel Summit 2012 so here is what I use. > > http://www.csn.ul.ie/~mel/projects/mmtests/ > http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz > > There are a number of stock configurations stored in configs/. For example > config-global-dhp__pagealloc-performance runs a number of tests that > may be able to identify performance regressions or gains in the page > allocator. Similarly there network and scheduler configs. There are also > more complex options. config-global-dhp__parallelio-memcachetest will run > memcachetest in the foreground while doing IO of different sizes in the > background to measure how much unrelated IO affects the throughput of an > in-memory database. > > This release is also a little rough and the extraction scripts could > have been tidier but they were mostly written in an airport and for the > most part they work as advertised. I'll fix bugs as according as they are > brought to my attention. > > The stats reporting still needs work because while some tests know how > to make a better estimate of mean by filtering outliers it is not being > handled consistently and the methodology needs work. I know filtering > statistics like this is a major flaw in the methodology but the decision > was made in this case in the interest of the benchmarks with unstable > results completing in a reasonable time. Hi Gorman, Could MMTests 0.07 auto download related packages for different distributions? Regards, Chen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757176Ab2JZIAL (ORCPT ); Fri, 26 Oct 2012 04:00:11 -0400 Received: from cantor2.suse.de ([195.135.220.15]:49175 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754104Ab2JZIAJ (ORCPT ); Fri, 26 Oct 2012 04:00:09 -0400 Date: Fri, 26 Oct 2012 09:00:05 +0100 From: Mel Gorman To: Ni zhan Chen Cc: Linux-MM , LKML Subject: Re: MMTests 0.06 Message-ID: <20121026080004.GA15591@suse.de> References: <20121012145114.GZ29125@suse.de> <508A4060.4060800@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <508A4060.4060800@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 26, 2012 at 03:48:48PM +0800, Ni zhan Chen wrote: > On 10/12/2012 10:51 PM, Mel Gorman wrote: > >MMTests 0.06 is a configurable test suite that runs a number of common > >workloads of interest to MM developers. There are multiple additions > >all but in many respects the most useful will be automatic package > >installation. The package names are based on openSUSE but it's easy to > >create mappings in bin/install-depends where the package names differ. The > >very basics of monitoring NUMA efficiency is there as well and the autonuma > >benchmark has a test. The stats it reports for NUMA need significant > >improvement but for the most part that should be straight forward. > > > >Changelog since v0.05 > >o Automatically install packages (need name mappings for other distros) > >o Add benchmark for autonumabench > >o Add support for benchmarking NAS with MPI > >o Add pgbench for autonumabench (may need a bit more work) > >o Upgrade postgres version to 9.2.1 > >o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains > >o Alter mailserver config to finish in a reasonable time > >o Add monitor for perf sched > >o Add moinitor that gathers ftrace information with trace-cmd > >o Add preliminary monitors for NUMA stats (very basic) > >o Specify ftrace events to monitor from config file > >o Remove the bulk of whats left of VMRegress > >o Convert shellpacks to a template format to auto-generate boilerplate code > >o Collect lock_stat information if enabled > >o Run multiple iterations of aim9 > >o Add basic regression tests for Cross Memory Attach > >o Copy with preempt being enabled in highalloc stres tests > >o Have largedd cope with a missing large file to work with > >o Add a monitor-only mode to just capture logs > >o Report receive-side throughput in netperf for results > > > >At LSF/MM at some point a request was made that a series of tests > >be identified that were of interest to MM developers and that could be > >used for testing the Linux memory management subsystem. There is renewed > >interest in some sort of general testing framework during discussions for > >Kernel Summit 2012 so here is what I use. > > > >http://www.csn.ul.ie/~mel/projects/mmtests/ > >http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz > > > >There are a number of stock configurations stored in configs/. For example > >config-global-dhp__pagealloc-performance runs a number of tests that > >may be able to identify performance regressions or gains in the page > >allocator. Similarly there network and scheduler configs. There are also > >more complex options. config-global-dhp__parallelio-memcachetest will run > >memcachetest in the foreground while doing IO of different sizes in the > >background to measure how much unrelated IO affects the throughput of an > >in-memory database. > > > >This release is also a little rough and the extraction scripts could > >have been tidier but they were mostly written in an airport and for the > >most part they work as advertised. I'll fix bugs as according as they are > >brought to my attention. > > > >The stats reporting still needs work because while some tests know how > >to make a better estimate of mean by filtering outliers it is not being > >handled consistently and the methodology needs work. I know filtering > >statistics like this is a major flaw in the methodology but the decision > >was made in this case in the interest of the benchmarks with unstable > >results completing in a reasonable time. > > Hi Gorman, > > Could MMTests 0.07 auto download related packages for different > distributions? > Sure. Look at bin/install-depends and you'll see near the top of the file this declaration. my %package_map = ( "debian::zlib-devel" => "zlib1g-dev", "debian::gcc-fortran" => "gfortran", "debian::gcc-c++" => "g++", "debian::diffutils" => "diff", ); This says for example that the zlib-devel package in openSUSE is called zlib1g-dev in Debian. Send me the mappings of the different packages for your distribution or edit this file yourself, send me the patch and I'll add them. -- Mel Gorman SUSE Labs