MMTests 0.06

MMTests 0.06

[Mel Gorman]

MMTests 0.06 is a configurable test suite that runs a number of common
workloads of interest to MM developers. There are multiple additions
all but in many respects the most useful will be automatic package
installation. The package names are based on openSUSE but it's easy to
create mappings in bin/install-depends where the package names differ. The
very basics of monitoring NUMA efficiency is there as well and the autonuma
benchmark has a test. The stats it reports for NUMA need significant
improvement but for the most part that should be straight forward.

Changelog since v0.05
o Automatically install packages (need name mappings for other distros)
o Add benchmark for autonumabench
o Add support for benchmarking NAS with MPI
o Add pgbench for autonumabench (may need a bit more work)
o Upgrade postgres version to 9.2.1
o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains
o Alter mailserver config to finish in a reasonable time
o Add monitor for perf sched
o Add moinitor that gathers ftrace information with trace-cmd
o Add preliminary monitors for NUMA stats (very basic)
o Specify ftrace events to monitor from config file
o Remove the bulk of whats left of VMRegress
o Convert shellpacks to a template format to auto-generate boilerplate code
o Collect lock_stat information if enabled
o Run multiple iterations of aim9
o Add basic regression tests for Cross Memory Attach
o Copy with preempt being enabled in highalloc stres tests
o Have largedd cope with a missing large file to work with
o Add a monitor-only mode to just capture logs
o Report receive-side throughput in netperf for results

At LSF/MM at some point a request was made that a series of tests
be identified that were of interest to MM developers and that could be
used for testing the Linux memory management subsystem. There is renewed
interest in some sort of general testing framework during discussions for
Kernel Summit 2012 so here is what I use.

http://www.csn.ul.ie/~mel/projects/mmtests/
http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz

There are a number of stock configurations stored in configs/.  For example
config-global-dhp__pagealloc-performance runs a number of tests that
may be able to identify performance regressions or gains in the page
allocator. Similarly there network and scheduler configs. There are also
more complex options. config-global-dhp__parallelio-memcachetest will run
memcachetest in the foreground while doing IO of different sizes in the
background to measure how much unrelated IO affects the throughput of an
in-memory database.

This release is also a little rough and the extraction scripts could
have been tidier but they were mostly written in an airport and for the
most part they work as advertised. I'll fix bugs as according as they are
brought to my attention.

The stats reporting still needs work because while some tests know how
to make a better estimate of mean by filtering outliers it is not being
handled consistently and the methodology needs work. I know filtering
statistics like this is a major flaw in the methodology but the decision
was made in this case in the interest of the benchmarks with unstable
results completing in a reasonable time.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: MMTests 0.06

[Ezequiel Garcia]

[-- Attachment #1: Type: text/plain, Size: 3569 bytes --]

Hi,

On Fri, Oct 12, 2012 at 11:51 AM, Mel Gorman <mgorman@suse.de> wrote:
> MMTests 0.06 is a configurable test suite that runs a number of common
> workloads of interest to MM developers. There are multiple additions
> all but in many respects the most useful will be automatic package
> installation. The package names are based on openSUSE but it's easy to
> create mappings in bin/install-depends where the package names differ. The
> very basics of monitoring NUMA efficiency is there as well and the autonuma
> benchmark has a test. The stats it reports for NUMA need significant
> improvement but for the most part that should be straight forward.
>
> Changelog since v0.05
> o Automatically install packages (need name mappings for other distros)
> o Add benchmark for autonumabench
> o Add support for benchmarking NAS with MPI
> o Add pgbench for autonumabench (may need a bit more work)
> o Upgrade postgres version to 9.2.1
> o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains
> o Alter mailserver config to finish in a reasonable time
> o Add monitor for perf sched
> o Add moinitor that gathers ftrace information with trace-cmd
> o Add preliminary monitors for NUMA stats (very basic)
> o Specify ftrace events to monitor from config file
> o Remove the bulk of whats left of VMRegress
> o Convert shellpacks to a template format to auto-generate boilerplate code
> o Collect lock_stat information if enabled
> o Run multiple iterations of aim9
> o Add basic regression tests for Cross Memory Attach
> o Copy with preempt being enabled in highalloc stres tests
> o Have largedd cope with a missing large file to work with
> o Add a monitor-only mode to just capture logs
> o Report receive-side throughput in netperf for results
>
> At LSF/MM at some point a request was made that a series of tests
> be identified that were of interest to MM developers and that could be
> used for testing the Linux memory management subsystem. There is renewed
> interest in some sort of general testing framework during discussions for
> Kernel Summit 2012 so here is what I use.
>
> http://www.csn.ul.ie/~mel/projects/mmtests/
> http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz
>
> There are a number of stock configurations stored in configs/.  For example
> config-global-dhp__pagealloc-performance runs a number of tests that
> may be able to identify performance regressions or gains in the page
> allocator. Similarly there network and scheduler configs. There are also
> more complex options. config-global-dhp__parallelio-memcachetest will run
> memcachetest in the foreground while doing IO of different sizes in the
> background to measure how much unrelated IO affects the throughput of an
> in-memory database.
>
> This release is also a little rough and the extraction scripts could
> have been tidier but they were mostly written in an airport and for the
> most part they work as advertised. I'll fix bugs as according as they are
> brought to my attention.
>
> The stats reporting still needs work because while some tests know how
> to make a better estimate of mean by filtering outliers it is not being
> handled consistently and the methodology needs work. I know filtering
> statistics like this is a major flaw in the methodology but the decision
> was made in this case in the interest of the benchmarks with unstable
> results completing in a reasonable time.
>

FWIW, I found a minor problem with sudo and yum incantation when trying this.

I'm attaching a patch.

Hope it helps,

    Ezequiel

[-- Attachment #2: yum-fix-install-option.patch --]
[-- Type: application/octet-stream, Size: 740 bytes --]

diff -Naur mmtests-0.06-mmtests-0.01/bin/install-depends mmtests-0.06-mmtests-0.01-mine/bin/install-depends
--- mmtests-0.06-mmtests-0.01/bin/install-depends	2012-10-12 11:28:42.000000000 -0300
+++ mmtests-0.06-mmtests-0.01-mine/bin/install-depends	2012-10-24 16:59:41.029417193 -0300
@@ -11,7 +11,7 @@
 	my $binary = $_[0];
 
 	open(BODGE, "which $binary 2>/dev/null |") || die("Failed to open pipe to which");
-	my $line = <BODGE>;
+	chomp(my $line = <BODGE>);
 	close(BODGE);
 	return $line;
 }
@@ -26,7 +26,7 @@
 my %package_bin = (
 	"suse"   => "zypper install -y",
 	"debian" => "apt-get install -y --force-yes",
-	"redhat" => "yum install -i",
+	"redhat" => "yum install -y",
 );
 
 if ($ENV{"ASSUME_PACKAGE_INSTALLED"} eq "yes") {

Re: MMTests 0.06

[Mel Gorman]

On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote:
> > The stats reporting still needs work because while some tests know how
> > to make a better estimate of mean by filtering outliers it is not being
> > handled consistently and the methodology needs work. I know filtering
> > statistics like this is a major flaw in the methodology but the decision
> > was made in this case in the interest of the benchmarks with unstable
> > results completing in a reasonable time.
> >
> 
> FWIW, I found a minor problem with sudo and yum incantation when trying this.
> 
> I'm attaching a patch.
> 

Thanks very much. I've picked it up and it'll be in MMTests 0.07.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: MMTests 0.06

[Ni zhan Chen]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

On 10/25/2012 06:20 PM, Mel Gorman wrote:
> On Wed, Oct 24, 2012 at 05:14:31PM -0300, Ezequiel Garcia wrote:
>>> The stats reporting still needs work because while some tests know how
>>> to make a better estimate of mean by filtering outliers it is not being
>>> handled consistently and the methodology needs work. I know filtering
>>> statistics like this is a major flaw in the methodology but the decision
>>> was made in this case in the interest of the benchmarks with unstable
>>> results completing in a reasonable time.
>>>
>> FWIW, I found a minor problem with sudo and yum incantation when trying this.
>>
>> I'm attaching a patch.
>>
> Thanks very much. I've picked it up and it'll be in MMTests 0.07.

Hi Gorman,

Could MMTests 0.07 auto download related packages for different 
distributions?

Regards,
Chen

>


[-- Attachment #2: Type: text/html, Size: 2016 bytes --]

Re: MMTests 0.06

[Ni zhan Chen]

On 10/12/2012 10:51 PM, Mel Gorman wrote:
> MMTests 0.06 is a configurable test suite that runs a number of common
> workloads of interest to MM developers. There are multiple additions
> all but in many respects the most useful will be automatic package
> installation. The package names are based on openSUSE but it's easy to
> create mappings in bin/install-depends where the package names differ. The
> very basics of monitoring NUMA efficiency is there as well and the autonuma
> benchmark has a test. The stats it reports for NUMA need significant
> improvement but for the most part that should be straight forward.
>
> Changelog since v0.05
> o Automatically install packages (need name mappings for other distros)
> o Add benchmark for autonumabench
> o Add support for benchmarking NAS with MPI
> o Add pgbench for autonumabench (may need a bit more work)
> o Upgrade postgres version to 9.2.1
> o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains
> o Alter mailserver config to finish in a reasonable time
> o Add monitor for perf sched
> o Add moinitor that gathers ftrace information with trace-cmd
> o Add preliminary monitors for NUMA stats (very basic)
> o Specify ftrace events to monitor from config file
> o Remove the bulk of whats left of VMRegress
> o Convert shellpacks to a template format to auto-generate boilerplate code
> o Collect lock_stat information if enabled
> o Run multiple iterations of aim9
> o Add basic regression tests for Cross Memory Attach
> o Copy with preempt being enabled in highalloc stres tests
> o Have largedd cope with a missing large file to work with
> o Add a monitor-only mode to just capture logs
> o Report receive-side throughput in netperf for results
>
> At LSF/MM at some point a request was made that a series of tests
> be identified that were of interest to MM developers and that could be
> used for testing the Linux memory management subsystem. There is renewed
> interest in some sort of general testing framework during discussions for
> Kernel Summit 2012 so here is what I use.
>
> http://www.csn.ul.ie/~mel/projects/mmtests/
> http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz
>
> There are a number of stock configurations stored in configs/.  For example
> config-global-dhp__pagealloc-performance runs a number of tests that
> may be able to identify performance regressions or gains in the page
> allocator. Similarly there network and scheduler configs. There are also
> more complex options. config-global-dhp__parallelio-memcachetest will run
> memcachetest in the foreground while doing IO of different sizes in the
> background to measure how much unrelated IO affects the throughput of an
> in-memory database.
>
> This release is also a little rough and the extraction scripts could
> have been tidier but they were mostly written in an airport and for the
> most part they work as advertised. I'll fix bugs as according as they are
> brought to my attention.
>
> The stats reporting still needs work because while some tests know how
> to make a better estimate of mean by filtering outliers it is not being
> handled consistently and the methodology needs work. I know filtering
> statistics like this is a major flaw in the methodology but the decision
> was made in this case in the interest of the benchmarks with unstable
> results completing in a reasonable time.

Hi Gorman,

Could MMTests 0.07 auto download related packages for different 
distributions?

Regards,
Chen


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: MMTests 0.06

[Mel Gorman]

On Fri, Oct 26, 2012 at 03:48:48PM +0800, Ni zhan Chen wrote:
> On 10/12/2012 10:51 PM, Mel Gorman wrote:
> >MMTests 0.06 is a configurable test suite that runs a number of common
> >workloads of interest to MM developers. There are multiple additions
> >all but in many respects the most useful will be automatic package
> >installation. The package names are based on openSUSE but it's easy to
> >create mappings in bin/install-depends where the package names differ. The
> >very basics of monitoring NUMA efficiency is there as well and the autonuma
> >benchmark has a test. The stats it reports for NUMA need significant
> >improvement but for the most part that should be straight forward.
> >
> >Changelog since v0.05
> >o Automatically install packages (need name mappings for other distros)
> >o Add benchmark for autonumabench
> >o Add support for benchmarking NAS with MPI
> >o Add pgbench for autonumabench (may need a bit more work)
> >o Upgrade postgres version to 9.2.1
> >o Upgrade kernel verion used for kernbench to 3.0 for newer toolchains
> >o Alter mailserver config to finish in a reasonable time
> >o Add monitor for perf sched
> >o Add moinitor that gathers ftrace information with trace-cmd
> >o Add preliminary monitors for NUMA stats (very basic)
> >o Specify ftrace events to monitor from config file
> >o Remove the bulk of whats left of VMRegress
> >o Convert shellpacks to a template format to auto-generate boilerplate code
> >o Collect lock_stat information if enabled
> >o Run multiple iterations of aim9
> >o Add basic regression tests for Cross Memory Attach
> >o Copy with preempt being enabled in highalloc stres tests
> >o Have largedd cope with a missing large file to work with
> >o Add a monitor-only mode to just capture logs
> >o Report receive-side throughput in netperf for results
> >
> >At LSF/MM at some point a request was made that a series of tests
> >be identified that were of interest to MM developers and that could be
> >used for testing the Linux memory management subsystem. There is renewed
> >interest in some sort of general testing framework during discussions for
> >Kernel Summit 2012 so here is what I use.
> >
> >http://www.csn.ul.ie/~mel/projects/mmtests/
> >http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.06-mmtests-0.01.tar.gz
> >
> >There are a number of stock configurations stored in configs/.  For example
> >config-global-dhp__pagealloc-performance runs a number of tests that
> >may be able to identify performance regressions or gains in the page
> >allocator. Similarly there network and scheduler configs. There are also
> >more complex options. config-global-dhp__parallelio-memcachetest will run
> >memcachetest in the foreground while doing IO of different sizes in the
> >background to measure how much unrelated IO affects the throughput of an
> >in-memory database.
> >
> >This release is also a little rough and the extraction scripts could
> >have been tidier but they were mostly written in an airport and for the
> >most part they work as advertised. I'll fix bugs as according as they are
> >brought to my attention.
> >
> >The stats reporting still needs work because while some tests know how
> >to make a better estimate of mean by filtering outliers it is not being
> >handled consistently and the methodology needs work. I know filtering
> >statistics like this is a major flaw in the methodology but the decision
> >was made in this case in the interest of the benchmarks with unstable
> >results completing in a reasonable time.
> 
> Hi Gorman,
> 
> Could MMTests 0.07 auto download related packages for different
> distributions?
> 

Sure. Look at bin/install-depends and you'll see near the top of the file
this declaration.

my %package_map = (
        "debian::zlib-devel"  => "zlib1g-dev",
        "debian::gcc-fortran" => "gfortran",
        "debian::gcc-c++"     => "g++",
        "debian::diffutils"   => "diff",
);

This says for example that the zlib-devel package in openSUSE is called
zlib1g-dev in Debian. Send me the mappings of the different packages for your
distribution or edit this file yourself, send me the patch and I'll add them.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>