Re: OLS2003 Performance BOF Proposals

Re: OLS2003 Performance BOF Proposals

[Nivedita Singhvi]

> I would very much appreciate comments (even one-liners) on any  
> community interest in these two OLS Performance BoF sessions.  I  
> believe the topics are dissimilar and relevant enough to justify both:

> PROPOSAL FOR LINUX BENCHMARK AUTOMATION
> This BOF will include a discussion on Linux benchmark automation. We  
> will discuss the features needed to provide an effective benchmark  
> automation process for Linux. This will include, defining the  
> configuration, input files, benchmark execution, output files, etc.  We  
> will also discuss the types of benchmarks that are tailored for rapid  
> execution and results analysis, for maximum development impact.
> 
> PROPOSAL FOR LINUX PERFORMANCE
> Linux changes occur very quickly in the open source community. There is  
> a strong need to quickly collect and share performance data and  
> analysis. However, there may be some instances where good, quality  
> performance data collection and analysis take longer than the short  
> turnaround required for maximum impact regarding newly released  
> patches. We plan to discuss the most effective methodology for  
> impacting Linux performance in a rapidly changing Linux open source  
> community environment.

While the first is fairly clear on what the content might be,
its not very clear what the second is referring to..At first
I figured they were going to be discussing techniques like
how to automate the benchmark process to make it faster
(which would overlap with the first, I imagine?), but if not,
is it going to be about which benchmarks to run etc? or how to
avoid benchmark legalese miseries that can make them a big
headache?? :)

thanks,
Nivedita

OLS2003 Performance BOF Proposals

[Sandra K Johnson]

Nivedita,

Ruth Forester, Partha Narayanan and I are working on these proposals. I've
answered your questions below. We appreciate any additional comments or
community interest. Thanks.

> From: Nivedita Singhvi <niv@us.ibm.com>
> Date: Sat Mar 8, 2003  1:15:34 PM US/Pacific
> To: rsf@flying-dove.com, linux-kernel@vger.kernel.org
> Subject: Re:  OLS2003 Performance BOF Proposals
> Reply-To: niv@us.ibm.com
>
>> I would very much appreciate comments (even one-liners) on any
>> community interest in these two OLS Performance BoF sessions.  I
>> believe the topics are dissimilar and relevant enough to justify >>
both:
>
>> PROPOSAL FOR LINUX BENCHMARK AUTOMATION
>> This BOF will include a discussion on Linux benchmark automation. We
>> will discuss the features needed to provide an effective benchmark
>> automation process for Linux. This will include, defining the
>> configuration, input files, benchmark execution, output files, etc.
>> We  will also discuss the types of benchmarks that are tailored for
>> rapid  execution and results analysis, for maximum development >>
impact.

>> PROPOSAL FOR LINUX PERFORMANCE
>> Linux changes occur very quickly in the open source community. There
>> is  a strong need to quickly collect and share performance data and
>> analysis. However, there may be some instances where good, quality
>> performance data collection and analysis take longer than the short
>> turnaround required for maximum impact regarding newly released
>> patches. We plan to discuss the most effective methodology for
>> impacting Linux performance in a rapidly changing Linux open source
>> community environment.
>
> While the first is fairly clear on what the content might be,
> its not very clear what the second is referring to..At first
> I figured they were going to be discussing techniques like
> how to automate the benchmark process to make it faster
> (which would overlap with the first, I imagine?), but if not,
> is it going to be about which benchmarks to run etc? or how to
> avoid benchmark legalese miseries that can make them a big
> headache?? :)
>

This second proposal is for a more general discussion
on Linux performance. We do not plan to focus on
benchmark automation, etc. The plan to discuss the
tradeoffs between getting good, reliable, reproducible
performance data and the need to post performance data
regarding patches quickly. This may not be an issue for
small microbenchmarks; however, it may be an issue for
larger macrobenchmarks, that my require more than 24
hours to generate good, reliable data. We want to focus
on amenable solutions for this issue, as well as other issues.
This would include some discussion on various types of
benchmarks, but little or no discussion on benchmark restrictions.

We have revised the 2nd proposal as follows:

PROPOSAL FOR LINUX PERFORMANCE

Linux changes occur very quickly in the open source community. There
 is a strong need to quickly collect and share performance data and
 analysis. However, there may be some instances where good, quality
 performance data collection and analysis take longer than the short
 turnaround required for maximum impact regarding newly released
 patches. For example, some macro benchmarks require more than
24 hours to configure, collect, and analyze performance data. This
varies across benchmarks, and the associated targeted workloads.
Given this tradeoff between the need to quickly have performance
data for newly created patches, and the need to have good, reliable,
reproducible, data that may take time to generate and analyze, we
plan to discuss the most effective methodology for impacting Linux
performance in a rapidly changing Linux open source community
environment.

> thanks,
> Nivedita
>



Sandra K. Johnson, Ph.D.
Linux Technology Center

OLS2003 Performance BOF Proposals

[Ruth Forester]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

Everyone,

I would very much appreciate comments (even one-liners) on any  
community interest in these two OLS Performance BoF sessions.  I  
believe the topics are dissimilar and relevant enough to justify both:
------------------------------------------------------------------------ 
-------------------------------------------
PROPOSAL FOR LINUX BENCHMARK AUTOMATION
This BOF will include a discussion on Linux benchmark automation. We  
will discuss the features needed to provide an effective benchmark  
automation process for Linux. This will include, defining the  
configuration, input files, benchmark execution, output files, etc.  We  
will also discuss the types of benchmarks that are tailored for rapid  
execution and results analysis, for maximum development impact.

PROPOSAL FOR LINUX PERFORMANCE
Linux changes occur very quickly in the open source community. There is  
a strong need to quickly collect and share performance data and  
analysis. However, there may be some instances where good, quality  
performance data collection and analysis take longer than the short  
turnaround required for maximum impact regarding newly released  
patches. We plan to discuss the most effective methodology for  
impacting Linux performance in a rapidly changing Linux open source  
community environment.
------------------------------------------------------------------------ 
---------------------------------------------
Please reply immediately so we can quickly submit them to OLS?
Thanks for your (speedy) replies!

ruth
Ruth Forester, Linux Performance LTC
rsf@flying-dove.com
notes: rsf@us.ibm.com
IBM Linux Technology Center
Beaverton, Oregon

[-- Attachment #2: Type: text/enriched, Size: 1693 bytes --]

Everyone,


I would very much appreciate comments (even one-liners) on any
community interest in these two OLS Performance BoF sessions.  I
believe the topics are dissimilar and relevant enough to justify both:  

-------------------------------------------------------------------------------------------------------------------

<smaller>PROPOSAL FOR LINUX BENCHMARK AUTOMATION

This BOF will include a discussion on Linux benchmark automation. We
will discuss the features needed to provide an effective benchmark
automation process for Linux. This will include, defining the
configuration, input files, benchmark execution, output files, etc. 
We will also discuss the types of benchmarks that are tailored for
rapid execution and results analysis, for maximum development impact.


PROPOSAL FOR LINUX PERFORMANCE

Linux changes occur very quickly in the open source community. There
is a strong need to quickly collect and share performance data and
analysis. However, there may be some instances where good, quality
performance data collection and analysis take longer than the short
turnaround required for maximum impact regarding newly released
patches. We plan to discuss the most effective methodology for
impacting Linux performance in a rapidly changing Linux open source
community environment.

</smaller>---------------------------------------------------------------------------------------------------------------------

Please reply immediately so we can quickly submit them to OLS? 

Thanks for your (speedy) replies! 


ruth

Ruth Forester, Linux Performance LTC

<smaller>rsf@flying-dove.com

notes: rsf@us.ibm.com

IBM Linux Technology Center

Beaverton, Oregon</smaller>

Re: OLS2003 Performance BOF Proposals

[Craig Thomas]

I think both can be combined into one.  The first seems to outline
the way to run a benchmark (multiple runs, std dev, variance, etc)
and a list of micro benchmarks.  The second seems to handle the larger
performance tests such as large database loads, very long running tests,
etc.

I wonder if you can include a discussion of monitoring tools for
performance data collection as part of the first BOF?  We have found
that for 2.5 the packages report different numbers when
comparing between sysstat, vmstat, ziostat, and iostat.


On Sat, 2003-03-08 at 10:51, Ruth Forester wrote:
> Everyone,
> 
> 
> I would very much appreciate comments (even one-liners) on any
> community interest in these two OLS Performance BoF sessions.  I
> believe the topics are dissimilar and relevant enough to justify both:  
> 
> ----------------------------------------------------------------------
> 
> This BOF will include a discussion on Linux benchmark automation. We
> will discuss the features needed to provide an effective benchmark
> automation process for Linux. This will include, defining the
> configuration, input files, benchmark execution, output files, etc. 
> We will also discuss the types of benchmarks that are tailored for
> rapid execution and results analysis, for maximum development impact.
> 
> 
> PROPOSAL FOR LINUX PERFORMANCE
> 
> Linux changes occur very quickly in the open source community. There
> is a strong need to quickly collect and share performance data and
> analysis. However, there may be some instances where good, quality
> performance data collection and analysis take longer than the short
> turnaround required for maximum impact regarding newly released
> patches. We plan to discuss the most effective methodology for
> impacting Linux performance in a rapidly changing Linux open source
> community environment.
> 
> --------------------------------------------------------
> 
> Please reply immediately so we can quickly submit them to OLS? 
> 
> Thanks for your (speedy) replies! 
> 
> 
> ruth
> 
> Ruth Forester, Linux Performance LTC
> 
> <smaller>rsf@flying-dove.com
> 
> notes: rsf@us.ibm.com
> 
> IBM Linux Technology Center
> 
> Beaverton, Oregon</smaller>
-- 
Craig Thomas <craiger@osdl.org>
OSDL