Best tests to measure Kernel Performance

Best tests to measure Kernel Performance

[Victor Rodriguez]

Hi

Despite the fact that this is not a well formulated question. I wonder
what tests could be a good subset to measure the performance of the
kernel . I have some approaches like phoronix does here :

http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1

I am sure postmark/ John the ripper/ Apache are good candidates but I
want to ask the community if there is some specific test that you
recommend

Thanks a lot for the help

Regards

Victor Rodriguez

Best tests to measure Kernel Performance

[Greg KH]

On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
> Hi
> 
> Despite the fact that this is not a well formulated question. I wonder
> what tests could be a good subset to measure the performance of the
> kernel . I have some approaches like phoronix does here :
> 
> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
> 
> I am sure postmark/ John the ripper/ Apache are good candidates but I
> want to ask the community if there is some specific test that you
> recommend

It depends on what you want to test, specifically.  The "kernel" isn't a
very specific thing, what most of those tests test is the speed of the
hardware, not specifically the kernel itself.

good luck,

greg k-h

Best tests to measure Kernel Performance

[Valdis.Kletnieks at vt.edu]

On Tue, 01 Dec 2015 18:45:51 -0600, Victor Rodriguez said:
> Despite the fact that this is not a well formulated question.

Exactly why it's hard to give a good answer.  It *really* depends on
what aspect you're interested in - I/O bandwidth, throughput, latency,
or other.  What I currently do at $DAYJOB is worry how fast I can get
a cluster of servers to flip blocks to and from disk to 40Gbit network
interfaces, and we've found a combo of iozone's and 'tar xf linux-2.6.32.tar.gz'
running at the same time is a good stand-in for our production workload.

Of course, your mileage will vary. :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20151201/52c82001/attachment.bin 

Best tests to measure Kernel Performance

[Victor Rodriguez]

On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
>> Hi
>>
>> Despite the fact that this is not a well formulated question. I wonder
>> what tests could be a good subset to measure the performance of the
>> kernel . I have some approaches like phoronix does here :
>>
>> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
>>
>> I am sure postmark/ John the ripper/ Apache are good candidates but I
>> want to ask the community if there is some specific test that you
>> recommend
>
> It depends on what you want to test, specifically.  The "kernel" isn't a
> very specific thing, what most of those tests test is the speed of the
> hardware, not specifically the kernel itself.
>
> good luck,
>
> greg k-h

Thanks for the feedback . You are right they test the speed of the HW
however I have seen that when there is a change in the kernel for
network the performance of apache is changed, which make total sense .

I think that LTSI should have kind of a test suite with significant
test that could help the developer to detect those perf changes. Is
very common that one as OS developer make a change in one package (
important one as the kernel ) and do not check how this affect the
performance of the OS ( I know is too general , but we might show
BKM's)

I think this might be a good topic to discuss with the community and
we could came with a solid recommended test suite in the LTSI project.

Feedback more than welcome

Regards

Victor Rodriguez

Best tests to measure Kernel Performance

[Victor Rodriguez]

On Tue, Dec 1, 2015 at 7:38 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Tue, 01 Dec 2015 18:45:51 -0600, Victor Rodriguez said:
>> Despite the fact that this is not a well formulated question.
>
> Exactly why it's hard to give a good answer.  It *really* depends on
> what aspect you're interested in - I/O bandwidth, throughput, latency,
> or other.  What I currently do at $DAYJOB is worry how fast I can get
> a cluster of servers to flip blocks to and from disk to 40Gbit network
> interfaces, and we've found a combo of iozone's and 'tar xf linux-2.6.32.tar.gz'
> running at the same time is a good stand-in for our production workload.
>
> Of course, your mileage will vary. :)

Iozone is fine and I think you give me a good reason to turn my eyes
into it again . I decided not to check it due to the fact that is a
micro bench and sometimes users want numbers of something they can
feel ( how fast can I de compres , how many transactions per second
can my DB manage , how fast can I build my product )

I was answering in Greg mail , that it might be good to came with a
BKM of tests for the kernel , a list of linked test ( if you move this
you will affect that , please test before commit ) , it might be just
an initial idea but sure for industry might be something helpful

Feedback more than welcome

Regards

Victor Rodriguez

Best tests to measure Kernel Performance

[Greg KH]

On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
> On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
> >> Hi
> >>
> >> Despite the fact that this is not a well formulated question. I wonder
> >> what tests could be a good subset to measure the performance of the
> >> kernel . I have some approaches like phoronix does here :
> >>
> >> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
> >>
> >> I am sure postmark/ John the ripper/ Apache are good candidates but I
> >> want to ask the community if there is some specific test that you
> >> recommend
> >
> > It depends on what you want to test, specifically.  The "kernel" isn't a
> > very specific thing, what most of those tests test is the speed of the
> > hardware, not specifically the kernel itself.
> >
> > good luck,
> >
> > greg k-h
> 
> Thanks for the feedback . You are right they test the speed of the HW
> however I have seen that when there is a change in the kernel for
> network the performance of apache is changed, which make total sense .

Maybe, maybe not, depending on if "apache" is cpu or hardware bound
(networking hardware has physical limits...) again, you have to be very
sure about exactly what you are wanting to test before using such a test
to try to "validate" anything other than just raw hardware speed.

Take a look at the "old" lmbench set of benchmarks for valid things that
a kernel change can affect, it's much different from what you might be
thinking of as a test.

> I think that LTSI should have kind of a test suite with significant
> test that could help the developer to detect those perf changes. Is
> very common that one as OS developer make a change in one package (
> important one as the kernel ) and do not check how this affect the
> performance of the OS ( I know is too general , but we might show
> BKM's)

WHat is "BKM"?

> I think this might be a good topic to discuss with the community and
> we could came with a solid recommended test suite in the LTSI project.

LTSI already has a "test suite" that is uses to test the releases,
what's wrong with that?  I'm sure the developers would be glad to add
any additional tests that you want added to it that you find missing and
useful.

Also note that the upstream kernel is tested by a huge test suite of
performance tests and static analysis tools for every commit in all
development branches by the wonderful 0-day bot system.  That's been
helping prevent regressions for a long time now.

thanks,

greg k-h

Best tests to measure Kernel Performance

[Valdis.Kletnieks at vt.edu]

On Wed, 02 Dec 2015 17:57:23 -0600, Victor Rodriguez said:

> Iozone is fine and I think you give me a good reason to turn my eyes
> into it again . I decided not to check it due to the fact that is a
> micro bench and sometimes users want numbers of something they can
> feel

I'm not sure you can call "see how fast the system can serve a terabyte from
disk over NFS" is a micro bench :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20151202/650482a4/attachment.bin 

Best tests to measure Kernel Performance

[Nicholas Mc Guire]

On Wed, Dec 02, 2015 at 04:36:50PM -0800, Greg KH wrote:
> On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
> > On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > > On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
> > >> Hi
> > >>
> > >> Despite the fact that this is not a well formulated question. I wonder
> > >> what tests could be a good subset to measure the performance of the
> > >> kernel . I have some approaches like phoronix does here :
> > >>
> > >> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
> > >>
> > >> I am sure postmark/ John the ripper/ Apache are good candidates but I
> > >> want to ask the community if there is some specific test that you
> > >> recommend
> > >
> > > It depends on what you want to test, specifically.  The "kernel" isn't a
> > > very specific thing, what most of those tests test is the speed of the
> > > hardware, not specifically the kernel itself.
> > >
> > > good luck,
> > >
> > > greg k-h
> > 
> > Thanks for the feedback . You are right they test the speed of the HW
> > however I have seen that when there is a change in the kernel for
> > network the performance of apache is changed, which make total sense .
> 
> Maybe, maybe not, depending on if "apache" is cpu or hardware bound
> (networking hardware has physical limits...) again, you have to be very
> sure about exactly what you are wanting to test before using such a test
> to try to "validate" anything other than just raw hardware speed.
> 
> Take a look at the "old" lmbench set of benchmarks for valid things that
> a kernel change can affect, it's much different from what you might be
> thinking of as a test.
>
We also still use lmbench as the usual first level of assessment as
it gives a lot of information about the change set impact on low-level
functions (system-calls, IPC, allocation...) was. It is much more precise
than trying to detect changes in complex applications that might only be making
a handful of a affected system call and thus look like
performance did not change while it actually did - just its in some
hard to reach corner case.

As with all testing - you need layers of testing to get a usable
picture of what is going on and lmbench is a good candidate for the
lowest level. Deducing system level changes from looking at complex 
application performance changes is alost impossible.

Specifically lmbench has a simple  make results; make rerun  which can give
a good overview of differences - but actually the tests default runs are 
only a small part of what the tests can uncover so looking at individual 
microbenchmarks to discover latency/bandwidth changes can be very helpful 
also to uncover odd hardware behavior.  

Some other low-level benchmarks we use are:
 rt-tests - scheduling, pi
 NetPIPE - network bandwidth
 bonnie++ - filesystem

thx!
hofrat
 

Best tests to measure Kernel Performance

[Victor Rodriguez]

On Wed, Dec 2, 2015 at 6:36 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
>> On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> > On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
>> >> Hi
>> >>
>> >> Despite the fact that this is not a well formulated question. I wonder
>> >> what tests could be a good subset to measure the performance of the
>> >> kernel . I have some approaches like phoronix does here :
>> >>
>> >> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
>> >>
>> >> I am sure postmark/ John the ripper/ Apache are good candidates but I
>> >> want to ask the community if there is some specific test that you
>> >> recommend
>> >
>> > It depends on what you want to test, specifically.  The "kernel" isn't a
>> > very specific thing, what most of those tests test is the speed of the
>> > hardware, not specifically the kernel itself.
>> >
>> > good luck,
>> >
>> > greg k-h
>>
>> Thanks for the feedback . You are right they test the speed of the HW
>> however I have seen that when there is a change in the kernel for
>> network the performance of apache is changed, which make total sense .
>
> Maybe, maybe not, depending on if "apache" is cpu or hardware bound
> (networking hardware has physical limits...) again, you have to be very
> sure about exactly what you are wanting to test before using such a test
> to try to "validate" anything other than just raw hardware speed.
>

Agree , you are right

> Take a look at the "old" lmbench set of benchmarks for valid things that
> a kernel change can affect, it's much different from what you might be
> thinking of as a test.
>

I will do

>> I think that LTSI should have kind of a test suite with significant
>> test that could help the developer to detect those perf changes. Is
>> very common that one as OS developer make a change in one package (
>> important one as the kernel ) and do not check how this affect the
>> performance of the OS ( I know is too general , but we might show
>> BKM's)
>
> WHat is "BKM"?
>

Sorry , too many years at intel :)

Best Known Method

>> I think this might be a good topic to discuss with the community and
>> we could came with a solid recommended test suite in the LTSI project.
>
> LTSI already has a "test suite" that is uses to test the releases,
> what's wrong with that?  I'm sure the developers would be glad to add
> any additional tests that you want added to it that you find missing and
> useful.

Do we have a subset of test inside LTSI test suite just for
performance ? What i am thinking is that in LTSI suite we could have
subset of tests:

-> Unit tests
-> Functional Tests
-> Performance tests

So if that could exist would be amazing for a OS development team.

Also to give the possibility to have CI natural in the LTSI test suite
:) . Imagine that :

-> when you have a new commit you can run specific test and you can
run the test you want or all the test suite without merging your
change
-> you could make a new release and then the Performance test be
executed and then the graphs show of ( are we better or worst ? )

We in Clear Linux ( as in many other projects ) are working to make
this a full CI for the entire OS , not just for Kernel , but would be
nice if LTSI system could have that and the kernel team of many
companies could use it as a full test suite / CI

It might be just a naive idea but I think it  would be very helpful

As always thanks a lot for the feedback Greg I lear a lot from this :)

Best Regards

Victor Rodriguez


> Also note that the upstream kernel is tested by a huge test suite of
> performance tests and static analysis tools for every commit in all
> development branches by the wonderful 0-day bot system.  That's been
> helping prevent regressions for a long time now.
>

> thanks,
>
> greg k-h

Best tests to measure Kernel Performance

[Victor Rodriguez]

On Thu, Dec 3, 2015 at 2:10 AM, Nicholas Mc Guire <der.herr@hofr.at> wrote:
> On Wed, Dec 02, 2015 at 04:36:50PM -0800, Greg KH wrote:
>> On Wed, Dec 02, 2015 at 05:50:30PM -0600, Victor Rodriguez wrote:
>> > On Tue, Dec 1, 2015 at 7:32 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> > > On Tue, Dec 01, 2015 at 06:45:51PM -0600, Victor Rodriguez wrote:
>> > >> Hi
>> > >>
>> > >> Despite the fact that this is not a well formulated question. I wonder
>> > >> what tests could be a good subset to measure the performance of the
>> > >> kernel . I have some approaches like phoronix does here :
>> > >>
>> > >> http://www.phoronix.com/scan.php?page=article&item=linux-41-byt&num=1
>> > >>
>> > >> I am sure postmark/ John the ripper/ Apache are good candidates but I
>> > >> want to ask the community if there is some specific test that you
>> > >> recommend
>> > >
>> > > It depends on what you want to test, specifically.  The "kernel" isn't a
>> > > very specific thing, what most of those tests test is the speed of the
>> > > hardware, not specifically the kernel itself.
>> > >
>> > > good luck,
>> > >
>> > > greg k-h
>> >
>> > Thanks for the feedback . You are right they test the speed of the HW
>> > however I have seen that when there is a change in the kernel for
>> > network the performance of apache is changed, which make total sense .
>>
>> Maybe, maybe not, depending on if "apache" is cpu or hardware bound
>> (networking hardware has physical limits...) again, you have to be very
>> sure about exactly what you are wanting to test before using such a test
>> to try to "validate" anything other than just raw hardware speed.
>>
>> Take a look at the "old" lmbench set of benchmarks for valid things that
>> a kernel change can affect, it's much different from what you might be
>> thinking of as a test.
>>
> We also still use lmbench as the usual first level of assessment as
> it gives a lot of information about the change set impact on low-level
> functions (system-calls, IPC, allocation...) was. It is much more precise
> than trying to detect changes in complex applications that might only be making
> a handful of a affected system call and thus look like
> performance did not change while it actually did - just its in some
> hard to reach corner case.
>
> As with all testing - you need layers of testing to get a usable
> picture of what is going on and lmbench is a good candidate for the
> lowest level. Deducing system level changes from looking at complex
> application performance changes is alost impossible.
>
> Specifically lmbench has a simple  make results; make rerun  which can give
> a good overview of differences - but actually the tests default runs are
> only a small part of what the tests can uncover so looking at individual
> microbenchmarks to discover latency/bandwidth changes can be very helpful
> also to uncover odd hardware behavior.
>
> Some other low-level benchmarks we use are:
>  rt-tests - scheduling, pi
>  NetPIPE - network bandwidth
>  bonnie++ - filesystem


Thanks a lot hofrat

I really appreciate all the help I think that is time to turn my eyes
to lmbench for sure as well to the tools you mention :)

Yes a lot of layers are necessary to measure the QA of an OS , we need
full image test as well as cloud tests ( since our OS is designed for
Cloud ) . lmbench will be amazing for low level

I really appreciate all the help


> thx!
> hofrat
>

Best tests to measure Kernel Performance

[Greg KH]

On Thu, Dec 03, 2015 at 10:51:09AM -0600, Victor Rodriguez wrote:
> Do we have a subset of test inside LTSI test suite just for
> performance ?

First off, please go look at the test suite that LTSI uses for testing
before asking this type of thing, I think that will answer all of your
questions already...

> We in Clear Linux ( as in many other projects ) are working to make
> this a full CI for the entire OS , not just for Kernel , but would be
> nice if LTSI system could have that and the kernel team of many
> companies could use it as a full test suite / CI

LTSI is just about the kernel, not the "full system".  If you wish to
help work on something larger, wonderful, but please be aware of the
resources that LTSI currently has for any "new" work (i.e. very
limited.)

thanks,

greg k-h