Re: [LSF/MM/BPF TOPIC] A common project for file system performance testing

["Theodore Tso" <tytso@mit.edu>]

I think this is definitely an interesting topic.  One thing that I
think we should consider is requirements (or feature requests, if
we're talking about an existing code base) to make it easier to run
performance testing.

A) Separate out the building of the benchamrks from the running of
    said benchmarks.  My pattern is to build a test appliance which
    can be uploaded to the system under test (SUT) which has all of
    the necessary dependencies that might be used (e.g., fio, dbench,
    etc.) precompiled.  The SUT might be a VM, or a device where
    running a compiler is prohibited by security policy (e.g., a
    machine in a data center), or a device which doesn't have a
    compiler installed, and/or where running the compiler would be
    slow and painful (e.g., an Android device).

B) Separate out fetching the benchmark components from the building.
   This might be because an enterprise might have local changes, so
   they want to use a version of these tools from a local repo.  It
   also could be that security policy prohibits downloading software
   from the network in an automated process, and there is a
   requirement that any softare to be built in the build environment
   has to reviewed by one or more human beings.

C) A modular way of storing the results.  I like to run my file system
   tests in a VM, which is deleted as soon as the test run is
   completed.  This significantly reduces the cost since the cost of
   the VM is only paid when a test is active.  But that means that the
   performance runs should not be assumed to be stored on the local
   file system where the benchmarks are run, but instead, the results
   should ideally be stored in some kind of flat file (ala Junit and
   Kunit files) which can then be collated in some kind of centralized
   store.

D) A standardized way of specifying the hardware configuration of the
   SUT.  This might include using VM's hosted at a hyperscale because
   of the cost advantage, and because very often, the softare defined
   storage in cloud VM's don't necessarily act like traditional HDD's
   or flash devices.)

I'll note that one of the concerns of running performance tests using
a VM is the noisy neighbor problem.  That is, what if the behavior of
other VM's on the host affects the performance of the test VM?  This
may vary depending on whether CPU or memory is subject to
overprovisioning (which may vary depending on the VM type).  There are
also VM types where all of the resources are dedicated to a single VM.

One thing that would be useful would be to have people running
benchmarks to run the exact same configuration (kernel version,
benchmark software versions, etc.) multiple times at different times
on the same VM type, so the variability of the benchmark results can
be measured.

Yes, this is a bit more work, but the benefits of using VM's, where
you don't have to maintain hardware, and deal with hard drive
failings, etc., means that some people might find the cost/benefits
tradeoffs to be appealing.

Cheers,

					- Ted