Re: [LSF/MM/BPF TOPIC] A common project for file system performance testing

[Hans Holmberg <Hans.Holmberg@wdc.com>]

On 18/02/2026 16:31, Theodore Tso wrote:
> I think this is definitely an interesting topic.  One thing that I
> think we should consider is requirements (or feature requests, if
> we're talking about an existing code base) to make it easier to run
> performance testing.
> 
> A) Separate out the building of the benchamrks from the running of
>     said benchmarks.  My pattern is to build a test appliance which
>     can be uploaded to the system under test (SUT) which has all of
>     the necessary dependencies that might be used (e.g., fio, dbench,
>     etc.) precompiled.  The SUT might be a VM, or a device where
>     running a compiler is prohibited by security policy (e.g., a
>     machine in a data center), or a device which doesn't have a
>     compiler installed, and/or where running the compiler would be
>     slow and painful (e.g., an Android device).
> 
> B) Separate out fetching the benchmark components from the building.
>    This might be because an enterprise might have local changes, so
>    they want to use a version of these tools from a local repo.  It
>    also could be that security policy prohibits downloading software
>    from the network in an automated process, and there is a
>    requirement that any softare to be built in the build environment
>    has to reviewed by one or more human beings.
> 
> C) A modular way of storing the results.  I like to run my file system
>    tests in a VM, which is deleted as soon as the test run is
>    completed.  This significantly reduces the cost since the cost of
>    the VM is only paid when a test is active.  But that means that the
>    performance runs should not be assumed to be stored on the local
>    file system where the benchmarks are run, but instead, the results
>    should ideally be stored in some kind of flat file (ala Junit and
>    Kunit files) which can then be collated in some kind of centralized
>    store.


Yeah, I think splitting things up in modules/parts would be really
beneficial.

Just like blktests and fstests mainly focus on providing good tests
(with well defined ways for starting test runs and providing results),
we could have a module or a project that just defines a bunch of useful
workloads.

Setting up benchmarking runs and analyzing & presenting results could
be done by other modules as there'll be different preferences and needs
for those, depending on use case, ci-system etc.


> 
> D) A standardized way of specifying the hardware configuration of the
>    SUT.  This might include using VM's hosted at a hyperscale because
>    of the cost advantage, and because very often, the softare defined
>    storage in cloud VM's don't necessarily act like traditional HDD's
>    or flash devices.)
> 
> I'll note that one of the concerns of running performance tests using
> a VM is the noisy neighbor problem.  That is, what if the behavior of
> other VM's on the host affects the performance of the test VM?  This
> may vary depending on whether CPU or memory is subject to
> overprovisioning (which may vary depending on the VM type).  There are
> also VM types where all of the resources are dedicated to a single VM.
> 
> One thing that would be useful would be to have people running
> benchmarks to run the exact same configuration (kernel version,
> benchmark software versions, etc.) multiple times at different times
> on the same VM type, so the variability of the benchmark results can
> be measured.
> 
> Yes, this is a bit more work, but the benefits of using VM's, where
> you don't have to maintain hardware, and deal with hard drive
> failings, etc., means that some people might find the cost/benefits
> tradeoffs to be appealing.
> 
> Cheers,
> 
> 					- Ted
> 
>