Re: [LSF/MM/BPF TOPIC] Long Duration Stress Testing Filesystems

[Dave Chinner <david@fromorbit.com>]

On Tue, Feb 04, 2025 at 02:09:39PM -0800, Darrick J. Wong wrote:
> On Tue, Feb 04, 2025 at 11:38:45AM -0800, Boris Burkov wrote:
> > On Mon, Feb 03, 2025 at 11:53:43AM -0800, Darrick J. Wong wrote:
> > > On Mon, Feb 03, 2025 at 10:55:19AM -0800, Boris Burkov wrote:
> > Which kind of gets back to what I was getting at in the first place. I
> > don't know enough about xfs to fully grok what the various
> > configurations do to the test (I imagine they enable various features
> > you want to validate under the soak), but I imagine there are still more
> > nasty things to do to the system in parallel.
> 
> Probably, but we've never really dug into that.  Dave might get there
> with check-parallel but I don't have 64p systems to spare right now.
> 
> As for configurations -- yeah, that's how we deal with the combinatoric
> explosion of mkfs options.  Run a lot of different weird configs in
> parallel with a fleet of VMs.  It's too bad that sort of implies that we
> all have to work for cloud vendors.

Well, that's one of the issues I'm addressing with check-parallel.

When a full auto run takes 10 minutes, a single developer can
iterate a significant chunk of the configuration matrix on a single
machine in a few hours with a single check-parallel command.

The functionality is already there to do this - if we define all
the configs that are to be tested via config section definitions,
check-parallel will iterate them all in one go.

That's the way I want to run testing - testing mkfs defaults with
the auto group is a ten minute smoke test that will catch most
regressions in new code. That "full auto" smoke test is now faster
than my typical think-code-build-deploy cycle time. Perfect.

Now running half a dozen common configs (e.g. each of the LTS-kernel
related mkfs defaults) for better coverage becomes a "run it while
I'm at lunch/in a meeting" exercise. IT can be done multiple times a
day, and interrupting it to start again with a new build is no
longer a big deal.

End-of-day/overnight testing has a long enough duration (12+ hours)
to exercise /several dozen/ fs configs. That's more than enough
testing to drown a typical developer in things that need analysis
and/or fixing.

All on one local machine built using cheap commodity parts.

The cloud is a lie.

-Dave.
-- 
Dave Chinner
david@fromorbit.com