Re: [PATCH 09/12] generic/251: constrain runtime via time/load/soak factors

["Darrick J. Wong" <djwong@kernel.org>]

On Tue, Nov 19, 2024 at 12:45:05PM +1100, Dave Chinner wrote:
> On Mon, Nov 18, 2024 at 03:03:43PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > On my test fleet, this test can run for well in excess of 20 minutes:
> > 
> >    613 generic/251
> >    616 generic/251
> >    624 generic/251
> >    630 generic/251
> >    634 generic/251
> >    652 generic/251
> >    675 generic/251
> >    749 generic/251
> >    777 generic/251
> >    808 generic/251
> >    832 generic/251
> >    946 generic/251
> >   1082 generic/251
> >   1221 generic/251
> >   1241 generic/251
> >   1254 generic/251
> >   1305 generic/251
> >   1366 generic/251
> >   1646 generic/251
> >   1936 generic/251
> >   1952 generic/251
> >   2358 generic/251
> >   4359 generic/251
> >   5325 generic/251
> >  34046 generic/251
> > 
> > because it hardcodes 20 threads and 10 copies.  It's not great to have a
> > test that results in a significant fraction of the total test runtime.
> > Fix the looping and load on this test to use LOAD and TIME_FACTOR to
> > scale up its operations, along with the usual SOAK_DURATION override.
> > That brings the default runtime down to less than a minute.
> > 
> > Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
> 
> Question for you: Does your $here directory contain a .git subdir?
> 
> One of the causes of long runtime for me has been that $here might
> only contain 30MB of files, but the .git subdir balloons to several
> hundred MB over time, resulting is really long runtimes because it's
> copying GBs of data from the .git subdir.
> 
> I have this patch in my tree:
> 
> --- a/tests/generic/251
> +++ b/tests/generic/251
> @@ -175,9 +175,12 @@ nproc=20
>  # Copy $here to the scratch fs and make coipes of the replica.  The fstests
>  # output (and hence $seqres.full) could be in $here, so we need to snapshot
>  # $here before computing file checksums.
> +#
> +# $here/* as the files to copy so we avoid any .git directory that might be
> +# much, much larger than the rest of the fstests source tree we are copying.
>  content=$SCRATCH_MNT/orig
>  mkdir -p $content
> -cp -axT $here/ $content/
> +cp -ax $here/* $content/
> 
>  mkdir -p $tmp
> 
> And that's made the runtime drop from (typically) 10-15 minutes
> down to around 5 minutes....
> 
> Does this have any impact on the runtime on your test systems?

Nope, I do vpath builds (sort of) so there's no .git history getting
sucked up by generic/251.  The fstests directory on the test VMs is
~34MB spread across ~4800 files.

--D

> -Dave.
> 
> -- 
> Dave Chinner
> david@fromorbit.com
>