Re: [PATCH 09/12] generic/251: constrain runtime via time/load/soak factors

Darrick J. Wong
2024-11-19T15:50:22Z

On Tue, Nov 19, 2024 at 12:45:05PM +1100, Dave Chinner wrote:
> On Mon, Nov 18, 2024 at 03:03:43PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> >
> > On my test fleet, this test can run for well in excess of 20 minutes:
> >
> > 613 generic/251
> > 616 generic/251
> > 624 generic/251
> > 630 generic/251
> > 634 generic/251
> > 652 generic/251
> > 675 generic/251
> > 749 generic/251
> > 777 generic/251
> > 808 generic/251
> > 832 generic/251
> > 946 generic/251
> > 1082 generic/251
> > 1221 generic/251
> > 1241 generic/251
> > 1254 generic/251
> > 1305 generic/251
> > 1366 generic/251
> > 1646 generic/251
> > 1936 generic/251
> > 1952 generic/251
> > 2358 generic/251
> > 4359 generic/251
> > 5325 generic/251
> > 34046 generic/251
> >
> > because it hardcodes 20 threads and 10 copies. It's not great to have a
> > test that results in a significant fraction of the total test runtime.
> > Fix the looping and load on this test to use LOAD and TIME_FACTOR to
> > scale up its operations, along with the usual SOAK_DURATION override.
> > That brings the default runtime down to less than a minute.
> >
> > Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
>
> Question for you: Does your $here directory contain a .git subdir?
>
> One of the causes of long runtime for me has been that $here might
> only contain 30MB of files, but the .git subdir balloons to several
> hundred MB over time, resulting is really long runtimes because it's
> copying GBs of data from the .git subdir.
>
> I have this patch in my tree:
>
> --- a/tests/generic/251
> +++ b/tests/generic/251
> @@ -175,9 +175,12 @@ nproc=20
> # Copy $here to the scratch fs and make coipes of the replica. The fstests
> # output (and hence $seqres.full) could be in $here, so we need to snapshot
> # $here before computing file checksums.
> +#
> +# $here/* as the files to copy so we avoid any .git directory that might be
> +# much, much larger than the rest of the fstests source tree we are copying.
> content=$SCRATCH_MNT/orig
> mkdir -p $content
> -cp -axT $here/ $content/
> +cp -ax $here/* $content/
>
> mkdir -p $tmp
>
> And that's made the runtime drop from (typically) 10-15 minutes
> down to around 5 minutes....
>
> Does this have any impact on the runtime on your test systems?

Nope, I do vpath builds (sort of) so there's no .git history getting
sucked up by generic/251. The fstests directory on the test VMs is
~34MB spread across ~4800 files.

--D

> -Dave.
>
> --
> Dave Chinner
> david@fromorbit.com
>