[LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ojaswin Mujoo]

Greetings,

This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
a proposal on centralizing filesystem and device configurations within xfstests
and maybe a further discussion on some of the open ideas listed by Ted here [3].
More details are mentioned below.

** Background ** 
There was a discussion last year at LSFMM [1] about creating a central fs-config
store, that can then be used by anyone for testing different FS
features/configurations. This can also bring an awareness among other developers
and testers on what is being actively maintained by FS maintainers. We recently
posted an RFC [2] for centralizing filesystem configuration which is under
review. The next step we are considering is to centralize device configurations
within xfstests itself. In line with this, Ted also suggested a similar idea (in
point A) [3], where he proposed specifying the device size for the TEST and
SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
improve the overall runtime of xfstests.

Recently Dave introduced a feature [4] to run the xfs and generic tests in
parallel. This patch creates the TEST and SCRATCH devices at runtime without
requiring them to be specified in any config file. However, at this stage, the
automatic device initialization appears to be somewhat limited. We believe that
centralizing device configuration could help enhance this functionality as well.

** Proposal ** 
We would like to propose a discussion at LSFMM on two key features: central
fsconfig and central device-config within xfstests. We can explore how the
fsconfig feature can be utilized, and by then, we aim to have a PoC for central
device-config feature, which we think can also be discussed in more detail. At
this point, we are hoping to get a PoC working with loop devices by default. It
will be good to hear from other developers, maintainers, and testers about their
thoughts and suggestions on these two features.

Additionally, we would like to thank Ted for listing several features he uses in
his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
reducing the burden of maintaining custom test scripts and wrappers around
xfstests, we can also discuss essential features that could be integrated
directly into xfstests, whether from Ted's list or suggestions from others.

Thoughts and suggestions are welcome.

** References **
[1] https://lore.kernel.org/all/87h6h4sopf.fsf@doe.com/
[2] https://lore.kernel.org/all/9a6764237b900f40e563d8dee2853f1430245b74.1736496620.git.nirjhar.roy.lists@gmail.com/
[3] https://lore.kernel.org/all/20250110163859.GB1514771@mit.edu/
[4] https://lore.kernel.org/all/20241127045403.3665299-1-david@fromorbit.com/

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Dave Chinner]

On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
> Greetings,
> 
> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
> a proposal on centralizing filesystem and device configurations within xfstests
> and maybe a further discussion on some of the open ideas listed by Ted here [3].
> More details are mentioned below.
> 
> ** Background ** 
> There was a discussion last year at LSFMM [1] about creating a central fs-config
> store, that can then be used by anyone for testing different FS
> features/configurations. This can also bring an awareness among other developers
> and testers on what is being actively maintained by FS maintainers. We recently
> posted an RFC [2] for centralizing filesystem configuration which is under
> review. The next step we are considering is to centralize device configurations
> within xfstests itself. In line with this, Ted also suggested a similar idea (in
> point A) [3], where he proposed specifying the device size for the TEST and
> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
> improve the overall runtime of xfstests.
> 
> Recently Dave introduced a feature [4] to run the xfs and generic tests in
> parallel. This patch creates the TEST and SCRATCH devices at runtime without
> requiring them to be specified in any config file. However, at this stage, the
> automatic device initialization appears to be somewhat limited. We believe that
> centralizing device configuration could help enhance this functionality as well.

Right, the point of check-parallel is to take away the need to
specify devices completely.  I've already added support for the
LOGWRITES_DEV, and I'm in the process of adding LOGDEV and RTDEV
support for both test and scratch devices. At this point, the need
for actual actual device specification in the config files goes
away.

What I am expecting to need is a set of fields that specify the
*size* of the devices so that the hard-coded image file sizes in
the check-parallel script go away.

From there, I intend to have check-parallel iterate config file run
sections itself, rather than have it run them internally to check.
That way check is only ever invoked by check-parallel with all the
devices completely set up.

Hence a typical host independent config file would look like:

TEST_DEV_SIZE=10g
TEST_RTDEV_size=10g
TEST_LOGDEV_SIZE=128m
SCRATCH_DEV_SIZE=20g
SCRATCH_RTDEV_size=20g
SCRATCH_LOGDEV_SIZE=512m
LOGWRITES_DEV_SIZE=2g

[xfs]
FSTYP=xfs
MKFS_OPTIONS="-b size=4k"
TEST_FS_MOUNT_OPTIONS=
MOUNT_OPTIONS=
USE_EXTERNAL=

[xfs-rmapbt]
MKFS_OPTIONS="-b size=4k -m rmapbt=1"

[xfs-noreflink]
MKFS_OPTIONS="-b size=4k -m reflink=0"

[xfs-n64k]
MKFS_OPTIONS="-b size=4k -n size=64k"

[xfs-ext]
MKFS_OPTIONS="-b size=4k"
USE_EXTERNAL=yes

[ext4]
FSTYP="ext4"
MKFS_OPTIONS=
USE_EXTERNAL=

[btrfs]
FSTYP="btrfs"
.....


IOWs, all that is different from system to system is the device size
setup. The actual config sections under test (e.g. [xfs]) never need
to change from host to host, nor environment to environment. i.e.
"xfs-n64k" runs the same config filesystem test on every system,
everywhere...

> ** Proposal ** 
> We would like to propose a discussion at LSFMM on two key features: central

I'm not going to be at LSFMM, so please discuss this on the list via
email as we'd normally do so. LSFMM discussions are exclusionary
whilst, OTOH, the mailing list is inclusive...

> fsconfig and central device-config within xfstests. We can explore how the
> fsconfig feature can be utilized, and by then, we aim to have a PoC for central
> device-config feature, which we think can also be discussed in more detail. At
> this point, we are hoping to get a PoC working with loop devices by default. It
> will be good to hear from other developers, maintainers, and testers about their
> thoughts and suggestions on these two features.

I don't really see a need for a new centralised config setup. With
the above, we can acheived a "zero-config" goal with the existing
config file formats and infrastructure. All that we need to do is
update the default config file in the repo to contain a section for
each of the "standard" test configs we want to define....

> Additionally, we would like to thank Ted for listing several features he uses in
> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
> reducing the burden of maintaining custom test scripts and wrappers around
> xfstests, we can also discuss essential features that could be integrated
> directly into xfstests, whether from Ted's list or suggestions from others.

On of my goals with check-parallel is to completely remove the need
for end users to configure fstests. i.e. all you need to do is point
it at a directory, tell it which filesystem to test, and it "just
runs" with all the defaults that come direct from the fstests
repository...

It is also worth keeping in mind that check-parallel can be run with
a single runner thread, in which case a single check instance runs
all tests serially. i.e. we can make check-parallel emulate existing
check behaviour exactly, except it uses all the
auto-config/auto-setup stuff that comes along with check-parallel...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

Dave Chinner <david@fromorbit.com> writes:

> On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
>> Greetings,
>> 
>> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
>> a proposal on centralizing filesystem and device configurations within xfstests
>> and maybe a further discussion on some of the open ideas listed by Ted here [3].
>> More details are mentioned below.
>> 
>> ** Background ** 
>> There was a discussion last year at LSFMM [1] about creating a central fs-config
>> store, that can then be used by anyone for testing different FS
>> features/configurations. This can also bring an awareness among other developers
>> and testers on what is being actively maintained by FS maintainers. We recently
>> posted an RFC [2] for centralizing filesystem configuration which is under
>> review. The next step we are considering is to centralize device configurations
>> within xfstests itself. In line with this, Ted also suggested a similar idea (in
>> point A) [3], where he proposed specifying the device size for the TEST and
>> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
>> improve the overall runtime of xfstests.
>> 
>> Recently Dave introduced a feature [4] to run the xfs and generic tests in
>> parallel. This patch creates the TEST and SCRATCH devices at runtime without
>> requiring them to be specified in any config file. However, at this stage, the
>> automatic device initialization appears to be somewhat limited. We believe that
>> centralizing device configuration could help enhance this functionality as well.
>
> Right, the point of check-parallel is to take away the need to
> specify devices completely.  I've already added support for the
> LOGWRITES_DEV, and I'm in the process of adding LOGDEV and RTDEV
> support for both test and scratch devices. At this point, the need
> for actual actual device specification in the config files goes
> away.
>
> What I am expecting to need is a set of fields that specify the
> *size* of the devices so that the hard-coded image file sizes in
> the check-parallel script go away.
>
> From there, I intend to have check-parallel iterate config file run
> sections itself, rather than have it run them internally to check.
> That way check is only ever invoked by check-parallel with all the
> devices completely set up.

Yes, this sounds good. This is what we were anticipating too.
Thanks for sharing.

>
> Hence a typical host independent config file would look like:

The work being proposed by us here was to make this config file
centralized within xfstests itself for both fsconfig and device-config.
This saves us from defining each of this section within local.config file
and can be used by passing cmdling arguments to invoke a given section
directly. 

e.g.

    ./check -c configs/xfs/64k -g auto

There have been cases where testers and others have requested info to
know about - 
- What different FS config options to test,
- What gets tested by the Maintainers,  
- Is there a common place where I can find MKFS and MOUNT options which
  I should be testing for my FS/feature testing. 

That is the reason, I think, centralizing fsconfig option can be
helpful. I remember bringing this idea in our last LSFMM-2024, where you
mentioned that - let's see the RFC [1] and maybe then we can discuss more :).
Here is the RFC for the same. There are some additional improvements in
that series, but it mainly adds fsconfig option.

[1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/

>
> TEST_DEV_SIZE=10g
> TEST_RTDEV_size=10g
> TEST_LOGDEV_SIZE=128m
> SCRATCH_DEV_SIZE=20g
> SCRATCH_RTDEV_size=20g
> SCRATCH_LOGDEV_SIZE=512m
> LOGWRITES_DEV_SIZE=2g
>

For centralizing device-configs idea, I was hoping we could mention above
in "configs/devices/loop" config file. This can be picked up by ./check
too if local.config hasn't been provided with these options.

> [xfs]
> FSTYP=xfs
> MKFS_OPTIONS="-b size=4k"
> TEST_FS_MOUNT_OPTIONS=
> MOUNT_OPTIONS=
> USE_EXTERNAL=
>
> [xfs-rmapbt]
> MKFS_OPTIONS="-b size=4k -m rmapbt=1"
>
> [xfs-noreflink]
> MKFS_OPTIONS="-b size=4k -m reflink=0"
>
> [xfs-n64k]
> MKFS_OPTIONS="-b size=4k -n size=64k"
>
> [xfs-ext]
> MKFS_OPTIONS="-b size=4k"
> USE_EXTERNAL=yes
>
> [ext4]
> FSTYP="ext4"
> MKFS_OPTIONS=
> USE_EXTERNAL=
>
> [btrfs]
> FSTYP="btrfs"
> .....

Above all fs configs could be added to configs/{ext4|xfs|btrfs}/... 
Than this can be used in 2 ways.. 

1. ./check -c configs/xfs/4k,configs/xfs/rmapbt,configs/ext4/4k ... 

2. Or we may still pass fsconfig via local.config file.. e.g. 

# both configs/xfs/4k or xfs/4k can be used here
[xfs]
FS_CONFIG_OPTION=configs/xfs/4k

[xfs-rmapbt]
FS_CONFIG_OPTION=xfs/rmapbt

[xfs-noreflink]
FS_CONFIG_OPTION=xfs/noreflink

[xfs-n64k]
FS_CONFIG_OPTION=xfs/64k

[xfs-ext]
FS_CONFIG_OPTION=xfs/4k
USE_EXTERNAL=yes

[ext4]
FS_CONFIG_OPTION=ext4/4k


>
>
> IOWs, all that is different from system to system is the device size
> setup. The actual config sections under test (e.g. [xfs]) never need
> to change from host to host, nor environment to environment. i.e.
> "xfs-n64k" runs the same config filesystem test on every system,
> everywhere...
>

Right. So it's also useful if those configs can stay in configs/<fs>/**
as well.


>> ** Proposal ** 
>> We would like to propose a discussion at LSFMM on two key features: central
>
> I'm not going to be at LSFMM, so please discuss this on the list via
> email as we'd normally do so. LSFMM discussions are exclusionary
> whilst, OTOH, the mailing list is inclusive...
>

Sure I am happy to discuss this on mailing list too! It will be good to
know from others as well, who do FS testing from time to time, on whether
having a central fsconfig store makes their life easy or not? 
And from other FS maintainers, on whether adding their test fsconfigs to
configs/<fs>/** will save them from maintaining it in their custom CI
wrappers or not? 

>> fsconfig and central device-config within xfstests. We can explore how the
>> fsconfig feature can be utilized, and by then, we aim to have a PoC for central
>> device-config feature, which we think can also be discussed in more detail. At
>> this point, we are hoping to get a PoC working with loop devices by default. It
>> will be good to hear from other developers, maintainers, and testers about their
>> thoughts and suggestions on these two features.
>
> I don't really see a need for a new centralised config setup. With
> the above, we can acheived a "zero-config" goal with the existing
> config file formats and infrastructure. All that we need to do is
> update the default config file in the repo to contain a section for
> each of the "standard" test configs we want to define....
>

Could you please take look at the shared RFC [1] once? I believe it will
be useful to have this central to xfstests for reasons I mentioned
above. This also gets us to zero-config setup with almost no need to
configure anything. This helps other testers & subsystem maintainers to
know what configs are being tested by maintainers, which they can use
for their FS feature testing too.
We can than directly issue -

           ./check -c <fs>/<config> -g quick

[1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/


>> Additionally, we would like to thank Ted for listing several features he uses in
>> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
>> reducing the burden of maintaining custom test scripts and wrappers around
>> xfstests, we can also discuss essential features that could be integrated
>> directly into xfstests, whether from Ted's list or suggestions from others.
>
> On of my goals with check-parallel is to completely remove the need
> for end users to configure fstests. i.e. all you need to do is point
> it at a directory, tell it which filesystem to test, and it "just
> runs" with all the defaults that come direct from the fstests
> repository...
>

Right. Centralizing fsconfigs & device configs is also doing the same.
In fact once we have configs/devices/loop config file, then we don't
need to even create local.config file (for most cases I believe).

./check and ./check_parallel can be passed these config files and
xfstests infra will create loop devices, run the tests and later do the
cleanups.

e.g. maybe something like this... 
      ./check -c configs/<fs>/config>,configs/devices/loop -g quick

OR
      ./check -c <fs>/config>,devices/loop -g quick
    


> It is also worth keeping in mind that check-parallel can be run with
> a single runner thread, in which case a single check instance runs
> all tests serially. i.e. we can make check-parallel emulate existing
> check behaviour exactly, except it uses all the
> auto-config/auto-setup stuff that comes along with check-parallel...
>

Sure thanks Dave for the review.

> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Dave Chinner]

On Tue, Feb 04, 2025 at 12:44:00PM +0530, Ritesh Harjani wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
> >> Greetings,
> >> 
> >> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
> >> a proposal on centralizing filesystem and device configurations within xfstests
> >> and maybe a further discussion on some of the open ideas listed by Ted here [3].
> >> More details are mentioned below.
> >> 
> >> ** Background ** 
> >> There was a discussion last year at LSFMM [1] about creating a central fs-config
> >> store, that can then be used by anyone for testing different FS
> >> features/configurations. This can also bring an awareness among other developers
> >> and testers on what is being actively maintained by FS maintainers. We recently
> >> posted an RFC [2] for centralizing filesystem configuration which is under
> >> review. The next step we are considering is to centralize device configurations
> >> within xfstests itself. In line with this, Ted also suggested a similar idea (in
> >> point A) [3], where he proposed specifying the device size for the TEST and
> >> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
> >> improve the overall runtime of xfstests.
> >> 
> >> Recently Dave introduced a feature [4] to run the xfs and generic tests in
> >> parallel. This patch creates the TEST and SCRATCH devices at runtime without
> >> requiring them to be specified in any config file. However, at this stage, the
> >> automatic device initialization appears to be somewhat limited. We believe that
> >> centralizing device configuration could help enhance this functionality as well.
> >
> > Right, the point of check-parallel is to take away the need to
> > specify devices completely.  I've already added support for the
> > LOGWRITES_DEV, and I'm in the process of adding LOGDEV and RTDEV
> > support for both test and scratch devices. At this point, the need
> > for actual actual device specification in the config files goes
> > away.
> >
> > What I am expecting to need is a set of fields that specify the
> > *size* of the devices so that the hard-coded image file sizes in
> > the check-parallel script go away.
> >
> > From there, I intend to have check-parallel iterate config file run
> > sections itself, rather than have it run them internally to check.
> > That way check is only ever invoked by check-parallel with all the
> > devices completely set up.
> 
> Yes, this sounds good. This is what we were anticipating too.
> Thanks for sharing.
> 
> >
> > Hence a typical host independent config file would look like:
> 
> The work being proposed by us here was to make this config file
> centralized within xfstests itself for both fsconfig and device-config.
> This saves us from defining each of this section within local.config file
> and can be used by passing cmdling arguments to invoke a given section
> directly. 
> 
> e.g.
> 
>     ./check -c configs/xfs/64k -g auto

This strikes me as re-implementing config sections with a different
file format.

> There have been cases where testers and others have requested info to
> know about - 
> - What different FS config options to test,
> - What gets tested by the Maintainers,  
> - Is there a common place where I can find MKFS and MOUNT options which
>   I should be testing for my FS/feature testing. 

Those are questions that documentation should answer - they are not
a reason for changing config file formats.

> That is the reason, I think, centralizing fsconfig option can be
> helpful. I remember bringing this idea in our last LSFMM-2024, where you
> mentioned that - let's see the RFC [1] and maybe then we can discuss more :).
> Here is the RFC for the same. There are some additional improvements in
> that series, but it mainly adds fsconfig option.
> 
> [1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/

Yeah, that's just making a tiny config file per config section, and
having to add a heap of parsing code to do that.

I find it much easier to manage a single config file where each
config is maybe only one or two lines than it is to have to find the
needle in a haystack of hundreds of tiny files that are almost all
alike.

> > TEST_DEV_SIZE=10g
> > TEST_RTDEV_size=10g
> > TEST_LOGDEV_SIZE=128m
> > SCRATCH_DEV_SIZE=20g
> > SCRATCH_RTDEV_size=20g
> > SCRATCH_LOGDEV_SIZE=512m
> > LOGWRITES_DEV_SIZE=2g
> >
> 
> For centralizing device-configs idea, I was hoping we could mention above
> in "configs/devices/loop" config file. This can be picked up by ./check
> too if local.config hasn't been provided with these options.

Great. How do you specify where the image files are going to be
hosted when no devices have been configured?

These sorts of zero-conf chicken/egg problems have already been
solved by check-parallel - I'm really not convinced that we need
to make check itself try to solve these. I designed check-parallel
the way I did because making check itself solve these problems is
.... convoluted and difficult.

If we need zero-conf, just use ./check-parallel....

> > [xfs]
> > FSTYP=xfs
> > MKFS_OPTIONS="-b size=4k"
> > TEST_FS_MOUNT_OPTIONS=
> > MOUNT_OPTIONS=
> > USE_EXTERNAL=
> >
> > [xfs-rmapbt]
> > MKFS_OPTIONS="-b size=4k -m rmapbt=1"
> >
> > [xfs-noreflink]
> > MKFS_OPTIONS="-b size=4k -m reflink=0"
> >
> > [xfs-n64k]
> > MKFS_OPTIONS="-b size=4k -n size=64k"
> >
> > [xfs-ext]
> > MKFS_OPTIONS="-b size=4k"
> > USE_EXTERNAL=yes
> >
> > [ext4]
> > FSTYP="ext4"
> > MKFS_OPTIONS=
> > USE_EXTERNAL=
> >
> > [btrfs]
> > FSTYP="btrfs"
> > .....
> 
> Above all fs configs could be added to configs/{ext4|xfs|btrfs}/... 
> Than this can be used in 2 ways.. 
> 
> 1. ./check -c configs/xfs/4k,configs/xfs/rmapbt,configs/ext4/4k ... 

Can't say I like using relative paths to specify the config we
should use. It's not an interface I'd choose over:

	./check -s xfs -g auto

Especially as the existing config section implementation does
exactly the same thing as the proposed config file farm....

> 2. Or we may still pass fsconfig via local.config file.. e.g. 
> 
> # both configs/xfs/4k or xfs/4k can be used here
> [xfs]
> FS_CONFIG_OPTION=configs/xfs/4k
> 
> [xfs-rmapbt]
> FS_CONFIG_OPTION=xfs/rmapbt
> 
> [xfs-noreflink]
> FS_CONFIG_OPTION=xfs/noreflink
> 
> [xfs-n64k]
> FS_CONFIG_OPTION=xfs/64k
> 
> [xfs-ext]
> FS_CONFIG_OPTION=xfs/4k
> USE_EXTERNAL=yes
> 
> [ext4]
> FS_CONFIG_OPTION=ext4/4k

How is that an improvement on the section setup we have right now?
Abstracting the config section options into a separate file is the
worst of both worlds - now I have to look up the section to find the
file and the section options, then lookup the file to see the
options that file contains.

It's like you've decided that the solution to centralised management
of configs must be "one config per file", and so every other way of
managing options needs to be reframed for that solution....

> > IOWs, all that is different from system to system is the device size
> > setup. The actual config sections under test (e.g. [xfs]) never need
> > to change from host to host, nor environment to environment. i.e.
> > "xfs-n64k" runs the same config filesystem test on every system,
> > everywhere...
> 
> Right. So it's also useful if those configs can stay in configs/<fs>/**
> as well.

Why is having hundreds of tiny single-config-only files
better than having all the configs in a single file that is
easily browsed and searched?

Honestly, I really don't see any advantage to re-implementing config
sections as a "file per config" object farm. Yes, you can store
information that way, but that doesn't make it an improvement over a
single file...

All that is needed is for the upstream repository to maintain a
config file with all the config sections defined that people need.
We don't need any new infrastructure to implement a "centralised
configs" feature - all we need is an agreement that upstream will
ship an update-to-date default config file instead of the ancient,
stale example.config/localhost.config files....

> > I don't really see a need for a new centralised config setup. With
> > the above, we can acheived a "zero-config" goal with the existing
> > config file formats and infrastructure. All that we need to do is
> > update the default config file in the repo to contain a section for
> > each of the "standard" test configs we want to define....
> 
> Could you please take look at the shared RFC [1] once? I believe it will
> be useful to have this central to xfstests for reasons I mentioned
> above. This also gets us to zero-config setup with almost no need to
> configure anything.

[1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/

There's nothing in that which provides zero-config. It still
requires all the devices to be specified by the user...

> This helps other testers & subsystem maintainers to
> know what configs are being tested by maintainers, which they can use
> for their FS feature testing too.
> We can than directly issue -
> 
>            ./check -c <fs>/<config> -g quick

As I've already said: this functionality already exists - they are
called config sections!

> >> Additionally, we would like to thank Ted for listing several features he uses in
> >> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
> >> reducing the burden of maintaining custom test scripts and wrappers around
> >> xfstests, we can also discuss essential features that could be integrated
> >> directly into xfstests, whether from Ted's list or suggestions from others.
> >
> > On of my goals with check-parallel is to completely remove the need
> > for end users to configure fstests. i.e. all you need to do is point
> > it at a directory, tell it which filesystem to test, and it "just
> > runs" with all the defaults that come direct from the fstests
> > repository...
> >
> 
> Right. Centralizing fsconfigs & device configs is also doing the same.
> In fact once we have configs/devices/loop config file, then we don't
> need to even create local.config file (for most cases I believe).
> 
> ./check and ./check_parallel can be passed these config files and
> xfstests infra will create loop devices, run the tests and later do the
> cleanups.

Please, no.

check-parallel already provides the external zero-conf
infrastructure that check needs to run without config. It solves the
chicken/egg problems that check itself has (e.g. how do you specify
the location of the loop device image files it needs to create for
zero-conf), and it does it without needing check or it's internal
infrastructure to be modified in any way.

I've already outlined how we can use check-parallel to run check in
it's traditional serial form, so there is absolutely no need to try
to make check jump through hoops to do zero-conf internally.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

Hi Dave,

Sorry about the delayed response on this. We gave some thoughts to this
and we believe, we could come to an agreement if we could extend the
current section approach itself. 

Dave Chinner <david@fromorbit.com> writes:

> On Tue, Feb 04, 2025 at 12:44:00PM +0530, Ritesh Harjani wrote:
>> Dave Chinner <david@fromorbit.com> writes:
>> 
>> > On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
>> >> Greetings,
>> >> 
>> >> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
>> >> a proposal on centralizing filesystem and device configurations within xfstests
>> >> and maybe a further discussion on some of the open ideas listed by Ted here [3].
>> >> More details are mentioned below.
>> >> 
>> >> ** Background ** 
>> >> There was a discussion last year at LSFMM [1] about creating a central fs-config
>> >> store, that can then be used by anyone for testing different FS
>> >> features/configurations. This can also bring an awareness among other developers
>> >> and testers on what is being actively maintained by FS maintainers. We recently
>> >> posted an RFC [2] for centralizing filesystem configuration which is under
>> >> review. The next step we are considering is to centralize device configurations
>> >> within xfstests itself. In line with this, Ted also suggested a similar idea (in
>> >> point A) [3], where he proposed specifying the device size for the TEST and
>> >> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
>> >> improve the overall runtime of xfstests.
>> >> 
>> >> Recently Dave introduced a feature [4] to run the xfs and generic tests in
>> >> parallel. This patch creates the TEST and SCRATCH devices at runtime without
>> >> requiring them to be specified in any config file. However, at this stage, the
>> >> automatic device initialization appears to be somewhat limited. We believe that
>> >> centralizing device configuration could help enhance this functionality as well.
>> >
>> > Right, the point of check-parallel is to take away the need to
>> > specify devices completely.  I've already added support for the
>> > LOGWRITES_DEV, and I'm in the process of adding LOGDEV and RTDEV
>> > support for both test and scratch devices. At this point, the need
>> > for actual actual device specification in the config files goes
>> > away.
>> >
>> > What I am expecting to need is a set of fields that specify the
>> > *size* of the devices so that the hard-coded image file sizes in
>> > the check-parallel script go away.
>> >
>> > From there, I intend to have check-parallel iterate config file run
>> > sections itself, rather than have it run them internally to check.
>> > That way check is only ever invoked by check-parallel with all the
>> > devices completely set up.
>> 
>> Yes, this sounds good. This is what we were anticipating too.
>> Thanks for sharing.
>> 
>> >
>> > Hence a typical host independent config file would look like:
>> 
>> The work being proposed by us here was to make this config file
>> centralized within xfstests itself for both fsconfig and device-config.
>> This saves us from defining each of this section within local.config file
>> and can be used by passing cmdling arguments to invoke a given section
>> directly. 
>> 
>> e.g.
>> 
>>     ./check -c configs/xfs/64k -g auto
>
> This strikes me as re-implementing config sections with a different
> file format.
>
>> There have been cases where testers and others have requested info to
>> know about - 
>> - What different FS config options to test,
>> - What gets tested by the Maintainers,  
>> - Is there a common place where I can find MKFS and MOUNT options which
>>   I should be testing for my FS/feature testing. 
>
> Those are questions that documentation should answer - they are not
> a reason for changing config file formats.
>
>> That is the reason, I think, centralizing fsconfig option can be
>> helpful. I remember bringing this idea in our last LSFMM-2024, where you
>> mentioned that - let's see the RFC [1] and maybe then we can discuss more :).
>> Here is the RFC for the same. There are some additional improvements in
>> that series, but it mainly adds fsconfig option.
>> 
>> [1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/
>
> Yeah, that's just making a tiny config file per config section, and
> having to add a heap of parsing code to do that.
>
> I find it much easier to manage a single config file where each
> config is maybe only one or two lines than it is to have to find the
> needle in a haystack of hundreds of tiny files that are almost all
> alike.
>
>> > TEST_DEV_SIZE=10g
>> > TEST_RTDEV_size=10g
>> > TEST_LOGDEV_SIZE=128m
>> > SCRATCH_DEV_SIZE=20g
>> > SCRATCH_RTDEV_size=20g
>> > SCRATCH_LOGDEV_SIZE=512m
>> > LOGWRITES_DEV_SIZE=2g
>> >
>> 
>> For centralizing device-configs idea, I was hoping we could mention above
>> in "configs/devices/loop" config file. This can be picked up by ./check
>> too if local.config hasn't been provided with these options.
>
> Great. How do you specify where the image files are going to be
> hosted when no devices have been configured?
>
> These sorts of zero-conf chicken/egg problems have already been
> solved by check-parallel - I'm really not convinced that we need
> to make check itself try to solve these. I designed check-parallel
> the way I did because making check itself solve these problems is
> .... convoluted and difficult.
>
> If we need zero-conf, just use ./check-parallel....
>
>> > [xfs]
>> > FSTYP=xfs
>> > MKFS_OPTIONS="-b size=4k"
>> > TEST_FS_MOUNT_OPTIONS=
>> > MOUNT_OPTIONS=
>> > USE_EXTERNAL=
>> >
>> > [xfs-rmapbt]
>> > MKFS_OPTIONS="-b size=4k -m rmapbt=1"
>> >
>> > [xfs-noreflink]
>> > MKFS_OPTIONS="-b size=4k -m reflink=0"
>> >
>> > [xfs-n64k]
>> > MKFS_OPTIONS="-b size=4k -n size=64k"
>> >
>> > [xfs-ext]
>> > MKFS_OPTIONS="-b size=4k"
>> > USE_EXTERNAL=yes
>> >
>> > [ext4]
>> > FSTYP="ext4"
>> > MKFS_OPTIONS=
>> > USE_EXTERNAL=
>> >
>> > [btrfs]
>> > FSTYP="btrfs"
>> > .....
>> 
>> Above all fs configs could be added to configs/{ext4|xfs|btrfs}/... 
>> Than this can be used in 2 ways.. 
>> 
>> 1. ./check -c configs/xfs/4k,configs/xfs/rmapbt,configs/ext4/4k ... 
>
> Can't say I like using relative paths to specify the config we
> should use. It's not an interface I'd choose over:
>
> 	./check -s xfs -g auto
>
> Especially as the existing config section implementation does
> exactly the same thing as the proposed config file farm....
>
>> 2. Or we may still pass fsconfig via local.config file.. e.g. 
>> 
>> # both configs/xfs/4k or xfs/4k can be used here
>> [xfs]
>> FS_CONFIG_OPTION=configs/xfs/4k
>> 
>> [xfs-rmapbt]
>> FS_CONFIG_OPTION=xfs/rmapbt
>> 
>> [xfs-noreflink]
>> FS_CONFIG_OPTION=xfs/noreflink
>> 
>> [xfs-n64k]
>> FS_CONFIG_OPTION=xfs/64k
>> 
>> [xfs-ext]
>> FS_CONFIG_OPTION=xfs/4k
>> USE_EXTERNAL=yes
>> 
>> [ext4]
>> FS_CONFIG_OPTION=ext4/4k
>
> How is that an improvement on the section setup we have right now?
> Abstracting the config section options into a separate file is the
> worst of both worlds - now I have to look up the section to find the
> file and the section options, then lookup the file to see the
> options that file contains.
>
> It's like you've decided that the solution to centralised management
> of configs must be "one config per file", and so every other way of
> managing options needs to be reframed for that solution....
>
>> > IOWs, all that is different from system to system is the device size
>> > setup. The actual config sections under test (e.g. [xfs]) never need
>> > to change from host to host, nor environment to environment. i.e.
>> > "xfs-n64k" runs the same config filesystem test on every system,
>> > everywhere...
>> 
>> Right. So it's also useful if those configs can stay in configs/<fs>/**
>> as well.
>
> Why is having hundreds of tiny single-config-only files
> better than having all the configs in a single file that is
> easily browsed and searched?
>
> Honestly, I really don't see any advantage to re-implementing config
> sections as a "file per config" object farm. Yes, you can store
> information that way, but that doesn't make it an improvement over a
> single file...
>
> All that is needed is for the upstream repository to maintain a
> config file with all the config sections defined that people need.
> We don't need any new infrastructure to implement a "centralised
> configs" feature - all we need is an agreement that upstream will
> ship an update-to-date default config file instead of the ancient,
> stale example.config/localhost.config files....
>

If we can create 1 config for every filesystem instead of creating a lot
of smaller config files. i.e.  
- configs/ext4/config.ext4
- configs/xfs/config.xfs

Each of above can contain sections like (e.g.)

[xfs-b4k]
MKFS_OPTIONS="-b size=4k"
ddUNT_OPTIdd    d=""dd

[xfs-b64k]
MKFS_OPTIONS="-b size=64k"
MOUNT_OPTIONS=""


Then during make we can merge all these configs into a common config file
i.e. configs/.all-section-configs. We can update the current check script to
look for either local.config file or configs/.all-section-configs file
for location the section passed in the command line. 

This will help solve all the listed problems:
1. We don't have to add a new parsing logic for configs
2. We don't need to create 1 file per config
3. We still can get all sections listed in one place under which check
script can parse.
4. Calling different filesystem sections from a common config file can work.

So as you mentioned calling something like below should work. 

./check -s xfs_4k -s ext4_4k -g quick

Hopefully this will require minimal changes to work. Does this sound
good to you?

It's better if we discuss device config later once the central config
feature design is agreed by then. 

Thanks a lot for your review!
-ritesh



>> > I don't really see a need for a new centralised config setup. With
>> > the above, we can acheived a "zero-config" goal with the existing
>> > config file formats and infrastructure. All that we need to do is
>> > update the default config file in the repo to contain a section for
>> > each of the "standard" test configs we want to define....
>> 
>> Could you please take look at the shared RFC [1] once? I believe it will
>> be useful to have this central to xfstests for reasons I mentioned
>> above. This also gets us to zero-config setup with almost no need to
>> configure anything.
>
> [1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/
>
> There's nothing in that which provides zero-config. It still
> requires all the devices to be specified by the user...
>
>> This helps other testers & subsystem maintainers to
>> know what configs are being tested by maintainers, which they can use
>> for their FS feature testing too.
>> We can than directly issue -
>> 
>>            ./check -c <fs>/<config> -g quick
>
> As I've already said: this functionality already exists - they are
> called config sections!
>
>> >> Additionally, we would like to thank Ted for listing several features he uses in
>> >> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
>> >> reducing the burden of maintaining custom test scripts and wrappers around
>> >> xfstests, we can also discuss essential features that could be integrated
>> >> directly into xfstests, whether from Ted's list or suggestions from others.
>> >
>> > On of my goals with check-parallel is to completely remove the need
>> > for end users to configure fstests. i.e. all you need to do is point
>> > it at a directory, tell it which filesystem to test, and it "just
>> > runs" with all the defaults that come direct from the fstests
>> > repository...
>> >
>> 
>> Right. Centralizing fsconfigs & device configs is also doing the same.
>> In fact once we have configs/devices/loop config file, then we don't
>> need to even create local.config file (for most cases I believe).
>> 
>> ./check and ./check_parallel can be passed these config files and
>> xfstests infra will create loop devices, run the tests and later do the
>> cleanups.
>
> Please, no.
>
> check-parallel already provides the external zero-conf
> infrastructure that check needs to run without config. It solves the
> chicken/egg problems that check itself has (e.g. how do you specify
> the location of the loop device image files it needs to create for
> zero-conf), and it does it without needing check or it's internal
> infrastructure to be modified in any way.
>
> I've already outlined how we can use check-parallel to run check in
> it's traditional serial form, so there is absolutely no need to try
> to make check jump through hoops to do zero-conf internally.
>
> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

Ritesh Harjani (IBM) <ritesh.list@gmail.com> writes:

> Hi Dave,
>
> Sorry about the delayed response on this. We gave some thoughts to this
> and we believe, we could come to an agreement if we could extend the
> current section approach itself. 
>
> Dave Chinner <david@fromorbit.com> writes:
>
>> On Tue, Feb 04, 2025 at 12:44:00PM +0530, Ritesh Harjani wrote:
>>> Dave Chinner <david@fromorbit.com> writes:
>>> 
>>> > On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
>>> >> Greetings,
>>> >> 
>>> >> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
>>> >> a proposal on centralizing filesystem and device configurations within xfstests
>>> >> and maybe a further discussion on some of the open ideas listed by Ted here [3].
>>> >> More details are mentioned below.
>>> >> 
>>> >> ** Background ** 
>>> >> There was a discussion last year at LSFMM [1] about creating a central fs-config
>>> >> store, that can then be used by anyone for testing different FS
>>> >> features/configurations. This can also bring an awareness among other developers
>>> >> and testers on what is being actively maintained by FS maintainers. We recently
>>> >> posted an RFC [2] for centralizing filesystem configuration which is under
>>> >> review. The next step we are considering is to centralize device configurations
>>> >> within xfstests itself. In line with this, Ted also suggested a similar idea (in
>>> >> point A) [3], where he proposed specifying the device size for the TEST and
>>> >> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
>>> >> improve the overall runtime of xfstests.
>>> >> 
>>> >> Recently Dave introduced a feature [4] to run the xfs and generic tests in
>>> >> parallel. This patch creates the TEST and SCRATCH devices at runtime without
>>> >> requiring them to be specified in any config file. However, at this stage, the
>>> >> automatic device initialization appears to be somewhat limited. We believe that
>>> >> centralizing device configuration could help enhance this functionality as well.
>>> >
>>> > Right, the point of check-parallel is to take away the need to
>>> > specify devices completely.  I've already added support for the
>>> > LOGWRITES_DEV, and I'm in the process of adding LOGDEV and RTDEV
>>> > support for both test and scratch devices. At this point, the need
>>> > for actual actual device specification in the config files goes
>>> > away.
>>> >
>>> > What I am expecting to need is a set of fields that specify the
>>> > *size* of the devices so that the hard-coded image file sizes in
>>> > the check-parallel script go away.
>>> >
>>> > From there, I intend to have check-parallel iterate config file run
>>> > sections itself, rather than have it run them internally to check.
>>> > That way check is only ever invoked by check-parallel with all the
>>> > devices completely set up.
>>> 
>>> Yes, this sounds good. This is what we were anticipating too.
>>> Thanks for sharing.
>>> 
>>> >
>>> > Hence a typical host independent config file would look like:
>>> 
>>> The work being proposed by us here was to make this config file
>>> centralized within xfstests itself for both fsconfig and device-config.
>>> This saves us from defining each of this section within local.config file
>>> and can be used by passing cmdling arguments to invoke a given section
>>> directly. 
>>> 
>>> e.g.
>>> 
>>>     ./check -c configs/xfs/64k -g auto
>>
>> This strikes me as re-implementing config sections with a different
>> file format.
>>
>>> There have been cases where testers and others have requested info to
>>> know about - 
>>> - What different FS config options to test,
>>> - What gets tested by the Maintainers,  
>>> - Is there a common place where I can find MKFS and MOUNT options which
>>>   I should be testing for my FS/feature testing. 
>>
>> Those are questions that documentation should answer - they are not
>> a reason for changing config file formats.
>>
>>> That is the reason, I think, centralizing fsconfig option can be
>>> helpful. I remember bringing this idea in our last LSFMM-2024, where you
>>> mentioned that - let's see the RFC [1] and maybe then we can discuss more :).
>>> Here is the RFC for the same. There are some additional improvements in
>>> that series, but it mainly adds fsconfig option.
>>> 
>>> [1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/
>>
>> Yeah, that's just making a tiny config file per config section, and
>> having to add a heap of parsing code to do that.
>>
>> I find it much easier to manage a single config file where each
>> config is maybe only one or two lines than it is to have to find the
>> needle in a haystack of hundreds of tiny files that are almost all
>> alike.
>>
>>> > TEST_DEV_SIZE=10g
>>> > TEST_RTDEV_size=10g
>>> > TEST_LOGDEV_SIZE=128m
>>> > SCRATCH_DEV_SIZE=20g
>>> > SCRATCH_RTDEV_size=20g
>>> > SCRATCH_LOGDEV_SIZE=512m
>>> > LOGWRITES_DEV_SIZE=2g
>>> >
>>> 
>>> For centralizing device-configs idea, I was hoping we could mention above
>>> in "configs/devices/loop" config file. This can be picked up by ./check
>>> too if local.config hasn't been provided with these options.
>>
>> Great. How do you specify where the image files are going to be
>> hosted when no devices have been configured?
>>
>> These sorts of zero-conf chicken/egg problems have already been
>> solved by check-parallel - I'm really not convinced that we need
>> to make check itself try to solve these. I designed check-parallel
>> the way I did because making check itself solve these problems is
>> .... convoluted and difficult.
>>
>> If we need zero-conf, just use ./check-parallel....
>>
>>> > [xfs]
>>> > FSTYP=xfs
>>> > MKFS_OPTIONS="-b size=4k"
>>> > TEST_FS_MOUNT_OPTIONS=
>>> > MOUNT_OPTIONS=
>>> > USE_EXTERNAL=
>>> >
>>> > [xfs-rmapbt]
>>> > MKFS_OPTIONS="-b size=4k -m rmapbt=1"
>>> >
>>> > [xfs-noreflink]
>>> > MKFS_OPTIONS="-b size=4k -m reflink=0"
>>> >
>>> > [xfs-n64k]
>>> > MKFS_OPTIONS="-b size=4k -n size=64k"
>>> >
>>> > [xfs-ext]
>>> > MKFS_OPTIONS="-b size=4k"
>>> > USE_EXTERNAL=yes
>>> >
>>> > [ext4]
>>> > FSTYP="ext4"
>>> > MKFS_OPTIONS=
>>> > USE_EXTERNAL=
>>> >
>>> > [btrfs]
>>> > FSTYP="btrfs"
>>> > .....
>>> 
>>> Above all fs configs could be added to configs/{ext4|xfs|btrfs}/... 
>>> Than this can be used in 2 ways.. 
>>> 
>>> 1. ./check -c configs/xfs/4k,configs/xfs/rmapbt,configs/ext4/4k ... 
>>
>> Can't say I like using relative paths to specify the config we
>> should use. It's not an interface I'd choose over:
>>
>> 	./check -s xfs -g auto
>>
>> Especially as the existing config section implementation does
>> exactly the same thing as the proposed config file farm....
>>
>>> 2. Or we may still pass fsconfig via local.config file.. e.g. 
>>> 
>>> # both configs/xfs/4k or xfs/4k can be used here
>>> [xfs]
>>> FS_CONFIG_OPTION=configs/xfs/4k
>>> 
>>> [xfs-rmapbt]
>>> FS_CONFIG_OPTION=xfs/rmapbt
>>> 
>>> [xfs-noreflink]
>>> FS_CONFIG_OPTION=xfs/noreflink
>>> 
>>> [xfs-n64k]
>>> FS_CONFIG_OPTION=xfs/64k
>>> 
>>> [xfs-ext]
>>> FS_CONFIG_OPTION=xfs/4k
>>> USE_EXTERNAL=yes
>>> 
>>> [ext4]
>>> FS_CONFIG_OPTION=ext4/4k
>>
>> How is that an improvement on the section setup we have right now?
>> Abstracting the config section options into a separate file is the
>> worst of both worlds - now I have to look up the section to find the
>> file and the section options, then lookup the file to see the
>> options that file contains.
>>
>> It's like you've decided that the solution to centralised management
>> of configs must be "one config per file", and so every other way of
>> managing options needs to be reframed for that solution....
>>
>>> > IOWs, all that is different from system to system is the device size
>>> > setup. The actual config sections under test (e.g. [xfs]) never need
>>> > to change from host to host, nor environment to environment. i.e.
>>> > "xfs-n64k" runs the same config filesystem test on every system,
>>> > everywhere...
>>> 
>>> Right. So it's also useful if those configs can stay in configs/<fs>/**
>>> as well.
>>
>> Why is having hundreds of tiny single-config-only files
>> better than having all the configs in a single file that is
>> easily browsed and searched?
>>
>> Honestly, I really don't see any advantage to re-implementing config
>> sections as a "file per config" object farm. Yes, you can store
>> information that way, but that doesn't make it an improvement over a
>> single file...
>>
>> All that is needed is for the upstream repository to maintain a
>> config file with all the config sections defined that people need.
>> We don't need any new infrastructure to implement a "centralised
>> configs" feature - all we need is an agreement that upstream will
>> ship an update-to-date default config file instead of the ancient,
>> stale example.config/localhost.config files....
>>
>
> If we can create 1 config for every filesystem instead of creating a lot
> of smaller config files. i.e.  
> - configs/ext4/config.ext4
> - configs/xfs/config.xfs
>
> Each of above can contain sections like (e.g.)
>
> [xfs-b4k]
> MKFS_OPTIONS="-b size=4k"
> ddUNT_OPTIdd    d=""dd

Sorry as you could see my key bindings got all messed up while trying to
delete this above line using "dd" and the mail got sent while trying to
fix it up. :(

However the previous email still has the same gist of what I wanted to
add. Please let me know your thoughts on using the section configs
approach for centralizing the filesystems configs.

I also wanted to add the reason for this work. Most of the wrappers if
we see anyways define extra configs for their testing e.g.

1. https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs
2. https://github.com/avocado-framework-tests/avocado-misc-tests/tree/master/fs/xfstests.py.data/ext4
3. https://github.com/linux-kdevops/kdevops/tree/4bffd17ac6a3ac757a26897a75ffa185d623a8e4/playbooks/roles/fstests/templates


We believe it would be easier if the default filesystem configs which
are of interest to FS maintainers, were added directly to the xfstests
repo. Additionally, it would also help if fstests could provide a way to
invoke these different configs/sections for testing without requiring
them to be redefined or copied in the local.config file each time.

Thanks
-ritesh


>
> [xfs-b64k]
> MKFS_OPTIONS="-b size=64k"
> MOUNT_OPTIONS=""
>
>
> Then during make we can merge all these configs into a common config file
> i.e. configs/.all-section-configs. We can update the current check script to
> look for either local.config file or configs/.all-section-configs file
> for location the section passed in the command line. 
>
> This will help solve all the listed problems:
> 1. We don't have to add a new parsing logic for configs
> 2. We don't need to create 1 file per config
> 3. We still can get all sections listed in one place under which check
> script can parse.
> 4. Calling different filesystem sections from a common config file can work.
>
> So as you mentioned calling something like below should work. 
>
> ./check -s xfs_4k -s ext4_4k -g quick
>
> Hopefully this will require minimal changes to work. Does this sound
> good to you?
>
> It's better if we discuss device config later once the central config
> feature design is agreed by then. 
>
> Thanks a lot for your review!
> -ritesh
>
>
>
>>> > I don't really see a need for a new centralised config setup. With
>>> > the above, we can acheived a "zero-config" goal with the existing
>>> > config file formats and infrastructure. All that we need to do is
>>> > update the default config file in the repo to contain a section for
>>> > each of the "standard" test configs we want to define....
>>> 
>>> Could you please take look at the shared RFC [1] once? I believe it will
>>> be useful to have this central to xfstests for reasons I mentioned
>>> above. This also gets us to zero-config setup with almost no need to
>>> configure anything.
>>
>> [1]: https://lore.kernel.org/fstests/cover.1736496620.git.nirjhar.roy.lists@gmail.com/
>>
>> There's nothing in that which provides zero-config. It still
>> requires all the devices to be specified by the user...
>>
>>> This helps other testers & subsystem maintainers to
>>> know what configs are being tested by maintainers, which they can use
>>> for their FS feature testing too.
>>> We can than directly issue -
>>> 
>>>            ./check -c <fs>/<config> -g quick
>>
>> As I've already said: this functionality already exists - they are
>> called config sections!
>>
>>> >> Additionally, we would like to thank Ted for listing several features he uses in
>>> >> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
>>> >> reducing the burden of maintaining custom test scripts and wrappers around
>>> >> xfstests, we can also discuss essential features that could be integrated
>>> >> directly into xfstests, whether from Ted's list or suggestions from others.
>>> >
>>> > On of my goals with check-parallel is to completely remove the need
>>> > for end users to configure fstests. i.e. all you need to do is point
>>> > it at a directory, tell it which filesystem to test, and it "just
>>> > runs" with all the defaults that come direct from the fstests
>>> > repository...
>>> >
>>> 
>>> Right. Centralizing fsconfigs & device configs is also doing the same.
>>> In fact once we have configs/devices/loop config file, then we don't
>>> need to even create local.config file (for most cases I believe).
>>> 
>>> ./check and ./check_parallel can be passed these config files and
>>> xfstests infra will create loop devices, run the tests and later do the
>>> cleanups.
>>
>> Please, no.
>>
>> check-parallel already provides the external zero-conf
>> infrastructure that check needs to run without config. It solves the
>> chicken/egg problems that check itself has (e.g. how do you specify
>> the location of the loop device image files it needs to create for
>> zero-conf), and it does it without needing check or it's internal
>> infrastructure to be modified in any way.
>>
>> I've already outlined how we can use check-parallel to run check in
>> it's traditional serial form, so there is absolutely no need to try
>> to make check jump through hoops to do zero-conf internally.
>>
>> -Dave.
>> -- 
>> Dave Chinner
>> david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Dave Chinner]

On Sat, Mar 01, 2025 at 06:39:57PM +0530, Ritesh Harjani wrote:
> > Why is having hundreds of tiny single-config-only files
> > better than having all the configs in a single file that is
> > easily browsed and searched?
> >
> > Honestly, I really don't see any advantage to re-implementing config
> > sections as a "file per config" object farm. Yes, you can store
> > information that way, but that doesn't make it an improvement over a
> > single file...
> >
> > All that is needed is for the upstream repository to maintain a
> > config file with all the config sections defined that people need.
> > We don't need any new infrastructure to implement a "centralised
> > configs" feature - all we need is an agreement that upstream will
> > ship an update-to-date default config file instead of the ancient,
> > stale example.config/localhost.config files....
> >
> 
> If we can create 1 config for every filesystem instead of creating a lot
> of smaller config files. i.e.  
> - configs/ext4/config.ext4
> - configs/xfs/config.xfs

Why are directories that contain a single file needed here?

> Each of above can contain sections like (e.g.)
> 
> [xfs-b4k]
> MKFS_OPTIONS="-b size=4k"
> ddUNT_OPTIdd    d=""dd
> 
> [xfs-b64k]
> MKFS_OPTIONS="-b size=64k"
> MOUNT_OPTIONS=""
> 
> 
> Then during make we can merge all these configs into a common config file
> i.e. configs/.all-section-configs. We can update the current check script to
> look for either local.config file or configs/.all-section-configs file
> for location the section passed in the command line. 

What does this complexity gain us?

> This will help solve all the listed problems:
> 1. We don't have to add a new parsing logic for configs

We don't need new config files and makefile/build time shenanigans
to do this.

> 2. We don't need to create 1 file per config

Ditto.

> 3. We still can get all sections listed in one place under which check
> script can parse.

Ditto.

> 4. Calling different filesystem sections from a common config file can work.

Yes, that's the whole point of have config sections: one config file
that supports lots of different test configurations!

> So as you mentioned calling something like below should work. 
> 
> ./check -s xfs_4k -s ext4_4k -g quick
> 
> Hopefully this will require minimal changes to work. Does this sound
> good to you?

You haven't explained why we need new infrastructure to do something
we can already do with the existing infrastructure. What problem are
you trying to solve that the current infrastructure does not handle?

i.e. we won't need to change the global config file very often once the
common configs are defined in it; it'll only get modified when
filesystems add new features that need specific mkfs or mount option
support to be added, and that's fairly rare.

Hence I still don't understand what new problem multiple config files
and new infrastructure to support them is supposed to solve...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

Dave Chinner <david@fromorbit.com> writes:

> On Sat, Mar 01, 2025 at 06:39:57PM +0530, Ritesh Harjani wrote:
>> > Why is having hundreds of tiny single-config-only files
>> > better than having all the configs in a single file that is
>> > easily browsed and searched?
>> >
>> > Honestly, I really don't see any advantage to re-implementing config
>> > sections as a "file per config" object farm. Yes, you can store
>> > information that way, but that doesn't make it an improvement over a
>> > single file...
>> >
>> > All that is needed is for the upstream repository to maintain a
>> > config file with all the config sections defined that people need.
>> > We don't need any new infrastructure to implement a "centralised
>> > configs" feature - all we need is an agreement that upstream will
>> > ship an update-to-date default config file instead of the ancient,
>> > stale example.config/localhost.config files....
>> >
>> 
>> If we can create 1 config for every filesystem instead of creating a lot
>> of smaller config files. i.e.  
>> - configs/ext4/config.ext4
>> - configs/xfs/config.xfs
>
> Why are directories that contain a single file needed here?
>

Agreed. Not needed we can have configs/ext4.config, configs/xfs.config
and so on...

>> Each of above can contain sections like (e.g.)
>> 
>> [xfs-b4k]
>> MKFS_OPTIONS="-b size=4k"
>> ddUNT_OPTIdd    d=""dd
>> 
>> [xfs-b64k]
>> MKFS_OPTIONS="-b size=64k"
>> MOUNT_OPTIONS=""
>> 
>> 
>> Then during make we can merge all these configs into a common config file
>> i.e. configs/.all-section-configs. We can update the current check script to
>> look for either local.config file or configs/.all-section-configs file
>> for location the section passed in the command line. 
>
> What does this complexity gain us?
>

Let me try and explain this below.


>> This will help solve all the listed problems:
>> 1. We don't have to add a new parsing logic for configs
>
> We don't need new config files and makefile/build time shenanigans
> to do this.
>

Same here.

>> 2. We don't need to create 1 file per config
>
> Ditto.
>
>> 3. We still can get all sections listed in one place under which check
>> script can parse.
>
> Ditto.
>
>> 4. Calling different filesystem sections from a common config file can work.
>
> Yes, that's the whole point of have config sections: one config file
> that supports lots of different test configurations!
>
>> So as you mentioned calling something like below should work. 
>> 
>> ./check -s xfs_4k -s ext4_4k -g quick
>> 
>> Hopefully this will require minimal changes to work. Does this sound
>> good to you?
>
> You haven't explained why we need new infrastructure to do something
> we can already do with the existing infrastructure. What problem are
> you trying to solve that the current infrastructure does not handle?
>
> i.e. we won't need to change the global config file very often once the
> common configs are defined in it; it'll only get modified when
> filesystems add new features that need specific mkfs or mount option
> support to be added, and that's fairly rare.
>
> Hence I still don't understand what new problem multiple config files
> and new infrastructure to support them is supposed to solve...


I will try and explain our reasoning here: 

1. Why have per-fs config file i.e. configs/ext4.config or 
configs/xfs.config...

Instead of 1 large config file it's easier if we have FS specific
sections in their own .config file. I agree we don't need configs/<fs>
directories for each filesystem. But it's much easier if we have
configs/<fs>.config with the necessary sections defined in it.  That
will be easy to maintain by their respective FS maintainers rather than
maintaining all sections defined in 1 large common config file.

2. Why then add the infrastructure to create a new common
configs/all-fs.config file during make?

This is a combined configs/all-fs.config file which need not be
maintained in git version control. It gets generated for our direct
use. This is also needed to run different cross filesystem tests from a
single ./check script. i.e. 

        ./check -s ext4_4k -s xfs_4k -g quick

(otherwise one cannot run ext4_4k and xfs_4k from a single ./check invocation)

I don't think this is too much burden for "make" to generate this file.
And it's easier than, for people to use configs/all-fs.config to run
cross filesystem tests (as mentioned above).

e.g. 
1. "make" will generate configs/all-fs.config
2. Define your devices.config in configs/devices.config
3. Then run 
   (. configs/devices.config; ./check -s ext4_4k -s xfs_4k -g quick)

Note:
One problem which I can think of with the sections approach as opposed
to multiple small config files (-c configs/ext4/4k.config) approach is
that, in sections approach we iterate over all the sections present in
the provided config file and search for the section passed in the
cmdline. Whereas in the small config files approach we were directly
passing the configuration options via a small config file. But hopefully
that linear search in locating the section is not as time consuming as
opposed to running the test itself.


Here is the sample change snippet which we are thinking of.
Thoughts?


From c227f6335d09c2b482b759ac4df38d87ccedff6d Mon Sep 17 00:00:00 2001
Message-Id: <c227f6335d09c2b482b759ac4df38d87ccedff6d.1741201277.git.ritesh.list@gmail.com>
From: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Date: Wed, 5 Mar 2025 16:33:52 +0530
Subject: [RFC] configs: Add support for central fs configs using sections

This adds support for central fs configs using sections. This allows
different FS maintainers to maintain per-fs config file with different
sections to test multiple filesystem configurations like 1k, 4k, quota
etc.

During make we combine all configs/<fs>.config file into
configs/all-fs.config which is not tracked by git. check script will by
default use configs/all-fs.config file if local.config or
configs/$(HOST).config file is not present. This patch also adds
configs/devices.config.example which can be used as an example to export
default settings for SCRATCH_MNT, TEST_DIR, TEST_DEV, SCRATCH_DEV etc.

After defining default device settings say in configs/devices.config,
one can use:

(. configs/devices.config; ./check -s xfs_4k tests/selftest/001)

Note: make clean will clean the configs/all-fs.config file.

Co-developed-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
---
 Makefile                       |  3 ++-
 common/config                  |  1 +
 configs/Makefile               | 26 ++++++++++++++++++++++++++
 configs/btrfs.config           |  4 ++++
 configs/devices.config.example |  9 +++++++++
 configs/ext4.config            |  9 +++++++++
 configs/xfs.config             |  9 +++++++++
 7 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100644 configs/Makefile
 create mode 100644 configs/btrfs.config
 create mode 100644 configs/devices.config.example
 create mode 100644 configs/ext4.config
 create mode 100644 configs/xfs.config

diff --git a/Makefile b/Makefile
index f955f0d3..490bb59a 100644
--- a/Makefile
+++ b/Makefile
@@ -41,8 +41,9 @@ endif
 
 LIB_SUBDIRS = include lib
 TOOL_SUBDIRS = ltp src m4 common tools
+CONFIGS_SUBDIRS = configs
 
-SUBDIRS = $(LIB_SUBDIRS) $(TOOL_SUBDIRS) $(TESTS_DIR)
+SUBDIRS = $(LIB_SUBDIRS) $(TOOL_SUBDIRS) $(TESTS_DIR) $(CONFIGS_SUBDIRS)
 
 default: include/builddefs
 ifeq ($(HAVE_BUILDDEFS), no)
diff --git a/common/config b/common/config
index 2afbda14..b4470951 100644
--- a/common/config
+++ b/common/config
@@ -554,6 +554,7 @@ known_hosts()
 {
 	[ "$HOST_CONFIG_DIR" ] || HOST_CONFIG_DIR=`pwd`/configs
 
+	[ -f $HOST_CONFIG_DIR/all-fs.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/all-fs.config
 	[ -f /etc/xfsqa.config ]             && export HOST_OPTIONS=/etc/xfsqa.config
 	[ -f $HOST_CONFIG_DIR/$HOST ]        && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST
 	[ -f $HOST_CONFIG_DIR/$HOST.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST.config
diff --git a/configs/Makefile b/configs/Makefile
new file mode 100644
index 00000000..e91bcca1
--- /dev/null
+++ b/configs/Makefile
@@ -0,0 +1,26 @@
+TOPDIR = ..
+include $(TOPDIR)/include/builddefs
+
+CONFIGS_DIR = $(TOPDIR)/configs
+ALL_FS_CONFIG = $(CONFIGS_DIR)/all-fs.config
+EXT4_CONFIG = $(CONFIGS_DIR)/ext4.config
+XFS_CONFIG = $(CONFIGS_DIR)/xfs.config
+BTRFS_CONFIG = $(CONFIGS_DIR)/btrfs.config
+
+DIRT = $(ALL_FS_CONFIG)
+default: $(ALL_FS_CONFIG)
+
+include $(BUILDRULES)
+
+# Generate all-fs.config by concatenating required configs/.config files
+$(ALL_FS_CONFIG): $(EXT4_CONFIG) $(XFS_CONFIG) $(BTRFS_CONFIG)
+	@echo "Generating $@"
+	echo "# all-fs sections" > $@
+	echo "[default]" >> $@
+	@echo "" >> $@
+	@cat $(EXT4_CONFIG) >> $@
+	@echo "" >> $@
+	@cat $(XFS_CONFIG) >> $@
+	@echo "" >> $@
+	@cat $(BTRFS_CONFIG) >> $@
+	@echo "" >> $@
diff --git a/configs/btrfs.config b/configs/btrfs.config
new file mode 100644
index 00000000..1d85e13d
--- /dev/null
+++ b/configs/btrfs.config
@@ -0,0 +1,4 @@
+[btrfs]
+ FSTYP=btrfs
+ MKFS_OPTIONS="-f "
+ MOUNT_OPTIONS=""
diff --git a/configs/devices.config.example b/configs/devices.config.example
new file mode 100644
index 00000000..48598a4b
--- /dev/null
+++ b/configs/devices.config.example
@@ -0,0 +1,9 @@
+export TEST_DIR=/mnt1/test
+export SCRATCH_MNT=/mnt1/scratch
+export TEST_DEV=/dev/loop0
+export SCRATCH_DEV=/dev/loop1
+export TEST_LOGDEV=/dev/loop2
+export SCRATCH_LOGDEV=/dev/loop3
+export TEST_RTDEV=/dev/loop4
+export SCRATCH_RTDEV=/dev/loop5
+export LOGWRITES_DEV=/dev/loop6
diff --git a/configs/ext4.config b/configs/ext4.config
new file mode 100644
index 00000000..d5ceba3c
--- /dev/null
+++ b/configs/ext4.config
@@ -0,0 +1,9 @@
+[ext4_1k]
+ FSTYP=ext4
+ MKFS_OPTIONS="-F -b 1024"
+ MOUNT_OPTIONS=""
+
+[ext4_4k]
+ FSTYP=ext4
+ MKFS_OPTIONS="-F -b 4096"
+ MOUNT_OPTIONS=""
diff --git a/configs/xfs.config b/configs/xfs.config
new file mode 100644
index 00000000..516f35f7
--- /dev/null
+++ b/configs/xfs.config
@@ -0,0 +1,9 @@
+[xfs_1k]
+ FSTYP=xfs
+ MKFS_OPTIONS="-f -bsize=1024"
+ MOUNT_OPTIONS=""
+
+[xfs_4k]
+ FSTYP=xfs
+ MKFS_OPTIONS="-f -bsize=4096"
+ MOUNT_OPTIONS=""
-- 
2.39.5

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Dave Chinner]

On Wed, Mar 05, 2025 at 09:13:32AM +0530, Ritesh Harjani wrote:
> Dave Chinner <david@fromorbit.com> writes:
> 
> > On Sat, Mar 01, 2025 at 06:39:57PM +0530, Ritesh Harjani wrote:
> >> > Why is having hundreds of tiny single-config-only files
> >> > better than having all the configs in a single file that is
> >> > easily browsed and searched?
> >> >
> >> > Honestly, I really don't see any advantage to re-implementing config
> >> > sections as a "file per config" object farm. Yes, you can store
> >> > information that way, but that doesn't make it an improvement over a
> >> > single file...
> >> >
> >> > All that is needed is for the upstream repository to maintain a
> >> > config file with all the config sections defined that people need.
> >> > We don't need any new infrastructure to implement a "centralised
> >> > configs" feature - all we need is an agreement that upstream will
> >> > ship an update-to-date default config file instead of the ancient,
> >> > stale example.config/localhost.config files....

.....

> > You haven't explained why we need new infrastructure to do something
> > we can already do with the existing infrastructure. What problem are
> > you trying to solve that the current infrastructure does not handle?
> >
> > i.e. we won't need to change the global config file very often once the
> > common configs are defined in it; it'll only get modified when
> > filesystems add new features that need specific mkfs or mount option
> > support to be added, and that's fairly rare.
> >
> > Hence I still don't understand what new problem multiple config files
> > and new infrastructure to support them is supposed to solve...
> 
> 
> I will try and explain our reasoning here: 
> 
> 1. Why have per-fs config file i.e. configs/ext4.config or 
> configs/xfs.config...
.....
> 2. Why then add the infrastructure to create a new common
> configs/all-fs.config file during make?
.....

These aren't problems that need to be solved. These are "solutions"
posed as a questions.

Let's look at 1):

> Instead of 1 large config file it's easier if we have FS specific
> sections in their own .config file.  I agree we don't need configs/<fs>
> directories for each filesystem. But it's much easier if we have
> configs/<fs>.config with the necessary sections defined in it.

I disagree with both these "it is easier" assertions.

That same argument was made for splitting up MAINTAINERS in the
kernel tree, which sees far more concurrent changes than a test
config file would in fstests. The "split files are easier to
use/maintain" argument wasn't persuasive there, and I don't really
see that this is any different. We just aren't going to have a lot
of change to common test configs once the initial set is defined
and committed...

> That
> will be easy to maintain by their respective FS maintainers rather than
> maintaining all sections defined in 1 large common config file.

Again, it is no more difficult to add a new section config for a new
btrfs config to a configs/default.config file than it is to add it
to configs/default-btrfs.config.

The config sections are already namespaced by naming convention
(i.e. ["FSTYP"-"config description"]), so the argument that we need
to add a config namespace to an already namespaced config setup
to make it "easier to manage" isn't convincing - it's a subjective
opinion.

I'm saying subjective analysis is insufficient justification for a
change, because the subjective analysis of the situation done by
different people can result in (and often does) completely opposed
stances. Both subjective opinions are as valid as each other, so the
only way to address the situation is to look at the technical merits
of the proposal. The requires all parties to understand the problem
that needs to be solved.

I still don't know what problem is solved by shipping lots of config
files and additional code, build infrastructure and CLI interfaces
to address.  I'm probably still missing something important, but I'm
not going to learn what that might be from subjective opinion
statements like "X will be easier if ...."

> This is a combined configs/all-fs.config file which need not be
> maintained in git version control. It gets generated for our direct
> use. This is also needed to run different cross filesystem tests from a
> single ./check script. i.e. 
> 
>         ./check -s ext4_4k -s xfs_4k -g quick
> 
> (otherwise one cannot run ext4_4k and xfs_4k from a single ./check invocation)

Well, yes, and therein lies the problem with this approach. Where do
custom configs go? Are you proposing that everyone with custom
configs will be forced to run or manage fstests in some new,
different way?

> I don't think this is too much burden for "make" to generate this file.
> And it's easier than, for people to use configs/all-fs.config to run
> cross filesystem tests (as mentioned above).
>
> e.g. 
> 1. "make" will generate configs/all-fs.config
> 2. Define your devices.config in configs/devices.config
> 3. Then run 
>    (. configs/devices.config; ./check -s ext4_4k -s xfs_4k -g quick)

<looks at code providec>

Yup, and now this is all ignored and doesn't work because the test
machine has a custom config setup in <hostname>.config and that
overrides using configs/all-fs.config.

That is not ideal.

Of course, we could add a "configs/local.configs" file for local
configs that get included via the make rule.

However, now we need both a per-machine configs/local.config to be
exist or be distributed at the fstests source code update time (i.e.
before build), as well as also needing an additional static
per-machine configs/devices.config to be defined before fstests is
run.

This is much more convoluted that setting up in
configs/<hostname>.config once at machine setup time and almost
never having to touch it again. The build time requirement also
makes it hard to install packaged fstests (e.g. in a rpm or deb)
because now there's a configure and build step needed after package
installation...

Part of the problem is that you've treated the fstests-provided
section definitions as exclusive w.r.t. local custom config
definitions.  i.e. We can't have both fstest defined sections and
custom sections at the same time.

This restriction essentially forces anyone with a custom config to
have to copy the built config file into their custom config file so
that they can run both fstests provided and custom configs in the
same test run.

That is not ideal.

Maybe this is an oversight, but I still don't know what problem you
are trying to solve and so I can't make any judgement on whether it
is a simple mistake or intended behaviour...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Darrick J. Wong]

On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
> Greetings,
> 
> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
> a proposal on centralizing filesystem and device configurations within xfstests
> and maybe a further discussion on some of the open ideas listed by Ted here [3].
> More details are mentioned below.
> 
> ** Background ** 
> There was a discussion last year at LSFMM [1] about creating a central fs-config
> store, that can then be used by anyone for testing different FS
> features/configurations. This can also bring an awareness among other developers
> and testers on what is being actively maintained by FS maintainers. We recently
> posted an RFC [2] for centralizing filesystem configuration which is under
> review. The next step we are considering is to centralize device configurations
> within xfstests itself. In line with this, Ted also suggested a similar idea (in
> point A) [3], where he proposed specifying the device size for the TEST and
> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
> improve the overall runtime of xfstests.
> 
> Recently Dave introduced a feature [4] to run the xfs and generic tests in
> parallel. This patch creates the TEST and SCRATCH devices at runtime without
> requiring them to be specified in any config file. However, at this stage, the
> automatic device initialization appears to be somewhat limited. We believe that
> centralizing device configuration could help enhance this functionality as well.
> 
> ** Proposal ** 
> We would like to propose a discussion at LSFMM on two key features: central
> fsconfig and central device-config within xfstests. We can explore how the
> fsconfig feature can be utilized, and by then, we aim to have a PoC for central
> device-config feature, which we think can also be discussed in more detail. At
> this point, we are hoping to get a PoC working with loop devices by default. It
> will be good to hear from other developers, maintainers, and testers about their
> thoughts and suggestions on these two features.
> 
> Additionally, we would like to thank Ted for listing several features he uses in
> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
> reducing the burden of maintaining custom test scripts and wrappers around
> xfstests, we can also discuss essential features that could be integrated
> directly into xfstests, whether from Ted's list or suggestions from others.
> 
> Thoughts and suggestions are welcome.

Considering all the questions downthread, I'm wondering, are you just
going to stuff all the known configs into a single configs/default file
and then modify known_hosts() to set HOST_OPTIONS to that?

	[ -f /etc/xfsqa.config ]             && export HOST_OPTIONS=/etc/xfsqa.config
	[ -f $HOST_CONFIG_DIR/default ]      && export HOST_OPTIONS=$HOST_CONFIG_DIR/default
	[ -f $HOST_CONFIG_DIR/$HOST ]        && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST
	[ -f $HOST_CONFIG_DIR/$HOST.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST.config

Then configs/default contains things like:

[xfs_nocrc]
MKFS_OPTIONS="-m crc=0"

Would that work for running configurations in this manner:

	./check -s xfs_nocrc -g all

?

(I am completely ignorant of config files and never use them.)

--D

> 
> ** References **
> [1] https://lore.kernel.org/all/87h6h4sopf.fsf@doe.com/
> [2] https://lore.kernel.org/all/9a6764237b900f40e563d8dee2853f1430245b74.1736496620.git.nirjhar.roy.lists@gmail.com/
> [3] https://lore.kernel.org/all/20250110163859.GB1514771@mit.edu/
> [4] https://lore.kernel.org/all/20241127045403.3665299-1-david@fromorbit.com/
> 

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

Dave Chinner <david@fromorbit.com> writes:

> On Wed, Mar 05, 2025 at 09:13:32AM +0530, Ritesh Harjani wrote:
>> Dave Chinner <david@fromorbit.com> writes:
>> 
>> > On Sat, Mar 01, 2025 at 06:39:57PM +0530, Ritesh Harjani wrote:
>> >> > Why is having hundreds of tiny single-config-only files
>> >> > better than having all the configs in a single file that is
>> >> > easily browsed and searched?
>> >> >
>> >> > Honestly, I really don't see any advantage to re-implementing config
>> >> > sections as a "file per config" object farm. Yes, you can store
>> >> > information that way, but that doesn't make it an improvement over a
>> >> > single file...
>> >> >
>> >> > All that is needed is for the upstream repository to maintain a
>> >> > config file with all the config sections defined that people need.
>> >> > We don't need any new infrastructure to implement a "centralised
>> >> > configs" feature - all we need is an agreement that upstream will
>> >> > ship an update-to-date default config file instead of the ancient,
>> >> > stale example.config/localhost.config files....
>
> .....
>
>> > You haven't explained why we need new infrastructure to do something
>> > we can already do with the existing infrastructure. What problem are
>> > you trying to solve that the current infrastructure does not handle?
>> >
>> > i.e. we won't need to change the global config file very often once the
>> > common configs are defined in it; it'll only get modified when
>> > filesystems add new features that need specific mkfs or mount option
>> > support to be added, and that's fairly rare.
>> >
>> > Hence I still don't understand what new problem multiple config files
>> > and new infrastructure to support them is supposed to solve...
>> 
>> 
>> I will try and explain our reasoning here: 
>> 
>> 1. Why have per-fs config file i.e. configs/ext4.config or 
>> configs/xfs.config...
> .....
>> 2. Why then add the infrastructure to create a new common
>> configs/all-fs.config file during make?
> .....
>
> These aren't problems that need to be solved. These are "solutions"
> posed as a questions.
>
> Let's look at 1):
>
>> Instead of 1 large config file it's easier if we have FS specific
>> sections in their own .config file.  I agree we don't need configs/<fs>
>> directories for each filesystem. But it's much easier if we have
>> configs/<fs>.config with the necessary sections defined in it.
>
> I disagree with both these "it is easier" assertions.
>
> That same argument was made for splitting up MAINTAINERS in the
> kernel tree, which sees far more concurrent changes than a test
> config file would in fstests. The "split files are easier to
> use/maintain" argument wasn't persuasive there, and I don't really
> see that this is any different. We just aren't going to have a lot
> of change to common test configs once the initial set is defined
> and committed...
>

Ok. 1 central config file for all fs sections then.
Not really fond of the idea, but I see your point. Since there isn't
going to be much of the modifications to this, maybe 1 file should do.

>> That
>> will be easy to maintain by their respective FS maintainers rather than
>> maintaining all sections defined in 1 large common config file.
>
> Again, it is no more difficult to add a new section config for a new
> btrfs config to a configs/default.config file than it is to add it
> to configs/default-btrfs.config.
>
> The config sections are already namespaced by naming convention
> (i.e. ["FSTYP"-"config description"]), so the argument that we need
> to add a config namespace to an already namespaced config setup
> to make it "easier to manage" isn't convincing - it's a subjective
> opinion.
>
> I'm saying subjective analysis is insufficient justification for a
> change, because the subjective analysis of the situation done by
> different people can result in (and often does) completely opposed
> stances. Both subjective opinions are as valid as each other, so the
> only way to address the situation is to look at the technical merits
> of the proposal. The requires all parties to understand the problem
> that needs to be solved.
>
> I still don't know what problem is solved by shipping lots of config
> files and additional code, build infrastructure and CLI interfaces
> to address.  I'm probably still missing something important, but I'm
> not going to learn what that might be from subjective opinion
> statements like "X will be easier if ...."
>
>> This is a combined configs/all-fs.config file which need not be
>> maintained in git version control. It gets generated for our direct
>> use. This is also needed to run different cross filesystem tests from a
>> single ./check script. i.e. 
>> 
>>         ./check -s ext4_4k -s xfs_4k -g quick
>> 
>> (otherwise one cannot run ext4_4k and xfs_4k from a single ./check invocation)
>
> Well, yes, and therein lies the problem with this approach. Where do
> custom configs go? Are you proposing that everyone with custom
> configs will be forced to run or manage fstests in some new,
> different way?
>
>> I don't think this is too much burden for "make" to generate this file.
>> And it's easier than, for people to use configs/all-fs.config to run
>> cross filesystem tests (as mentioned above).
>>
>> e.g. 
>> 1. "make" will generate configs/all-fs.config
>> 2. Define your devices.config in configs/devices.config
>> 3. Then run 
>>    (. configs/devices.config; ./check -s ext4_4k -s xfs_4k -g quick)
>
> <looks at code providec>
>
> Yup, and now this is all ignored and doesn't work because the test
> machine has a custom config setup in <hostname>.config and that
> overrides using configs/all-fs.config.
>
> That is not ideal.

That was intentionally put to not break any of the existing users custom
config setup.

>
> Of course, we could add a "configs/local.configs" file for local
> configs that get included via the make rule.
>
> However, now we need both a per-machine configs/local.config to be
> exist or be distributed at the fstests source code update time (i.e.
> before build), as well as also needing an additional static
> per-machine configs/devices.config to be defined before fstests is
> run.
>
> This is much more convoluted that setting up in
> configs/<hostname>.config once at machine setup time and almost
> never having to touch it again. The build time requirement also
> makes it hard to install packaged fstests (e.g. in a rpm or deb)
> because now there's a configure and build step needed after package
> installation...
>
> Part of the problem is that you've treated the fstests-provided
> section definitions as exclusive w.r.t. local custom config
> definitions.  i.e. We can't have both fstest defined sections and
> custom sections at the same time.
>
> This restriction essentially forces anyone with a custom config to
> have to copy the built config file into their custom config file so
> that they can run both fstests provided and custom configs in the
> same test run.
>

The current solution faces this problem because we were using
HOST_OPTIONS method to define the config file. That in fstests forces to
use only 1 config file which can be either local.config or <host>.config
or use configs/all-fs.config.

So the problem then is where do the users define their device settings. 

> That is not ideal.
>

Yes, I see the problem. Could you please suggest a better alternative
then?

One approach which I am thinking is to provide a custom -c option to
pass the section config file. That might require some changes in check
script. But the idea is that fstests will use more than 1 defined config
file i.e. it  will first look into it's HOST_OPTIONS provided config
file and also take into account -c <all-fs.config-file>, if passed from
cmdline, to find the additional section definitions. i.e. 

./check -c <all-fs.config-file-path> -s ext4_4k -s xfs_4k -g quick

I guess that will allow the users to use local.config or host.config as
is to define their device settings and also be able to use -c config
file for testing any additional sections.

Thoughts? 

-ritesh


> Maybe this is an oversight, but I still don't know what problem you
> are trying to solve and so I can't make any judgement on whether it
> is a simple mistake or intended behaviour...
>
> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

[Ritesh Harjani]

"Darrick J. Wong" <djwong@kernel.org> writes:

> On Sat, Feb 01, 2025 at 10:23:29PM +0530, Ojaswin Mujoo wrote:
>> Greetings,
>> 
>> This proposal is on behalf of Me, Nirjhar and Ritesh. We would like to submit
>> a proposal on centralizing filesystem and device configurations within xfstests
>> and maybe a further discussion on some of the open ideas listed by Ted here [3].
>> More details are mentioned below.
>> 
>> ** Background ** 
>> There was a discussion last year at LSFMM [1] about creating a central fs-config
>> store, that can then be used by anyone for testing different FS
>> features/configurations. This can also bring an awareness among other developers
>> and testers on what is being actively maintained by FS maintainers. We recently
>> posted an RFC [2] for centralizing filesystem configuration which is under
>> review. The next step we are considering is to centralize device configurations
>> within xfstests itself. In line with this, Ted also suggested a similar idea (in
>> point A) [3], where he proposed specifying the device size for the TEST and
>> SCRATCH devices to reduce costs (especially when using cloud infrastructure) and
>> improve the overall runtime of xfstests.
>> 
>> Recently Dave introduced a feature [4] to run the xfs and generic tests in
>> parallel. This patch creates the TEST and SCRATCH devices at runtime without
>> requiring them to be specified in any config file. However, at this stage, the
>> automatic device initialization appears to be somewhat limited. We believe that
>> centralizing device configuration could help enhance this functionality as well.
>> 
>> ** Proposal ** 
>> We would like to propose a discussion at LSFMM on two key features: central
>> fsconfig and central device-config within xfstests. We can explore how the
>> fsconfig feature can be utilized, and by then, we aim to have a PoC for central
>> device-config feature, which we think can also be discussed in more detail. At
>> this point, we are hoping to get a PoC working with loop devices by default. It
>> will be good to hear from other developers, maintainers, and testers about their
>> thoughts and suggestions on these two features.
>> 
>> Additionally, we would like to thank Ted for listing several features he uses in
>> his custom kvm-xfstests and gce-xfstests [3]. If there is an interest in further
>> reducing the burden of maintaining custom test scripts and wrappers around
>> xfstests, we can also discuss essential features that could be integrated
>> directly into xfstests, whether from Ted's list or suggestions from others.
>> 
>> Thoughts and suggestions are welcome.
>
> Considering all the questions downthread, I'm wondering, are you just
> going to stuff all the known configs into a single configs/default file
> and then modify known_hosts() to set HOST_OPTIONS to that?

In the last approach that's what we were doing. However since check
script only consider 1 exclusive config file for looking into into the
device settings and sections, that forces the users to pass device
settings separately. Not an elegant solution.

>
> 	[ -f /etc/xfsqa.config ]             && export HOST_OPTIONS=/etc/xfsqa.config
> 	[ -f $HOST_CONFIG_DIR/default ]      && export HOST_OPTIONS=$HOST_CONFIG_DIR/default
> 	[ -f $HOST_CONFIG_DIR/$HOST ]        && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST
> 	[ -f $HOST_CONFIG_DIR/$HOST.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST.config
>
> Then configs/default contains things like:
>
> [xfs_nocrc]
> MKFS_OPTIONS="-m crc=0"
>
> Would that work for running configurations in this manner:
>
> 	./check -s xfs_nocrc -g all
>
> ?

I am discussing another approach in my last response to Dave. i.e. Let's
maybe define 1 config file for all fs sections e.g. configs/all-fs.config. 
We then need to modify the check script to continue allowing the use of the
HOST_OPTIONS provided config file while also supporting an additional
config file passed via the -c option via cmdline.

That way users can continue to use their local.config file as is but can
also find additional sections to test if they pass 
-c configs/all-fs.config file.

i.e. 

     ./check -c configs/all-fs.config -s xfs_nocrc -g all

where all-fs.config would have defined the [xfs_nocrc] section.

>
> (I am completely ignorant of config files and never use them.)
>

So you have your custom local.config file defined somewhere which you
always use for testing? What's your setup like in terms of different fs
& device configurations to test?
(both your local setup for fstests testing and ci setup which you use)

> --D
>

Thanks Darrick for looking into this :)

-ritesh

>> 
>> ** References **
>> [1] https://lore.kernel.org/all/87h6h4sopf.fsf@doe.com/
>> [2] https://lore.kernel.org/all/9a6764237b900f40e563d8dee2853f1430245b74.1736496620.git.nirjhar.roy.lists@gmail.com/
>> [3] https://lore.kernel.org/all/20250110163859.GB1514771@mit.edu/
>> [4] https://lore.kernel.org/all/20241127045403.3665299-1-david@fromorbit.com/
>>