Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Josef Bacik <josef@toxicpanda.com>]

On Wed, Mar 09, 2022 at 05:28:28PM -0800, Luis Chamberlain wrote:
> On Wed, Mar 09, 2022 at 04:19:21PM -0500, Josef Bacik wrote:
> > On Wed, Mar 09, 2022 at 11:00:49AM -0800, Luis Chamberlain wrote:
> > > On Wed, Mar 09, 2022 at 01:49:18PM -0500, Josef Bacik wrote:
> > > > On Wed, Mar 09, 2022 at 10:41:53AM -0800, Luis Chamberlain wrote:
> > > > > On Tue, Mar 08, 2022 at 11:40:18AM -0500, Theodore Ts'o wrote:
> > > > > > One of my team members has been working with Darrick to set up a set
> > > > > > of xfs configs[1] recommended by Darrick, and she's stood up an
> > > > > > automated test spinner using gce-xfstests which can watch a git branch
> > > > > > and automatically kick off a set of tests whenever it is updated.
> > > > > 
> > > > > I think its important to note, as we would all know, that contrary to
> > > > > most other subsystems, in so far as blktests and fstests is concerned,
> > > > > simply passing a test once does not mean there is no issue given that
> > > > > some test can fail with a failure rate of 1/1,000 for instance.
> > > > > 
> > > > 
> > > > FWIW we (the btrfs team) have been running nightly runs of fstests against our
> > > > devel branch for over a year and tracking the results.
> > > 
> > > That's wonderful, what is your steady state goal? And do you have your
> > > configurations used public and also your baseline somehwere? I think
> > > this later aspect could be very useful to everyone.
> > > 
> > 
> > Yeah I post the results to http://toxicpanda.com, you can see the results from
> > the runs, and http://toxicpanda.com/performance/ has the nightly performance
> > numbers and graphs as well.
> 
> That's great!
> 
> But although this runs nightly, it seems this runs fstest *once* to
> ensure if there are no regressions. Is that right?
> 

Yup once per config, so 8 full fstest runs.

> > This was all put together to build into something a little more polished, but
> > clearly priorities being what they are this is as far as we've taken it.  For
> > configuration you can see my virt-scripts here
> > https://github.com/josefbacik/virt-scripts which are what I use to generate the
> > VM's to run xfstests in.
> > 
> > The kernel config I use is in there, I use a variety of btrfs mount options and
> > mkfs options, not sure how interesting those are for people outside of btrfs.
> 
> Extremely useful.
> 

[root@fedora-rawhide ~]# cat /xfstests-dev/local.config
[btrfs_normal_freespacetree]
TEST_DIR=/mnt/test
TEST_DEV=/dev/mapper/vg0-lv0
SCRATCH_DEV_POOL="/dev/mapper/vg0-lv7 /dev/mapper/vg0-lv6 /dev/mapper/vg0-lv5 /dev/mapper/vg0-lv4 /dev/mapper/vg0-lv3 /dev/mapper/vg0-lv2 /dev/mapper/vg0-lv1 "
SCRATCH_MNT=/mnt/scratch
LOGWRITES_DEV=/dev/mapper/vg0-lv8
PERF_CONFIGNAME=jbacik
MKFS_OPTIONS="-K -f -O ^no-holes"
MOUNT_OPTIONS="-o space_cache=v2"
FSTYP=btrfs

[btrfs_compress_freespacetree]
MOUNT_OPTIONS="-o compress=zlib,space_cache=v2"
MKFS_OPTIONS="-K -f -O ^no-holes"

[btrfs_normal]
TEST_DIR=/mnt/test
TEST_DEV=/dev/mapper/vg0-lv0
SCRATCH_DEV_POOL="/dev/mapper/vg0-lv9 /dev/mapper/vg0-lv8 /dev/mapper/vg0-lv7 /dev/mapper/vg0-lv6 /dev/mapper/vg0-lv5 /dev/mapper/vg0-lv4 /dev/mapper/vg0-lv3 /dev/mapper/vg0-lv2 /dev/mapper/vg0-lv1 "
SCRATCH_MNT=/mnt/scratch
LOGWRITES_DEV=/dev/mapper/vg0-lv10
PERF_CONFIGNAME=jbacik
MKFS_OPTIONS="-K -O ^no-holes -R ^free-space-tree"
MOUNT_OPTIONS="-o discard=async"

[btrfs_compression]
MOUNT_OPTIONS="-o compress=zstd,discard=async"
MKFS_OPTIONS="-K -O ^no-holes -R ^free-space-tree"

[kdave]
MKFS_OPTIONS="-K -O no-holes -R ^free-space-tree"
MOUNT_OPTIONS="-o discard,space_cache=v2"

[root@xfstests3 ~]# cat /xfstests-dev/local.config
[btrfs_normal_noholes]
TEST_DIR=/mnt/test
TEST_DEV=/dev/mapper/vg0-lv0
SCRATCH_DEV_POOL="/dev/mapper/vg0-lv9 /dev/mapper/vg0-lv8 /dev/mapper/vg0-lv7 /dev/mapper/vg0-lv6 /dev/mapper/vg0-lv5 /dev/mapper/vg0-lv4 /dev/mapper/vg0-lv3 /dev/mapper/vg0-lv2 /dev/mapper/vg0-lv1 "
SCRATCH_MNT=/mnt/scratch
LOGWRITES_DEV=/dev/mapper/vg0-lv10
PERF_CONFIGNAME=jbacik
MKFS_OPTIONS="-K -O no-holes -f -R ^free-space-tree"

[btrfs_compress_noholes]
MKFS_OPTIONS="-K -O no-holes -f -R ^free-space-tree"
MOUNT_OPTIONS="-o compress=lzo"

[btrfs_noholes_freespacetree]
MKFS_OPTIONS="-K -O no-holes -f"
MOUNT_OPTIONS="-o space_cache=v2"


> > Right now I have a box with ZNS drives waiting for me to set this up on so that
> > we can also be testing btrfs zoned support nightly, as well as my 3rd
> > RaspberryPi that I'm hoping doesn't blow up this time.
> 
> Great to hear you will be covering ZNS as well.
> 
> > I have another virt setup that uses btrfs snapshots to create a one off chroot
> > to run smoke tests for my development using virtme-run.  I want to replace the
> > libvirtd vms with virtme-run, however I've got about a 2x performance difference
> > between virtme-run and libvirtd that I'm trying to figure out, so right now all
> > the nightly test VM's are using libvirtd.
> > 
> > Long, long term the plan is to replace my janky home setup with AWS VM's that
> > can be fired from GitHub actions whenever we push branches, that way individual
> > developers can get results for their patches before they're merged, and we don't
> > have to rely on my terrible python+html for test results.
> 
> If you do move to AWS just keep in mind using loopback drives +
> truncated files *finds* more issues than not. So when I used AWS
> I got two spare nvme drives and used one to stuff the truncated
> files there.
> 

My plan was to get ones with attached storage and do the LVM thing I do for my
vms.

> > > Yes, everyone's test setup can be different, but this is why I went with
> > > a loopback/truncated file setup, it does find more issues and so far
> > > these have all been real.
> > > 
> > > It kind of begs the question if we should adopt something like kconfig
> > > on fstests to help enable a few test configs we can agree on. Thoughts?
> > > 
> > 
> > For us (and I imagine other fs'es) the kconfigs are not interesting, it's the
> > combo of different file system features that can be toggled on and off via mkfs
> > as well as different mount options.  For example I run all the different mkfs
> > features through normal mount options, and then again with compression turned
> > on.  Thanks,
> 
> So what I mean by kconfig is not the Linux kernel kconfig, but rather
> the kdevops kconfig options. kdevops essentially has a kconfig symbol
> per mkfs-param-mount config we test. And it runs *ones* guest per each
> of these. For example:
> 
> config FSTESTS_XFS_SECTION_REFLINK_1024
> 	bool "Enable testing section: xfs_reflink_1024"
> 	default y
> 	help
> 	  This will create a host to test the baseline of fstests using the
> 	  following configuration which enables reflink using 1024 byte block
> 	  size.
> 
> 	[xfs_reflink]
> 	MKFS_OPTIONS='-f -m reflink=1,rmapbt=1, -i sparse=1,'
> 	FSTYP=xfs
> 
> The other ones can be found here for XFS:
> 
> https://github.com/mcgrof/kdevops/blob/master/workflows/fstests/xfs/Kconfig
> 
> So indeed, exactly what you mean. What I'm getting at is that it would
> be good to construct these with the community. So it would beg the
> question if we should embrace for instance kconfig language to be
> able to configure fstests (yes I know it is xfstests but I think loose
> new people who tend to assume that xfstest is only for XFS, so I only
> always call it fstests).
> 

Got it, that's pretty cool, I pasted my configs above.  Once I figure out why
virtme is so much slower than libvirtd I'll give kdevops a try and see if I can
make it work for my setup.  Thanks,

Josef