From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18299C43334 for ; Thu, 7 Jul 2022 21:36:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236857AbiGGVgi (ORCPT ); Thu, 7 Jul 2022 17:36:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236801AbiGGVgh (ORCPT ); Thu, 7 Jul 2022 17:36:37 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91942380; Thu, 7 Jul 2022 14:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ttm6gGP6SLsd/ERoDNA6enC+GfvJnC9kynUKCW0O0ns=; b=yG/OHYUxpGtlw44gkc5XopLGK7 I6y30vKOkhOgcv+c8LaKNmQixg7SzUKOv8wSBBO+6LBT8YXJPQXcIR0qWJClh6qOzughOSEPsNMfy S8DJkDEk+QYlc/azK+k+pGgEtW9CrNp8CzMc1vh7pfA3wOIoQtKS6wSLiAAByTp2BbN5++4vl3VAC Zku/392MLRMD/b2b6Zeco5gTA+2lC0sHcilcbcxPhxpF/P1rFjeBgdu6UdzweR4E0XmzLUCeP2ogK RObaFJoTdvPJ1wAzXrh490Wnuns+OppbHZaPWd99DmY2EgLSK7UE7EuHzOSk9RlAx4688b2+ycQXa /fqLBcTw==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1o9ZAn-000Lbq-4A; Thu, 07 Jul 2022 21:36:33 +0000 Date: Thu, 7 Jul 2022 14:36:33 -0700 From: Luis Chamberlain To: Theodore Ts'o Cc: Matthew Wilcox , Zorro Lang , Amir Goldstein , linux-fsdevel , linux-block , pankydev8@gmail.com, Josef Bacik , jmeneghi@redhat.com, Jan Kara , Davidlohr Bueso , Dan Williams , Jake Edge , Klaus Jensen , fstests Subject: Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges Message-ID: References: <20220519112450.zbje64mrh65pifnz@zlang-mailbox> <20220519154419.ziy4esm4tgikejvj@zlang-mailbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: Luis Chamberlain Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Sat, Jul 02, 2022 at 01:01:22PM -0400, Theodore Ts'o wrote: > Note: I recommend that you skip using the loop device xfstests > strategy, which Luis likes to advocate. For the perspective of > *likely* regressions caused by the Folio patches, I claim they are > going to cause you more pain than they are worth. If there are some > strange Folio/loop device interactions, they aren't likely going to be > obvious/reproduceable failures that will cause pain to linux-next > testers. While it would be nice to find **all** possible bugs before > patches go usptream to Linus, if it slows down your development > velocity to near-standstill, it's not worth it. We have to be > realistic about things. Regressions with the loopback block driver can creep up and we used to be much worse, but we have gotten better at it. Certainly testing a loopback driver can mean running into a regression with the loopback driver. But some block driver must be used in the end. > What about other file systems? Well, first of all, xfstests only has > support for the following file systems: > > 9p btrfs ceph cifs exfat ext2 ext4 f2fs gfs glusterfs jfs msdos > nfs ocfs2 overlay pvfs2 reiserfs tmpfs ubifs udf vfat virtiofs xfs > > {kvm,gce}-xfstests supports these 16 file systems: > > 9p btrfs exfat ext2 ext4 f2fs jfs msdos nfs overlay reiserfs > tmpfs ubifs udf vfat xfs > > kdevops has support for these file systems: > > btrfs ext4 xfs Thanks for this list Ted! And so adding suport for a new filesystem in kdevops should be: * a kconfig symbol for the fs and then one per supported mkfs config option you want to support * a configuration file for it, this can be as elaborate to support different mkfs config options as we have for xfs [0] or one with just one or two mkfs config options [1]. The default is just shared information. [0] https://github.com/linux-kdevops/kdevops/blob/master/playbooks/roles/fstests/templates/xfs/xfs.config [1] https://github.com/linux-kdevops/kdevops/blob/master/playbooks/roles/fstests/templates/ext4/ext4.config > There are more complex things you could do, such as running a baseline > set of tests 500 times (as Luis suggests), I advocate 100 and I suggest that is a nice goal for enterprise kernels. I also personally advocate this confidence in a baseline for stable kernels if *I* am going to backport changes. > but I believe that for your > use case, it's not a good use of your time. You'd need to speed > several weeks finding *all* the flaky tests up front, especially if > you want to do this for a large set of file systems. It's much more > efficient to check if a suspetected test regression is really a flaky > test result when you come across them. Or you work with a test runner that has the list of known failures / flaky failures for a target configuration like using loopbacks already. And hence why I tend to attend to these for xfs, btrfs, and ext4 when I have time. My goal has been to work towards a baseline of at least 100 successful runs without failure tracking upstream. Luis