From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF22C43217 for ; Thu, 19 May 2022 14:58:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240102AbiESO6u (ORCPT ); Thu, 19 May 2022 10:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240059AbiESO6m (ORCPT ); Thu, 19 May 2022 10:58:42 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F23E9A5A89; Thu, 19 May 2022 07:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=bOAgmeR6UE8Jq8eTwtkBILkXF6ykn4b5YwRwZCX7b7E=; b=qEDHH7DSiP/DGEjXLjijAwKUfs ymY04FdevHDrpotuF87hsVzBvNWWYvNH5zQhntY0z8c7nSKJIfMAdBXaymRN048YEk4ndlrZoQD15 NANKGMUYqo0oGmL0hRZTeYr1ikIEIqYufdgH/Wjmb1HDJe04rDmC7aUODLhN7/6tL1XlIcZvB++v3 Hl/FBOF926zEo8TRYzn/bh0dk53P0Sdf1P+GULrCZa4uFykTTfJi2E8QxnXDQE6XC37QdYeMcDyu5 tMNcqhHGGHdChjWN5LkgL9kgSJyvavUF4uxYX5d1l93jC0JBM8P4vnDqS5HARbf6yK0jSYky7xUnn 1QKLksqA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nrhbj-00CoKR-FB; Thu, 19 May 2022 14:58:31 +0000 Date: Thu, 19 May 2022 15:58:31 +0100 From: Matthew Wilcox To: Zorro Lang Cc: Amir Goldstein , Luis Chamberlain , linux-fsdevel , linux-block , pankydev8@gmail.com, Theodore Tso , Josef Bacik , jmeneghi@redhat.com, Jan Kara , Davidlohr Bueso , Dan Williams , Jake Edge , Klaus Jensen , fstests Subject: Re: [RFC: kdevops] Standardizing on failure rate nomenclature for expunges Message-ID: References: <20220519112450.zbje64mrh65pifnz@zlang-mailbox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220519112450.zbje64mrh65pifnz@zlang-mailbox> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Thu, May 19, 2022 at 07:24:50PM +0800, Zorro Lang wrote: > Yes, we talked about this, but if I don't rememeber wrong, I recommended each > downstream testers maintain their own "testing data/config", likes exclude > list, failed ratio, known failures etc. I think they're not suitable to be > fixed in the mainline fstests. This assumes a certain level of expertise, which is a barrier to entry. For someone who wants to check "Did my patch to filesystem Y that I have never touched before break anything?", having non-deterministic tests run by default is bad. As an example, run xfstests against jfs. Hundreds of failures, including some very scary-looking assertion failures from the page allocator. They're (mostly) harmless in fact, just being a memory leak, but it makes xfstests useless for this scenario. Even for well-maintained filesystems like xfs which is regularly tested, I expect generic/270 and a few others to fail. They just do, and they're not an indication that *I* broke anything. By all means, we want to keep tests around which have failures, but they need to be restricted to people who have a level of expertise and interest in fixing long-standing problems, not people who are looking for regressions.