From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBC2C433EF for ; Wed, 9 Mar 2022 18:57:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235952AbiCIS6d (ORCPT ); Wed, 9 Mar 2022 13:58:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235281AbiCIS6c (ORCPT ); Wed, 9 Mar 2022 13:58:32 -0500 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1B096597 for ; Wed, 9 Mar 2022 10:57:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=380l+Gu5qIiiXNX/ArbUGoUCh7yNUhAFRcWP2zrXEf4=; b=vG5NBLGQlonAUGXQ+V6WFpLZzZ F/H+zyPQgIw3L58tjvbZqOTSlL322PzxhXKS0XvXj34q+K9+R0xzt5VMQn6WJqLAzDj1crIUur0cn yfK95xYdRyhTXWAqjhxC52UEuWSFDaZ9tYDu3W+CTrx029fOiNvsgiuPivqxqN0TEUVtQvGIVYs5Z rYt8K9xuoFiq9/kdGA1aN4Er/lg8Trxp/Isp4inP78UNy+BBJf9okAjHC4Xw97tnFH3aHaRqHVd/H XqbnbKXQrRSQr0NeGWfuc3Atw+TbcS8cNcAR7D9nBpKKxoSXb8RlQWmMYhE2bxpJMvdzXv+d2eDnO TT4ebcBQ==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1nS1Uy-00AA89-Ag; Wed, 09 Mar 2022 18:57:24 +0000 Date: Wed, 9 Mar 2022 10:57:24 -0800 From: Luis Chamberlain To: Sasha Levin Cc: Amir Goldstein , Greg KH , lsf-pc , linux-fsdevel , Jan Kara , Theodore Tso , "Darrick J. Wong" , Josef Bacik , Matthew Wilcox Subject: Re: [LSF/MM TOPIC] FS, MM, and stable trees Message-ID: References: <20190212170012.GF69686@sasha-vm> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: Luis Chamberlain Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Tue, Mar 08, 2022 at 02:06:57PM -0500, Sasha Levin wrote: > What we can't do is invest significant time into doing the testing work > ourselves for each and every subsystem in the kernel. I think this experience helps though, it gives you I think a better appreciation for what concerns we have to merge any fix and the effort and dilligence required to ensure we don't regress. I think the kernel-ci steady state goal takes this a bit further. > The testing rig I had is expensive, not even just time-wise but also > w.r.t the compute resources it required to operate, I suspect that most > of the bots that are running around won't dedicate that much resources > to each filesystem on a voluntary basis. Precicely because of the above is *why* one of *my* requirements for building a kernel-ci system was to be able to ensure I can run my tests regardless of what employer I am at, and easily ramp up. So I can use local virtualized solutions (KVM or virtualbox), or *any* cloud solution at will (AWS, GCE, Azure, OpenStack). And so kdevops enables all this using the same commands I posted before, using simple make target commands. Perhaps the one area that might interest folks is the test setup, using loopback drives and truncated files, if you find holes in this please let me know: https://github.com/mcgrof/kdevops/blob/master/docs/testing-with-loopback.md In my experience this setup just finds *more* issues, rather than less, and in my experience as well none of these issues found were bogus, they always lead to real bugs: https://github.com/mcgrof/kdevops/blob/master/docs/seeing-more-issues.md A test rig for a high kernel-ci steady state goal does require resources, time and effort. Fortunately I am now confident in the architecture behind the tests / automation though. So all that is really needed now is just a dedicated system to run these, agree what configs we'd test (I have some well defined and documented for XFS on kdevops through Kconfig, based on conversations we last had about stable testing), work with a public baseline to reflect this setup (I have public baselines already published for tons of kernels and for different filesystems), and then test candidate fixes. This later effort is still time consuming too. But with a proper ongoing rig running a kernel-ci, this becomes much easier and it is a much smoother sailing process. > I can comment on what I'm seeing with Google's COS distro: it's a > chicken-and-egg problem. It's hard to offer commercial support with the > current state of xfs, but on the other hand it's hard to improve the > state of xfs without a commercial party that would invest more > significant resources into it. This is the non-Enterprise argument to it. And yes. I agree, but it doesn't mean we can't resolve it. I think we just need to agree to a a dedicated test rig, test setup, and a public baseline might be a good next step. > Luckily there is an individual in Google who has picked up this work and > hopefully we will see something coming out of it very soon, but honestly > - we just got lucky. Groovy. Luis