From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE4D0C00140 for ; Wed, 10 Aug 2022 19:31:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233137AbiHJTbq (ORCPT ); Wed, 10 Aug 2022 15:31:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232960AbiHJTbm (ORCPT ); Wed, 10 Aug 2022 15:31:42 -0400 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48E2975389 for ; Wed, 10 Aug 2022 12:31:41 -0700 (PDT) Received: by mail-qt1-x82a.google.com with SMTP id l5so5566212qtv.4 for ; Wed, 10 Aug 2022 12:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc; bh=dLpGkLIiz/54cVWJRgggdlnjUABVaQiLzQFQiAlQPGA=; b=ouc3/Q4QmYB1bJ51O5HBWOL3LY/GeDOT4Zl4M2lOTIvMyM5yOFAcxdjShv7m+mw7st NkJxfNCPxm1koPC2HWtD9C8QKkXudV/HjezWm4QGzfYIPpJ3aY57nxKXWyl7ZTXX0dCu csKga0oGb8iJrnLMCk58S/UU5GQoeTdVd/FYyRQbgND37OHxNQYFFgDEKL91yNzxqaOZ /SR7cBPnwo4HClujH5IJYvqKz/2fserwC/SR3p4cyq+SHsw483ZCWWXd8t1uCkCP1FcZ IIkZjutHyYx5zsUmxMMPC4Fq0TXxMzmjcCuvOl3vy7RbhavW8V0c0NerNvZswYH47SFF h4zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc; bh=dLpGkLIiz/54cVWJRgggdlnjUABVaQiLzQFQiAlQPGA=; b=kO3iCXuR+TETX1fkPaJJBkN2SPehCcjbjrMTbcd2paIM48WdbjMzMqCBAxNy6UuuNw 34p84yAfRLn8AmrO153kGulDaL1EYZIzmPbwPVVKcdN/QI77Uow8sPnfG+nME+TSB2vr wfEuxrB/LoUiv5Ytiy2dxzdl8Do1TjBKG+OwTyrATOzgK7WvYSLC+QxqNjQC+zfJANo0 AMnhdLBbotvCtIaPDfN49dpkCZui0TQlpJJZkK+E3Us0KxO1iPcUE3MvLgjxq/Oq9/t3 uUmgGu1QplAlaEHmMqoV3OO4EekPT9rZUITTT1I7HoXuqUjPQQSyjNJ+zrZj93plVBAY aDRQ== X-Gm-Message-State: ACgBeo1/TlqVWnqU9SZaU2G4H6SlCgPsk9U+4odg9MuvS9vch+kn6LAH K9jFX5ZpBV/XLqlJAF2VVNNBCv2t70QS1Q== X-Google-Smtp-Source: AA6agR63H3khOeYEyAgxUArdGqdtq4HNQfyGWhenU0M9vtjj3U1R0WqUmrP9r2uH1h2GBlSYweiCeA== X-Received: by 2002:a05:622a:190:b0:31f:3999:b949 with SMTP id s16-20020a05622a019000b0031f3999b949mr25242898qtw.444.1660159900306; Wed, 10 Aug 2022 12:31:40 -0700 (PDT) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id bl35-20020a05620a1aa300b006b555509398sm397591qkb.136.2022.08.10.12.31.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 12:31:39 -0700 (PDT) Date: Wed, 10 Aug 2022 15:31:38 -0400 From: Josef Bacik To: Chris Murphy Cc: Btrfs BTRFS , Linux-RAID , linux-block@vger.kernel.org, linux-kernel Subject: Re: stalling IO regression in linux 5.12 Message-ID: References: <2220d403-e443-4e60-b7c3-d149e402c13e@www.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 10, 2022 at 02:42:40PM -0400, Chris Murphy wrote: > > > On Wed, Aug 10, 2022, at 2:33 PM, Chris Murphy wrote: > > On Wed, Aug 10, 2022, at 1:48 PM, Josef Bacik wrote: > > > >> To help narrow this down can you disable any IO controller you've got enabled > >> and see if you can reproduce? If you can sysrq+w is super helpful as it'll > >> point us in the next direction to look. Thanks, > > > > I'm not following, sorry. I can boot with > > systemd.unified_cgroup_hierarchy=0 to make sure it's all off, but we're > > not using an IO cgroup controllers specifically as far as I'm aware. > > OK yeah that won't work because the workload requires cgroup2 or it won't run. > Oh no I don't want cgroups completley off, just disable the io controller, so figure out which cgroup your thing is being run in, and then echo "-io" > /cgroup.subtree_control If you cat /sys/fs/cgroup/whatever/cgroup/cgroup.controllers and you see "io" in there keep doing the above in the next highest parent directory until io is no longer in there. Thanks, Josef