From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CCB2EE49A3 for ; Wed, 23 Aug 2023 06:04:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232787AbjHWGEW (ORCPT ); Wed, 23 Aug 2023 02:04:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231238AbjHWGEQ (ORCPT ); Wed, 23 Aug 2023 02:04:16 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80FABFB; Tue, 22 Aug 2023 23:04:14 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 0F1281F45A; Wed, 23 Aug 2023 06:04:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1692770653; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i8K0uqA75zgrqeyQqMOjA4vyh8El/ZLPyoR7z7nwmbY=; b=cVwNHGclP/hGyDqH09K5eBIX+15NxFuAWXP9obLtWgsIrqBFik+JVFV33qZj8egkv0XI0Q sUzwMNfxy6zm5sKHsJiQd699ZDKUOpnp3eko0oEJX6p6O/UoK53WGM8TjNZHcy1eIxrgP3 6I4QtU+HMax/BNSoUVAj8IYF4iPs4RA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1692770653; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i8K0uqA75zgrqeyQqMOjA4vyh8El/ZLPyoR7z7nwmbY=; b=dCjAtT+gGxw8t9YU35qqSEBvQBmVXVmwCzK64k9GxMCsAdxBGxgtknr0p2f5MeHYZEYB7s 1XH8fgkYjRTAhRCw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1779F13458; Wed, 23 Aug 2023 06:04:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id +z4gOVqh5WQ5GwAAMHmgww (envelope-from ); Wed, 23 Aug 2023 06:04:10 +0000 Message-ID: Date: Wed, 23 Aug 2023 08:04:10 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: Infiniate systemd loop when power off the machine with multiple MD RAIDs Content-Language: en-US To: Yu Kuai , Carlos Carvalho , Song Liu Cc: Bagas Sanjaya , Christoph Hellwig , AceLan Kao , Linux Kernel Mailing List , Linux Regressions , Linux RAID , "yukuai (C)" , "yangerkun@huawei.com" References: <028a21df-4397-80aa-c2a5-7c754560f595@gmail.com> <909a6669-01af-dbe7-b64b-84a5f3bc75a3@huaweicloud.com> From: Hannes Reinecke In-Reply-To: <909a6669-01af-dbe7-b64b-84a5f3bc75a3@huaweicloud.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org On 8/23/23 03:28, Yu Kuai wrote: > Hi, > > 在 2023/08/23 3:13, Carlos Carvalho 写道: >> Song Liu (song@kernel.org) wrote on Tue, Aug 22, 2023 at 03:56:04PM -03: >>> >From systemd code, i.e. function delete_md(), this error: >>> >>> [ 205.957004] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:6). >>> [ 205.964177] systemd-shutdown[1]: Could not stop MD /dev/md124p1: >>> Device or resource busy >>> >>> is most likely triggered by ioctl(STOP_ARRAY). >>> >>> And based on the code, I think the ioctl fails here: >>> >>>          if (cmd == STOP_ARRAY || cmd == STOP_ARRAY_RO) { >>>                  /* Need to flush page cache, and ensure no-one else >>> opens >>>                   * and writes >>>                   */ >>>                  mutex_lock(&mddev->open_mutex); >>>                  if (mddev->pers && atomic_read(&mddev->openers) > 1) { >>>                          mutex_unlock(&mddev->open_mutex); >>>                          err = -EBUSY; >>>                          goto out;        ////////////////////// HERE > > Yes, I suspect here as well, but I do some test with error injection to > gurantee that ioctl(STOP_ARRAY) always return -EBUSY, but I found that > system reboot didn't hang, it'll try a few times but eventually reboot > finished. > >>>                  } >>>                  if (test_and_set_bit(MD_CLOSING, &mddev->flags)) { >>>                          mutex_unlock(&mddev->open_mutex); >>>                          err = -EBUSY; >>>                          goto out; >>>                  } >>>                  did_set_md_closing = true; >>>                  mutex_unlock(&mddev->open_mutex); >>>                  sync_blockdev(bdev); >>>          } >> >> Probably. The problem is why doesn't it manage to flush the page >> cache? I find >> strange that the problem appears only when trying to stop the array, I >> get it >> when trying to umount the filesystem, where it also hangs because of >> the same >> reason. The kworker thread runs continuously using 100% cpu of only 1 >> core. > > The kworker do you mean the daemon thread or the sync thread?  runs > continuously using 100% cpu doen't sounds correct to me. > That sounds more like kworker waiting on a mutex to be released. Can you trigger a kdump while in that state? Or do a sysrq-T to get a stack trace of all running processes? That should tell you where exactly the 100% cpu time is being wasted. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman