From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0647C77B60 for ; Thu, 27 Apr 2023 02:21:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242745AbjD0CVi (ORCPT ); Wed, 26 Apr 2023 22:21:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242709AbjD0CVi (ORCPT ); Wed, 26 Apr 2023 22:21:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDD433C16 for ; Wed, 26 Apr 2023 19:20:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682562044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=GCPcaFdJ9SvoZ93cdbG0lFwwi9Thl1ubA5vESylOj3g=; b=XP42qjAwM3lzgyaYuWSOHp9o1lxcxWVCnK4zHL5ozhraMfCE3kEEhFu2FLnhY1mkJExG0g g9Q8bs6cxwLeFZmJ1r84a/ty8TWOLpLwVwMDEM1AD5lCjVHo6WZ3Ey42z4m5SRZKuaiwKs s0TZsrOtVsJfE/DOgx1LsyauQv8mNTQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-aRep1uVGPCu1PlVejb15nA-1; Wed, 26 Apr 2023 22:20:40 -0400 X-MC-Unique: aRep1uVGPCu1PlVejb15nA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5842B29A9D28; Thu, 27 Apr 2023 02:20:40 +0000 (UTC) Received: from ovpn-8-24.pek2.redhat.com (ovpn-8-24.pek2.redhat.com [10.72.8.24]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C35681121314; Thu, 27 Apr 2023 02:20:33 +0000 (UTC) Date: Thu, 27 Apr 2023 10:20:28 +0800 From: Ming Lei To: Theodore Ts'o , linux-ext4@vger.kernel.org Cc: ming.lei@redhat.com, Andreas Dilger , linux-block@vger.kernel.org, Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Dave Chinner , Eric Sandeen , Christoph Hellwig , Zhang Yi Subject: [ext4 io hang] buffered write io hang in balance_dirty_pages Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Hello Guys, I got one report in which buffered write IO hangs in balance_dirty_pages, after one nvme block device is unplugged physically, then umount can't succeed. Turns out it is one long-term issue, and it can be triggered at least since v5.14 until the latest v6.3. And the issue can be reproduced reliably in KVM guest: 1) run the following script inside guest: mkfs.ext4 -F /dev/nvme0n1 mount /dev/nvme0n1 /mnt dd if=/dev/zero of=/mnt/z.img& sleep 10 echo 1 > /sys/block/nvme0n1/device/device/remove 2) dd hang is observed and /dev/nvme0n1 is gone actually [root@ktest-09 ~]# ps -ax | grep dd 1348 pts/0 D 0:33 dd if=/dev/zero of=/mnt/z.img 1365 pts/0 S+ 0:00 grep --color=auto dd [root@ktest-09 ~]# cat /proc/1348/stack [<0>] balance_dirty_pages+0x649/0x2500 [<0>] balance_dirty_pages_ratelimited_flags+0x4c6/0x5d0 [<0>] generic_perform_write+0x310/0x4c0 [<0>] ext4_buffered_write_iter+0x130/0x2c0 [ext4] [<0>] new_sync_write+0x28e/0x4a0 [<0>] vfs_write+0x62a/0x920 [<0>] ksys_write+0xf9/0x1d0 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd [root@ktest-09 ~]# lsblk | grep nvme [root@ktest-09 ~]# BTW, my VM sets 2G ram, and the nvme disk size is 40GB. So far only observed on ext4 FS, not see it on XFS. I guess it isn't related with disk type, and not tried such test on other type of disks yet, but will do. Seems like dirty pages aren't cleaned after ext4 bio is failed in this situation? Thanks, Ming