From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A996C226D18 for ; Tue, 30 Jun 2026 03:26:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782789996; cv=none; b=qv2MFRGJDaxTjJyJ7RKEs19R6+Mm/JxifiCXhtYs0bDfBhUp9QeV9P3Rge5l98g4KcQCCqoN/O7jjzNXZgVRsmVnL1zrCkhHlO3qYZ45QEX7UnjYIJTcyLLxmr/a7mcAyjdXdykCbJ6r5aXSs9CvKURrH+B2H8W9EWLSKeCZ6uI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782789996; c=relaxed/simple; bh=xr0aRpHeV9ucIJytFA6D76S309gRgBByKC01ww6rxqI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=olSyVVUghg2NyHU7M9J2w44K3OAEwPnhVjjE12bKMIQqtf7evDgvKeA1+k7SzJbjWSMUiLTnbMVqsbuU5tnewjhaDY7gRl1zWSeV36Wo3QLOaLvG58FndLsLToT6bMFn4aUR7YgeLBO1/vu017czjfhJAZatbS/jq3Wc1l2lchM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ED7rxRy+; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ED7rxRy+" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-3800f0907d5so114609a91.0 for ; Mon, 29 Jun 2026 20:26:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782789995; x=1783394795; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=JAdJcLQ7bI7HEMxuph9geODYbnyOgAdf3zDMYO3qtbw=; b=ED7rxRy+iH2DhSpUaiuomo7+RCfsWq51RVPIL1KI85id6xYJXw7SfGTk52KOz+0ARK LWYPklh1TlScaOaLFiA6Xgm7yxA7N8frsuSeTdC1f4vCLHse2HImQytXIV0gQJqIQt90 GJGAeGHj79BMc8unFYxTfZOgLadK9NFf72dsL2XnObB8pPzxasq3/LVzTNvM8kiFhYLx VJthuBdzJ2c8tb46l2q8BnEzzWbTw8U3bOFwK76l30ys+zJxrp6Cc2wxjJxenFZCl9E0 W4R00azyRgdj97HKzUniR+knl3M3u5hAqdJL25QwW+wrqQYKLySeBOCwb3vaLsiDu68z NNrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782789995; x=1783394795; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=JAdJcLQ7bI7HEMxuph9geODYbnyOgAdf3zDMYO3qtbw=; b=M5iSAhnsRj6i/FFjrSxzYfNdzwQLPVQUA5ghaIVJlvWVhhQ0hEgtXYbzYmWzPU332v DVRkPPY6/ymY+RNVmxAwwK/r76vmWLRHQdAmpgIE5AbFRUiUCIDohLWJ0UCF4QX/u38m 0a2+lHdoq8Jrvh0L8OfAaG2gvcfpRnycCn0pUT3DXhaeHvxa8ty6TU5NJX1USjK948SA n+eSDdkrzN0EDRtpbLx8a6WYgYTC25iKvGz2b6b+E1a27sZw/G28GIvmjIZN7/Ud7dva 1C5Tig1ql8+wgULBQVu/QM3zmSPLzU7szcK+6lpqmnMxl4ARM3aJ16enUzGFoL9OASVn Vsyw== X-Forwarded-Encrypted: i=1; AHgh+RrUojcR+Rj8HCez7bICDyzj9RhaLEU/gseM3ZEaT0TiFoQnQ+DAOmccUhw2YDqwd08WeWhNZ1M89pBbREc=@vger.kernel.org X-Gm-Message-State: AOJu0YzxmirXKWOdmZIjv7pQH4wTARWt4kDUka1tQx8WBYSCr2QsS6Jn bJutb/15C/6Z68Id1S3nYPz3UgSQa3jp4IEWbazMdpE3XHq9E28NjUEQ X-Gm-Gg: AfdE7ckgh4bK75ynyrfvs7HMyspjPb0rn1wNWYjU7n2dpBtURGfpS6XpmdVcCu48YDf wPf9XEjSYCBH1PEEszjKjnKimb+4ehzpSE3rUZcgOokr81TzDFsJ+zO3GtDsAjQ3MGbNQ9tu/zk uX/XakLD4aRQ+NCc8fu/t3Kd9yjhvUAE6cksSt313oi6Yr9Bc1QpqjN6lb/+Mi9fnj5nK/B0KGb aSCeuzh38YdlqDA54zO/csLPlxVN9y8LrgZBVyQcyd0B9d8kVX/07vL2JS3r/RxQrtuDNqfBs28 GH7zwILtJTw5bH6jmLF8Jx/zzBbAnwpToo6JHGF3BAJA9uKijyCJ2Mepu6UAU7T6wG9uI2NOufL 7KBLrl8rM42vpzRqjTHp9X6siPtogrDgg0mhYn8Zhe4mFxSYfxZLIx2yokpe5tycPx6Kh1Tg2Ax Ip110SEyHO1OT+ X-Received: by 2002:a17:90b:52c7:b0:36a:5438:cff4 with SMTP id 98e67ed59e1d1-3805278b76emr944610a91.6.1782789994877; Mon, 29 Jun 2026 20:26:34 -0700 (PDT) Received: from aa-HP-ProDesk ([43.224.245.229]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-380610a9c14sm141907a91.7.2026.06.29.20.26.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jun 2026 20:26:34 -0700 (PDT) Date: Tue, 30 Jun 2026 11:26:31 +0800 From: Wenwu Hou To: Greg Kroah-Hartman Cc: Minchan Kim , Tejun Heo , LKML Subject: Re: [RFC PATCH] kernfs: release kernfs_mutex before the inode allocation Message-ID: References: <20211116194317.1430399-1-minchan@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Nov 16, 2021 at 08:49:46PM +0100, Greg Kroah-Hartman wrote: > On Tue, Nov 16, 2021 at 11:43:17AM -0800, Minchan Kim wrote: > > The kernfs implementation has big lock granularity(kernfs_rwsem) so > > every kernfs-based(e.g., sysfs, cgroup, dmabuf) fs are able to compete > > the lock. Thus, if one of userspace goes the sleep under holding > > the lock for a long time, rest of them should wait it. A example is > > the holder goes direct reclaim with the lock since it needs memory > > allocation. Let's fix it at common technique that release the lock > > and then allocate the memory. Fortunately, kernfs looks like have > > an refcount so I hope it's fine. > > > > Signed-off-by: Minchan Kim > > --- > > fs/kernfs/dir.c | 14 +++++++++++--- > > fs/kernfs/inode.c | 2 +- > > fs/kernfs/kernfs-internal.h | 1 + > > 3 files changed, 13 insertions(+), 4 deletions(-) > > What workload hits this lock to cause it to be noticable? > > There was a bunch of recent work in this area to make this much more > fine-grained, and the theoritical benchmarks that people created (adding > 10s of thousands of scsi disks at boot time) have gotten better. Hi, By 2026, the kernfs_rwsem has been split into per-fs. But the problem still exists. In a k8s cluster environment, there are a lot of processes reading /sys, for example: Container runtime: runc Prometheus Exporter: node_exporter If a tenant's container processes read /sys and sleep on the OOM (or direct reclaim) path, both runc and node_exporter will be blocked, which will cause the cluster worker node to become unschedulable. For example, the nodejs runtime reads /sys/devices/system/cpu/cpu%u/cpufreq/scaling_cur_freq [1] Jun 22 21:43:17 kernel: task:node state:D stack:0 pid:2621530 ppid:1062663 flags:0x00004002 Jun 22 21:43:17 kernel: Call Trace: Jun 22 21:43:17 kernel: Jun 22 21:43:17 kernel: __schedule+0x278/0x740 Jun 22 21:43:17 kernel: schedule+0x5a/0xd0 Jun 22 21:43:17 kernel: schedule_preempt_disabled+0x11/0x20 Jun 22 21:43:17 kernel: __mutex_lock.constprop.0+0x376/0x680 Jun 22 21:43:17 kernel: ? kvm_sched_clock_read+0xd/0x20 Jun 22 21:43:17 kernel: mem_cgroup_out_of_memory+0x53/0x150 Jun 22 21:43:17 kernel: try_charge_memcg+0x682/0x7d0 Jun 22 21:43:17 kernel: ? memcg_list_lru_alloc+0xa7/0x330 Jun 22 21:43:17 kernel: obj_cgroup_charge+0x70/0x170 Jun 22 21:43:17 kernel: slab_pre_alloc_hook.constprop.0+0xb6/0x1d0 Jun 22 21:43:17 kernel: ? alloc_inode+0x59/0xc0 Jun 22 21:43:17 kernel: kmem_cache_alloc_lru+0x54/0x2c0 Jun 22 21:43:17 kernel: alloc_inode+0x59/0xc0 Jun 22 21:43:17 kernel: iget_locked+0xe3/0x220 Jun 22 21:43:17 kernel: kernfs_get_inode+0x18/0x110 Jun 22 21:43:17 kernel: kernfs_iop_lookup+0x74/0xd0 Jun 22 21:43:17 kernel: __lookup_slow+0x82/0x130 Jun 22 21:43:17 kernel: walk_component+0xdb/0x150 Jun 22 21:43:17 kernel: link_path_walk.part.0.constprop.0+0x240/0x380 Jun 22 21:43:17 kernel: ? path_init+0x293/0x3d0 Jun 22 21:43:17 kernel: path_openat+0x85/0x2a0 Jun 22 21:43:17 kernel: ? seq_printf+0x8e/0xb0 Jun 22 21:43:17 kernel: do_filp_open+0xb4/0x160 Jun 22 21:43:17 kernel: ? __check_object_size.part.0+0x5e/0x130 Jun 22 21:43:17 kernel: do_sys_openat2+0x91/0xc0 Jun 22 21:43:17 kernel: __x64_sys_openat+0x53/0xa0 Jun 22 21:43:17 kernel: do_syscall_64+0x35/0x80 Jun 22 21:43:17 kernel: entry_SYSCALL_64_after_hwframe+0x78/0xe2 A rwsem writer is involved: Jun 22 21:46:17 kernel: task:kworker/u232:0 state:D stack:0 pid:2886179 ppid:2 flags:0x00004000 Jun 22 21:46:17 kernel: Workqueue: netns cleanup_net Jun 22 21:46:17 kernel: Call Trace: Jun 22 21:46:17 kernel: Jun 22 21:46:17 kernel: __schedule+0x278/0x740 Jun 22 21:46:17 kernel: ? psi_group_change+0x226/0x3d0 Jun 22 21:46:17 kernel: schedule+0x5a/0xd0 Jun 22 21:46:17 kernel: schedule_preempt_disabled+0x11/0x20 Jun 22 21:46:17 kernel: rwsem_down_write_slowpath+0x1e2/0x4f0 Jun 22 21:46:17 kernel: down_write+0x57/0x60 Jun 22 21:46:17 kernel: kernfs_remove_by_name_ns+0x38/0xc0 Jun 22 21:46:17 kernel: remove_files+0x2b/0x70 Jun 22 21:46:17 kernel: sysfs_remove_group+0x38/0x80 Jun 22 21:46:17 kernel: sysfs_remove_groups+0x29/0x50 Jun 22 21:46:17 kernel: ib_free_port_attrs+0x92/0x170 [ib_core] Jun 22 21:46:17 kernel: rdma_dev_exit_net+0x117/0x1d0 [ib_core] Jun 22 21:46:17 kernel: ops_exit_list+0x30/0x70 Jun 22 21:46:17 kernel: cleanup_net+0x273/0x430 Jun 22 21:46:17 kernel: process_one_work+0x18a/0x3a0 Jun 22 21:46:17 kernel: worker_thread+0x277/0x3a0 Jun 22 21:46:17 kernel: ? __pfx_worker_thread+0x10/0x10 Jun 22 21:46:17 kernel: kthread+0xe1/0x110 Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 Jun 22 21:46:17 kernel: ret_from_fork+0x2d/0x50 Jun 22 21:46:17 kernel: ? __pfx_kthread+0x10/0x10 Jun 22 21:46:17 kernel: ret_from_fork_asm+0x1b/0x30 Jun 22 21:46:17 kernel: In our real-world case, thousands of nodejs processes hit the cgroup memory limit and sleep on the OOM path, runc gets blocked, and consequently the node remains in an unschedulable state for many hours. Link: - [1] https://github.com/nodejs/node/blob/v22.19.0/deps/uv/src/unix/linux.c#L1889 > But in that work, no one could find a real benchmark or use case that > anyone could even notice this type of thing. What do you have that > shows this? > thanks, > > greg k-h