From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B70933A0E8E for ; Wed, 15 Apr 2026 08:28:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776241689; cv=none; b=RrVHcFSvvg5Ujb5gNQ75rvZXzEK0WlO5j0UM75QopdHhuv/XSXzTaPRo/6WTQzTssohx14H9pIyFZ1tXLJivSC7Fr2c7PIeMjUEIZLACnDjgPH5DaO+8RHpyPv401uewoRCVqaL8EZydHTcG9N+9hpqFUwVTZQ457a4OR4g/WaU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776241689; c=relaxed/simple; bh=KZ31+iVuJ3hjGiVu2o5PRH1lxsvTWH4MQ3mopx32FA4=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mkDY46MRrMhyq6w1AQSc2Z7cGlJ/HjkMSPfAzu0AmNa9+d31v2Wts7I1bvrqCxd2lcf+4UtsE61HX0VZJwTD7M3QqFXRceOKx3yNxOmWXAB8E7wEcWbBtRl9nNeZ3JYVzhTAmpERvlRwcxQ9RoQk16Bd4MfI2RqrXNNyuIHOQ7c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VJFpsSxf; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VJFpsSxf" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-43cfac48bc7so4488441f8f.0 for ; Wed, 15 Apr 2026 01:28:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1776241685; x=1776846485; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=0MPHkT8FkNcBtLY+Pw7K92GBLHfnQFXFDeYyX4HYwxI=; b=VJFpsSxfRwLVLraetQXuPhCXJ/73WCQRviZS2LD/YvMRtWLauyQGXyp7chBzw5o/BP j3sTArpIOvAeh/AG7QtkNU89seNBN/oyMgVxRbNK0veSNfOdF9dzaITomy1niersAG8K 1IOL8tKldhEl5E83l6KS2DT3CfxGMQbb2WMKtS3HRoCCHqOItQ/vZmGNSvY9y2rng3m7 5Sqf2yjGYtExqrYvXW10+YJ6z7eKwD90uUsfzkTXdl9rkysXQS5IQy1IC4eAo/cSGjee NIjeAFkelaFuwPD8LElF2e+4qpZINq3Rl44O0IyK44wj3FSAIqDlVoQcZx6GhiHIqPzE x/6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776241685; x=1776846485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=0MPHkT8FkNcBtLY+Pw7K92GBLHfnQFXFDeYyX4HYwxI=; b=ApVGWEi+4TexiuLBng/PW7VT1QWB1VVvor1Fg19SnONsdYA1/0TahxwFrLJBzHQjcz 5uyzPopdZ2Y58yLrwwHqJyz+EMYRZcx5UKLKRDcO6OKA/XR8mkOlLvZWsvg2g13R2W/u fvEKTngv0HdJIY/Zxytr/YV8nunkpJDpNK6QgXZAhom3SK2E+7yd++GH/1skXgcYJwAB hjSDRCLvVbx6+aY16jYjwETw09rCqfRw84EaPK8zEstMl9/vhb/GHEjD4e9vaUXe66I+ 4fqjcVi7kfK4E9RQh1TTCygBY+++SETyk4Yf9tHZyiI/jvVPNg8JNgSY/ufkefAgvSCq y+CA== X-Forwarded-Encrypted: i=1; AFNElJ9N+OZ7SuXgMlX+OlK/nBLxIQPGR+wenqRoz0vzsgp48SezWD5a+kvcfXrG0StBKB5wTh6kMcAu0Vp45uI=@vger.kernel.org X-Gm-Message-State: AOJu0YwtwbG/yH+nkNAYrefRm6MPJPb61BMqJvZAh8TGOrdKk0kajHQ/ 9hR3dVn3y6iJCQBI4xXimm3qyne24GgznkXKqA3nQDEIhTRa9SiydqM9 X-Gm-Gg: AeBDieuspgiekGwGjmwzd2CX/PLdVJ01GqcS7z/wkA4cxZK3AcYeNf1oMbZCz8ewu+/ lL3Wal+Ikmq9vTUdWfEwFjUfkRUCZwniiGqxQfTx88RtJng0MIzzKn4JS1vH9FuHE9dsWNZIbwm 9E5g8R4/fS+73X4oSRMOiVwuOsF/6um/81uGVTzwJK8MfYO7pexdevIUohkl/JeTH4kcWR4p65a 18hyTKGIHnb/AwSqQPi9WY1B+1VbaYKn+39pfOCtFk89Cwj5aea1HLgiDPhcpsU0GJQDu4XqBz0 V3nWirx9XyBcagNHy1j0dYcV7j+bMD5bNRDFJ5zhiuJdydMdUMTiBeRgILah6R4K49vbT9rLmVk EED3sT/E9ZCSU48d6RhRVgXRIUnZ67/Zkhxd2dj0P+EKovvkfmhZrB3cvvSQ25W7QC5Jkl+OUNb atw6eZ/09srl/xkR9KGj8YBj9jIlST3hqUuRvXAjEomjgBIvYeLmEX8twgsu4RXLxg X-Received: by 2002:a05:6000:268a:b0:43d:7403:4b60 with SMTP id ffacd0b85a97d-43d74034c72mr20655235f8f.3.1776241684906; Wed, 15 Apr 2026 01:28:04 -0700 (PDT) Received: from pumpkin (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43ead3564d8sm3573210f8f.10.2026.04.15.01.28.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Apr 2026 01:28:04 -0700 (PDT) Date: Wed, 15 Apr 2026 09:28:02 +0100 From: David Laight To: Dave Chinner Cc: Gao Xiang , Christoph Hellwig , Tal Zussman , Jens Axboe , "Matthew Wilcox (Oracle)" , Christian Brauner , "Darrick J. Wong" , Carlos Maiolino , Al Viro , Jan Kara , Bart Van Assche , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, Sandeep Dhavale Subject: Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support Message-ID: <20260415092802.5864c457@pumpkin> In-Reply-To: References: <20260409160243.1008358-1-hch@lst.de> <20260409160243.1008358-9-hch@lst.de> <7f0d072b-97a7-405f-bff5-d3819de2e3dd@linux.alibaba.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; arm-unknown-linux-gnueabihf) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Tue, 14 Apr 2026 10:58:16 +1000 Dave Chinner wrote: > On Sat, Apr 11, 2026 at 07:44:43AM +0800, Gao Xiang wrote: > > > > > > On 2026/4/11 06:11, Dave Chinner wrote: > > > On Thu, Apr 09, 2026 at 06:02:21PM +0200, Christoph Hellwig wrote: > > > > Commit 3fffb589b9a6 ("erofs: add per-cpu threads for decompression as an > > > > option") explains why workqueue aren't great for low-latency completion > > > > handling. Switch to a per-cpu kthread to handle it instead. This code > > > > is based on the erofs code in the above commit, but further simplified > > > > by directly using a kthread instead of a kthread_work. > > > > > > > > Signed-off-by: Christoph Hellwig > > > > > > Can we please not go back to the (bad) old days of individual > > > subsystems needing their own set of per-cpu kernel tasks just > > > sitting around idle most of of the time? The whole point of the > > > workqueue infrastructure was to get rid of this widely repeated > > > anti-pattern. > > > > > > If there's a latency problem with workqueue scheduling, then we > > > should be fixing that problem rather than working around it in every > > > subsystem that thinkgs it has a workqueue scheduling latency > > > issue... > > > > It has been "fixed" but never actually get fixed: > > https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.com > > > > and workqueues don't have any plan to introduce RT threads; > > They don't need to (or should) introduce RT threads. Per-cpu kernel > threads already get priority over normal user tasks on scheduling > decisions. However, they do not pre-empt running kernel tasks of > the same priority. > > In general, kernel threads should not use RT scheduling at all - if > the kernel uses RT prioprity tasks then that can interfere with user > scheduled RT tasks. This is especially true in this case where a > non-RT tasks issue the IO, and the IO completion is then scheduled > with RT priority. IOWs, any unprivileged user can now impact the > processing time available to, and the response latency of, other > RT scheduled tasks the system is running. But might not an IO for a use RT task be sat behind it in the completion queue? To avoid priority inversion you need to process the completions. The non-RT task won't be rescheduled to issue a later request. The only other way is to force that completion work be done in the context of the thread that issued the request. > > Tejun asked Sandeep if setting the workqueue thread priority to > -19 through sysfs (i.e. making them higher priority than normal > kernel threads) had the same effect on latency as using a dedicated > per-cpu RT task thread. THere was no followup. > > In theory, this should provide the same benefit, because what RT > scheduling is doing is pre-empting any user and kernel task that was > running when the interrupt was delivered to execute the completion > task immediately. > > Setting the workqueue to use kernel threads of a higher scheduler > prioirty should do the same thing, without the need to use dedicated > per-cpu RT threads. We had a related issue with the network stack's use of softint and threaded napi when trying to receive very high packet rate UDP traffic. (IIRC something like 500000 200byte RTP audio packets every second.) There is an absolute requirement to run the ethernet rx processing in order to avoid dropping packets. Now normally the code runs as 'softint' making it higher priority that any user process, but under load it switches to using 'normal priority' kernel threads which are basically low priority. Similarly 'threaded napi' uses 'normal priority' threads. The only we made it work was to manually change the threads to be a low 'RT FIFO' priority so they got scheduled in preference to all user processes. A system may need some user RT threads with a lower priority than these kernel threads and others with a higher priority. So defaulting to a middling RT priority may be best. David