From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FB2829992A for ; Wed, 8 Apr 2026 23:20:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775690447; cv=none; b=hFX//MHGC8T4LSKWyDMAzQyUcVgxhvOY6An8sUnSfVV0aW+FgG9CYlgdwYeXXSWChGsemfiVbiqnK6QpTQf6hU6MbhBlvxVD1KjqsYCz8nQqbkixz0WBVvcruqwgDzSBujzkZ+W9pzYAJaKEE7k/n4wi71Mo8TpY0EoZYeDq03s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775690447; c=relaxed/simple; bh=J4zLxB3Z2srZjMFeA8I9iSK+nmXGlPhU9//6QvCOc0g=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type; b=LyKDwuGv9RagFZvQtDGQLPase0qiGmv8grD+DOPDulLYgf0gqLlYGLikIKaTehTEy//lPUM8Lpx6VulZ90GBdNZcVcGsjkhhX8qID4ApTMRtqw2dxIEf84q/qUEinPDgE/i/VfbhUtpR7l9SlflHrxllxpjPOtbmHbQimhcoplA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net; spf=pass smtp.mailfrom=telus.net; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b=bC4KhB0O; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=telus.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=telus.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=telus.net header.i=@telus.net header.b="bC4KhB0O" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-82cf96ca071so179682b3a.1 for ; Wed, 08 Apr 2026 16:20:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telus.net; s=google; t=1775690446; x=1776295246; darn=vger.kernel.org; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=wpmt9+5/VizQ4eFYNxSHC3AChLls0N+VfX9KfH3AYDc=; b=bC4KhB0Oel0g2U4q1rM2GnSbwMqggoEx+IcjXdLgDciXZSyRKL1M+Wp/y4Ep+uw4hc 0HiEuH3a08HLTW+NUnaDwkDfGs8WGMSwo1BaEaVhjpZYETLRXpi8uVGqRvhFUaBHwBRv 07oGWcwmDcUCv5eDMSa9HK1A+9tJYK2D+yPYO927QyNqDy49D+dPTp+GaeeXHscWXvmx eRT6RXil9WHBCKIpfx1mx8c7PgkrP4ZrMLqEFnaPB4q20szIJilY5GS63EvgJyN5S99M zqjTmwsLXpFvgPff9UW/F2EjvXNf7nXwsR8iDtJRjOioi990vdyZVv1n++YfRd73muOr Vqlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775690446; x=1776295246; h=thread-index:content-language:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wpmt9+5/VizQ4eFYNxSHC3AChLls0N+VfX9KfH3AYDc=; b=F8oUU25o2YMLK8SR99+l2pWMJfhnZXTHnUXUdc0Jhr12bc6+wuB1SydM9LCkIGwKT/ 5q2P/uln6A3/CvTDSLLsODSBqPVEa/dBiE91dbxjm6JQYWBywD+yyh2Ft95NlZzFNEw+ r9qP2d+63vIdDnAvhICJqpm+gEbghu1GqpEw3Xp3cpyaVMgaCNOuwiTA6UDmNGNj8nu0 SVD4pAuQArMUWaNPXSRsVfmW8Hp76BmhQdpz/hJW9mAaGs6OOBFZSOWpv1xDdwKbXeim kR2vPT+/bpLAvmY4H6cjd3iXw//wzQ+r7UAVFDt7uZ15C12Bf/ebp+LxGR16wy+ToH2k SDFg== X-Forwarded-Encrypted: i=1; AJvYcCVEEvFsmm+geknVNG9JLcNxHgsRjZ8ym5h1Csou0/2BPXvAsUoCya8kyprTP8VgDa/5RAub2Fajv/x7+74=@vger.kernel.org X-Gm-Message-State: AOJu0YyGPDCp5x67lzJookHGzifBDz0J0GPpqW++1Tyhqg5RPCNfO+xf n+0iQt5VFjHO44AsbbjqtCXxyBTVPZ8SIvUbMr7iyuqoGAy9x2kXoKXG+/bXLgodb20= X-Gm-Gg: AeBDievOeI2uBl2U1TyubWDzprNZbZ82FQouYScT90P12XmlyEeQtIRG//83xyoHDmW OKmVkQCgid1Gqfobu6LCWSgFDItrfu9PPsHFZz+NF1XPAKSYSagLGSY4soFqJK/noWn4X/bXB5D ct2QCDUQBYtE8Wq3+TOsybUHOon/nAOVQL/NWhFzqqRkKHvEEB79/PlCUcxr2H1E8RuvW2CYNKc kthSln3+fqy8ihYMLtoAa6VDa8SDfCwEOl2BVMj+goE9g983NUYSzydud61zb0DsVMzfGfFOUD5 l6DJBbx8GQv2H9PB7DS0z+kp479FyHijE9wWBBXPFyiPCjjscn/QyZU8Ai3VkrjpQdMmkHXKqV+ TPpDYXg/REHG93uI+WQ/5BmIdnonrvvTRMN36ahne+mcQwGY6Lhs0RtoBpmX3698FBKUb4k0rdK Wt/L1XCi2JF2xSk5Po++pxRh+mZRBJw7baSyySL3q0XgMLeZLO7ZNcXohXV+RB6jM0TWRDUZ0+h nucCQ== X-Received: by 2002:a05:6a00:1acb:b0:82c:d9d0:f482 with SMTP id d2e1a72fcca58-82dd8ad4a09mr1148941b3a.46.1775690445745; Wed, 08 Apr 2026 16:20:45 -0700 (PDT) Received: from DougS18 (s66-183-142-209.bc.hsia.telus.net. [66.183.142.209]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82cf9ca75dfsm21840109b3a.60.2026.04.08.16.20.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2026 16:20:44 -0700 (PDT) From: "Doug Smythies" To: "'Yu Kuai'" Cc: "'Nilay Shroff'" , "'Bart Van Assche'" , "'Jens Axboe'" , "Doug Smythies" , Subject: REGRESSION BISECTED: mq-deadline: covert to use request_queue->async_depth Date: Wed, 8 Apr 2026 16:20:46 -0700 Message-ID: <005201dcc7ae$54a33010$fde99030$@telus.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AdzHokCduPDPM6HiSVaCxdHfjQvhMA== Hi, I observed a couple of regressions in a workflow (described further below). I have not started to investigate the first regression yet. The first regression was of less magnitude and was introduced somewhere between kernels 6.18 and 6.19-rc1. For the second regression, I bisected the kernel, and then double checked the result: > commit 988bb1b9ededab9aed83df8c1f5be0232b71ded3 > Author: Yu Kuai > Date: Tue Feb 3 16:19:47 2026 +0800 > mq-deadline: covert to use request_queue->async_depth > > In downstream kernel, we test with mq-deadline with many fio workloads, and > we found a performance regression after commit 39823b47bbd4 > ... Workflow: My version of "critical-jobs", an attempt to do similar to the non-free SPECjbb critical-jOPS. The data base file is 500 Gigabytes. Each requested job does 2 record lookups and a tiny bit of work for each lookup. 340 jobs per second are requested, but the system can only handle an average of about 231 jobs per second. Jobs keep being launched until the maximum number of processes of 1000 is hit, at which time the jobs rate is throttled back. The requested test execution time is extended such that all jobs are completed. The test measures latency between the launch of the job and completion. The issue is that some jobs seem to get forgotten about for a long long time. Processor: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz, 6 cores 12 CPUs. HWP: Enabled. Test Disk: HDD Seagate Ironwolf Pro16TB NAS OS: Ubuntu 24.04.4 LTS (server, no GUI). Other than the test, the system is very idle. Kernels used for this email: bad: As of commit 988bb1b9eded called "kuai" With 988bb1b9eded reverted, called "kuai-revert" I.E.: 2d7cf26b0d4b (HEAD -> kuai) Revert "mq-deadline: covert to use request_queue->async_depth" 988bb1b9eded mq-deadline: covert to use request_queue->async_depth 8cbe62f4d8c3 kyber: covert to use request_queue->async_depth f98afe4f31bb blk-mq: add a new queue sysfs attribute async_depth Results: kuai: 340.0 Requested jobs per second. 230.9 Actual jobs per second. 110.4 Execution time (Seconds) 13.8 job latency, 90th percentile (Seconds), 85% regression 21.9 job latency, 95th percentile, 144% regression 40.7 job latency, 99th percentile, 235% regression 99.2 maximum job latency, 313% regression kuai-revert (reference): 340.0 Requested jobs per second. 231.8 Actual jobs per second. 110.0 Execution time (Seconds) 7.47 job latency, 90th percentile (Seconds) 8.96 job latency, 95th percentile 12.14 job latency, 99th percentile 24.01 maximum job latency I have yet to investigate the first regression, but just for readers potential interest, results for a kernel before 6.19-rc1: 340.0 Requested jobs per second. 233.2 Actual jobs per second. 109.4 Execution time (Seconds). 5.7 job latency, 90th percentile (Seconds) 6.32 job latency, 95th percentile 7.37 job latency, 99th percentile 12.60 maximum job latency