From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fra-out-011.esa.eu-central-1.outbound.mail-perimeter.amazon.com (fra-out-011.esa.eu-central-1.outbound.mail-perimeter.amazon.com [52.28.197.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76DF118A6CF for ; Wed, 4 Feb 2026 22:49:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=52.28.197.132 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770245348; cv=none; b=pSHlOtzUVBeAEoxd7XhpYLimnnQ7gRrZr3CD3FezzVZTXgvNXE+AyPqjqTG4VO9knKjPT+fExdp9o/Z9ORXzYLpoxpp1WN/9pvtyw8pvq5SZAXiAiyvELI4tVRDyEpueNzoaVr4Slnc2SuEr+i4USiKBVlFc1Z/uNtCNstAnqHY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770245348; c=relaxed/simple; bh=crLbp634SS/tabQ5xCp4XmmWI1s5JJgLzag/DRLDbCc=; h=Subject:Message-ID:Date:MIME-Version:To:CC:References:From: In-Reply-To:Content-Type; b=sglESPf7Li9yFEQ80EOeLID+/Ta40KcbB10sgo9Dy4ygbx9eCtd7TjvnRO1PgmtQy5ThLmQb5LkqMhNcgG3fETH6URV4RFway3IMcS6eImJQM19e/AuUfu/bMCLZRoz91OVUJZ1J0tTVxqmzwMiuJlrUUbk6ngoyN8TinA5+vfA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=BtCWGrr3; arc=none smtp.client-ip=52.28.197.132 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="BtCWGrr3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1770245347; x=1801781347; h=message-id:date:mime-version:to:cc:references:from: in-reply-to:content-transfer-encoding:subject; bh=zg56rUbfPoM07EGUifSD1z3MkrkoMx8OG5GIGGIU9jk=; b=BtCWGrr3GSk3m8N0kKEtCMgPufUow9JEaIdF12IGudjhggc4wOnVK0zj ifwrPz5Z84HlyoVBMTIgD16s7jN/+tEm5Tadz+kFv/vEBLI82iNzxgHyc qVCUNTSE+zqMnNPWlDyK1+lgpUjUoYU+HII5nNBB1RNVj2FWfr719X+cb oNCRWxaWoCjeW4DnN4IdBUxEHSGSEIucPCuB6QqaJby/qDAbR7nO0GZjO oMTfK88svYY5weOxzqLusX9vfNZvWK88AigByw70qESeREjIYu/4EraGK a5CJ1g/89rVWE7ZLkpHKaIZeQZgJOfLzEiMI/vNogd1AefcXJj4fPMiTZ g==; X-CSE-ConnectionGUID: 9ZdCViT1QNOx7l376t83qQ== X-CSE-MsgGUID: cEAnwxEOQYGzTfJRtP4djw== X-IronPort-AV: E=Sophos;i="6.21,273,1763424000"; d="scan'208";a="8849075" Subject: Re: [PATCH 4/4] sched/fair: Proportional newidle balance Received: from ip-10-6-11-83.eu-central-1.compute.internal (HELO smtpout.naws.eu-central-1.prod.farcaster.email.amazon.dev) ([10.6.11.83]) by internal-fra-out-011.esa.eu-central-1.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2026 22:48:50 +0000 Received: from EX19MTAEUC001.ant.amazon.com [54.240.197.225:5560] by smtpin.naws.eu-central-1.prod.farcaster.email.amazon.dev [10.0.37.34:2525] with esmtp (Farcaster) id 456c1b36-3447-49b2-968f-a13d7a56c669; Wed, 4 Feb 2026 22:48:49 +0000 (UTC) X-Farcaster-Flow-ID: 456c1b36-3447-49b2-968f-a13d7a56c669 Received: from EX19D018EUA004.ant.amazon.com (10.252.50.85) by EX19MTAEUC001.ant.amazon.com (10.252.51.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.35; Wed, 4 Feb 2026 22:48:49 +0000 Received: from [192.168.1.220] (10.106.83.33) by EX19D018EUA004.ant.amazon.com (10.252.50.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.35; Wed, 4 Feb 2026 22:48:48 +0000 Message-ID: <92f30c7e-a0e2-4c50-9ae8-f97a915e2c8d@amazon.com> Date: Wed, 4 Feb 2026 22:48:48 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Peter Zijlstra CC: Mario Roy , Chris Mason , "Joseph Salisbury" , Adam Li , Josh Don , , , , , , , , , References: <3ef2a178-db96-4a2c-9e74-44ba81bef030@amazon.com> <20260127084412.GV171111@noisy.programming.kicks-ass.net> <20260129091937.GY166857@noisy.programming.kicks-ass.net> <683b4860-1076-43b8-8fb9-ca8ac26cd8f4@amazon.com> <20260202105144.GC1282955@noisy.programming.kicks-ass.net> <6237f4ee-6b07-4d79-8ee7-04e49a4e99b7@amazon.com> <20260204132710.GK2995752@noisy.programming.kicks-ass.net> <20260204140552.GF3016024@noisy.programming.kicks-ass.net> Content-Language: en-US From: "Mohamed Abuelfotoh, Hazem" In-Reply-To: <20260204140552.GF3016024@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: EX19D009EUA004.ant.amazon.com (10.252.50.97) To EX19D018EUA004.ant.amazon.com (10.252.50.85) On 04/02/2026 14:05, Peter Zijlstra wrote: >> I will try to >> come-up with standalone reproduction steps that can be used to investigate >> this memcached regression. Meanwhile we will share the fio regression >> reproduction steps that I mentioned in my previous update. This should be >> much simpler in steps and can be done on a single machine. > > Thanks! I have a few machines with a 'spare' nvme drive to run things > on, hopefully that is sufficient. It looks like the previously reported fio regression has been fully mitigated by the proposed patch [1]. I verified this on both 6.18.5 & 6.12.66. I will try to come-up with standalone reproduction for the memcached regression to make it easier for debugging. **fio regression reproduction environment** AWS EC2 instance: c5ad.24xlarge 96 vCPUs = 48 Cores with HT 12 CCDs Memory : 192 GiB SSD Disk space: 1900 GiB SSD Disk Max write IOPS: 180K SSD Disk Max Write B.W: 760 MB/sec Below are the results of three different runs. 6.18.5 6.18.5_revert 6.18.5 with the revert of 1b9c118fe318 ("sched/fair: Proportional newidle balance") 6.18.5_proposed 6.18.5 with patch[1] --------------------------------------------------------------- Version 6.18.5 # sudo fio --time_based --name=benchmark --size=50G --runtime=60 --filename=/dev/nvme1n1 --ioengine=psync --randrepeat=0 --iodepth=1 --fsync=64 --invalidate=1 --verify=0 --verify_fatal=0 --blocksize=4k --group_reporting --rw=randwrite --numjobs=4 Run status group 0 (all jobs): WRITE: bw=478MiB/s (501MB/s), 478MiB/s-478MiB/s (501MB/s-501MB/s), io=28.0GiB (30.1GB), run=60003-60003msec ---------------------------------------------------------------- Version 6.18.5_revert # sudo fio --time_based --name=benchmark --size=50G --runtime=60 --filename=/dev/nvme1n1 --ioengine=psync --randrepeat=0 --iodepth=1 --fsync=64 --invalidate=1 --verify=0 --verify_fatal=0 --blocksize=4k --group_reporting --rw=randwrite --numjobs=4 Run status group 0 (all jobs): WRITE: bw=549MiB/s (575MB/s), 549MiB/s-549MiB/s (575MB/s-575MB/s), io=32.2GiB (34.5GB), run=60002-60002msec ----------------------------------------------------------------- Version 6.18.5_proposed # sudo fio --time_based --name=benchmark --size=50G --runtime=60 --filename=/dev/nvme1n1 --ioengine=psync --randrepeat=0 --iodepth=1 --fsync=64 --invalidate=1 --verify=0 --verify_fatal=0 --blocksize=4k --group_reporting --rw=randwrite --numjobs=4 Run status group 0 (all jobs): WRITE: bw=551MiB/s (578MB/s), 551MiB/s-551MiB/s (578MB/s-578MB/s), io=32.3GiB (34.7GB), run=60003-60003msec [1] https://lore.kernel.org/all/20260127151748.GA1079264@noisy.programming.kicks-ass.net/T/#u