From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D61C33D7D96 for ; Thu, 2 Apr 2026 12:52:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775134345; cv=none; b=XS0Z4AE9ZcLkmuLlhLDD0fsEDolxmrBU1HDYFdqzxNVs5V5jozuCYXcHcI+t1xAVyeBEWJOQ9lv31I1iaAqzey1xap0n19rUtnu/X7enO9g3WUnbtlmUpac4ncjk3/H0X5/nCB1Mv1PLYKhJGdY/iNbssP3BrLQVZdMrgv3KX0g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775134345; c=relaxed/simple; bh=o1ZCYwcFc8lRBKP856YOVeSOHiTUCIiilJIC6oIentI=; h=From:To:Cc:Subject:In-Reply-To:Date:Message-ID:References; b=ui27AhhQ2f5LclETi1x97fehEFXe2KFRfoSv+E3pVEH+hldNuYkImzQsUpmvlep1ijklNWfVp2wsWWBZTCx7/tHJXiKmYF+J29EtSbCrksqY14ygrn0ifrCOcsX9HKOeRu1r/nJ+vVDecwCoGezPG5Zg7gyKmPz/hw3dWofz3hg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=s3oi11MO; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="s3oi11MO" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-35d8e548a05so793742a91.1 for ; Thu, 02 Apr 2026 05:52:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775134339; x=1775739139; darn=vger.kernel.org; h=references:message-id:date:in-reply-to:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=TQ3lrWVKO87z8vhsStJXZkDwRc6VLmdZN5o7clgXULk=; b=s3oi11MOqnDG32fuaoFARK5ha79/g9zCsdwtFdWWCZWIHsSvBTQKJz+lE3UO8jhWAD rKT1TDd0zuNlFDCgX4/hTDnx5OJ0HdNeXxQTkPZkP60W+/pnd47sLSht8Yy/nHD/NAkY VBcB4qockof2HeEFkojTvsaDg9br6MfJoDmw40gjFgQs5aVc9Xyx8fegOQGLu2X/15B7 k4U3Kp4KqWLlRqUyIqq+CvxN/gen40shZHd8hJ/IWcxJSWJt86SebzPTnoqIcs2o2RQP vwu57W2vCS9LBuB3M8qKuG3bH9QhNn0ELeUKyvlU5Bnen4r4EvNfts6PCWaMtVZbvXeC RGWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775134339; x=1775739139; h=references:message-id:date:in-reply-to:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TQ3lrWVKO87z8vhsStJXZkDwRc6VLmdZN5o7clgXULk=; b=U3yEcFewzOgwVSdzI27kYN6v3lOHLSOEd/3/3fyXeRUEvThVpgKJJU0jtD+7wOZMPX AoLE8a2XIUjjrg7/tk4gfXEa/Mvw3oOk2zj7qBkEHtlVR7My/pox8qpEjx8kqKDODcko 6Wc6zaWXhfHGDDyFpCY5pONy0LzIQDDrUmLtrnWIR/YtgtPlSDjxG6Orw5w5wkNt6wXo lJSIy5m2pBUVW2QcCkTRrZq+Uo20LyprmE6AAwldnEORKMqEUidGWhi05vnSr0slSNpX pN0j0XqgJomUAsunxUOnbpLKKEJZWh6zlE/RULtQtNXvQjrYkmKmBcIRNwRPyWAZJsuH Htdw== X-Forwarded-Encrypted: i=1; AJvYcCXmbH/31A9M8p/tQfL5NlwxlUBd58MwaVKXFOcrJWVP/9cvgUttPVWlluOD0sjBuphvzCn+YbeGy2fupZY=@vger.kernel.org X-Gm-Message-State: AOJu0YytOJu5hSc9QMzhsGtDTHnXX1NVm1WtzYLnH8nA9QCuEdVDBSi0 ppF15vBaz47pWeXGaNAB7GNVkxvNgzr9Z7WPkEPQjlvnbHH7OpmvPFGS X-Gm-Gg: AeBDievQXTTjqauXuPLsHdwH4XnKZBUEEsQPuenRlwP9hZqYAvzwIafqb8qAyL8AsDB 5A0v/PyfX3TWzNrIN3SB+owmHyx3Kfin9F+OAvXOYNl8fIA+IC8yBwB/N1O0rkFJFOJvLCasj8b +6oIZWUDYryxqlbH6xFPuM4+TKfguKwnZiBzfNDzTb6OanR3HqDw55SusfJ3z7O6L49EC+bJuPq PkZNJ02yxNVwQx+YuHF4+3zIYoQOXzSKXrCBN3mbHs2J3ZcuZK1xu3UHm5aF8o3k36VIIySk8LW jLRAaF7o+ajE/1kBjVdNtNKSILk3m84Sx3rVmgpwhhZYOAjPYS3MNpc/g67NmspqPCnRzDZjZr1 Q2EL6m8Kmly3+tjUGICAousF6Sffx65FTE3JpUuTfsm9apPjOBcafroe3fFA0yjWYr5CgS8PRMJ H0nwiqVbRTvl6BZONJtxf0lw== X-Received: by 2002:a17:90b:1850:b0:35d:9c39:4dc9 with SMTP id 98e67ed59e1d1-35dc6f8a3famr6709877a91.23.1775134339387; Thu, 02 Apr 2026 05:52:19 -0700 (PDT) Received: from pve-server ([49.205.216.49]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-35dbb8932ecsm3838360a91.5.2026.04.02.05.52.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Apr 2026 05:52:18 -0700 (PDT) From: Ritesh Harjani (IBM) To: Jeff Layton , Alexander Viro , Christian Brauner , Jan Kara , "Matthew Wilcox (Oracle)" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Mike Snitzer , Chuck Lever Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 1/4] mm: fix IOCB_DONTCACHE write performance with rate-limited writeback In-Reply-To: <09672fa10c77d4fbfa1a13ea16aedf79d23fd8f8.camel@kernel.org> Date: Thu, 02 Apr 2026 18:10:32 +0530 Message-ID: References: <20260401-dontcache-v1-0-1f5746fab47a@kernel.org> <20260401-dontcache-v1-1-1f5746fab47a@kernel.org> <09672fa10c77d4fbfa1a13ea16aedf79d23fd8f8.camel@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Jeff Layton writes: > On Thu, 2026-04-02 at 10:13 +0530, Ritesh Harjani wrote: >> Jeff Layton writes: >> >> > IOCB_DONTCACHE calls filemap_flush_range() with nr_to_write=LONG_MAX >> > on every write, which flushes all dirty pages in the written range. >> > Under concurrent writers this creates severe serialization on the >> > writeback submission path, causing throughput to collapse to ~47% of >> > buffered I/O with multi-second tail latency. >> >> Yes, between concurrent writers, I agree with the theory. >> >> >> > Even single-client >> > sequential writes suffer: on a 512GB file with 256GB RAM, the >> > aggressive flushing triggers dirty throttling that limits throughput >> > to 575 MB/s vs 1442 MB/s with rate-limited writeback. >> >> I am not sure if this 2.5x performance penalty in a "single" sequential Sorry my bad.. I mis-understood this 2.5x delta at first. So in a single sequential write case, what this patch is mainly improving is from unpatched RWF_DONTCACHE (1179 MB/s) to patched RWF_DONTCACHE (1453 MB/s) = ~23% improvement. So the below theory which I was talking about was from this delta perspective i.e. comparing unpatched v/s patched RWF_DONTCACHE mode. >> writer is due to throttling logic. On giving it some thoughts, I suspect >> if this is because, the submission side and the completion side both >> takes the xa_lock and hence could be contending on that. >> >> For e.g. since this patch skips doing the flush the second time, (note >> that writeback is active when the same writer dirtied the page during >> previous write), this allows the writer to do more work of writing data >> to page cache pages, instead of waiting on the xa_lock which the >> completion callback could be holding (folio_end_writeback() -> folio_end_dropbehind()) >> >> If I see Peak Dirty data from the link you shared [1] in single writer case... >> >> Mode MB/s p50 (ms) p99 (ms) p99.9 (ms) Peak Dirty Peak Cache >> dontcache (unpatched) 1179 3.2 103.3 170.9 14 MB 4.7 GB >> dontcache (patched) 1453 5.4 43.8 57.4 36 GB 45 GB >> >> ... this too shows that the submission side is writing more dirty pages, >> then the completion side able to write it... >> >> I suspect this contention (between submission and completion) could more >> in IOCB_DONTCACHE case, since the completion side also removes the folio >> from the page cache within the same xa_lock, which is not the same with >> normal buffered writes. >> >> Maybe a perf callgraph showing the contention would be nicer thing to add >> here [1] ;). >> >> [1]: https://markdownpastebin.com/?id=96249deb897a401ba32acbce05312dcc >> > > That's an interesting point. > > The theory I've been operating on is that the flusher thread ends up > squatting on the xa_lock for a while when memory gets tight, and that > blocks other readers and writers. Staying ahead of the dirty limits and > limiting the amount of flush work that each writer does alleviates > contention for that lock and that's what improves the performance. > That's right for comparison between buffered write against RWF_DONTCACHE. But what I meant in above was for the improvement from 1179 MB/s to 1453 MB/s could be accounted to less contention on xa_lock on patched version v/s unpatched version for single write sequential testcase. > You're right though. I'll plan to play around with perf and see if I > can confirm the theory. > Yes, thanks, that will be nice to have! -ritesh