[PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Julian Sun]

Foreign dirty flushing handles cases where a dirtied folio's memcg and
the memcg owning its inode wb differ.

wbc_detach_inode() notes that "concurrent write sharing of an inode is
expected to be very rare". This expectation does not hold for bdev inodes.
A bdev inode and mapping are shared by buffered users of the block device,
while the inode has a single wb owner. On ext4, buffer-cache folios used
by metadata and journal I/O can therefore be charged to different memcgs
while sharing the same mapping. Normal bdev activity can thus produce
frequent folio/wb ownership mismatches.

This was reproduced on cgroup v2 with the memory and I/O controllers
enabled, using ext4 on a loop device. jbd2 repeatedly generated
track_foreign_dirty events for bdev folios charged to unrelated memcgs.
When one of those memcgs entered dirty throttling, the resulting record led
to a flush_foreign event and queued writeback with reason=foreign_flush for
the bdev inode's root wb.

Skip foreign dirty tracking when the folio mapping's host inode is on the
blockdev pseudo superblock. This prevents bdev-originated records from
triggering later foreign flushes.

The tradeoff is that dirty throttling in the folio's memcg no longer uses
bdev dirtiness to queue writeback on the bdev inode's owner wb. Those
folios remain subject to normal writeback. This is preferable because a
single bdev wb owner does not identify which cgroup is responsible for a
mapping shared by buffered bdev users. Flushing it can write back data
unrelated to the throttled memcg. Dirty accounting, normal writeback, and
foreign dirty tracking for non-bdev inodes remain unchanged.

Fixes: 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")
Signed-off-by: Julian Sun <sunjunchao@bytedance.com>
---
 mm/memcontrol.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1271d390b617..6e8f7ff7d44a 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3898,6 +3898,15 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
 	u64 oldest_at = now;
 	int oldest = -1;
 	int i;
+	struct address_space *mapping = folio_mapping(folio);
+	struct inode *bdev_inode = mapping ? mapping->host : NULL;
+
+	/*
+	 * Bdev inodes are usually shared by many memcgs so foreign
+	 * tracking leads to frequent flushes which is counterproductive.
+	 */
+	if (bdev_inode && sb_is_blkdev_sb(bdev_inode->i_sb))
+		return;
 
 	trace_track_foreign_dirty(folio, wb);
 
-- 
2.39.5

Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Andrew Morton]

On Thu,  3 Sep 2026 16:33:03 +0800 Julian Sun <sunjunchao@bytedance.com> wrote:

> Foreign dirty flushing handles cases where a dirtied folio's memcg and
> the memcg owning its inode wb differ.
> 
> wbc_detach_inode() notes that "concurrent write sharing of an inode is
> expected to be very rare". This expectation does not hold for bdev inodes.
> A bdev inode and mapping are shared by buffered users of the block device,
> while the inode has a single wb owner. On ext4, buffer-cache folios used
> by metadata and journal I/O can therefore be charged to different memcgs
> while sharing the same mapping. Normal bdev activity can thus produce
> frequent folio/wb ownership mismatches.
> 
> This was reproduced on cgroup v2 with the memory and I/O controllers
> enabled, using ext4 on a loop device. jbd2 repeatedly generated
> track_foreign_dirty events for bdev folios charged to unrelated memcgs.
> When one of those memcgs entered dirty throttling, the resulting record led
> to a flush_foreign event and queued writeback with reason=foreign_flush for
> the bdev inode's root wb.
> 
> Skip foreign dirty tracking when the folio mapping's host inode is on the
> blockdev pseudo superblock. This prevents bdev-originated records from
> triggering later foreign flushes.
> 
> The tradeoff is that dirty throttling in the folio's memcg no longer uses
> bdev dirtiness to queue writeback on the bdev inode's owner wb. Those
> folios remain subject to normal writeback. This is preferable because a
> single bdev wb owner does not identify which cgroup is responsible for a
> mapping shared by buffered bdev users. Flushing it can write back data
> unrelated to the throttled memcg. Dirty accounting, normal writeback, and
> foreign dirty tracking for non-bdev inodes remain unchanged.

When fixing something, please always describe the userspace-visible
runtime effects.

I'm thinking probably "unnecessary I/O"?  If so, has this been
quantified/measured?  If not, can the effects be guessed about?

> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3898,6 +3898,15 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
>  	u64 oldest_at = now;
>  	int oldest = -1;
>  	int i;
> +	struct address_space *mapping = folio_mapping(folio);
> +	struct inode *bdev_inode = mapping ? mapping->host : NULL;
> +
> +	/*
> +	 * Bdev inodes are usually shared by many memcgs so foreign
> +	 * tracking leads to frequent flushes which is counterproductive.
> +	 */
> +	if (bdev_inode && sb_is_blkdev_sb(bdev_inode->i_sb))
> +		return;
>  
>  	trace_track_foreign_dirty(folio, wb);
>  
> -- 
> 2.39.5

Re: [External] Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Julian Sun]

On 9/4/26 3:34 AM, Andrew Morton wrote:
> On Thu,  3 Sep 2026 16:33:03 +0800 Julian Sun <sunjunchao@bytedance.com> wrote:
> 
>> Foreign dirty flushing handles cases where a dirtied folio's memcg and
>> the memcg owning its inode wb differ.
>>
>> wbc_detach_inode() notes that "concurrent write sharing of an inode is
>> expected to be very rare". This expectation does not hold for bdev inodes.
>> A bdev inode and mapping are shared by buffered users of the block device,
>> while the inode has a single wb owner. On ext4, buffer-cache folios used
>> by metadata and journal I/O can therefore be charged to different memcgs
>> while sharing the same mapping. Normal bdev activity can thus produce
>> frequent folio/wb ownership mismatches.
>>
>> This was reproduced on cgroup v2 with the memory and I/O controllers
>> enabled, using ext4 on a loop device. jbd2 repeatedly generated
>> track_foreign_dirty events for bdev folios charged to unrelated memcgs.
>> When one of those memcgs entered dirty throttling, the resulting record led
>> to a flush_foreign event and queued writeback with reason=foreign_flush for
>> the bdev inode's root wb.
>>
>> Skip foreign dirty tracking when the folio mapping's host inode is on the
>> blockdev pseudo superblock. This prevents bdev-originated records from
>> triggering later foreign flushes.
>>
>> The tradeoff is that dirty throttling in the folio's memcg no longer uses
>> bdev dirtiness to queue writeback on the bdev inode's owner wb. Those
>> folios remain subject to normal writeback. This is preferable because a
>> single bdev wb owner does not identify which cgroup is responsible for a
>> mapping shared by buffered bdev users. Flushing it can write back data
>> unrelated to the throttled memcg. Dirty accounting, normal writeback, and
>> foreign dirty tracking for non-bdev inodes remain unchanged.
> 
> When fixing something, please always describe the userspace-visible
> runtime effects.

Sorry, I overlooked this aspect.
> 
> I'm thinking probably "unnecessary I/O"?  If so, has this been
> quantified/measured?  If not, can the effects be guessed about?

We suspect that a large amount of prematurely triggered foreign 
writeback consumes device writeback bandwidth, thereby degrading the 
performance of the entire writeback path.

On one production machine, the total nr_pages of the foreign-flush works 
was 630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of 
memory. Although nr_pages does not represent the number of dirty pages 
that will ultimately be written back, the actual number of pages written 
back may still be very large because the corresponding wb was 
continuously accumulating dirty pages.>
>> --- a/mm/memcontrol.c
>> +++ b/mm/memcontrol.c
>> @@ -3898,6 +3898,15 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
>>   	u64 oldest_at = now;
>>   	int oldest = -1;
>>   	int i;
>> +	struct address_space *mapping = folio_mapping(folio);
>> +	struct inode *bdev_inode = mapping ? mapping->host : NULL;
>> +
>> +	/*
>> +	 * Bdev inodes are usually shared by many memcgs so foreign
>> +	 * tracking leads to frequent flushes which is counterproductive.
>> +	 */
>> +	if (bdev_inode && sb_is_blkdev_sb(bdev_inode->i_sb))
>> +		return;
>>   
>>   	trace_track_foreign_dirty(folio, wb);
>>   
>> -- 
>> 2.39.5

Thanks,
-- 
Julian Sun <sunjunchao@bytedance.com>

Re: [External] Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Jan Kara]

On Fri 04-09-26 11:39:23, Julian Sun wrote:
> On 9/4/26 3:34 AM, Andrew Morton wrote:
> > On Thu,  3 Sep 2026 16:33:03 +0800 Julian Sun <sunjunchao@bytedance.com> wrote:
> > 
> > > Foreign dirty flushing handles cases where a dirtied folio's memcg and
> > > the memcg owning its inode wb differ.
> > > 
> > > wbc_detach_inode() notes that "concurrent write sharing of an inode is
> > > expected to be very rare". This expectation does not hold for bdev inodes.
> > > A bdev inode and mapping are shared by buffered users of the block device,
> > > while the inode has a single wb owner. On ext4, buffer-cache folios used
> > > by metadata and journal I/O can therefore be charged to different memcgs
> > > while sharing the same mapping. Normal bdev activity can thus produce
> > > frequent folio/wb ownership mismatches.
> > > 
> > > This was reproduced on cgroup v2 with the memory and I/O controllers
> > > enabled, using ext4 on a loop device. jbd2 repeatedly generated
> > > track_foreign_dirty events for bdev folios charged to unrelated memcgs.
> > > When one of those memcgs entered dirty throttling, the resulting record led
> > > to a flush_foreign event and queued writeback with reason=foreign_flush for
> > > the bdev inode's root wb.
> > > 
> > > Skip foreign dirty tracking when the folio mapping's host inode is on the
> > > blockdev pseudo superblock. This prevents bdev-originated records from
> > > triggering later foreign flushes.
> > > 
> > > The tradeoff is that dirty throttling in the folio's memcg no longer uses
> > > bdev dirtiness to queue writeback on the bdev inode's owner wb. Those
> > > folios remain subject to normal writeback. This is preferable because a
> > > single bdev wb owner does not identify which cgroup is responsible for a
> > > mapping shared by buffered bdev users. Flushing it can write back data
> > > unrelated to the throttled memcg. Dirty accounting, normal writeback, and
> > > foreign dirty tracking for non-bdev inodes remain unchanged.
> > 
> > When fixing something, please always describe the userspace-visible
> > runtime effects.
> 
> Sorry, I overlooked this aspect.
> > 
> > I'm thinking probably "unnecessary I/O"?  If so, has this been
> > quantified/measured?  If not, can the effects be guessed about?
> 
> We suspect that a large amount of prematurely triggered foreign writeback
> consumes device writeback bandwidth, thereby degrading the performance of
> the entire writeback path.
> 
> On one production machine, the total nr_pages of the foreign-flush works was
> 630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of memory.
> Although nr_pages does not represent the number of dirty pages that will
> ultimately be written back, the actual number of pages written back may
> still be very large because the corresponding wb was continuously
> accumulating dirty pages.

As Tejun also wrote, it would be good to really quantify these effects -
best by demonstrating how this change improved writeback behavior (and we
are not so much interested in the number of submitted writeback works but
more in the observed load on the storage or completion of some workload).
The problem with your solution is that it has its downsided as well
(possible stalls in dirty throttling) so it should be very clear what are
we gaining to outweight this risk.

In principle I agree that a lot of submitted writeback works can degrade
writeback behavior - a lot of workloads has some set of folios that is
practically permanently dirty and if you are pushing writeback too much, we
will be just writing these folios over and over again without useful work
really done. Also if there's comparably small amount of folios dirty, the
IO resulting from writeback is generally less efficient. But how severe
these effects can get in case you describe is not clear...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Julian Sun]

On 9/4/26 5:00 PM, Jan Kara wrote:
> On Fri 04-09-26 11:39:23, Julian Sun wrote:
>> On 9/4/26 3:34 AM, Andrew Morton wrote:
>>> On Thu,  3 Sep 2026 16:33:03 +0800 Julian Sun <sunjunchao@bytedance.com> wrote:
>>>
>>>> Foreign dirty flushing handles cases where a dirtied folio's memcg and
>>>> the memcg owning its inode wb differ.
>>>>
>>>> wbc_detach_inode() notes that "concurrent write sharing of an inode is
>>>> expected to be very rare". This expectation does not hold for bdev inodes.
>>>> A bdev inode and mapping are shared by buffered users of the block device,
>>>> while the inode has a single wb owner. On ext4, buffer-cache folios used
>>>> by metadata and journal I/O can therefore be charged to different memcgs
>>>> while sharing the same mapping. Normal bdev activity can thus produce
>>>> frequent folio/wb ownership mismatches.
>>>>
>>>> This was reproduced on cgroup v2 with the memory and I/O controllers
>>>> enabled, using ext4 on a loop device. jbd2 repeatedly generated
>>>> track_foreign_dirty events for bdev folios charged to unrelated memcgs.
>>>> When one of those memcgs entered dirty throttling, the resulting record led
>>>> to a flush_foreign event and queued writeback with reason=foreign_flush for
>>>> the bdev inode's root wb.
>>>>
>>>> Skip foreign dirty tracking when the folio mapping's host inode is on the
>>>> blockdev pseudo superblock. This prevents bdev-originated records from
>>>> triggering later foreign flushes.
>>>>
>>>> The tradeoff is that dirty throttling in the folio's memcg no longer uses
>>>> bdev dirtiness to queue writeback on the bdev inode's owner wb. Those
>>>> folios remain subject to normal writeback. This is preferable because a
>>>> single bdev wb owner does not identify which cgroup is responsible for a
>>>> mapping shared by buffered bdev users. Flushing it can write back data
>>>> unrelated to the throttled memcg. Dirty accounting, normal writeback, and
>>>> foreign dirty tracking for non-bdev inodes remain unchanged.
>>>
>>> When fixing something, please always describe the userspace-visible
>>> runtime effects.
>>
>> Sorry, I overlooked this aspect.
>>>
>>> I'm thinking probably "unnecessary I/O"?  If so, has this been
>>> quantified/measured?  If not, can the effects be guessed about?
>>
>> We suspect that a large amount of prematurely triggered foreign writeback
>> consumes device writeback bandwidth, thereby degrading the performance of
>> the entire writeback path.
>>
>> On one production machine, the total nr_pages of the foreign-flush works was
>> 630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of memory.
>> Although nr_pages does not represent the number of dirty pages that will
>> ultimately be written back, the actual number of pages written back may
>> still be very large because the corresponding wb was continuously
>> accumulating dirty pages.
> 
> As Tejun also wrote, it would be good to really quantify these effects -
> best by demonstrating how this change improved writeback behavior (and we
> are not so much interested in the number of submitted writeback works but
> more in the observed load on the storage or completion of some workload).
> The problem with your solution is that it has its downsided as well
> (possible stalls in dirty throttling) so it should be very clear what are
> we gaining to outweight this risk.
> 
> In principle I agree that a lot of submitted writeback works can degrade
> writeback behavior - a lot of workloads has some set of folios that is
> practically permanently dirty and if you are pushing writeback too much, we
> will be just writing these folios over and over again without useful work
> really done. Also if there's comparably small amount of folios dirty, the
> IO resulting from writeback is generally less efficient. But how severe
> these effects can get in case you describe is not clear...

Thanks for the suggestion. Following your point about repeatedly writing hot
pages, I tested buffered overwrites alongside a concurrent direct-I/O workload.
Skipping bdev tracking reduced actual writes to the overwritten file by 69.6%
and the concurrent workload's completion time by 14.4%.

I also reproduced the downside you described: a metadata-heavy workload
completed 31-33% more slowly with the patch, including final sync. So these
results do not establish that blanket skipping is a net improvement.

I've sent the detailed test setup and results in a separate reply.

https://lore.kernel.org/cgroups/apnYouiUINQR-98e@slm.duckdns.org/T/#mdc87094720abce0162c14cdf8f3d0ab218eea686
> 
> 								Honza

Thanks,
-- 
Julian Sun <sunjunchao@bytedance.com>

Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Tejun Heo]

Hello,

On Thu, Sep 03, 2026 at 04:33:03PM +0800, Julian Sun wrote:
> Foreign dirty flushing handles cases where a dirtied folio's memcg and
> the memcg owning its inode wb differ.
> 
> wbc_detach_inode() notes that "concurrent write sharing of an inode is
> expected to be very rare". This expectation does not hold for bdev inodes.
> A bdev inode and mapping are shared by buffered users of the block device,
> while the inode has a single wb owner. On ext4, buffer-cache folios used
> by metadata and journal I/O can therefore be charged to different memcgs
> while sharing the same mapping. Normal bdev activity can thus produce
> frequent folio/wb ownership mismatches.
> 
> This was reproduced on cgroup v2 with the memory and I/O controllers
> enabled, using ext4 on a loop device. jbd2 repeatedly generated
> track_foreign_dirty events for bdev folios charged to unrelated memcgs.
> When one of those memcgs entered dirty throttling, the resulting record led
> to a flush_foreign event and queued writeback with reason=foreign_flush for
> the bdev inode's root wb.

Was this reproduced from an actual workload with adverse effects? If so, can
you please explain the workload and effects with concrete details?

> Skip foreign dirty tracking when the folio mapping's host inode is on the
> blockdev pseudo superblock. This prevents bdev-originated records from
> triggering later foreign flushes.

If you do this, tho, that means now cgroups that accumulated a lot of
metadata writes on ext4 and crunched for memory don't have a way to relieve
the pressure outside of periodic or other lucky flushes. ie. I have a hard
time judging whether this is net plus or not without learning more about how
this patch came to be.

Thanks.

-- 
tejun

Re: [External] Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Julian Sun]

On 9/4/26 4:29 AM, Tejun Heo wrote:
> Hello,
> 
> On Thu, Sep 03, 2026 at 04:33:03PM +0800, Julian Sun wrote:
>> Foreign dirty flushing handles cases where a dirtied folio's memcg and
>> the memcg owning its inode wb differ.
>>
>> wbc_detach_inode() notes that "concurrent write sharing of an inode is
>> expected to be very rare". This expectation does not hold for bdev inodes.
>> A bdev inode and mapping are shared by buffered users of the block device,
>> while the inode has a single wb owner. On ext4, buffer-cache folios used
>> by metadata and journal I/O can therefore be charged to different memcgs
>> while sharing the same mapping. Normal bdev activity can thus produce
>> frequent folio/wb ownership mismatches.
>>
>> This was reproduced on cgroup v2 with the memory and I/O controllers
>> enabled, using ext4 on a loop device. jbd2 repeatedly generated
>> track_foreign_dirty events for bdev folios charged to unrelated memcgs.
>> When one of those memcgs entered dirty throttling, the resulting record led
>> to a flush_foreign event and queued writeback with reason=foreign_flush for
>> the bdev inode's root wb.
> 
> Was this reproduced from an actual workload with adverse effects? If so, can
> you please explain the workload and effects with concrete details?

Yes, this can be reproduced on many production machines. On one 
production machine, we observed 102 wb works with 
WB_REASON_FOREIGN_FLUSH as their reason, 101 of which belonged to the 
same wb. This wb was the owner of the bdev inode, and dirty pages were 
continuously being generated for it.

The logic here is that, Once a memcg has recorded the owner wb of a bdev 
inode as foreign, any task in that memcg that reaches the throttling 
path in balance_dirty_pages() may queue one WB_REASON_FOREIGN_FLUSH work 
item for each recently recorded foreign wb, up to four in total, without 
checking whether those target wbs caused the current dirty throttling.

On this machine, the total nr_pages of the foreign-flush works was 
630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of 
memory. Although nr_pages does not represent the number of dirty pages 
that will ultimately be written back, the actual number of pages written 
back may still be very large because the corresponding wb was 
continuously accumulating dirty pages.

> 
>> Skip foreign dirty tracking when the folio mapping's host inode is on the
>> blockdev pseudo superblock. This prevents bdev-originated records from
>> triggering later foreign flushes.
> 
> If you do this, tho, that means now cgroups that accumulated a lot of
> metadata writes on ext4 and crunched for memory don't have a way to relieve
> the pressure outside of periodic or other lucky flushes. ie. I have a hard
> time judging whether this is net plus or not without learning more about how
> this patch came to be.

How about this approach? Before queuing a WB_REASON_FOREIGN_FLUSH work 
item, check the target wb and avoid queuing another one if it already 
has an unfinished WB_REASON_FOREIGN_FLUSH work item.

This approach can eliminate a large number of duplicate foreign flushes, 
but one issue remains: a memcg may dirty only a small number of pages in 
a bdev inode, and when that memcg enters dirty throttling, the 
throttling may be completely unrelated to the wb that owns the bdev 
inode, yet a foreign flush is still queued to that wb. This problem 
becomes more pronounced when the wb has a large number of dirty pages. 
Perhaps we should track the number of foreign pages in the memcg and 
avoid issuing a foreign flush when the number is below a certain 
percentage?

> 
> Thanks.
> 

Thanks,
-- 
Julian Sun <sunjunchao@bytedance.com>

Re: [External] Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Tejun Heo]

Hello,

On Fri, Sep 04, 2026 at 12:34:27PM +0800, Julian Sun wrote:
...
> > Was this reproduced from an actual workload with adverse effects? If so, can
> > you please explain the workload and effects with concrete details?
> 
> Yes, this can be reproduced on many production machines. On one production
> machine, we observed 102 wb works with WB_REASON_FOREIGN_FLUSH as their
> reason, 101 of which belonged to the same wb. This wb was the owner of the
> bdev inode, and dirty pages were continuously being generated for it.

This is the mechanism triggering.

> The logic here is that, Once a memcg has recorded the owner wb of a bdev
> inode as foreign, any task in that memcg that reaches the throttling path in
> balance_dirty_pages() may queue one WB_REASON_FOREIGN_FLUSH work item for
> each recently recorded foreign wb, up to four in total, without checking
> whether those target wbs caused the current dirty throttling.
> 
> On this machine, the total nr_pages of the foreign-flush works was
> 630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of memory.
> Although nr_pages does not represent the number of dirty pages that will
> ultimately be written back, the actual number of pages written back may
> still be very large because the corresponding wb was continuously
> accumulating dirty pages.

Is this necessarily an adverse effect tho? These are all metadata flushes,
right? When they're gonna get written might change but do the extra flushes
change how much is going to be written? If so, how? If not, are the extra
flushes adding noticeable overhead in terms of cpu or io?

> > > Skip foreign dirty tracking when the folio mapping's host inode is on the
> > > blockdev pseudo superblock. This prevents bdev-originated records from
> > > triggering later foreign flushes.
> > 
> > If you do this, tho, that means now cgroups that accumulated a lot of
> > metadata writes on ext4 and crunched for memory don't have a way to relieve
> > the pressure outside of periodic or other lucky flushes. ie. I have a hard
> > time judging whether this is net plus or not without learning more about how
> > this patch came to be.
> 
> How about this approach? Before queuing a WB_REASON_FOREIGN_FLUSH work item,
> check the target wb and avoid queuing another one if it already has an
> unfinished WB_REASON_FOREIGN_FLUSH work item.
> 
> This approach can eliminate a large number of duplicate foreign flushes, but
> one issue remains: a memcg may dirty only a small number of pages in a bdev
> inode, and when that memcg enters dirty throttling, the throttling may be
> completely unrelated to the wb that owns the bdev inode, yet a foreign flush
> is still queued to that wb. This problem becomes more pronounced when the wb
> has a large number of dirty pages. Perhaps we should track the number of
> foreign pages in the memcg and avoid issuing a foreign flush when the number
> is below a certain percentage?

Yeah, maybe, but I'm kinda having a hard time evaluating anything as the
mental picture I have of the problem is too incomplete. Please fill us in.

Thanks.

-- 
tejun

Re: [External] Re: [PATCH v2] writeback, memcg: skip foreign dirty tracking for bdev inodes

[Julian Sun]

On 9/4/26 2:36 PM, Tejun Heo wrote:
> Hello,
> 
> On Fri, Sep 04, 2026 at 12:34:27PM +0800, Julian Sun wrote:
> ...
>>> Was this reproduced from an actual workload with adverse effects? If so, can
>>> you please explain the workload and effects with concrete details?
>>
>> Yes, this can be reproduced on many production machines. On one production
>> machine, we observed 102 wb works with WB_REASON_FOREIGN_FLUSH as their
>> reason, 101 of which belonged to the same wb. This wb was the owner of the
>> bdev inode, and dirty pages were continuously being generated for it.
> 
> This is the mechanism triggering.
> 
>> The logic here is that, Once a memcg has recorded the owner wb of a bdev
>> inode as foreign, any task in that memcg that reaches the throttling path in
>> balance_dirty_pages() may queue one WB_REASON_FOREIGN_FLUSH work item for
>> each recently recorded foreign wb, up to four in total, without checking
>> whether those target wbs caused the current dirty throttling.
>>
>> On this machine, the total nr_pages of the foreign-flush works was
>> 630,681,285, or about 2.4 TiB, while the machine had only 400 GiB of memory.
>> Although nr_pages does not represent the number of dirty pages that will
>> ultimately be written back, the actual number of pages written back may
>> still be very large because the corresponding wb was continuously
>> accumulating dirty pages.
> 
> Is this necessarily an adverse effect tho? These are all metadata flushes,
> right? 

No. The foreign dirty records in this case are triggered by bdev inode
folios used for metadata and journal I/O, but the resulting writeback
work is not restricted to those folios or even to the bdev inode.
cgroup_writeback_by_id() queues writeback for the _entire_ target wb,
so it may write any eligible dirty inode attached to that wb.

> When they're gonna get written might change but do the extra flushes
> change how much is going to be written? If so, how? If not, are the extra
> flushes adding noticeable overhead in terms of cpu or io?

Yes, in our test, the extra foreign flushes increased device writes and
noticeably degraded the concurrent workload's performance. These flushes
also wrote ordinary file data in the target wb, not just bdev metadata.
More frequent writeback of the continuously overwritten file reduced
overwrite coalescing, generating more device I/O for the same logical
writes and leaving less bandwidth for the concurrent workload.

I tested both kernels in a QEMU/KVM VM with 4 vCPUs, 8 GiB RAM, cgroup v2
and ext4. The kernels use the same base and configuration; ordinary
periodic writeback remains enabled.

For the overwrite test, virtual NVMe write bandwidth is capped at 200
MiB/s. fio repeatedly overwrites a 256 MiB file in the root wb using
buffered I/O. Four memory- and I/O-limited memcgs generate metadata and
buffered writes, triggering dirty throttling and foreign flushes to that
wb. Concurrent fio measures completion of 16 GiB of sequential direct
writes on the same filesystem.

Across five pairs, both kernels completed the same 9 GiB of buffered
overwrites per run. Skipping bdev tracking reduced mean actual device
writes for that file from 5.60 GiB to 1.70 GiB (69.6%), including final
sync. Mean foreground completion time fell from 104.2 s to 89.1 s (14.4%),
bandwidth rose from 157.3 to 183.8 MiB/s, and p99 completion latency fell
from 1434 ms to 480 ms.

However, a separate metadata test confirmed the downside. With uncapped
storage and memory.high=256 MiB, a memcg creates 60,000 files paced at
4,000 files/s, each with a distinct 128-byte xattr, alongside occasional
buffered data writes. The xattrs occupy external 4 KiB blocks in this
setup. Across two warning-free pairs, skipping bdev tracking made
completion 31-33% slower including final sync, while device writes remained
around 524 MiB.

The metadata regression means these results do not establish a net benefit
from blanket skipping. Whether foreign flushes caused our production stalls
remains unconfirmed.

Core fio commands:

     fio --name=hotspot --filename="$dir/hot.data" --rw=write \
         --bs=64K --ioengine=psync --direct=0 --thread=1 \
         --size=256M --io_size=9216M --rate=64M \
         --refill_buffers=1 --invalidate=0 --eta=never \
         --output-format=json --output="$out/target.json" &
     hot_pid=$!


     fio --name=foreground --filename="$dir/foreground.data" \
         --rw=write --bs=1M --ioengine=libaio --iodepth=32 \
         --direct=1 --thread=1 --invalidate=0 --size=4096M \
         --io_size=16384M --eta=never --output-format=json \
         --output="$out/foreground.json"


The full five-pair overwrite log follows: base=original, skip=skip bdev
tracking, hot_GiB=device writes for the repeatedly overwritten file
including final sync.

[2026-09-05 17:52:10] RESULTS=/tmp/cgwb-20260905-175145; WARNING: reformats /dev/nvme5n1; 5 pair(s), limit=200 MiB/s
[2026-09-05 17:52:10] round-001-base: booting
[2026-09-05 17:52:20] round-001-base: running workload
[2026-09-05 17:55:36] round-001-skip: booting
[2026-09-05 17:55:45] round-001-skip: running workload
[2026-09-05 17:58:57] ROUND 1: kind     seconds    MiB/s      IOPS   mean_ms    p99_ms   hot_GiB WARN
[2026-09-05 17:58:57]              base    104.574   156.67    156.67    203.82   1434.45     5.862    0
[2026-09-05 17:58:57]              skip     89.125   183.83    183.83    173.62    480.25     1.656    0
[2026-09-05 17:58:57] skip vs base: bandwidth +17.33%, time -14.77%, hotspot IO -71.75%
[2026-09-05 17:58:57] round-002-skip: booting
[2026-09-05 17:59:06] round-002-skip: running workload
[2026-09-05 18:02:18] round-002-base: booting
[2026-09-05 18:02:27] round-002-base: running workload
[2026-09-05 18:05:42] ROUND 2: kind     seconds    MiB/s      IOPS   mean_ms    p99_ms   hot_GiB WARN
[2026-09-05 18:05:42]              base    103.533   158.25    158.25    201.75   1434.45     5.493    0
[2026-09-05 18:05:42]              skip     89.124   183.83    183.83    173.65    480.25     1.730    0
[2026-09-05 18:05:42] skip vs base: bandwidth +16.17%, time -13.92%, hotspot IO -68.50%
[2026-09-05 18:05:42] round-003-base: booting
[2026-09-05 18:05:52] round-003-base: running workload
[2026-09-05 18:09:06] round-003-skip: booting
[2026-09-05 18:09:15] round-003-skip: running workload
[2026-09-05 18:12:28] ROUND 3: kind     seconds    MiB/s      IOPS   mean_ms    p99_ms   hot_GiB WARN
[2026-09-05 18:12:28]              base    104.648   156.56    156.56    203.96   1434.45     5.334    1
[2026-09-05 18:12:28]              skip     89.243   183.59    183.59    173.92    480.25     1.824    0
[2026-09-05 18:12:28] skip vs base: bandwidth +17.26%, time -14.72%, hotspot IO -65.80%
[2026-09-05 18:12:28] WARN: runtime diagnostics present; inspect each data/dmesg before using this pair.
[2026-09-05 18:12:28] round-004-skip: booting
[2026-09-05 18:12:37] round-004-skip: running workload
[2026-09-05 18:15:50] round-004-base: booting
[2026-09-05 18:16:00] round-004-base: running workload
[2026-09-05 18:19:15] ROUND 4: kind     seconds    MiB/s      IOPS   mean_ms    p99_ms   hot_GiB WARN
[2026-09-05 18:19:15]              base    103.565   158.20    158.20    201.84   1434.45     5.497    0
[2026-09-05 18:19:15]              skip     89.124   183.83    183.83    173.57    480.25     1.648    0
[2026-09-05 18:19:15] skip vs base: bandwidth +16.20%, time -13.94%, hotspot IO -70.01%
[2026-09-05 18:19:15] round-005-base: booting
[2026-09-05 18:19:24] round-005-base: running workload
[2026-09-05 18:22:41] round-005-skip: booting
[2026-09-05 18:22:50] round-005-skip: running workload
[2026-09-05 18:26:03] ROUND 5: kind     seconds    MiB/s      IOPS   mean_ms    p99_ms   hot_GiB WARN
[2026-09-05 18:26:03]              base    104.567   156.68    156.68    203.78   1434.45     5.830    1
[2026-09-05 18:26:03]              skip     89.035   184.02    184.02    173.47    480.25     1.648    0
[2026-09-05 18:26:03] skip vs base: bandwidth +17.44%, time -14.85%, hotspot IO -71.72%
[2026-09-05 18:26:03] WARN: runtime diagnostics present; inspect each data/dmesg before using this pair.
[2026-09-05 18:26:03] DONE: 5 pair(s); all test VMs stopped

> 
>>>> Skip foreign dirty tracking when the folio mapping's host inode is on the
>>>> blockdev pseudo superblock. This prevents bdev-originated records from
>>>> triggering later foreign flushes.
>>>
>>> If you do this, tho, that means now cgroups that accumulated a lot of
>>> metadata writes on ext4 and crunched for memory don't have a way to relieve
>>> the pressure outside of periodic or other lucky flushes. ie. I have a hard
>>> time judging whether this is net plus or not without learning more about how
>>> this patch came to be.
>>
>> How about this approach? Before queuing a WB_REASON_FOREIGN_FLUSH work item,
>> check the target wb and avoid queuing another one if it already has an
>> unfinished WB_REASON_FOREIGN_FLUSH work item.
>>
>> This approach can eliminate a large number of duplicate foreign flushes, but
>> one issue remains: a memcg may dirty only a small number of pages in a bdev
>> inode, and when that memcg enters dirty throttling, the throttling may be
>> completely unrelated to the wb that owns the bdev inode, yet a foreign flush
>> is still queued to that wb. This problem becomes more pronounced when the wb
>> has a large number of dirty pages. Perhaps we should track the number of
>> foreign pages in the memcg and avoid issuing a foreign flush when the number
>> is below a certain percentage?
> 
> Yeah, maybe, but I'm kinda having a hard time evaluating anything as the
> mental picture I have of the problem is too incomplete. Please fill us in.
> 
> Thanks.
> 

Thanks,
-- 
Julian Sun <sunjunchao@bytedance.com>