From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A3E5CD1288 for ; Wed, 3 Apr 2024 06:56:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7295F6B0085; Wed, 3 Apr 2024 02:56:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D8CE6B0088; Wed, 3 Apr 2024 02:56:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A1106B0089; Wed, 3 Apr 2024 02:56:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3C6EC6B0085 for ; Wed, 3 Apr 2024 02:56:31 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9C4138082B for ; Wed, 3 Apr 2024 06:56:30 +0000 (UTC) X-FDA: 81967312140.18.1A41CBB Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf14.hostedemail.com (Postfix) with ESMTP id E0156100002 for ; Wed, 3 Apr 2024 06:56:26 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712127389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Klsl7WQx1YpN4Xl52rpt9b5zwwy0T7Ndr1uFL1ASdLU=; b=qdFdVPvLUtwKqb3z2uRpSNeEapneW8gWMFRL3B5t4u8ZB08tjZu7tLHR6JCnZVnug5NZ1h 6pMS9xK+YowttJKn3YZbcBZwfbE04pnpWv3CpBc3rZQPYeRKqgxaHoHYTlwxa/djKxozeD JTBnMXu9lgaKiuF+gFjqlVnBn8e2ITI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712127389; a=rsa-sha256; cv=none; b=LUxADnp2V8k7gkHhtG2uAjQao41RMA4Aa0qM7DzeEorUCjosKG/749YAqRpyxMUzoG3oGf iGH+PWlXcy3/re+iNl/P2ATh1DCUEfLfkcz0/OZTEdU22WycRRinWLD4zyVdyHrzenuLNc vWM5VxFZBlecLPtJTwpglX8+y98ZnHM= Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4V8bBN75G4z4f3nJQ for ; Wed, 3 Apr 2024 14:56:12 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.252]) by mail.maildlp.com (Postfix) with ESMTP id 5E8501A0393 for ; Wed, 3 Apr 2024 14:56:21 +0800 (CST) Received: from [10.174.178.129] (unknown [10.174.178.129]) by APP3 (Coremail) with SMTP id _Ch0CgA39p2S_QxmsaQcIw--.36319S2; Wed, 03 Apr 2024 14:56:20 +0800 (CST) Subject: Re: [PATCH v2 0/6] Improve visibility of writeback To: Kent Overstreet Cc: akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, tj@kernel.org, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org References: <20240327155751.3536-1-shikemeng@huaweicloud.com> From: Kemeng Shi Message-ID: <6d83cf77-a0eb-3ee7-78f2-dae45562a6aa@huaweicloud.com> Date: Wed, 3 Apr 2024 14:56:18 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=gbk Content-Transfer-Encoding: 7bit X-CM-TRANSID:_Ch0CgA39p2S_QxmsaQcIw--.36319S2 X-Coremail-Antispam: 1UD129KBjvJXoWxCFW8uryfZr15Kr4kKryrXrb_yoWrtw4kpF Z5A3W5tr48Zr1xArnaka4aqryYy3y0qFy3Xr97ZFWIyrn0gF15tFykW3yFyF15Zr9xAry3 ZrsxZrykKr1vvaUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUvIb4IE77IF4wAFF20E14v26r4j6ryUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcVAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7Mxk0xIA0c2IE e2xFo4CEbIxvr21l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxV Aqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r1q 6r43MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6x kF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWrZr1j6s0DMIIF0xvEx4A2jsIE 14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf 9x07UWE__UUUUU= X-CM-SenderInfo: 5vklyvpphqwq5kxd4v5lfo033gof0z/ X-Rspamd-Queue-Id: E0156100002 X-Rspam-User: X-Stat-Signature: wkk3qzynjh9cnzim1r7p3x5zjqqbznxk X-Rspamd-Server: rspam01 X-HE-Tag: 1712127386-685116 X-HE-Meta: U2FsdGVkX18eHp5UB4rTPsP711Lh5yK3ghn8sq+4wpM01xK6gG5tDaNw39rUze5UBuG7SePlMy1XAlSiXlI9MtuERK82yU6gTiMQLzOJ04mdAFkiY2IAu4gr81xwRWOOPHzbSeqFurKX3eOTL0Wdf5XRARsIZvy3RC9qrObJ/f6B/LxlBZrremgXFZfQf9Q5C0bNRkPi37t1u01ZR/tpaTFdpO+bhlCtI5Ot9xNcU/nkF94CAxz8jmtx6YeYvUrIPY+scnWXkEjEh7azWpU27kuwGjhqLwNonQheIHrl1XuNyliO5kojVGEkqfbkH05uAeI3+J0/JAST0c/Z6nN6j+PRhIKkTWMGM72DqMDwh39IchWBxDjHWXADAmB0uaQy+pCGGGCPzldtUT9mFslRysFQQV3/gOM/cBVGViJ7Zj68/OLYOxoU8l3w9JaDVX8X31yxvB/KV+L8TrMI0z+SLSMJJRuyCRIh9710ad+vTdSTbBdb5NQpIO5oOuHPwQJ2lfLOch978Y3LKznwLN2UY5aZffhyqcSv52qKvRpaolpZNhs4+SEPMQEJ4hZVaJcT9zcc2TnGh8WVtw8s6NkhIjUBqNJGYUmxalU8ajrwl5oDgMxVJvgEawDaASVTOsbFdINQYqSjfMiZOjQBJzF7TRvPfFqOTnFpe+2znEBXOco8362bzTluD8lCma8f+1f1H5KxkBWdUfYEmiB4cgnLHhuxddAbGYPJcS8SAAyWyVQ7nniP53+dyHD/UQUSQ9fFwNTJ2hRxJyClYyfZYPz9oNGUS4/3XDETl8XNr7gw85URKkZW2ifUZdazDkQiI0rweQjMlmbNv9g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: on 3/29/2024 3:24 AM, Kent Overstreet wrote: > On Wed, Mar 27, 2024 at 11:57:45PM +0800, Kemeng Shi wrote: >> v1->v2: >> -Send cleanup to wq_monitor.py separately. >> -Add patch to avoid use after free of bdi. >> -Rename wb_calc_cg_thresh to cgwb_calc_thresh as Tejun suggested. >> -Use rcu walk to avoid use after free. >> -Add debug output to each related patches. >> >> This series tries to improve visilibity of writeback. Patch 1 make >> /sys/kernel/debug/bdi/xxx/stats show writeback info of whole bdi >> instead of only writeback info in root cgroup. Patch 2 add a new >> debug file /sys/kernel/debug/bdi/xxx/wb_stats to show per wb writeback >> info. Patch 4 add wb_monitor.py to monitor basic writeback info >> of running system, more info could be added on demand. Rest patches >> are some random cleanups. More details can be found in respective >> patches. Thanks! >> This series is on top of patchset [1]. >> >> [1] https://lore.kernel.org/lkml/20240123183332.876854-1-shikemeng@huaweicloud.com/T/#mc6455784a63d0f8aa1a2f5aff325abcdf9336b76 > > Not bad > Hi Kent, > I've been trying to improve our ability to debug latency issues - stalls > of all sorts. While you're looking at all this code, do you think you > could find some places to collect useful latency numbers? I would like to do it to collect more useful info for writeback. > > fs/bcachefs/time_stats.c has some code that's going to be moving out to > lib/ at some point, after I switch it to MAD; if you could hook that up > as well to a few points we could see at a glance if there are stalls > happening in the writeback path. I see that Tejun recommend to use bpf. I don't know much about bpf and new approach in time_stats.c. For me personly, I think that it's better to use the new approach after the work of moving code to lib/ is merged. Then I would like to submit a patchset to discuss of using it in writeback. Would this make sense to you. Look forward to your reply! Thanks, Kemeng > >> >> Following domain hierarchy is tested: >> global domain (320G) >> / \ >> cgroup domain1(10G) cgroup domain2(10G) >> | | >> bdi wb1 wb2 >> >> /* all writeback info of bdi is successfully collected */ >> # cat /sys/kernel/debug/bdi/252:16/stats: >> BdiWriteback: 448 kB >> BdiReclaimable: 1303904 kB >> BdiDirtyThresh: 189914124 kB >> DirtyThresh: 195337564 kB >> BackgroundThresh: 32516508 kB >> BdiDirtied: 3591392 kB >> BdiWritten: 2287488 kB >> BdiWriteBandwidth: 322248 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 2 >> b_dirty_time: 0 >> bdi_list: 1 >> state: 1 >> >> /* per wb writeback info is collected */ >> # cat /sys/kernel/debug/bdi/252:16/wb_stats: >> cat wb_stats >> WbCgIno: 1 >> WbWriteback: 0 kB >> WbReclaimable: 0 kB >> WbDirtyThresh: 0 kB >> WbDirtied: 0 kB >> WbWritten: 0 kB >> WbWriteBandwidth: 102400 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 0 >> b_dirty_time: 0 >> state: 1 >> WbCgIno: 4284 >> WbWriteback: 448 kB >> WbReclaimable: 818944 kB >> WbDirtyThresh: 3096524 kB >> WbDirtied: 2266880 kB >> WbWritten: 1447936 kB >> WbWriteBandwidth: 214036 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 1 >> b_dirty_time: 0 >> state: 5 >> WbCgIno: 4325 >> WbWriteback: 224 kB >> WbReclaimable: 819392 kB >> WbDirtyThresh: 2920088 kB >> WbDirtied: 2551808 kB >> WbWritten: 1732416 kB >> WbWriteBandwidth: 201832 kBps >> b_dirty: 0 >> b_io: 0 >> b_more_io: 1 >> b_dirty_time: 0 >> state: 5 >> >> /* monitor writeback info */ >> # ./wb_monitor.py 252:16 -c >> writeback reclaimable dirtied written avg_bw >> 252:16_1 0 0 0 0 102400 >> 252:16_4284 672 820064 9230368 8410304 685612 >> 252:16_4325 896 819840 10491264 9671648 652348 >> 252:16 1568 1639904 19721632 18081952 1440360 >> >> >> writeback reclaimable dirtied written avg_bw >> 252:16_1 0 0 0 0 102400 >> 252:16_4284 672 820064 9230368 8410304 685612 >> 252:16_4325 896 819840 10491264 9671648 652348 >> 252:16 1568 1639904 19721632 18081952 1440360 >> ... >> >> Kemeng Shi (6): >> writeback: protect race between bdi release and bdi_debug_stats_show >> writeback: collect stats of all wb of bdi in bdi_debug_stats_show >> writeback: support retrieving per group debug writeback stats of bdi >> writeback: add wb_monitor.py script to monitor writeback info on bdi >> writeback: rename nr_reclaimable to nr_dirty in balance_dirty_pages >> writeback: define GDTC_INIT_NO_WB to null >> >> include/linux/writeback.h | 1 + >> mm/backing-dev.c | 203 ++++++++++++++++++++++++++++++---- >> mm/page-writeback.c | 31 ++++-- >> tools/writeback/wb_monitor.py | 172 ++++++++++++++++++++++++++++ >> 4 files changed, 378 insertions(+), 29 deletions(-) >> create mode 100644 tools/writeback/wb_monitor.py >> >> -- >> 2.30.0 >> >