From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8195CD1283 for ; Thu, 28 Mar 2024 20:13:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4006D6B0092; Thu, 28 Mar 2024 16:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B0EF6B009B; Thu, 28 Mar 2024 16:13:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 279B26B009D; Thu, 28 Mar 2024 16:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 05D926B0092 for ; Thu, 28 Mar 2024 16:13:33 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AC133A0265 for ; Thu, 28 Mar 2024 20:13:32 +0000 (UTC) X-FDA: 81947547864.05.6F0FC56 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf26.hostedemail.com (Postfix) with ESMTP id 9A96314001E for ; Thu, 28 Mar 2024 20:13:30 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AtovYqU3; spf=pass (imf26.hostedemail.com: domain of htejun@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=htejun@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711656810; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TJh0w+6LNIpaseEpVPREtMFaluSVUJYpCPOmmi1HWNg=; b=3+Cw++9obAFRXN339Yw5c1BDxjy6MXCUtKm23zbsqIGSiHtGaWCBDrG78gai7Zk4M8oGE6 q/iI5/vMJuZCse8DD7rW82beAiO3jSpIVPs7berIwmMCGoVirq2ov+mq7ZP6OpPfW+LQTF Dcyln+zA3MqhRX7soIaJXRJpAwGZ5Hk= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AtovYqU3; spf=pass (imf26.hostedemail.com: domain of htejun@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=htejun@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711656810; a=rsa-sha256; cv=none; b=8goNsYWvrq/qTn1avjofdPrE7+MumbnLWqxewcjRCNn3thUbudZswORzCOPOSfy7xgZcIg 0wmZzb63K17D6s86Le3gDsKc1ArcyMzpUxBSflO6pbp60YhI1PjbAtaHW3ehIIJGvu90BO paLhkBs1Nzqzy0f5aalTgWrrG2SN4LY= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1e0ae065d24so12611635ad.1 for ; Thu, 28 Mar 2024 13:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711656809; x=1712261609; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=TJh0w+6LNIpaseEpVPREtMFaluSVUJYpCPOmmi1HWNg=; b=AtovYqU3xC1BKZKjxm8ZrLUM/xGJZ4f67c35sDcAM5q+fk3nVzncEQMrQc2lJedqdw tVNh7mmdu/0xM05UKcb5Wm3ZKVecgKhgFD7vUueEkTu1flg5waUikCS3Yw4OFSqD8ZS7 xKPybNaqactVMF767jgQCgCyXqTL4q2xddCwJKcXpQRm2Bs4ipDsKnD1pV5tGd432oCd KGADm6ho3blX7QE9DxSnkAF9xDY7tTcHfRIbtdn1Tnx6H3XDo/o4aJ1quGuBbmU3nsve HW2niRTk/lfJr8/v2FMmeXsz+2T49NwS88qCopCT5rBF7hF6yOqnVqXQ9zyiCZwHkh3m tpKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711656809; x=1712261609; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TJh0w+6LNIpaseEpVPREtMFaluSVUJYpCPOmmi1HWNg=; b=GQ2v/GuJYwMf9iQcpdOq5t6/ZoaUWMpi78624j2fGPyr/neZLei28owt0A/BHxWx13 6CvwOUskzb1FGT0GexDWqHUuOEzRn1jTqnDnmLZDdQ2bD9S9x27ODiZ/AqIJVddIJclh n7/w7DijqWDGmeXwlISUu0m3CSCCaFPaN3+SgxGBEu5ItK3tVhwo1gPN6HGM+/w6ElMf 7qP0oGQDftn3bDb78kV0gMxJdxSvP1EjN/oJ+B56GxPEr2p0SeAFI7C/KO/s3fXjhaxl 5hpOo19L1F4a1kEkJSUW/bvgwb0TstWD74npZyIFgHqewLcdetQXsj/uiNcL0sPW3vMq c+QA== X-Forwarded-Encrypted: i=1; AJvYcCUqCjO2TWgKxvKq4xHsbyO0AE5fqvH/XCbEbPkg7hu+9nPx3IWRIZJu5Df5Pu9+wmH6SKbSnDy8t1ujUMOPDHK7lPY= X-Gm-Message-State: AOJu0YyOGIduBCnfNTO41ToTdMbvAopHYyHTs4Y2j36Q0CTdNhk11e+I EhiUV8pCdK6NyVnbeUjq6gRksJLGJWWR3yCoUaIJdJewI2Qur4Vp X-Google-Smtp-Source: AGHT+IF2RZEilfGlheBdjw01iR0oba703+q/sHwMzEEvWDchg6pVds9Ll4DvJwEw0oTnZsBiTFQzQg== X-Received: by 2002:a17:902:8341:b0:1de:fd12:1929 with SMTP id z1-20020a170902834100b001defd121929mr485333pln.55.1711656809385; Thu, 28 Mar 2024 13:13:29 -0700 (PDT) Received: from localhost (dhcp-141-239-158-86.hawaiiantel.net. [141.239.158.86]) by smtp.gmail.com with ESMTPSA id f21-20020a170902e99500b001e208707048sm2041353plb.117.2024.03.28.13.13.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Mar 2024 13:13:29 -0700 (PDT) Date: Thu, 28 Mar 2024 10:13:27 -1000 From: Tejun Heo To: Kent Overstreet Cc: Kemeng Shi , akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240327155751.3536-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 9A96314001E X-Rspam-User: X-Stat-Signature: 9f9z4zxfoin8f8espktyyjxazdfjycxw X-Rspamd-Server: rspam01 X-HE-Tag: 1711656810-812837 X-HE-Meta: U2FsdGVkX1/6ymV4sASiElHdgHvCd5mxUZdLEu9VUuiwWA9Y3ZB5DzRnj+TDmcCVUyB6lvvVuZX4BjYHSxyRz9I3NqiyfsO5uBXzv/vpJMGDNIAJ9X3r1t2mhH3aPcTDDhmNrtzLLvYm0taxOyxbJek71dmEMK0cJ2i2WpnLM6aCjrGT3Q0ojsk2/NrUgLARvz4FXshs0lbPcjhV68hBS8uqsGGtYGQY/ANvhIgCYh4SS7JETwytYU1mT/kUOfS4MUOOABQNAQZ9A8L4TiK7NLei4kppQwRwLGLdyLj6m5CgobIjhDKgrcdtGPjpRCceZ1HYX0bENZOiN3QpgNvsSEdQJZB+ucY4aAbVBO/DBNMTpAjvEeMijbqNYZQNFq15DXttee8kcHjkuzQzU/KOEmh2MUYIQD/XqiV4CAudpn54rlLulyTs29KZRJ8HuPqQjGobn0fZBi4oAYxqpY96f1XkdJeFP4caoMGUXB0NmggxJz4kIt1/8TloJDW1fOwqFZN+yha0OpbR0skRSQoKWnYfr0k7FEKO30kjlXHsxgNWifUbUtoFvjiMMxmESTj1cLWmvdaLVii18kTCzMNxmEMQt0jppSLkekYXE1hZ7h8XDBXc1kX4Br6cmFB9W5v4ZaBAKOJRC2sVxUev6i2CsTqtuRGwgyb+XxwAbpuv5MLboq0ISQpHatYEDjOCQH+R/tI10tHk8tGHtU5PdtD8HOq9C5JlfCBOi+MlISbcQymnGNc+U0DMGmIzymg19LKt2h/SYKzws0YPG+jPB3VkUQOaN+ir1+N9ZR6dlNz7Hq068bYDTHiuON3+1M4gnwJA2NyIemLsR8n7Zb9WZ4d3tQK7T3hXHncRxa6uB87N3egx2m9XB2q0mxts74MTnW7zeu9om46Jc+JxKzJMatl9PXDUOeGNZK7d47ew9Qf8ztgpS8cMPyQfpm+8PU4DrTWq9T34LrhfeDTZdexKo+A d7O1C9Cu 0XcVwmdf8EucXMkuxu506SIJl9WllrizQvv+jlIaXjc4gYXngOkuaxzKze9Yx51kRNpM1tcEmvXlNLDxIZUhkpvnfNOZgTeBigEAhSClL2EmX7Er1BmX5gY1hrg5sTgLsxrkrilzoyLjM3iDHIuIoF/tZKF+fA1Nvo1Ak9+VJoFwDtZc2flhYxuNGEQqI5msLB+PIzsx5uwp/8txIUZebSI6fPzCdxLOHHhw7aCTDOM2MsMndrEQ/HeR3Qj+G3hT4fE924mmuxFgY3Y1aZsT49YQIzNu26m0sF4epw00mc7n3R8QUw/CLB9N2UcUbH7tPEDVa0wmEjqY7VXjgNU9w2NymDv5sMtdBm0qTiWdCPCdfe66HfPh4Ab+Tu1DE/0ur0KTzSDkjypbyZ52gReb2g05Q/DhY5XDHfGSnuyZW41npkALZl8rPj0Hem+PVZC2YHHXY8SPbHTdJJ1LmJ96C4HKGfzCkOZ71eyLYzak3/DFPw50XoH8N8cgV4a+9VGyIE8sY X-Bogosity: Ham, tests=bogofilter, spamicity=0.040293, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, On Thu, Mar 28, 2024 at 03:55:32PM -0400, Kent Overstreet wrote: > > On Thu, Mar 28, 2024 at 03:40:02PM -0400, Kent Overstreet wrote: > > > Collecting latency numbers at various key places is _enormously_ useful. > > > The hard part is deciding where it's useful to collect; that requires > > > intimate knowledge of the code. Once you're defining those collection > > > poitns statically, doing it with BPF is just another useless layer of > > > indirection. > > > > Given how much flexibility helps with debugging, claiming it useless is a > > stretch. > > Well, what would it add? It depends on the case but here's an example. If I'm seeing occasional tail latency spikes, I'd want to know whether there's any correation with specific types or sizes of IOs and if so who's issuing them and why. With BPF, you can detect those conditions to tag and capture where exactly those IOs are coming from and aggregate the result however you like across thousands of machines in production without anyone noticing. That's useful, no? Also, actual percentile disribution is almost always a lot more insightful than more coarsely aggregated numbers. We can't add all that to fixed infra. In most cases not because runtime overhead would be too hight but because the added interface and code complexity and maintenance overhead isn't justifiable given how niche, adhoc and varied these use cases get. > > > The time stats stuff I wrote is _really_ cheap, and you really want this > > > stuff always on so that you've actually got the data you need when > > > you're bughunting. > > > > For some stats and some use cases, always being available is useful and > > building fixed infra for them makes sense. For other stats and other use > > cases, flexibility is pretty useful too (e.g. what if you want percentile > > distribution which is filtered by some criteria?). They aren't mutually > > exclusive and I'm not sure bdi wb instrumentation is on top of enough > > people's minds. > > > > As for overhead, BPF instrumentation can be _really_ cheap too. We often run > > these programs per packet. > > The main things I want are just > - elapsed time since last writeback IO completed, so we can see at a > glance if it's stalled > - time stats on writeback io initiation to completion > > The main value of this one will be tracking down tail latency issues and > finding out where in the stack they originate. Yeah, I mean, if always keeping those numbers around is useful for wide enough number of users and cases, sure, go ahead and add fixed infra. I'm not quite sure bdi wb stats fall in that bucket given how little attention it usually gets. Thanks. -- tejun