From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E3C1C54E5D for ; Tue, 19 Mar 2024 14:23:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B1FB6B008A; Tue, 19 Mar 2024 10:23:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8609D6B008C; Tue, 19 Mar 2024 10:23:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 728116B0092; Tue, 19 Mar 2024 10:23:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 625C96B008A for ; Tue, 19 Mar 2024 10:23:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1FA39141083 for ; Tue, 19 Mar 2024 14:23:52 +0000 (UTC) X-FDA: 81914007504.23.ABF0F92 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf27.hostedemail.com (Postfix) with ESMTP id 41ECA40009 for ; Tue, 19 Mar 2024 14:23:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=Zaur+9xp; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf27.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710858230; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0jo5XV6mn0FIvuqG3IAFW3tUxkVMyLg5UBql45gg1w4=; b=ZwJfDpqnsWN0E1F7Rbwis/NsLqUAL/nDPyHmljATxlMMz5GdGV4K1hBkX64HpOSRYb479g o332PNxpthTNsi4nFyusLDbyPkpwJezgY/STAP/F/wf568/vnfLYxTW97bFjQl/H8nkOPK zL2RL8X11ovpY8g2q+E92rKfWS1w5KY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=Zaur+9xp; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf27.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710858230; a=rsa-sha256; cv=none; b=8GAql1wJVZymlBT2b9sxqW4wlr5T+h6w2e0UOo/F+HGmacQSI0xIABKjbkVy6IDIzmQR0k cvEaSTWoWfN5I4CEQ71kNqybtWv+hD5a5aRUzOhCSxCdniTfCLp90Oc4PQgK+ydLTIZDN6 Y1NYYzGcOL8lMD5xo+b49pReG1Zop+w= Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-430c63d4da9so17217961cf.0 for ; Tue, 19 Mar 2024 07:23:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710858229; x=1711463029; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=0jo5XV6mn0FIvuqG3IAFW3tUxkVMyLg5UBql45gg1w4=; b=Zaur+9xpBb4hAjI+3oMMZsci6w2Wls9mSFI/vfqBp6UDsv/zZ3hcOtr+Dsf0o5cUVe yaXyB6fny5xuoQ6uNKVcBIJ/fm6GE0Re4gdOWpf0xr/Bs6+TJe3+60s/iQxt5xwP3EfV ++VxV0MNgreUbbeFpYiw9QHfLisfXMHLpetTeQHVJi3AQQdmnyc3dw5MtWeqHGY8/oWj u8IqaHiuk7aYLuJjH5MDyiP/+CS3Z1lv37c5irgc+nzsqT2j6e7Xhxvkx+rOoJJsPAMv +RLiQXeaTrFJiRUp0oHgHNcZFxpvAUGSZuiz5splzjaqIM2L/5j0lVledNGWBDJLcI7S IlNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710858229; x=1711463029; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0jo5XV6mn0FIvuqG3IAFW3tUxkVMyLg5UBql45gg1w4=; b=NekKOdzvAJ1TihVWzQDg5jhSkbL0FngisjNjf7X/qvtApnlBR6a9W/mMfw2hmppRtC uLXL5M/znXBpq1cWBaoJ+xIVZuYuou5cTo+KYBnoUjOQ/mWso1RpcyWkYqRmHornygnH ktzrWj4d4vPhF+m8rxWHNhoYsZB/JhItK/B4N0XlC0Ew7+ygP3ufqClMCJ1Rec67AhLz gIi3Zc54gwMWmMU1HjQR4vm8xjHmU/461tPX+nguvHAo2HeEmAeDoP3ORT8O66oHMYTq Em+KajaOEZpq4PZHl0OWB5vr1Xpy/5mvYEYYIaSyCVu/p+oWGCYIdo0bnzC1i9JgkEaE 8rUQ== X-Forwarded-Encrypted: i=1; AJvYcCXbNeEvtugUJ7DzvWoSe9FVmd34hbvMlW4x6MmftN5a9vZ/NAAgA45Gpc/xH0sNoRRP9TwtNEhKj8N96I9R4ktZXhw= X-Gm-Message-State: AOJu0YyCj5BYYfMF4smOq9agkknY3gWDh9rFxUC21633bVVOjWfSpWqR 2skB4Oezy1lAJllhEbJNRtoqZW+QPFLpE9ff5UDDbG65hXz0dY5zJvCBS+nqntdduR8AeEUUznz Ys2s2q2/KVgS21IlWPAMOHs7jvOt4pi6c2g8QJA== X-Google-Smtp-Source: AGHT+IGYBrtWZkZbtgvRzNPVu1B+8JDnsX07phI5PUGbd+e0ih3lWeVmaZ4LMbeEje3pQV2LuO9GQYfJpojQAFjApa0= X-Received: by 2002:ac8:59d0:0:b0:430:b97c:6a36 with SMTP id f16-20020ac859d0000000b00430b97c6a36mr14589850qtf.0.1710858229337; Tue, 19 Mar 2024 07:23:49 -0700 (PDT) MIME-Version: 1.0 References: <20240314145457.1106299-1-pasha.tatashin@soleen.com> <20240318134055.c2f6b29bb6eb73ec93bf7079@linux-foundation.org> In-Reply-To: <20240318134055.c2f6b29bb6eb73ec93bf7079@linux-foundation.org> From: Pasha Tatashin Date: Tue, 19 Mar 2024 10:23:12 -0400 Message-ID: Subject: Re: [PATCH v2] vmstat: Keep count of the maximum page reached by the kernel stack To: Andrew Morton Cc: jpoimboe@kernel.org, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 41ECA40009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: wyb4nmstry15oz91fwd1nfghqgatck68 X-HE-Tag: 1710858230-755693 X-HE-Meta: U2FsdGVkX19yn/jx5u/v5UXciyskpvVsPPyz8yHWeqdMaFfxoDpbCUCOhJVU0MjaWQ/NH6XJteXz0csceded9O01YjDuR6J37hK62UoGT+f4kyhTUUUYzJTsNfAfrazVX756pwljNW1UmlFft3KQgYLtlHC13NUTJAccOHxSqCRXnYM49u8LDN7OtWTH6XbECyWwNTIeedC3VNyrWV1smYFj0QuXPS65r1TSDqH/hqJOiMUU8+qnTKETYq7Yru+yL3JxEoqAaYzT4pWUjISXPSVYEfqROeCV1xgZp4ApJ1vHo+bQ8KTQyyHI5F/EmUWOsZsdGYle/lGwaUKUVuKw47uV+tuyIXJSH4+WbltZhndNR0ze05pEPJWPuPH3/bmrrdu+Z6WgfNiq/m/nWfNFOM8IPcZKFfXJNdPw4fzBFtEcbfEzHZFkutGeyZcFxF5Mz5/YKNQV2z9Bn1TFu41l5SfWhWgIUVCTb4hQDnq+7H/ZDId/rXSuF/InheCMUo6d2LkEIci3aLFO9kc56Z3wi+s5F+y5LPG45/jWFyEKgdeotNgx3qTLk3ciDLrJ5IaiVKkU8YVxP7vehKBXteIzw1iYMemB1BN+vSeY504+JSoKs7IvbN/XDA3KOlE4jGhdAnHq96a/nfmxuWIRSAU/Nqmkne1J7s/01aC38fkm4LxJyiLdeGTDVPibuycMfD3hmZNGisp/UZDSzYd9SXPqAvnGgqbpFJMOtWuLvDPmWo5mER0elen2MYgfCOazg0S05aA0wXX2vT0q8PxjoP/ZWMEwLbr7Wv66Bzghe71xeQ+gLqNB/52oiuDy3e4pYBGOmXO4W+gCSj5Tp4gBpg03emS8stPpuWo8OtVAYrNFSYZSfw7mzBtB/nvHOPnoAYiMyTVFfIa65jI3ulGf0EaB2ev7VtTy6qVxT8GzOxi/z0f5eWVXAkLPcKj93tdJAPuMKI6gI4DWLRGQ5H1j5Yf x7oO0PTw ZAcoBYKu6J57x4U5l/17g2ODd83tNK/0MSqBCkc/c/bFbU2CciW23zQF4dWXTQvw0MVNLQpHpKwgnorhYUcFZxpTuwx+L9EaIA4w6rT2iv+UWzdAZTwP9ZK5thCXo0BD0JpxY0784gdEqMtxFWjf31iNFCcpF3JNDnBsbSB5dwGn8hiz6lzvvb24Jm2r5E7NNs1p+d+GLGhzn2colcxq3wdJoM94dph8sBpZsUgnwqn0j7NlkkOUaZ3g9Cwd+xY19KqlSLep/e8/+CO+TNXWTDCA753Eee3KIcrCYuhsVddOUuKd4K8QTUJ1/YL7sPl13d5UtTUXRc/LbIKYkPY49f9XpUp1uyaQHd2+s/pbPtR1xWEw98pbU3/vXbzdwFPiGo7NHckH+GOpi6yv8GNu6z5s72A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Mar 18, 2024 at 4:40=E2=80=AFPM Andrew Morton wrote: > > On Thu, 14 Mar 2024 14:54:57 +0000 Pasha Tatashin wrote: > > > CONFIG_DEBUG_STACK_USAGE provides a mechanism to determine the minimum > > amount of memory left in a stack. Every time a new low-memory record is > > reached, a message is printed to the console. > > > > However, this doesn't reveal how many pages within each stack were > > actually used. Introduce a mechanism that keeps count the number of > > times each of the stack's pages were reached: > > > > $ grep kstack /proc/vmstat > > kstack_page_1 19974 > > kstack_page_2 94 > > kstack_page_3 0 > > kstack_page_4 0 > > > > In the above example, out of 20,068 threads that exited on this > > machine, only 94 reached the second page of their stack, and none > > touched pages three or four. > > > > In fleet environments with millions of machines, this data can help > > optimize kernel stack sizes. > > We really should have somewhere to document vmstat things. We really should have a documentation for both procfs and sysfs versions of these files: $ wc -l /proc/vmstat 177 /proc/vmstat $ wc -l /sys/devices/system/node/node0/vmstat 61 /sys/devices/system/node/node0/vmstat Some of the counters are shared between the two (where procfs contains machine global view), and some such as vm_event are only part of /proc/vmstat. All of that requires a documentation somewhere under Documentation/mm/vmstat.rst. We must explain that this is not a stable API, as these counters depend on the kernel internal implementation. However, there are so many of them, that it will take some effort to do the initial explanation of all of them. > > --- a/include/linux/vm_event_item.h > > +++ b/include/linux/vm_event_item.h > > @@ -153,10 +153,39 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSW= POUT, > > VMA_LOCK_ABORT, > > VMA_LOCK_RETRY, > > VMA_LOCK_MISS, > > +#endif > > +#ifdef CONFIG_DEBUG_STACK_USAGE > > + KSTACK_PAGE_1, > > + KSTACK_PAGE_2, > > +#if THREAD_SIZE >=3D (4 * PAGE_SIZE) > > + KSTACK_PAGE_3, > > + KSTACK_PAGE_4, > > +#endif > > +#if THREAD_SIZE > (4 * PAGE_SIZE) > > + KSTACK_PAGE_REST, > > +#endif > > #endif > > NR_VM_EVENT_ITEMS > > }; > > This seems a rather cumbersome way to produce a kind of histogram. I > wonder if there should be a separate pseudo file for this. If you would like, the #if for stack size can be removed, I added them not to print kstack_page_3 and kstack_page_4 on order 1 stacks where there are only two-pages. This series shows the frequency of reaching each of these pages by the existing threads. > And there may be a call for extending this. For example I can forsee > people wanting to know "hey, which process did that", in which case Which process did that (to some extent) would be what is printed out by: CONFIG_DEBUG_STACK_USAGE when it finds the new record size stack. Other than that, we could use tracing to find the callers that cause these deep stacks. However, the information provided by this patch can help to figure out if tracing is needed or not. For example, if it is known that the third or forth pages are extremely rarely used, say 0.00001% of time, they could have a special API for deep stack, and save half of the stack memory in the fleet. Thank you, Pasha