From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96A32C6FA8E for ; Thu, 2 Mar 2023 17:27:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D55636B0078; Thu, 2 Mar 2023 12:27:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D06126B007B; Thu, 2 Mar 2023 12:27:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCE5F6B007D; Thu, 2 Mar 2023 12:27:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AD4CA6B0078 for ; Thu, 2 Mar 2023 12:27:18 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3B731AB50B for ; Thu, 2 Mar 2023 17:27:18 +0000 (UTC) X-FDA: 80524639356.09.DC6041C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 229DD1A0016 for ; Thu, 2 Mar 2023 17:27:15 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KfUoTxoe; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677778036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=taZnUIskUDKhma9jU5HVtD2d3fqkdXORT/OzHkjNDbY=; b=UyRGlZpi3RfaHkn5M/z3A4QpKJtu3NYGSyT45ErnVFN57VAhNTNonSHQtoKGDVNMg9ajjj aAjNlNW/l9K/7u4wB8jzcVEWhBQ00K0FjHW152lTxOsqJxx1B9QHD4c9eIhW/S+er64QCE dhMXK+KkLbElZ3xyNjTpHf4J1OVVHC8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KfUoTxoe; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677778036; a=rsa-sha256; cv=none; b=hqLGfSf94qiGOFFO4Sf8dR3ZZJFueB2QX7N43CBmRINGCmCBqeQKZ40rCwRqsiuxUf/InP mLKaubHyiHEQtIjmEzby0EV9e3Veloxay3XEiapilV2HBrjUS0Be7FyVpa9AFyf+QquJ4t 44ett6vn2rJmWuKjvMvwoFLCNqKMcSE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677778035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=taZnUIskUDKhma9jU5HVtD2d3fqkdXORT/OzHkjNDbY=; b=KfUoTxoegwJUw8vjSGBKPoC1S9dfd3gYZDbk+drQSQgq4/BYeawOMfE433k7FIEE8T/8FR nIgSdrgm3chseXyM9VaE4WnbQb49W9BBvWXsNR0f2Ne5tyf3hwpNm6IoSreH72jaDZegGs 0Okw760p+e0gJ4rfQ/yn6jaFCavErXs= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-581-ejdr8D0VPXO91nwKdEuwkw-1; Thu, 02 Mar 2023 12:27:14 -0500 X-MC-Unique: ejdr8D0VPXO91nwKdEuwkw-1 Received: by mail-qt1-f198.google.com with SMTP id l12-20020ac84a8c000000b003bfe751a7fdso14785qtq.1 for ; Thu, 02 Mar 2023 09:27:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677778034; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=taZnUIskUDKhma9jU5HVtD2d3fqkdXORT/OzHkjNDbY=; b=efILqkeCjIwPdbio8bM2S/01Kx+EagJhEIrVJ9kCD5rb7KZ3gY00m+ZvgzAFg/7Rb9 0h6Tty6A/lPmLRTWegYcn1Bd4T1tYQg2ygIBukyfAv76Wxjj1sMQWNyLrzXTqSE0oOmG GuZ0UzN+h0lgf6KBr/EudIAoGvJoWjtIWgVIj8H0Bh/pjG46dOcosd0IyuSt+UsQYhMU GFu9tOGbw9lownl6JOiDPePDgLKY65biGDiA9Gt/ra2N/UwysBu4+BrNhQJMOwf92aDi qxmAKZvz+GiOI8k/JLKL7w2Kpfipgyq+UL7mHfHKx23mPZZXFdPcXHTzfBG2dC2Sik2F OCeA== X-Gm-Message-State: AO0yUKXEdlUiJWuBRsBMdD4mpczdurXfEGdriP7hd4nF+QTY7b6N3FHr /WJ8uipwqkkLJavwWd3EWFc2JBck+YznNmSSai8bRNggPiH5pAfxE+FB8sXkgO7pefMYthPghRv rxaZIpscUdlI= X-Received: by 2002:a0c:9a43:0:b0:56c:222d:427a with SMTP id q3-20020a0c9a43000000b0056c222d427amr19519713qvd.1.1677778033870; Thu, 02 Mar 2023 09:27:13 -0800 (PST) X-Google-Smtp-Source: AK7set9uOP3CD3qJVsAZhk5lIgft5Ka5qkWjcQx7eXp+HakmNI2Ev035+Qw0dV9io636wOMCj/WfDQ== X-Received: by 2002:a0c:9a43:0:b0:56c:222d:427a with SMTP id q3-20020a0c9a43000000b0056c222d427amr19519688qvd.1.1677778033532; Thu, 02 Mar 2023 09:27:13 -0800 (PST) Received: from x1n (bras-base-aurron9127w-grc-56-70-30-145-63.dsl.bell.ca. [70.30.145.63]) by smtp.gmail.com with ESMTPSA id t190-20020a3746c7000000b0074235745fdasm31572qka.58.2023.03.02.09.27.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Mar 2023 09:27:12 -0800 (PST) Date: Thu, 2 Mar 2023 12:27:11 -0500 From: Peter Xu To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 01/11] mm/vmstat: remove remote node draining Message-ID: References: <20230209150150.380060673@redhat.com> <20230209153204.656996515@redhat.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: dfcftaf9j7p76ni4snjpp4rh4r6f71ru X-Rspamd-Queue-Id: 229DD1A0016 X-HE-Tag: 1677778035-388680 X-HE-Meta: U2FsdGVkX19PhDkXC64/y71ffHAc5VfbVPyJ6ZqjmrSOopiXLMsG1wDsG9rduqegqRkHmMT8kRtXkyYhxNt5dpUGMmkAyM8rpQvuBgBOZ5+xec95TTgWxoyjL8hU8VDSshdxsJgYApKXjqLA4Xmt1hs4JrYXLTtxHt8QfpJfohyhiLJ0tSMJp8aeDrDs0ziB8fSoJBVmdFzaEWSwZmtJMwdQQNqYFBhi/7pT0bNgpA9z1xugnnokcGDFmhwSmCQ5QGT7wQWIbrRZjczwT6glPBwuYbIaDnt4nh3A6cHzWtn52oN92sNT4+mxujmc5s67RP2TrYnLYvjXTo7jDfYSNHz16l0a4759k+e1JJS6M1aEabLA/lVolHNQlBpnZvlERCPiLbTQ1lw7tIYnLPxyKmLrM9hQy2Poh4Uv2NdE9tfuTpfNFqge/B1qSAwYJ3lsHkeoR0eSNqLxD+8PDJeGxYNqsLLMRwdiMKVimj4AYct/S3wUIgQAFtjRPxV0kukc+5QWEE3v6QnRFkU1vCcWLj0cFkqWEJUkDF2jlFU8nviEdYEN6MRJzjnYb+LHS302OHuUdFhkGmywRtGN+YaTGUizqNj7VZXj/BBTrXtE2IXmGvK4HX/C/852nhor8eYyRLtzJUWapkt/yWX8HA3J/xAb3prT5l049C6Dq+lELj/BL/l22ywe7DM/+awEb203Fwn34UE/95oDo+BaG1dtjHz32+aHegHMhxetGffJ3b4UT808GcpRMzs6IX4KRrypxehIQUgoXDoDv632nw6x+onVmqL4/0uBsshgnp23VgVKNbSQDxRE1pncWEHnodU9KkBlaT2uNHlgrfMhcuv3lqiyr6doDMz0bdlmvSxr94ueV8n/hAIDb/pTBbZ8d2vcAe0OVkCdigPgAe1k6h6PUqe7JJBOnpLrIHABeLFman4g/ftzzApFpp7kUI5P5AVa5GeUqbW62oiHnn8P7lS LOo3iu5p 0EEHi93ElaQoVYNrs5LPb017KYvZRpKtyCW1Vq0X30KXZ21yr6q7eYpkmXASnDxMMpQPmZ/BD8LmZ50unZXJH2s//08HEphmH3kax+M/kQQ4i6ZQf6vbUn+HKQi8MMe8IY72vE92VLDXV5foP/KkTVMyOcQLzO7qO7sIqqY8u/zT9a0UIQTbsjOmDZepbaqpNpwm0pMTH1ri7w01BtMI29HMr7s90rNT/6YBOgotacI9hyUKovae1U3qyMImV5KLxBYlTR+8MunCvMnQYvEWrhYnZ36saTzvVnL7p3w+3vsSES+i7awbw8tlFXJbkNxUA9rg8QawABfGtVNmW+3nFQM6IOBlNk9MBO+m5Y7XfNMQIgLUBNo7SjSP7x6yOXqMKxqYtu0Q0jKxjWevmM+suoWJy/f84LJxmCsj2MXEUhCJTij/wvj4N10QRtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 02, 2023 at 12:21:15PM -0500, Peter Xu wrote: > On Thu, Feb 09, 2023 at 12:01:51PM -0300, Marcelo Tosatti wrote: > > Draining of pages from the local pcp for a remote zone was necessary > > since: > > > > "Note that remote node draining is a somewhat esoteric feature that is > > required on large NUMA systems because otherwise significant portions > > of system memory can become trapped in pcp queues. The number of pcp is > > determined by the number of processors and nodes in a system. A system > > with 4 processors and 2 nodes has 8 pcps which is okay. But a system > > with 1024 processors and 512 nodes has 512k pcps with a high potential > > for large amount of memory being caught in them." > > How about mentioning more details on where does this come from? > > afaict it's from commit 4037d45 since 2007. > > So I digged that out mostly because I want to know why we did flush pcp at > all during vmstat update. It already sounds weird to me but I could have > been missing important details. > > The rational I had here is refresh_cpu_vm_stats(true) is mostly being > called by the shepherd afaict, while: > > (1) The frequency of that interval is defined as sysctl_stat_interval, > which has nothing yet to do with pcp pages but only stat at least in > the name of it, and, > > (2) vmstat_work is only queued if need_update() here: > > for_each_online_cpu(cpu) { > struct delayed_work *dw = &per_cpu(vmstat_work, cpu); > > if (!delayed_work_pending(dw) && need_update(cpu)) > queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); > > cond_resched(); > } > > need_update() tells us "we should flush vmstats", nothing it tells > about "we should flush pcp list".. > > I looked into the 2007 commit, besides what Marcelo quoted, I do see > there's a major benefit of reusing cache lines, quotting from the commit: > > Move the node draining so that is is done when the vm statistics > are updated. At that point we are already touching all the > cachelines with the pagesets of a processor. > > However I didn't see why it's rational to flush pcp list when vmstat needs > flushing either. I also don't know whether that "cacheline locality" hold > true or not, because I saw that the pcp page list is split from vmstats > since 2021: > > commit 28f836b6777b6f42dce068a40d83a891deaaca37 > Author: Mel Gorman > Date: Mon Jun 28 19:41:38 2021 -0700 > > mm/page_alloc: split per cpu page lists and zone stats > > I'm not even sure my A-b or R-b worth anything at all here, just offer > something I got from git archaeology so maybe helpful to readers and > reasoning to this patch. The correctness of archaeology needs help from > others (Christoph and Gel?).. I would just say if there's anything useful > or correct may worth collect some into the commit log. > > So from what I can tell this patch makes sense. One thing I forgot to mention, which may be a slight abi change, is that I think the pcp page drain is also triggered by /proc/PID/refresh_vm_stats (even though again I don't see why flushing pcp is strictly needed). It's just that I don't know whether there's potential user app that can leverage this. The worst case is we can drain pcp list for refresh_vm_stats procfs explicitly, but I'm not sure whether it'll be worthwhile either, probably just to be safe. -- Peter Xu