From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF10AC83F2C for ; Mon, 4 Sep 2023 14:55:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29A998E0003; Mon, 4 Sep 2023 10:55:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24A1B8D0001; Mon, 4 Sep 2023 10:55:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 112C18E0003; Mon, 4 Sep 2023 10:55:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 01FCB8D0001 for ; Mon, 4 Sep 2023 10:55:45 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C8F821CA168 for ; Mon, 4 Sep 2023 14:55:45 +0000 (UTC) X-FDA: 81199214250.16.5FEFD8A Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf04.hostedemail.com (Postfix) with ESMTP id D735540011 for ; Mon, 4 Sep 2023 14:55:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=L0jEowZU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693839344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qMYqlhxNIvQnoz2i2gTHdbFl4bwBHBPtaOUi7bDs8bs=; b=HFSUrE6hvHTfaHLAMiKk8BybZPOlQ318yzzx8uaAw+TBG29ZTjuuQOj51rJLVq2od6jKs1 GgziIn1EuecUONBqk//K/YeSYrG0JW4QvijlrqZlPyiWqr9qvNSLxN6dvmu4WVvRC1mQlF YRdyWYf2aMVviDy6s9TvjJugs4opxZQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=L0jEowZU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693839344; a=rsa-sha256; cv=none; b=X9XcnJvW/VW22hi8s/Ov7gu2idtC5kV9O0cKJ9Ur/4nQD3FHthNw9zw7wdOeBVk4wk8O9w 2Kc66bk6EUVVQnMl0MnWnWkQcntN/VHEa2ewvCldNqvifGEfqgMhjxYRGBEmZlLIClQCH2 AWuWelY25hhU6Qj7mD/9efOonfPonwk= Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2b703a0453fso23217731fa.3 for ; Mon, 04 Sep 2023 07:55:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693839342; x=1694444142; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=qMYqlhxNIvQnoz2i2gTHdbFl4bwBHBPtaOUi7bDs8bs=; b=L0jEowZU0qONwIB+/uo8wF09RY96038RGJCLxBjJU6UeacRD1pbfwN4RLYJnSp85U5 GKfH6Z/bq8rurA4NQukC4W+3SCrnmSEwU71tjgdvAGbb7m6Icknr45A/TuUe+7+twE1q UFx2mKlQjiudEmmCHpnFZY/jLlhTAVwL7dk5PViAgh1a2xp18A0Qpm89Q42rlV31Uo8n l3DzHwCCizOtjslpeTr42puhSE4+/FXxhrO5iw8A6EwnUbG5BRdw91QK6Ye3aB4Sl4jO rOTiWTAGCEudTIOY4H3RStQJBg4VxZtgisdHlndBjfpcVg+dNxrkZvg4YEw6xc15c7/B BAhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693839342; x=1694444142; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qMYqlhxNIvQnoz2i2gTHdbFl4bwBHBPtaOUi7bDs8bs=; b=O0aMJVlm8KdPvy3Q/LbTQnvXRXrU+d1Wy+TlYxAeRTXZxVHzA77rysDeuURwDniRhU a0TYeiyed1Ls/5cIYOALEGEsmvstWL0tEppuKnE6kwWUgKj0j1yje635atIm2WOXOwZj vU+HwqyYup9D1vf0HigkxRZrsNt3g7ffWdBiAf+eH/L9g00xl7Z9Q7VXu1BqrAsQ0lPt egxdzKNAiT0AZa59VpXnc9KgT0faFWqv+ESsbzByTeBX5WM5314DkfJm7EdfcTYXc5mp WWnQKA/S5AqRg/77xUMw/IaYjhZ1nQJItvWfpBNb/xTrjNZoO9NPnL/bmroG06r9yREz A82Q== X-Gm-Message-State: AOJu0YzFSAXTHnLsyfX7qFAgf+TTIkoSrHUzWqKPNjOuLyLPXaoaKUs5 pygz2bRUyUxGR5PVbrUSPYY= X-Google-Smtp-Source: AGHT+IHYMVpp2doc7lurqwDdugR0o+F59rLyi+qulPer88rRAn8DqEoDwXp+MwCCiJ6Zf5JxUpI46Q== X-Received: by 2002:a05:651c:158:b0:2bd:1732:c10b with SMTP id c24-20020a05651c015800b002bd1732c10bmr7333565ljd.34.1693839341809; Mon, 04 Sep 2023 07:55:41 -0700 (PDT) Received: from pc636 (host-90-235-20-237.mobileonline.telia.com. [90.235.20.237]) by smtp.gmail.com with ESMTPSA id w20-20020a2e3014000000b002bcb89e92dcsm2178791ljw.6.2023.09.04.07.55.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Sep 2023 07:55:41 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Mon, 4 Sep 2023 16:55:38 +0200 To: Andrew Morton Cc: linux-mm@kvack.org, Andrew Morton , LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Oleksiy Avramchenko Subject: Re: [PATCH v2 0/9] Mitigate a vmap lock contention v2 Message-ID: References: <20230829081142.3619-1-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230829081142.3619-1-urezki@gmail.com> X-Rspamd-Queue-Id: D735540011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 73irc9hitkjpkdeew5p9oscihbongarz X-HE-Tag: 1693839343-422939 X-HE-Meta: U2FsdGVkX1+QVJHlBQaouhaM6g3M0SbgCzKa+sG0/2A/7YkwZZx92sz52WlXm8Ft8ffaAfhoSRvAE8dQ6oagmfQo6b/bwDVvTHlTJr1zvhX8MrgXTZ2a6j1I9O667V5DZWijPsi1nV+qMdrljbmvEd/g0wLcG6ZfUAHLrGMOkUS2I0MnuaXPI2Tdo5Y2ms4jNwqgK41DSR9fGr4F4i5MWRv3+XZztoj/av2NtHPBKPO2f8n9wU+oHhFIuAZdCHMmmhuuUZgWjdBZdHc+5Ed/K2MRkehAq2f8yp3LJ+tX7sBUQArEKi8nc4IfEKP15YDAY3C8WPC+WT00NEBKiDWcyxSTE8NX3Ft4ehC7exr7psPhzPRzPSQ/doGQYPc++3hi1vj5kZ26rzLdhJYZ3BfFuAcAJSXrArE2+UTe3bCcZFnA4r2HCrbc8xWoW1VL3GATvWTo/cZiaBBf/dYxyJpAg1mWNCiogYbWzNqDZqm/8JfPf6HU7xhS5YksL6aY2fQRHEdPp03lFi7EDKTE8ITQYlVSNAPb1HXeZT0n3G7IXpxLQh3W+K1IfJJBA3ue8jU5CcIH6RGYswmQWsLAjnbqCiMO4xSn3JNuJZZhmN8EiExWeDEEAUzAfMQpNqQ1mqmj52BwucD7SjeWmCiTrCZ3WLQtgGHg1gZZU4Zazr0NLXYbpCL2khCcraIqlfQBYbWCTesfXjsRxhYmajr2nE6efXtWh63JA71bnJvwBiOVi2i0Mdpv/5z+7Mje4GAg4T8Rvg+PqYYCzXbbuXGhiBouo5E2ELSCbfKaBfECAagSRIQkgRzoVOlV2/REpjiPd5o+7eKnX1FBLNh+QKCAyA/gUIeX1sl+cZTzgm7IUB+OKuOn/90jv4R2L06UfDQmv6q8BIYlKbBlAkZ09iqpt9zpc47eB+1nHs466iqqnRqvgvJFskRLQZPYMm+iQsYuZ9rSgzFDUc34Ex5ktF3hYEh VgfsYhun 3S6o/0eS5g1tnfydIz1ZCFauhcvdQ0WF2VScIvDsw9Kn1qC9FBuvw7yl/IGiAaKxc05yA4QAOfSYviEo40CBwBeo4nxWdaMYJrkvC9nhdM5KTBpSuYJswsDMxjkFhiRVnlpwfdJXgp/jPnWXdlqVk/bpJx30sG0zbh+iOvgJ85nCqQOHT3FM0TmuQ4eOVOwfKmh1b3Yi5/m7vpAsFyFUTuLj5Ps34qiaW4z+5illNNdNtwT+DgQQCWB+BOBoACbBlOrIJOcqwAAly1WCvKPcl1LgzRfSmUsQQPooe+MApVnFHMC7/BIiZLxStIK6Q43rRgtcwkWo8kMYtUoZ7FfG0bqUV+OUG0nXHjjJKMsTT/A7H820N1RbxQ9IOc0p0qByt7bps8cAdix9wV0Mwl+j26OFv2MoFW5bjKVUPjLRqAIzxqw087iHQUYc/MW/UQFFlQxUUo6OulJ0JQ/zbh/H2SStfU4cew2wdP7DNq6pY/LM6MZz/SiE0vUnTWkfy44gIcxQGV6PzF5BUhVGGJQ3fxZbe+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, Andrew! > Hello, folk! > > This is the v2, the series which tends to minimize the vmap > lock contention. It is based on the tag: v6.5-rc6. Here you > can find a documentation about it: > > wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf > > even though it is a bit outdated(it follows v1), it still gives a > good overview on the problem and how it can be solved. On demand > and by request i can update it. > > The v1 is here: https://lore.kernel.org/linux-mm/ZIAqojPKjChJTssg@pc636/T/ > > Delta v1 -> v2: > - open coded locking; > - switch to array of nodes instead of per-cpu definition; > - density is 2 cores per one node(not equal to number of CPUs); > - VAs first go back(free path) to an owner node and later to > a global heap if a block is fully freed, nid is saved in va->flags; > - add helpers to drain lazily-freed areas faster, if high pressure; > - picked al Reviewed-by. > > Test on AMD Ryzen Threadripper 3970X 32-Core Processor: > sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64 > > > 94.17% 0.90% [kernel] [k] _raw_spin_lock > 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath > 74.69% 0.25% [kernel] [k] __vmalloc_node_range > 72.64% 0.01% [kernel] [k] __get_vm_area_node > 72.04% 0.89% [kernel] [k] alloc_vmap_area > 42.17% 0.00% [kernel] [k] vmalloc > 32.53% 0.00% [kernel] [k] __vmalloc_node > 24.91% 0.25% [kernel] [k] vfree > 24.32% 0.01% [kernel] [k] remove_vm_area > 22.63% 0.21% [kernel] [k] find_unlink_vmap_area > 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac > 14.35% 0.00% [kernel] [k] ret_from_fork_asm > 14.35% 0.00% [kernel] [k] ret_from_fork > 14.35% 0.00% [kernel] [k] kthread > > vs > > 74.32% 2.42% [kernel] [k] __vmalloc_node_range > 69.58% 0.01% [kernel] [k] vmalloc > 54.21% 1.17% [kernel] [k] __alloc_pages_bulk > 48.13% 47.91% [kernel] [k] clear_page_orig > 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f > 32.06% 0.00% [kernel] [k] ret_from_fork_asm > 32.06% 0.00% [kernel] [k] ret_from_fork > 32.06% 0.00% [kernel] [k] kthread > 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 > 22.98% 4.16% [kernel] [k] vfree > 14.36% 0.28% [kernel] [k] __get_vm_area_node > 13.43% 3.35% [kernel] [k] alloc_vmap_area > 10.86% 0.04% [kernel] [k] remove_vm_area > 8.89% 2.75% [kernel] [k] _raw_spin_lock > 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 > 6.65% 1.37% [kernel] [k] free_unref_page > 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath > > > On smaller systems, for example, 8xCPU Hikey960 board the > contention is not that high and is approximately ~16 percent. > > Uladzislau Rezki (Sony) (9): > mm: vmalloc: Add va_alloc() helper > mm: vmalloc: Rename adjust_va_to_fit_type() function > mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c > mm: vmalloc: Remove global vmap_area_root rb-tree > mm: vmalloc: Remove global purge_vmap_area_root rb-tree > mm: vmalloc: Offload free_vmap_area_lock lock > mm: vmalloc: Support multiple nodes in vread_iter > mm: vmalloc: Support multiple nodes in vmallocinfo > mm: vmalloc: Set nr_nodes/node_size based on CPU-cores > > mm/vmalloc.c | 929 +++++++++++++++++++++++++++++++++++++-------------- > 1 file changed, 683 insertions(+), 246 deletions(-) > > -- > 2.30.2 > It would be good if this series somehow could be tested having some runtime from the people. So far there was a warning from the test robot: https://lore.kernel.org/lkml/202308292228.RRrGUYyB-lkp@intel.com/T/#m397b3834cb3b7a0a53b8dffb3624384c8e278007 urezki@pc638:~/data/raid0/coding/linux.git$ git diff diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 08990f630c21..7105d7bcd37e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4778,7 +4778,7 @@ static void vmap_init_free_space(void) * |<--------------------------------->| */ for (busy = vmlist; busy; busy = busy->next) { - if (busy->addr - vmap_start > 0) { + if ((unsigned long) busy->addr - vmap_start > 0) { free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (!WARN_ON_ONCE(!free)) { free->va_start = vmap_start; urezki@pc638:~/data/raid0/coding/linux.git$ This extra patch has to be applied to fix the warning. >From my side i have tested it as much as i can. Can it be plugged into linux-next to get some runtime? Or is there any other way you prefer to go? Thank you in advance! -- Uladzislau Rezki