From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA07AC77B7A for ; Tue, 30 May 2023 21:18:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231330AbjE3VSo (ORCPT ); Tue, 30 May 2023 17:18:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229580AbjE3VSn (ORCPT ); Tue, 30 May 2023 17:18:43 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92821C0 for ; Tue, 30 May 2023 14:18:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 25F2162F3A for ; Tue, 30 May 2023 21:18:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F97BC433EF; Tue, 30 May 2023 21:18:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1685481521; bh=twB0MqwpH7Oejb8YvVLCq7d2rh4D2IU0rW4ycr0vnr0=; h=Date:To:From:Subject:From; b=BNL5x9SSUbJkbmGj6nvBFj0LfgZNRcPharfIv45mSmoHCY8imYQ+K3yd4Ktcr6FiE iieAG3d7NkxzD6EAM9wFb+1X535RsFHtxto086YUaC47EBjew6TaQUQDAzj4FzvC/E gHBsMEieznrvTiGlXaTT89Q42HtquunYByLjLE1k= Date: Tue, 30 May 2023 14:18:40 -0700 To: mm-commits@vger.kernel.org, yuzhao@google.com, ying.huang@intel.com, willy@infradead.org, ryan.roberts@arm.com, kirill.shutemov@linux.intel.com, fengwei.yin@intel.com, akpm@linux-foundation.org From: Andrew Morton Subject: + thp-avoid-lock-when-check-whether-thp-is-in-deferred-list.patch added to mm-unstable branch Message-Id: <20230530211841.6F97BC433EF@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The patch titled Subject: THP: avoid lock when check whether THP is in deferred list has been added to the -mm mm-unstable branch. Its filename is thp-avoid-lock-when-check-whether-thp-is-in-deferred-list.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/thp-avoid-lock-when-check-whether-thp-is-in-deferred-list.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Yin Fengwei Subject: THP: avoid lock when check whether THP is in deferred list Date: Sat, 29 Apr 2023 16:27:58 +0800 free_transhuge_page() acquires split queue lock then check whether the THP was added to deferred list or not. It brings high deferred queue lock contention. It's safe to check whether the THP is in deferred list or not without holding the deferred queue lock in free_transhuge_page() because when code hit free_transhuge_page(), there is no one tries to add the folio to _deferred_list. Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 63.02% 0.01% page_fault1_pro [kernel.kallsyms] [k] free_transhuge_page - 63.01% free_transhuge_page + 62.91% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Link: https://lkml.kernel.org/r/20230429082759.1600796-2-fengwei.yin@intel.com Signed-off-by: Yin Fengwei Acked-by: Kirill A. Shutemov Reviewed-by: "Huang, Ying" Cc: Matthew Wilcox Cc: Ryan Roberts Cc: Yu Zhao Signed-off-by: Andrew Morton --- mm/huge_memory.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) --- a/mm/huge_memory.c~thp-avoid-lock-when-check-whether-thp-is-in-deferred-list +++ a/mm/huge_memory.c @@ -2792,12 +2792,19 @@ void free_transhuge_page(struct page *pa struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; - list_del(&folio->_deferred_list); + /* + * At this point, there is no one trying to add the folio to + * deferred_list. If folio is not in deferred_list, it's safe + * to check without acquiring the split_queue_lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + if (!list_empty(&folio->_deferred_list)) { + ds_queue->split_queue_len--; + list_del(&folio->_deferred_list); + } + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); } _ Patches currently in -mm which might be from fengwei.yin@intel.com are thp-avoid-lock-when-check-whether-thp-is-in-deferred-list.patch