From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B1EC4363D for ; Tue, 22 Sep 2020 07:23:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9501F2145D for ; Tue, 22 Sep 2020 07:23:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9501F2145D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB9CB6B00D6; Tue, 22 Sep 2020 03:23:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6A2D6B00D7; Tue, 22 Sep 2020 03:23:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A6DA900012; Tue, 22 Sep 2020 03:23:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id 846806B00D6 for ; Tue, 22 Sep 2020 03:23:29 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4148C1EF1 for ; Tue, 22 Sep 2020 07:23:29 +0000 (UTC) X-FDA: 77289856938.18.pain23_371355c2714b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 20BF1100EC699 for ; Tue, 22 Sep 2020 07:23:29 +0000 (UTC) X-HE-Tag: pain23_371355c2714b X-Filterd-Recvd-Size: 4104 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Tue, 22 Sep 2020 07:23:28 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id A7A26AC79; Tue, 22 Sep 2020 07:24:03 +0000 (UTC) Date: Tue, 22 Sep 2020 08:23:24 +0100 From: Mel Gorman To: Yafang Shao Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Linux MM Subject: Re: [PATCH] mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED Message-ID: <20200922072324.GJ3117@suse.de> References: <20200921014317.73915-1-laoar.shao@gmail.com> <20200921223430.GI3117@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 22, 2020 at 10:12:31AM +0800, Yafang Shao wrote: > On Tue, Sep 22, 2020 at 6:34 AM Mel Gorman wrote: > > > > On Mon, Sep 21, 2020 at 09:43:17AM +0800, Yafang Shao wrote: > > > Our users reported that there're some random latency spikes when their RT > > > process is running. Finally we found that latency spike is caused by > > > FADV_DONTNEED. Which may call lru_add_drain_all() to drain LRU cache on > > > remote CPUs, and then waits the per-cpu work to complete. The wait time > > > is uncertain, which may be tens millisecond. > > > That behavior is unreasonable, because this process is bound to a > > > specific CPU and the file is only accessed by itself, IOW, there should > > > be no pagecache pages on a per-cpu pagevec of a remote CPU. That > > > unreasonable behavior is partially caused by the wrong comparation of the > > > number of invalidated pages and the number of the target. For example, > > > if (count < (end_index - start_index + 1)) > > > The count above is how many pages were invalidated in the local CPU, and > > > (end_index - start_index + 1) is how many pages should be invalidated. > > > The usage of (end_index - start_index + 1) is incorrect, because they > > > are virtual addresses, which may not mapped to pages. We'd better use > > > inode->i_data.nrpages as the target. > > > > > > > How does that work if the invalidation is for a subset of the file? > > > > I realized it as well. There are some solutions to improve it. > > Option 1, take the min as the target. > - if (count < (end_index - start_index + 1)) { > + target = min_t(unsigned long, inode->i_data.nrpages, > + end_index - start_index + 1); > + if (count < target) { > lru_add_drain_all(); > > Option 2, change the prototype of invalidate_mapping_pages and then > check how many pages were skipped. > > + struct invalidate_stat { > + unsigned long skipped; // how many pages were skipped > + unsigned long invalidated; // how many pages were invalidated > +}; > > - unsigned long invalidate_mapping_pages(struct address_space *mapping, > +unsigned long invalidate_mapping_pages(struct address_space *mapping, > struct invalidate_stat *stat, > That would involve updating each caller and the struct is unnecessarily heavy. Create one that returns via **nr_lruvec. For invalidate_mapping_pages, pass in NULL as nr_lruvec. Create a new helper for fadvise that accepts nr_lruvec. In the common helper, account for pages that are likely on an LRU and count them in nr_lruvec if !NULL. Update fadvise to drain only if pages were skipped that were on the lruvec. That should also deal with the case where holes have been punched between start and end. -- Mel Gorman SUSE Labs