From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79BDBC433F5 for ; Mon, 23 May 2022 09:29:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18EE16B0005; Mon, 23 May 2022 05:29:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1198B6B0006; Mon, 23 May 2022 05:29:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F22EE6B0007; Mon, 23 May 2022 05:29:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E13316B0005 for ; Mon, 23 May 2022 05:29:49 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A803C34547 for ; Mon, 23 May 2022 09:29:49 +0000 (UTC) X-FDA: 79496485698.02.CD7110C Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf13.hostedemail.com (Postfix) with ESMTP id 24D4A2001A for ; Mon, 23 May 2022 09:29:19 +0000 (UTC) Received: by mail-wm1-f43.google.com with SMTP id k126so8350431wme.2 for ; Mon, 23 May 2022 02:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=message-id:date:mime-version:user-agent:subject:to:cc:references :from:in-reply-to:content-transfer-encoding; bh=Pnpr494hnLQiQgSniXG4dedtD9vAIlrv2YCFBq6/bJk=; b=RT0d0o+rGT9eb8OtGLbDP+vie/YBmOdJAGORMRmiTnzK739Xyadin/nUD6vaMhvk9o IsUks9JtWQvggrD1p6QMAGVACssRzUdHNUiyjHY/YY+jORUQjWrW9thr0UIIXpJ07U9C Ia43er3d5Bkq1EQ33yngwcoBal3e5KWHNw4paiSoQomLtU8xIjZ8COxgWDhuxu19j/v+ Ymb7H06vcg8dF83QJhUt37Ex7EArAbHwKmpqZYZzaUjX/3BunO/I3gmiq22q9mtg53V+ ingo0Wbjw8vJWqfhl+cMs01B9pr/zT4B4ACkIh8b+FR+VJV5lQefpFrStl2h+lKtS4DK 0k8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :to:cc:references:from:in-reply-to:content-transfer-encoding; bh=Pnpr494hnLQiQgSniXG4dedtD9vAIlrv2YCFBq6/bJk=; b=KSI0R7ayi+Pc1pyBzFyCnO/NL1S1kG07e1NzALxUh/VbBDTHaaWdoNeNqF4zmvXogm uA7ZlM2vLkaKzTI199MDwKHxgvs6NJqV2BqtVau3yO+SQjTfLC2PUg6w5c9IOV0aT6EC +fxQmji55P0sx/kqjHMtfmtqedjujpADECH4x9ex8Cd49HRDl4W5FZmW/75Z7M0Ts9ho UUMSO22XHMFZG7hvyT3XApl29N7MD/ofXi7t8vjLZGjTvVGQp6DFft67FUmT/AmrjPXw UK2W7tFUztPfwjE9EB/fFYhhgjlKcmAf08DpyG9d67z4ModZVyw+fuVDIV1fAcD4ND6B wT1g== X-Gm-Message-State: AOAM5336Gepk6+t93HukdubM5+QmKQSv+gjFNuq0m8/W7/ruqbeb3fga pUwVfj4I03HsgfImOG+eqKUN4A== X-Google-Smtp-Source: ABdhPJzu9lHDhyv33m/66/hyW0RNfwUhvW0Nk8KCnXzLmMGU7ILI6RCWM/GKo9PrhdNlG1HugBtDlQ== X-Received: by 2002:a1c:a4c3:0:b0:397:3bf0:d14d with SMTP id n186-20020a1ca4c3000000b003973bf0d14dmr12783340wme.186.1653298183704; Mon, 23 May 2022 02:29:43 -0700 (PDT) Received: from ?IPV6:2a00:1098:3142:14:3110:d736:2a7:6aff? ([2a00:1098:3142:14:3110:d736:2a7:6aff]) by smtp.gmail.com with ESMTPSA id l16-20020a1c7910000000b003972dcfb614sm9260131wme.14.2022.05.23.02.29.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 23 May 2022 02:29:43 -0700 (PDT) Message-ID: <58cb7fbb-d317-83e6-0427-d3f3944b24b8@raspberrypi.com> Date: Mon, 23 May 2022 10:29:42 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: vchiq: Performance regression since 5.18-rc1 To: Stefan Wahren , paulmck@kernel.org Cc: Marcelo Tosatti , Andrew Morton , Nicolas Saenz Julienne , Borislav Petkov , Minchan Kim , Mel Gorman , Juri Lelli , Thomas Gleixner , Sebastian Andrzej Siewior , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linux ARM , regressions@lists.linux.dev, riel@surriel.com, viro@zeniv.linux.org.uk References: <77d6d498-7dd9-03eb-60f2-d7e682bb1b20@i2se.com> <20220521234616.GO1790663@paulmck-ThinkPad-P17-Gen-1> <20220523044818.GS1790663@paulmck-ThinkPad-P17-Gen-1> From: Phil Elwell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 24D4A2001A X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=raspberrypi.com header.s=google header.b=RT0d0o+r; spf=pass (imf13.hostedemail.com: domain of phil@raspberrypi.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=phil@raspberrypi.com; dmarc=pass (policy=none) header.from=raspberrypi.com X-Stat-Signature: 4wtyrpqhazd1hxpe1bssxmxu1tc73ffj X-HE-Tag: 1653298159-854392 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Stefan, On 23/05/2022 07:19, Stefan Wahren wrote: > Hi Paul, > > Am 23.05.22 um 06:48 schrieb Paul E. McKenney: >> On Sun, May 22, 2022 at 05:11:36PM +0200, Stefan Wahren wrote: >>> Hi Paul, >>> >>> Am 22.05.22 um 01:46 schrieb Paul E. McKenney: >>>> On Sun, May 22, 2022 at 01:22:00AM +0200, Stefan Wahren wrote: >>>>> Hi, >>>>> >>>>> while testing the staging/vc04_services/interface/vchiq_arm driver with my >>>>> Raspberry Pi 3 B+ (multi_v7_defconfig) i noticed a huge performance >>>>> regression since [ff042f4a9b050895a42cae893cc01fa2ca81b95c] mm: >>>>> lru_cache_disable: replace work queue synchronization with synchronize_rcu >>>>> >>>>> Usually i run "vchiq_test -f 1" to see the driver is still working [1]. >>>>> >>>>> Before commit: >>>>> >>>>> real    0m1,500s >>>>> user    0m0,068s >>>>> sys    0m0,846s >>>>> >>>>> After commit: >>>>> >>>>> real    7m11,449s >>>>> user    0m2,049s >>>>> sys    0m0,023s >>>>> >>>>> Best regards >>>>> >>>>> [1] - https://github.com/raspberrypi/userland >>>> Please feel free to try the patch shown below.  Or the pair of patches >>>> from Rik here: >>>> >>>> https://lore.kernel.org/lkml/20220218183114.2867528-2-riel@surriel.com/ >>>> https://lore.kernel.org/lkml/20220218183114.2867528-3-riel@surriel.com/ >>> I tried your patch and Rik's patches but in both cases vchiq_test runs 7 >>> minutes instead of ~ 1 second. >> That is surprising.  Do you boot with rcupdate.rcu_normal=1? > No, not explicit. >>    That would >> nullify my patch, but I would expect that Rik's patch would still provide >> increased performance even in that case. > I will retest with a fresh SD card image. >> >> Could you please characterize where the slowdown is occurring? > > Unfortunately i don't have a deep insight into driver and vchiq_test tool. Just > a user view. > > Do you think an strace would be a good starting point? > > @Phil Any advices to analyse this issue? Sending many small control packets: vchiq_test -c 1 10000 essentially tests interrupt latency. Using a small number of large bulk transfers: vchiq_test -b 10000 1 becomes a test of how long it takes to lock down pages. It also tests DMA transfer speeds, but since the DMA is run by the firmware (which you aren't changing), I think you can rule that. You may also find it helpful to include "force_turbo=1" in config.txt for more predictable results. By the way, running our 5.18-rc7-based branch on a 3B+ I'm not seeing any performance problems: pi@raspberrypi:~$ time vchiq_test -f 1 Functional test - iters:1 ======== iteration 1 ======== Testing bulk transfer for alignment. Testing bulk transfer at PAGE_SIZE. real 0m0.512s user 0m0.042s sys 0m0.165s Phil