From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0755C433EF for ; Sun, 26 Jun 2022 03:29:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fcKeZzqFcb1U9iKiGNDFva37e4W9kSFkasrbjt8Mghw=; b=Ul/YPo9nImvyWw oD7auebAPwCP+mFbGaoMlDsBd6/1cLqjkMGkb9A6h/8Je3vRydAaU2k/grvs3nK1Oo9udjUFLRK5+ Kuf9cR1riJJ56nVKkTrZ0fQVJmThcT6TzWn1xUoRz9DWrYYaeUOqDSzoftm028egaI9oO1/3evnMC VCN78u8M1zgd5uy3ybxeiRdgcFmvQEDU6XC6cS/dS1ObYszavLpje60AIw8oyOvDczSa1puYEBckz AAOGsSUl/VwbXCKtXM9YnhxVlnESj12gaHy6Ld+gCs4kuJDXJCNeuZcu8CzjZiLry+8GsWl8CGAZi M9pppdb3vnkx2qOrTzAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o5Iwt-009H6y-F3; Sun, 26 Jun 2022 03:28:35 +0000 Received: from mail-oi1-x231.google.com ([2607:f8b0:4864:20::231]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o5Iwq-009H5v-HZ for linux-arm-kernel@lists.infradead.org; Sun, 26 Jun 2022 03:28:34 +0000 Received: by mail-oi1-x231.google.com with SMTP id w193so8645723oie.5 for ; Sat, 25 Jun 2022 20:28:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=RqOSrVav6dU2nmRFVG/BMhdhMSZeEt0YoIOWTqo4LME=; b=z8nHIm7hfTZjmF+GyKjb5qIaDqyBCc5A3ktrVAsyewCbQW0C7pFns4zRluMgHKcsm4 Y3uHHVsJafCt9Rzyto3Ddns1pAOe72z5jn6XwJ8Nyrv6aOGZI+7PzzkMuZnJ1mKJGV1+ stczT3ZQeTkbD+n5Fac39JuvInaOEQBFUZv1+b3kI5URwgAj1J188zfhIj+lx4S7AnLl VRPfwpTU0AyDokK7yQIqMUcQ5DXx92JKYW79XYJjjj8yF9TukI1KO/EZSbC5W7mU7Hcm 5jpTGnsom8IvRT/L5ELl/DxinIuTMi8uRorP2sloPBVsBt1z4sJN05jQjgoLCv33pmxf z/Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=RqOSrVav6dU2nmRFVG/BMhdhMSZeEt0YoIOWTqo4LME=; b=U7/NbpYJRBhIMQ4uxpDWkU04+4S6TJJZ7NgUq2nkTEz9VVExToykCioiNcJJ+hTOhc TBTGSA8wKGxgoUxNCcT2CPyZ9rM3HUyrVMDAm9U/Hdg9x4EFHr2xUmRJ1kofi9f09cWo rxOzv/HiMRLM3W9Tswq4zQLhiJuYJK4h9G94+ZWPqMGBYTQIxGgrnojfkev49dw3YREb mNAJ1y80kVoJk3VipVjpkjm4Mi5KuzWJQFzHV5wu1N6NVO01m3UeKOhVcuy7FAvKcrbN xQx7s10MSd0HWGhK6+WqDPsUE1zuGn4BEZoRe7G2GEbAr35uLXKGaKXNn3KxMYh013SL jtiQ== X-Gm-Message-State: AJIora+6XK/Fg7UsV0PhOsYLJUVTkrFz8MzYbrXwGrMLSpFfVkdp80u1 TfWRjmEzsn7cNt9Xo3BHF+CEkQ== X-Google-Smtp-Source: AGRyM1tx8+6+Zu+4IZgFzgGg5Ks/WomGiGglgk8urzN6S7v9axXx6pFjUKoTIgXW9WArCc36O0/sSw== X-Received: by 2002:aca:bfc6:0:b0:32e:b45d:bd72 with SMTP id p189-20020acabfc6000000b0032eb45dbd72mr7083019oif.274.1656214109737; Sat, 25 Jun 2022 20:28:29 -0700 (PDT) Received: from builder.lan (104-57-184-186.lightspeed.austtx.sbcglobal.net. [104.57.184.186]) by smtp.gmail.com with ESMTPSA id i13-20020a056808054d00b0032ed4543c36sm3456972oig.36.2022.06.25.20.28.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 20:28:29 -0700 (PDT) Date: Sat, 25 Jun 2022 22:28:27 -0500 From: Bjorn Andersson To: Krzysztof Kozlowski Cc: Rajendra Nayak , Andy Gross , Georgi Djakov , Rob Herring , Catalin Marinas , Will Deacon , linux-arm-msm@vger.kernel.org, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Thara Gopinath Subject: Re: [PATCH v4 4/4] arm64: dts: qcom: sdm845: Add CPU BWMON Message-ID: References: <20220601101140.170504-1-krzysztof.kozlowski@linaro.org> <20220601101140.170504-5-krzysztof.kozlowski@linaro.org> <64eb52ee-b3ac-3d94-cfce-ceb1c88dddb6@linaro.org> <042cb765-113b-9335-edae-595addf50dd0@quicinc.com> <23320e3c-40c3-12bb-0a1c-7e659a1961f2@linaro.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <23320e3c-40c3-12bb-0a1c-7e659a1961f2@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220625_202832_696301_77694F5E X-CRM114-Status: GOOD ( 33.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu 23 Jun 07:58 CDT 2022, Krzysztof Kozlowski wrote: > On 23/06/2022 08:48, Rajendra Nayak wrote: > >>>> diff --git a/arch/arm64/boot/dts/qcom/sdm845.dtsi b/arch/arm64/boot/dts/qcom/sdm845.dtsi > >>>> index 83e8b63f0910..adffb9c70566 100644 > >>>> --- a/arch/arm64/boot/dts/qcom/sdm845.dtsi > >>>> +++ b/arch/arm64/boot/dts/qcom/sdm845.dtsi > >>>> @@ -2026,6 +2026,60 @@ llcc: system-cache-controller@1100000 { > >>>> interrupts = ; > >>>> }; > >>>> > >>>> + pmu@1436400 { > >>>> + compatible = "qcom,sdm845-cpu-bwmon"; > >>>> + reg = <0 0x01436400 0 0x600>; > >>>> + > >>>> + interrupts = ; > >>>> + > >>>> + interconnects = <&gladiator_noc MASTER_APPSS_PROC 3 &mem_noc SLAVE_EBI1 3>, > >>>> + <&osm_l3 MASTER_OSM_L3_APPS &osm_l3 SLAVE_OSM_L3>; > >>>> + interconnect-names = "ddr", "l3c"; > >>> > >>> Is this the pmu/bwmon instance between the cpu and caches or the one between the caches and DDR? > >> > >> To my understanding this is the one between CPU and caches. > > > > Ok, but then because the OPP table lists the DDR bw first and Cache bw second, isn't the driver > > ending up comparing the bw values thrown by the pmu against the DDR bw instead of the Cache BW? > > I double checked now and you're right. > > > Atleast with my testing on sc7280 I found this to mess things up and I always was ending up at > > higher OPPs even while the system was completely idle. Comparing the values against the Cache bw > > fixed it.(sc7280 also has a bwmon4 instance between the cpu and caches and a bwmon5 between the cache > > and DDR) > > In my case it exposes different issue - under performance. Somehow the > bwmon does not report bandwidth high enough to vote for high bandwidth. > > After removing the DDR interconnect and bandwidth OPP values I have for: > sysbench --threads=8 --time=60 --memory-total-size=20T --test=memory > --memory-block-size=4M run > > 1. Vanilla: 29768 MB/s > 2. Vanilla without CPU votes: 8728 MB/s > 3. Previous bwmon (voting too high): 32007 MB/s > 4. Fixed bwmon 24911 MB/s > Bwmon does not vote for maximum L3 speed: > bwmon report 9408 MB/s (thresholds set: <9216000 15052801> > ) > osm l3 aggregate 14355 MBps -> 897 MHz, level 7, bw 14355 MBps > > Maybe that's just problem with missing governor which would vote for > bandwidth rounding up or anticipating higher needs. > > >>> Depending on which one it is, shouldn;t we just be scaling either one and not both the interconnect paths? > >> > >> The interconnects are the same as ones used for CPU nodes, therefore if > >> we want to scale both when scaling CPU, then we also want to scale both > >> when seeing traffic between CPU and cache. > > > > Well, they were both associated with the CPU node because with no other input to decide on _when_ > > to scale the caches and DDR, we just put a mapping table which simply mapped a CPU freq to a L3 _and_ > > DDR freq. So with just one input (CPU freq) we decided on what should be both the L3 freq and DDR freq. > > > > Now with 2 pmu's, we have 2 inputs, so we can individually scale the L3 based on the cache PMU > > counters and DDR based on the DDR PMU counters, no? > > > > Since you said you have plans to add the other pmu support as well (bwmon5 between the cache and DDR) > > how else would you have the OPP table associated with that pmu instance? Would you again have both the > > L3 and DDR scale based on the inputs from that bwmon too? > > Good point, thanks for sharing. I think you're right. I'll keep only the > l3c interconnect path. > If I understand correctly, <&osm_l3 MASTER_OSM_L3_APPS &osm_l3 SLAVE_OSM_L3> relates to the L3 cache speed, which sits inside the CPU subsystem. As such traffic hitting this cache will not show up in either bwmon instance. The path <&gladiator_noc MASTER_APPSS_PROC 3 &mem_noc SLAVE_EBI1 3> affects the DDR frequency. So the traffic measured by the cpu-bwmon would be the CPU subsystems traffic that missed the L1/L2/L3 caches and hits the memory bus towards DDR. If this is the case it seems to make sense to keep the L3 scaling in the opp-tables for the CPU and make bwmon only scale the DDR path. What do you think? Regards, Bjorn _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel