From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2F4FC83F1C for ; Mon, 28 Aug 2023 14:15:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230103AbjH1OO3 (ORCPT ); Mon, 28 Aug 2023 10:14:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231743AbjH1OOV (ORCPT ); Mon, 28 Aug 2023 10:14:21 -0400 Received: from mout-p-102.mailbox.org (mout-p-102.mailbox.org [IPv6:2001:67c:2050:0:465::102]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F68AF7; Mon, 28 Aug 2023 07:14:17 -0700 (PDT) Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4RZCGv27R1z9sr6; Mon, 28 Aug 2023 16:14:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oltmanns.dev; s=MBO0001; t=1693232055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+DO/l19fra2OrFIuBGZ5yUwxn75lwFHs9VDz/dQrv/o=; b=imCfI3Fl0CU6sxkurcZarAexLBD/sw/LLM0LSo6Esc/GeST0+FK1cfygO5a2HcEnwfkNEi hFwUwk0hHFU8cRjeVEUbZod44/7kwx3HpCnFxomHJr4m85AjXX/n6epx5B0tpt5BJ0xgn+ Nl8g6jnmVlh/+fv+SDKCHzlokI+qZ9h9C7dex3HUj2m6lNc44py+pyb83qvsXyxQXXimvd f+Nl52q9f0wtYeJodZ1dlzpuxG6QVt7T+8KmtC/tg2NORSKJVJb6okvtDbhwqDuxF6X8ak uztQy/lOw4qVyus8nZN/wV+MOcXO7Dk2Ls/Kf1IPjUUmj+9qoMM6vX99TPj1Uw== References: <20230825-pll-mipi_keep_rate-v1-0-35bc43570730@oltmanns.dev> <87ledzqhwx.fsf@oltmanns.dev> <878r9yb21b.fsf@oltmanns.dev> From: Frank Oltmanns To: Maxime Ripard Cc: Michael Turquette , Stephen Boyd , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , David Airlie , Daniel Vetter , Ondrej Jirman , Icenowy Zheng , linux-clk@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-sunxi@lists.linux.dev, dri-devel@lists.freedesktop.org, Icenowy Zheng Subject: Re: [PATCH 0/3] Make Allwinner A64's pll-mipi keep its rate when parent rate changes In-reply-to: Date: Mon, 28 Aug 2023 16:14:06 +0200 Message-ID: <87msybuudt.fsf@oltmanns.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4RZCGv27R1z9sr6 Precedence: bulk List-ID: X-Mailing-List: linux-clk@vger.kernel.org On 2023-08-28 at 10:25:01 +0200, Maxime Ripard wrote: > On Sat, Aug 26, 2023 at 11:12:16AM +0200, Frank Oltmanns wrote: >> >> On 2023-08-25 at 17:07:58 +0200, Frank Oltmanns wro= te: >> > Thank you for your feedback, Maxime! >> > >> > On 2023-08-25 at 10:13:53 +0200, Maxime Ripard wr= ote: >> >> [[PGP Signed Part:Undecided]] >> >> Hi, >> >> >> >> On Fri, Aug 25, 2023 at 07:36:36AM +0200, Frank Oltmanns wrote: >> >>> I would like to make the Allwinner A64's pll-mipi to keep its rate w= hen >> >>> its parent's (pll-video0) rate changes. Keeping pll-mipi's rate is >> >>> required, to let the A64 drive both an LCD and HDMI display at the s= ame >> >>> time, because both have pll-video0 as an ancestor. >> >>> >> >>> PATCH 1 adds this functionality as a feature into the clk framework = (new >> >>> flag: CLK_KEEP_RATE). >> >>> >> >>> Cores that use this flag, store a rate as req_rate when it or one of= its >> >>> descendants requests a new rate. >> >>> >> >>> That rate is then restored in the clk_change_rate recursion, which w= alks >> >>> through the tree. It will reach the flagged core (e.g. pll-mipi) aft= er >> >>> the parent's rate (e.g. pll-video0) has already been set to the new >> >>> rate. It will then call determine_rate (which requests the parent's >> >>> current, i.e. new, rate) to determine a rate that is close to the >> >>> flagged core's previous rate. Afterward it will re-calculate the rat= es >> >>> for the flagged core's subtree. >> >> >> >> I don't think it's the right way forward. It makes the core logic more >> >> complicated, for something that is redundant with the notifiers >> >> mechanism that has been the go-to for that kind of things so far. >> > >> > Yeah, that was my initial idea as well. But I couldn't get it to work. >> > See details below. >> > >> > Do you have an example of a clock that restores its previous rate after >> > the parent rate has changed? I've looked left and right, but to me it >> > seems that notifiers are mainly used for setting clocks into some kind >> > of "safe mode" prior to the rate change. Examples: >> > >> > sunxi-ng: >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/sunxi-ng/c= cu_mux.c#L273 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/sunxi-ng/c= cu_common.c#L60 >> > >> > but also others: >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/at91/clk-m= aster.c#L248 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/meson/meso= n8b.c#L3755 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/qcom/clk-c= pu-8996.c#L546 >> > >> >> It's not really obvious to me why the notifiers don't work there. >> >> >> >>> This work is inspired by an out-of-tree patchset [1] [2] [3]. >> >>> Unfortunately, the patchset uses clk_set_rate() in a notifier callba= ck, >> >>> which the following comment on clk_notifier_register() forbids: "The >> >>> callbacks associated with the notifier must not re-enter into the clk >> >>> framework by calling any top-level clk APIs." [4] Furthermore, that >> >>> out-of-tree patchset no longer works with the current linux-next, >> >>> because setting pll-mipi is now also resetting pll-video0 [5]. >> >> >> >> Is it because of the "The callbacks associated with the notifier must >> >> not re-enter into the clk framework by calling any top-level clk APIs= ." >> >> comment? >> > >> > I don't think that's the reason. I'm fairly certain that the problem i= s, >> > that pll-mipi tries to set the parent rate. Maybe it should check if t= he >> > parent is locked, before determining a rate that requires the parent >> > rate to change. =F0=9F=A4=94 Currently, it only calls clk_hw_can_set_r= ate_parent() >> > which only checks the flag, but does not check if it is really possible >> > to change the parent's rate. >> > >> > Regardless, please don't prematurely dismiss my proposal. It has the >> > advantage that it is not specific for sunxi-ng, but could be used for >> > other drivers as well. Maybe there other instances of exclusive locks >> > today where the CLK_KEEP_RATE flag might work equally well. =F0=9F=A4= =B7 >> > >> >> If so, I think the thing we should emphasize is that it's about *any >> >> top-level clk API*, as in clk_set_rate() or clk_set_parent(). >> >> >> >> The issue is that any consumer-facing API is taking the clk_prepare l= ock >> >> and thus we would have reentrancy. But we're a provider there, and no= ne >> >> of the clk_hw_* functions are taking that lock. Neither do our own fu= nction. >> >> >> >> So we could call in that notifier our set_rate callback directly, or = we >> >> could create a clk_hw_set_rate() function. >> >> >> >> The first one will create cache issue between the actual rate that the >> >> common clock framework is running and the one we actually enforced, b= ut >> >> we could create a function to flush the CCF cache. >> >> >> >> The second one is probably simpler. >> > >> > I'm probably missing something, because I don't think this would work. >> > For reference, this is our tree: >> > >> > pll-video0 >> > hdmi-phy-clk >> > hdmi >> > tcon1 >> > pll-mipi >> > tcon0 >> > tcon-data-clock >> > >> > When pll-video0's rate is changed (e.g. because a HDMI monitor is >> > plugged in), the rates of the complete subtree for pll-video0 are >> > recalculated, including tcon0 and tcon-data-clock. The rate of tcon0 is >> > based on the rate that was recalculated for pll-mipi, which - in turn - >> > was of course recalculated based on the pll-video0's new rate. These >> > values are stored by the clk framework in a private struct. They are >> > calculated before actually performing any rate changes. >> > >> > So, if a notifier sets pll-mipi's rate to something else than was >> > previously recalculated, the clk framework would still try to set tcon0 >> > to the value that it previously calculated. >> > >> > So, we would have to recalculate pll-mipi's subtree after changing its >> > rate (that's what PATCH 1 is doing). >> >> Sorry, I forgot that this actually was possible by flagging pll-mipi >> with CLK_RECALC_NEW_RATES. But the real problem I was fighting with when >> trying to use the notifiers is something else. >> >> Initially, pll-video0 is set by the bootloader. In my case uboot sets it >> to 294 MHz. pll-mipi is set to 588 MHz. >> >> Afterward, there are actually two types of calls for setting pll-mipi in >> my scenario: >> 1. during boot when tcon-data-clock is set to drive the LCD panel >> 2. when the HDMI cable is plugged in > > Not really. Both of those clocks can change (or not) at any point in > time. What triggers the rate set is a modeset which might never happen > (if the display driver or output is disabled, if the fbdev emulation is > disabled or if there's never a compositor starting) or possibly happen > each frame on both output for all you know. Ok, thank you for the explanation and I apologize for not having the terminology straight. This would mean that in my description above there are two modesets that trigger the rate set 1. for the tcon-data-clock on boot when the internal display is activated and 2. for hdmi when the external monitor is activated. For reference: In my scenario I'm using a pinephone which has an internal LCD display and an HDMI connector (not supported in mainline). I understand that there could be more. Let's put a pin in that, because my understandig is, that this is not the relevant part here. >> In the first case, the rate for pll-mipi is based on the rate that >> tcon-data-clock requests. In that case, we do not want to restore the >> previous rate. >> >> In the second case, pll-mipi should try to remain running at the >> previous rate (the one that was requested by tcon-data-clock). That's >> the reason for setting core->req_rate in PATCH 1. >> >> Unfortunately, the notifier does not provide us with enough context to >> distinguish the two cases. > > I don't think any piece of code will be able to, really. In the first case, setting the pll-mipi clock starts from the bottom (tcon-data-clock) and uses clk_calc_new_rates() to get to the top-most clock that needs and can be changed. On it's way up to pll-video0 it passes pll-mipi. My proposal (PATCH 1) is to use that moment to store the rate in req_rate. In contrast, in the second case, setting the pll-mipi clock starts from the top. pll-video0's rate is changed and therefore a new rate is propagate for the whole subtree where, on its way down, it passes pll-mipi. My proposal (PATCH 1) is to use that moment to restore the rate from req_rate that was previously (see pragraph above) set. Since I did not see a way to achieve this using notifiers (and you seem to agree), I chose to propose a different path. > Your definition of CLK_KEEP_RATE is that it will "try to keep rate, if > parent rate changes" > > What happens if it fails, possibly because of rounding like you > mentioned already? Maybe "keep" is too strong of a word. I'm sorry for the confusion my poor choice of wording has caused. What I would like if for a clock to go back as closely as possible to the previous rate. And this is what PATCH 1 does by using the clocks determine_rate (or round_rate) operation. > Fundamentally, the problem is that you need different rates on two > subtrees, and we set both to have CLK_SET_RATE_PARENT and allow both to > change the parent rate if needed. This reads to me as if you are emphasizing the word "both" here. I'm aware that you know, but I state it here for the benefit of others: Up until kernel 6.5 only hdmi resets pll-video0. pll-mipi does not set pll-video0. This has changed in clk-next. Now also pll-mipi sets the parent rate. Icenowy's patches are aimed at (and work for) up to kernel 6.5. They restore pll-mipi's rate after pll-video0 has been changed by hdmi. > What would happen if we force pll-video0 to a known, fixed, value and > remove CLK_SET_RATE_PARENT from both the pll-mipi and hdmi clocks? Approximately three months ago, I wrote "one could argue that pll-video0 should be set to 297MHz at boot", to which Jernej responded: "Ideally, clock rate setting code should be immune on "initial" values, set by bootloader or default values. If it's not, then it should be improved in the way that it is.": https://lore.kernel.org/linux-clk/4831731.31r3eYUQgx@jernej-laptop/#t That's what got me startet on the whole process of allowing pll-mipi to set it's parent instead of simply setting it to a known rate of 297 MHz. Should I resume the other (297MHz) investigation? I had another idea, but don't know how/if that's possible: Maybe we could use a notifier to notify pll-mipi (or even better: tcon-data-clock) and use some mechanism to defer calling clk_set_rate() to a point in time when the whole process of setting the clocks is complete. Best regards, Frank > > Maxime From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46AFDC83F12 for ; Mon, 28 Aug 2023 14:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date: In-reply-to:Subject:Cc:To:From:References:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FhiB9WGjqBfN7UaEx186RaVQ8rhcIuV+YlQT2ZBTbhs=; b=Dlg0ATX5pn3zKQbTO+A9j67gvX u7UEuheKHnfl4spD3R2oqPO3mfaz/USc5mJXOLe1kyJUrKrU2nCmo8Y6edrVabESktbeJsCV10sg3 wVzVAJQJWOI35BDk7DaIu7y2Fw2pIdQmXjHtLfVmx66eNN/inJrNPsal7ykNkC+VDzyZRKvyZFMVz r4bTgDB6Xq+N8sfg8tYj+H8Im8YObFQZ+WmYKvJDJvnJ0m370cfA8fN0JwTtyifm4h/2tMUxhDp/K f6Yl34QTDY4BGLON9wCfNu2mfFPBxXwJ1k8SbePUuf0gNhWmfeZQJm4I1GxQPOcfxl7/BR60FJcJj 0qy9lw/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qad0b-009hEU-39; Mon, 28 Aug 2023 14:14:25 +0000 Received: from mout-p-102.mailbox.org ([80.241.56.152]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qad0Y-009hDh-21 for linux-arm-kernel@lists.infradead.org; Mon, 28 Aug 2023 14:14:25 +0000 Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4RZCGv27R1z9sr6; Mon, 28 Aug 2023 16:14:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oltmanns.dev; s=MBO0001; t=1693232055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+DO/l19fra2OrFIuBGZ5yUwxn75lwFHs9VDz/dQrv/o=; b=imCfI3Fl0CU6sxkurcZarAexLBD/sw/LLM0LSo6Esc/GeST0+FK1cfygO5a2HcEnwfkNEi hFwUwk0hHFU8cRjeVEUbZod44/7kwx3HpCnFxomHJr4m85AjXX/n6epx5B0tpt5BJ0xgn+ Nl8g6jnmVlh/+fv+SDKCHzlokI+qZ9h9C7dex3HUj2m6lNc44py+pyb83qvsXyxQXXimvd f+Nl52q9f0wtYeJodZ1dlzpuxG6QVt7T+8KmtC/tg2NORSKJVJb6okvtDbhwqDuxF6X8ak uztQy/lOw4qVyus8nZN/wV+MOcXO7Dk2Ls/Kf1IPjUUmj+9qoMM6vX99TPj1Uw== References: <20230825-pll-mipi_keep_rate-v1-0-35bc43570730@oltmanns.dev> <87ledzqhwx.fsf@oltmanns.dev> <878r9yb21b.fsf@oltmanns.dev> From: Frank Oltmanns To: Maxime Ripard Cc: Michael Turquette , Stephen Boyd , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , David Airlie , Daniel Vetter , Ondrej Jirman , Icenowy Zheng , linux-clk@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-sunxi@lists.linux.dev, dri-devel@lists.freedesktop.org, Icenowy Zheng Subject: Re: [PATCH 0/3] Make Allwinner A64's pll-mipi keep its rate when parent rate changes In-reply-to: Date: Mon, 28 Aug 2023 16:14:06 +0200 Message-ID: <87msybuudt.fsf@oltmanns.dev> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4RZCGv27R1z9sr6 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_071422_898667_588EEB7D X-CRM114-Status: GOOD ( 61.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Ck9uIDIwMjMtMDgtMjggYXQgMTA6MjU6MDEgKzAyMDAsIE1heGltZSBSaXBhcmQgPG1yaXBhcmRA a2VybmVsLm9yZz4gd3JvdGU6Cj4gT24gU2F0LCBBdWcgMjYsIDIwMjMgYXQgMTE6MTI6MTZBTSAr MDIwMCwgRnJhbmsgT2x0bWFubnMgd3JvdGU6Cj4+Cj4+IE9uIDIwMjMtMDgtMjUgYXQgMTc6MDc6 NTggKzAyMDAsIEZyYW5rIE9sdG1hbm5zIDxmcmFua0BvbHRtYW5ucy5kZXY+IHdyb3RlOgo+PiA+ IFRoYW5rIHlvdSBmb3IgeW91ciBmZWVkYmFjaywgTWF4aW1lIQo+PiA+Cj4+ID4gT24gMjAyMy0w OC0yNSBhdCAxMDoxMzo1MyArMDIwMCwgTWF4aW1lIFJpcGFyZCA8bXJpcGFyZEBrZXJuZWwub3Jn PiB3cm90ZToKPj4gPj4gW1tQR1AgU2lnbmVkIFBhcnQ6VW5kZWNpZGVkXV0KPj4gPj4gSGksCj4+ ID4+Cj4+ID4+IE9uIEZyaSwgQXVnIDI1LCAyMDIzIGF0IDA3OjM2OjM2QU0gKzAyMDAsIEZyYW5r IE9sdG1hbm5zIHdyb3RlOgo+PiA+Pj4gSSB3b3VsZCBsaWtlIHRvIG1ha2UgdGhlIEFsbHdpbm5l ciBBNjQncyBwbGwtbWlwaSB0byBrZWVwIGl0cyByYXRlIHdoZW4KPj4gPj4+IGl0cyBwYXJlbnQn cyAocGxsLXZpZGVvMCkgcmF0ZSBjaGFuZ2VzLiBLZWVwaW5nIHBsbC1taXBpJ3MgcmF0ZSBpcwo+ PiA+Pj4gcmVxdWlyZWQsIHRvIGxldCB0aGUgQTY0IGRyaXZlIGJvdGggYW4gTENEIGFuZCBIRE1J IGRpc3BsYXkgYXQgdGhlIHNhbWUKPj4gPj4+IHRpbWUsIGJlY2F1c2UgYm90aCBoYXZlIHBsbC12 aWRlbzAgYXMgYW4gYW5jZXN0b3IuCj4+ID4+Pgo+PiA+Pj4gUEFUQ0ggMSBhZGRzIHRoaXMgZnVu Y3Rpb25hbGl0eSBhcyBhIGZlYXR1cmUgaW50byB0aGUgY2xrIGZyYW1ld29yayAobmV3Cj4+ID4+ PiBmbGFnOiBDTEtfS0VFUF9SQVRFKS4KPj4gPj4+Cj4+ID4+PiBDb3JlcyB0aGF0IHVzZSB0aGlz IGZsYWcsIHN0b3JlIGEgcmF0ZSBhcyByZXFfcmF0ZSB3aGVuIGl0IG9yIG9uZSBvZiBpdHMKPj4g Pj4+IGRlc2NlbmRhbnRzIHJlcXVlc3RzIGEgbmV3IHJhdGUuCj4+ID4+Pgo+PiA+Pj4gVGhhdCBy YXRlIGlzIHRoZW4gcmVzdG9yZWQgaW4gdGhlIGNsa19jaGFuZ2VfcmF0ZSByZWN1cnNpb24sIHdo aWNoIHdhbGtzCj4+ID4+PiB0aHJvdWdoIHRoZSB0cmVlLiBJdCB3aWxsIHJlYWNoIHRoZSBmbGFn Z2VkIGNvcmUgKGUuZy4gcGxsLW1pcGkpIGFmdGVyCj4+ID4+PiB0aGUgcGFyZW50J3MgcmF0ZSAo ZS5nLiBwbGwtdmlkZW8wKSBoYXMgYWxyZWFkeSBiZWVuIHNldCB0byB0aGUgbmV3Cj4+ID4+PiBy YXRlLiBJdCB3aWxsIHRoZW4gY2FsbCBkZXRlcm1pbmVfcmF0ZSAod2hpY2ggcmVxdWVzdHMgdGhl IHBhcmVudCdzCj4+ID4+PiBjdXJyZW50LCBpLmUuIG5ldywgcmF0ZSkgdG8gZGV0ZXJtaW5lIGEg cmF0ZSB0aGF0IGlzIGNsb3NlIHRvIHRoZQo+PiA+Pj4gZmxhZ2dlZCBjb3JlJ3MgcHJldmlvdXMg cmF0ZS4gQWZ0ZXJ3YXJkIGl0IHdpbGwgcmUtY2FsY3VsYXRlIHRoZSByYXRlcwo+PiA+Pj4gZm9y IHRoZSBmbGFnZ2VkIGNvcmUncyBzdWJ0cmVlLgo+PiA+Pgo+PiA+PiBJIGRvbid0IHRoaW5rIGl0 J3MgdGhlIHJpZ2h0IHdheSBmb3J3YXJkLiBJdCBtYWtlcyB0aGUgY29yZSBsb2dpYyBtb3JlCj4+ ID4+IGNvbXBsaWNhdGVkLCBmb3Igc29tZXRoaW5nIHRoYXQgaXMgcmVkdW5kYW50IHdpdGggdGhl IG5vdGlmaWVycwo+PiA+PiBtZWNoYW5pc20gdGhhdCBoYXMgYmVlbiB0aGUgZ28tdG8gZm9yIHRo YXQga2luZCBvZiB0aGluZ3Mgc28gZmFyLgo+PiA+Cj4+ID4gWWVhaCwgdGhhdCB3YXMgbXkgaW5p dGlhbCBpZGVhIGFzIHdlbGwuIEJ1dCBJIGNvdWxkbid0IGdldCBpdCB0byB3b3JrLgo+PiA+IFNl ZSBkZXRhaWxzIGJlbG93Lgo+PiA+Cj4+ID4gRG8geW91IGhhdmUgYW4gZXhhbXBsZSBvZiBhIGNs b2NrIHRoYXQgcmVzdG9yZXMgaXRzIHByZXZpb3VzIHJhdGUgYWZ0ZXIKPj4gPiB0aGUgcGFyZW50 IHJhdGUgaGFzIGNoYW5nZWQ/IEkndmUgbG9va2VkIGxlZnQgYW5kIHJpZ2h0LCBidXQgdG8gbWUg aXQKPj4gPiBzZWVtcyB0aGF0IG5vdGlmaWVycyBhcmUgbWFpbmx5IHVzZWQgZm9yIHNldHRpbmcg Y2xvY2tzIGludG8gc29tZSBraW5kCj4+ID4gb2YgInNhZmUgbW9kZSIgcHJpb3IgdG8gdGhlIHJh dGUgY2hhbmdlLiBFeGFtcGxlczoKPj4gPgo+PiA+IHN1bnhpLW5nOgo+PiA+IGh0dHBzOi8vZWxp eGlyLmJvb3RsaW4uY29tL2xpbnV4L3Y2LjQuMTEvc291cmNlL2RyaXZlcnMvY2xrL3N1bnhpLW5n L2NjdV9tdXguYyNMMjczCj4+ID4gaHR0cHM6Ly9lbGl4aXIuYm9vdGxpbi5jb20vbGludXgvdjYu NC4xMS9zb3VyY2UvZHJpdmVycy9jbGsvc3VueGktbmcvY2N1X2NvbW1vbi5jI0w2MAo+PiA+Cj4+ ID4gYnV0IGFsc28gb3RoZXJzOgo+PiA+IGh0dHBzOi8vZWxpeGlyLmJvb3RsaW4uY29tL2xpbnV4 L3Y2LjQuMTEvc291cmNlL2RyaXZlcnMvY2xrL2F0OTEvY2xrLW1hc3Rlci5jI0wyNDgKPj4gPiBo dHRwczovL2VsaXhpci5ib290bGluLmNvbS9saW51eC92Ni40LjExL3NvdXJjZS9kcml2ZXJzL2Ns ay9tZXNvbi9tZXNvbjhiLmMjTDM3NTUKPj4gPiBodHRwczovL2VsaXhpci5ib290bGluLmNvbS9s aW51eC92Ni40LjExL3NvdXJjZS9kcml2ZXJzL2Nsay9xY29tL2Nsay1jcHUtODk5Ni5jI0w1NDYK Pj4gPgo+PiA+PiBJdCdzIG5vdCByZWFsbHkgb2J2aW91cyB0byBtZSB3aHkgdGhlIG5vdGlmaWVy cyBkb24ndCB3b3JrIHRoZXJlLgo+PiA+Pgo+PiA+Pj4gVGhpcyB3b3JrIGlzIGluc3BpcmVkIGJ5 IGFuIG91dC1vZi10cmVlIHBhdGNoc2V0IFsxXSBbMl0gWzNdLgo+PiA+Pj4gVW5mb3J0dW5hdGVs eSwgdGhlIHBhdGNoc2V0IHVzZXMgY2xrX3NldF9yYXRlKCkgaW4gYSBub3RpZmllciBjYWxsYmFj aywKPj4gPj4+IHdoaWNoIHRoZSBmb2xsb3dpbmcgY29tbWVudCBvbiBjbGtfbm90aWZpZXJfcmVn aXN0ZXIoKSBmb3JiaWRzOiAiVGhlCj4+ID4+PiBjYWxsYmFja3MgYXNzb2NpYXRlZCB3aXRoIHRo ZSBub3RpZmllciBtdXN0IG5vdCByZS1lbnRlciBpbnRvIHRoZSBjbGsKPj4gPj4+IGZyYW1ld29y ayBieSBjYWxsaW5nIGFueSB0b3AtbGV2ZWwgY2xrIEFQSXMuIiBbNF0gRnVydGhlcm1vcmUsIHRo YXQKPj4gPj4+IG91dC1vZi10cmVlIHBhdGNoc2V0IG5vIGxvbmdlciB3b3JrcyB3aXRoIHRoZSBj dXJyZW50IGxpbnV4LW5leHQsCj4+ID4+PiBiZWNhdXNlIHNldHRpbmcgcGxsLW1pcGkgaXMgbm93 IGFsc28gcmVzZXR0aW5nIHBsbC12aWRlbzAgWzVdLgo+PiA+Pgo+PiA+PiBJcyBpdCBiZWNhdXNl IG9mIHRoZSAiVGhlIGNhbGxiYWNrcyBhc3NvY2lhdGVkIHdpdGggdGhlIG5vdGlmaWVyIG11c3QK Pj4gPj4gbm90IHJlLWVudGVyIGludG8gdGhlIGNsayBmcmFtZXdvcmsgYnkgY2FsbGluZyBhbnkg dG9wLWxldmVsIGNsayBBUElzLiIKPj4gPj4gY29tbWVudD8KPj4gPgo+PiA+IEkgZG9uJ3QgdGhp bmsgdGhhdCdzIHRoZSByZWFzb24uIEknbSBmYWlybHkgY2VydGFpbiB0aGF0IHRoZSBwcm9ibGVt IGlzLAo+PiA+IHRoYXQgcGxsLW1pcGkgdHJpZXMgdG8gc2V0IHRoZSBwYXJlbnQgcmF0ZS4gTWF5 YmUgaXQgc2hvdWxkIGNoZWNrIGlmIHRoZQo+PiA+IHBhcmVudCBpcyBsb2NrZWQsIGJlZm9yZSBk ZXRlcm1pbmluZyBhIHJhdGUgdGhhdCByZXF1aXJlcyB0aGUgcGFyZW50Cj4+ID4gcmF0ZSB0byBj aGFuZ2UuIPCfpJQgQ3VycmVudGx5LCBpdCBvbmx5IGNhbGxzIGNsa19od19jYW5fc2V0X3JhdGVf cGFyZW50KCkKPj4gPiB3aGljaCBvbmx5IGNoZWNrcyB0aGUgZmxhZywgYnV0IGRvZXMgbm90IGNo ZWNrIGlmIGl0IGlzIHJlYWxseSBwb3NzaWJsZQo+PiA+IHRvIGNoYW5nZSB0aGUgcGFyZW50J3Mg cmF0ZS4KPj4gPgo+PiA+IFJlZ2FyZGxlc3MsIHBsZWFzZSBkb24ndCBwcmVtYXR1cmVseSBkaXNt aXNzIG15IHByb3Bvc2FsLiBJdCBoYXMgdGhlCj4+ID4gYWR2YW50YWdlIHRoYXQgaXQgaXMgbm90 IHNwZWNpZmljIGZvciBzdW54aS1uZywgYnV0IGNvdWxkIGJlIHVzZWQgZm9yCj4+ID4gb3RoZXIg ZHJpdmVycyBhcyB3ZWxsLiBNYXliZSB0aGVyZSBvdGhlciBpbnN0YW5jZXMgb2YgZXhjbHVzaXZl IGxvY2tzCj4+ID4gdG9kYXkgd2hlcmUgdGhlIENMS19LRUVQX1JBVEUgZmxhZyBtaWdodCB3b3Jr IGVxdWFsbHkgd2VsbC4g8J+ktwo+PiA+Cj4+ID4+IElmIHNvLCBJIHRoaW5rIHRoZSB0aGluZyB3 ZSBzaG91bGQgZW1waGFzaXplIGlzIHRoYXQgaXQncyBhYm91dCAqYW55Cj4+ID4+IHRvcC1sZXZl bCBjbGsgQVBJKiwgYXMgaW4gY2xrX3NldF9yYXRlKCkgb3IgY2xrX3NldF9wYXJlbnQoKS4KPj4g Pj4KPj4gPj4gVGhlIGlzc3VlIGlzIHRoYXQgYW55IGNvbnN1bWVyLWZhY2luZyBBUEkgaXMgdGFr aW5nIHRoZSBjbGtfcHJlcGFyZSBsb2NrCj4+ID4+IGFuZCB0aHVzIHdlIHdvdWxkIGhhdmUgcmVl bnRyYW5jeS4gQnV0IHdlJ3JlIGEgcHJvdmlkZXIgdGhlcmUsIGFuZCBub25lCj4+ID4+IG9mIHRo ZSBjbGtfaHdfKiBmdW5jdGlvbnMgYXJlIHRha2luZyB0aGF0IGxvY2suIE5laXRoZXIgZG8gb3Vy IG93biBmdW5jdGlvbi4KPj4gPj4KPj4gPj4gU28gd2UgY291bGQgY2FsbCBpbiB0aGF0IG5vdGlm aWVyIG91ciBzZXRfcmF0ZSBjYWxsYmFjayBkaXJlY3RseSwgb3Igd2UKPj4gPj4gY291bGQgY3Jl YXRlIGEgY2xrX2h3X3NldF9yYXRlKCkgZnVuY3Rpb24uCj4+ID4+Cj4+ID4+IFRoZSBmaXJzdCBv bmUgd2lsbCBjcmVhdGUgY2FjaGUgaXNzdWUgYmV0d2VlbiB0aGUgYWN0dWFsIHJhdGUgdGhhdCB0 aGUKPj4gPj4gY29tbW9uIGNsb2NrIGZyYW1ld29yayBpcyBydW5uaW5nIGFuZCB0aGUgb25lIHdl IGFjdHVhbGx5IGVuZm9yY2VkLCBidXQKPj4gPj4gd2UgY291bGQgY3JlYXRlIGEgZnVuY3Rpb24g dG8gZmx1c2ggdGhlIENDRiBjYWNoZS4KPj4gPj4KPj4gPj4gVGhlIHNlY29uZCBvbmUgaXMgcHJv YmFibHkgc2ltcGxlci4KPj4gPgo+PiA+IEknbSBwcm9iYWJseSBtaXNzaW5nIHNvbWV0aGluZywg YmVjYXVzZSBJIGRvbid0IHRoaW5rIHRoaXMgd291bGQgd29yay4KPj4gPiBGb3IgcmVmZXJlbmNl LCB0aGlzIGlzIG91ciB0cmVlOgo+PiA+Cj4+ID4gICAgIHBsbC12aWRlbzAKPj4gPiAgICAgICAg aGRtaS1waHktY2xrCj4+ID4gICAgICAgIGhkbWkKPj4gPiAgICAgICAgdGNvbjEKPj4gPiAgICAg ICAgcGxsLW1pcGkKPj4gPiAgICAgICAgICAgdGNvbjAKPj4gPiAgICAgICAgICAgICAgdGNvbi1k YXRhLWNsb2NrCj4+ID4KPj4gPiBXaGVuIHBsbC12aWRlbzAncyByYXRlIGlzIGNoYW5nZWQgKGUu Zy4gYmVjYXVzZSBhIEhETUkgbW9uaXRvciBpcwo+PiA+IHBsdWdnZWQgaW4pLCB0aGUgcmF0ZXMg b2YgdGhlIGNvbXBsZXRlIHN1YnRyZWUgZm9yIHBsbC12aWRlbzAgYXJlCj4+ID4gcmVjYWxjdWxh dGVkLCBpbmNsdWRpbmcgdGNvbjAgYW5kIHRjb24tZGF0YS1jbG9jay4gVGhlIHJhdGUgb2YgdGNv bjAgaXMKPj4gPiBiYXNlZCBvbiB0aGUgcmF0ZSB0aGF0IHdhcyByZWNhbGN1bGF0ZWQgZm9yIHBs bC1taXBpLCB3aGljaCAtIGluIHR1cm4gLQo+PiA+IHdhcyBvZiBjb3Vyc2UgcmVjYWxjdWxhdGVk IGJhc2VkIG9uIHRoZSBwbGwtdmlkZW8wJ3MgbmV3IHJhdGUuIFRoZXNlCj4+ID4gdmFsdWVzIGFy ZSBzdG9yZWQgYnkgdGhlIGNsayBmcmFtZXdvcmsgaW4gYSBwcml2YXRlIHN0cnVjdC4gVGhleSBh cmUKPj4gPiBjYWxjdWxhdGVkIGJlZm9yZSBhY3R1YWxseSBwZXJmb3JtaW5nIGFueSByYXRlIGNo YW5nZXMuCj4+ID4KPj4gPiBTbywgaWYgYSBub3RpZmllciBzZXRzIHBsbC1taXBpJ3MgcmF0ZSB0 byBzb21ldGhpbmcgZWxzZSB0aGFuIHdhcwo+PiA+IHByZXZpb3VzbHkgcmVjYWxjdWxhdGVkLCB0 aGUgY2xrIGZyYW1ld29yayB3b3VsZCBzdGlsbCB0cnkgdG8gc2V0IHRjb24wCj4+ID4gdG8gdGhl IHZhbHVlIHRoYXQgaXQgcHJldmlvdXNseSBjYWxjdWxhdGVkLgo+PiA+Cj4+ID4gU28sIHdlIHdv dWxkIGhhdmUgdG8gcmVjYWxjdWxhdGUgcGxsLW1pcGkncyBzdWJ0cmVlIGFmdGVyIGNoYW5naW5n IGl0cwo+PiA+IHJhdGUgKHRoYXQncyB3aGF0IFBBVENIIDEgaXMgZG9pbmcpLgo+Pgo+PiBTb3Jy eSwgSSBmb3Jnb3QgdGhhdCB0aGlzIGFjdHVhbGx5IHdhcyBwb3NzaWJsZSBieSBmbGFnZ2luZyBw bGwtbWlwaQo+PiB3aXRoIENMS19SRUNBTENfTkVXX1JBVEVTLiBCdXQgdGhlIHJlYWwgcHJvYmxl bSBJIHdhcyBmaWdodGluZyB3aXRoIHdoZW4KPj4gdHJ5aW5nIHRvIHVzZSB0aGUgbm90aWZpZXJz IGlzIHNvbWV0aGluZyBlbHNlLgo+Pgo+PiBJbml0aWFsbHksIHBsbC12aWRlbzAgaXMgc2V0IGJ5 IHRoZSBib290bG9hZGVyLiBJbiBteSBjYXNlIHVib290IHNldHMgaXQKPj4gdG8gMjk0IE1Iei4g cGxsLW1pcGkgaXMgc2V0IHRvIDU4OCBNSHouCj4+Cj4+IEFmdGVyd2FyZCwgdGhlcmUgYXJlIGFj dHVhbGx5IHR3byB0eXBlcyBvZiBjYWxscyBmb3Igc2V0dGluZyBwbGwtbWlwaSBpbgo+PiBteSBz Y2VuYXJpbzoKPj4gIDEuIGR1cmluZyBib290IHdoZW4gdGNvbi1kYXRhLWNsb2NrIGlzIHNldCB0 byBkcml2ZSB0aGUgTENEIHBhbmVsCj4+ICAyLiB3aGVuIHRoZSBIRE1JIGNhYmxlIGlzIHBsdWdn ZWQgaW4KPgo+IE5vdCByZWFsbHkuIEJvdGggb2YgdGhvc2UgY2xvY2tzIGNhbiBjaGFuZ2UgKG9y IG5vdCkgYXQgYW55IHBvaW50IGluCj4gdGltZS4gV2hhdCB0cmlnZ2VycyB0aGUgcmF0ZSBzZXQg aXMgYSBtb2Rlc2V0IHdoaWNoIG1pZ2h0IG5ldmVyIGhhcHBlbgo+IChpZiB0aGUgZGlzcGxheSBk cml2ZXIgb3Igb3V0cHV0IGlzIGRpc2FibGVkLCBpZiB0aGUgZmJkZXYgZW11bGF0aW9uIGlzCj4g ZGlzYWJsZWQgb3IgaWYgdGhlcmUncyBuZXZlciBhIGNvbXBvc2l0b3Igc3RhcnRpbmcpIG9yIHBv c3NpYmx5IGhhcHBlbgo+IGVhY2ggZnJhbWUgb24gYm90aCBvdXRwdXQgZm9yIGFsbCB5b3Uga25v dy4KCk9rLCB0aGFuayB5b3UgZm9yIHRoZSBleHBsYW5hdGlvbiBhbmQgSSBhcG9sb2dpemUgZm9y IG5vdCBoYXZpbmcgdGhlCnRlcm1pbm9sb2d5IHN0cmFpZ2h0LiBUaGlzIHdvdWxkIG1lYW4gdGhh dCBpbiBteSBkZXNjcmlwdGlvbiBhYm92ZSB0aGVyZQphcmUgdHdvIG1vZGVzZXRzIHRoYXQgdHJp Z2dlciB0aGUgcmF0ZSBzZXQKIDEuIGZvciB0aGUgdGNvbi1kYXRhLWNsb2NrIG9uIGJvb3Qgd2hl biB0aGUgaW50ZXJuYWwgZGlzcGxheSBpcwogICAgYWN0aXZhdGVkIGFuZAogMi4gZm9yIGhkbWkg d2hlbiB0aGUgZXh0ZXJuYWwgbW9uaXRvciBpcyBhY3RpdmF0ZWQuCgpGb3IgcmVmZXJlbmNlOiBJ biBteSBzY2VuYXJpbyBJJ20gdXNpbmcgYSBwaW5lcGhvbmUgd2hpY2ggaGFzIGFuCmludGVybmFs IExDRCBkaXNwbGF5IGFuZCBhbiBIRE1JIGNvbm5lY3RvciAobm90IHN1cHBvcnRlZCBpbiBtYWlu bGluZSkuCgpJIHVuZGVyc3RhbmQgdGhhdCB0aGVyZSBjb3VsZCBiZSBtb3JlLiBMZXQncyBwdXQg YSBwaW4gaW4gdGhhdCwgYmVjYXVzZQpteSB1bmRlcnN0YW5kaWcgaXMsIHRoYXQgdGhpcyBpcyBu b3QgdGhlIHJlbGV2YW50IHBhcnQgaGVyZS4KCj4+IEluIHRoZSBmaXJzdCBjYXNlLCB0aGUgcmF0 ZSBmb3IgcGxsLW1pcGkgaXMgYmFzZWQgb24gdGhlIHJhdGUgdGhhdAo+PiB0Y29uLWRhdGEtY2xv Y2sgcmVxdWVzdHMuIEluIHRoYXQgY2FzZSwgd2UgZG8gbm90IHdhbnQgdG8gcmVzdG9yZSB0aGUK Pj4gcHJldmlvdXMgcmF0ZS4KPj4KPj4gSW4gdGhlIHNlY29uZCBjYXNlLCBwbGwtbWlwaSBzaG91 bGQgdHJ5IHRvIHJlbWFpbiBydW5uaW5nIGF0IHRoZQo+PiBwcmV2aW91cyByYXRlICh0aGUgb25l IHRoYXQgd2FzIHJlcXVlc3RlZCBieSB0Y29uLWRhdGEtY2xvY2spLiBUaGF0J3MKPj4gdGhlIHJl YXNvbiBmb3Igc2V0dGluZyBjb3JlLT5yZXFfcmF0ZSBpbiBQQVRDSCAxLgo+Pgo+PiBVbmZvcnR1 bmF0ZWx5LCB0aGUgbm90aWZpZXIgZG9lcyBub3QgcHJvdmlkZSB1cyB3aXRoIGVub3VnaCBjb250 ZXh0IHRvCj4+IGRpc3Rpbmd1aXNoIHRoZSB0d28gY2FzZXMuCj4KPiBJIGRvbid0IHRoaW5rIGFu eSBwaWVjZSBvZiBjb2RlIHdpbGwgYmUgYWJsZSB0bywgcmVhbGx5LgoKSW4gdGhlIGZpcnN0IGNh c2UsIHNldHRpbmcgdGhlIHBsbC1taXBpIGNsb2NrIHN0YXJ0cyBmcm9tIHRoZSBib3R0b20KKHRj b24tZGF0YS1jbG9jaykgYW5kIHVzZXMgY2xrX2NhbGNfbmV3X3JhdGVzKCkgdG8gZ2V0IHRvIHRo ZSB0b3AtbW9zdApjbG9jayB0aGF0IG5lZWRzIGFuZCBjYW4gYmUgY2hhbmdlZC4gT24gaXQncyB3 YXkgdXAgdG8gcGxsLXZpZGVvMCBpdApwYXNzZXMgcGxsLW1pcGkuIE15IHByb3Bvc2FsIChQQVRD SCAxKSBpcyB0byB1c2UgdGhhdCBtb21lbnQgdG8gc3RvcmUKdGhlIHJhdGUgaW4gcmVxX3JhdGUu CgpJbiBjb250cmFzdCwgaW4gdGhlIHNlY29uZCBjYXNlLCBzZXR0aW5nIHRoZSBwbGwtbWlwaSBj bG9jayBzdGFydHMgZnJvbQp0aGUgdG9wLiBwbGwtdmlkZW8wJ3MgcmF0ZSBpcyBjaGFuZ2VkIGFu ZCB0aGVyZWZvcmUgYSBuZXcgcmF0ZSBpcwpwcm9wYWdhdGUgZm9yIHRoZSB3aG9sZSBzdWJ0cmVl IHdoZXJlLCBvbiBpdHMgd2F5IGRvd24sIGl0IHBhc3NlcwpwbGwtbWlwaS4gTXkgcHJvcG9zYWwg KFBBVENIIDEpIGlzIHRvIHVzZSB0aGF0IG1vbWVudCB0byByZXN0b3JlIHRoZQpyYXRlIGZyb20g cmVxX3JhdGUgdGhhdCB3YXMgcHJldmlvdXNseSAoc2VlIHByYWdyYXBoIGFib3ZlKSBzZXQuCgpT aW5jZSBJIGRpZCBub3Qgc2VlIGEgd2F5IHRvIGFjaGlldmUgdGhpcyB1c2luZyBub3RpZmllcnMg KGFuZCB5b3Ugc2VlbQp0byBhZ3JlZSksIEkgY2hvc2UgdG8gcHJvcG9zZSBhIGRpZmZlcmVudCBw YXRoLgoKPiBZb3VyIGRlZmluaXRpb24gb2YgQ0xLX0tFRVBfUkFURSBpcyB0aGF0IGl0IHdpbGwg InRyeSB0byBrZWVwIHJhdGUsIGlmCj4gcGFyZW50IHJhdGUgY2hhbmdlcyIKPgo+IFdoYXQgaGFw cGVucyBpZiBpdCBmYWlscywgcG9zc2libHkgYmVjYXVzZSBvZiByb3VuZGluZyBsaWtlIHlvdQo+ IG1lbnRpb25lZCBhbHJlYWR5PwoKTWF5YmUgImtlZXAiIGlzIHRvbyBzdHJvbmcgb2YgYSB3b3Jk LiBJJ20gc29ycnkgZm9yIHRoZSBjb25mdXNpb24gbXkKcG9vciBjaG9pY2Ugb2Ygd29yZGluZyBo YXMgY2F1c2VkLgoKV2hhdCBJIHdvdWxkIGxpa2UgaWYgZm9yIGEgY2xvY2sgdG8gZ28gYmFjayBh cyBjbG9zZWx5IGFzIHBvc3NpYmxlIHRvCnRoZSBwcmV2aW91cyByYXRlLiBBbmQgdGhpcyBpcyB3 aGF0IFBBVENIIDEgZG9lcyBieSB1c2luZyB0aGUgY2xvY2tzCmRldGVybWluZV9yYXRlIChvciBy b3VuZF9yYXRlKSBvcGVyYXRpb24uCgo+IEZ1bmRhbWVudGFsbHksIHRoZSBwcm9ibGVtIGlzIHRo YXQgeW91IG5lZWQgZGlmZmVyZW50IHJhdGVzIG9uIHR3bwo+IHN1YnRyZWVzLCBhbmQgd2Ugc2V0 IGJvdGggdG8gaGF2ZSBDTEtfU0VUX1JBVEVfUEFSRU5UIGFuZCBhbGxvdyBib3RoIHRvCj4gY2hh bmdlIHRoZSBwYXJlbnQgcmF0ZSBpZiBuZWVkZWQuCgpUaGlzIHJlYWRzIHRvIG1lIGFzIGlmIHlv dSBhcmUgZW1waGFzaXppbmcgdGhlIHdvcmQgImJvdGgiIGhlcmUuIEknbQphd2FyZSB0aGF0IHlv dSBrbm93LCBidXQgSSBzdGF0ZSBpdCBoZXJlIGZvciB0aGUgYmVuZWZpdCBvZiBvdGhlcnM6IFVw CnVudGlsIGtlcm5lbCA2LjUgb25seSBoZG1pIHJlc2V0cyBwbGwtdmlkZW8wLiBwbGwtbWlwaSBk b2VzIG5vdCBzZXQKcGxsLXZpZGVvMC4gVGhpcyBoYXMgY2hhbmdlZCBpbiBjbGstbmV4dC4gTm93 IGFsc28gcGxsLW1pcGkgc2V0cyB0aGUKcGFyZW50IHJhdGUuCgpJY2Vub3d5J3MgcGF0Y2hlcyBh cmUgYWltZWQgYXQgKGFuZCB3b3JrIGZvcikgdXAgdG8ga2VybmVsIDYuNS4gVGhleQpyZXN0b3Jl IHBsbC1taXBpJ3MgcmF0ZSBhZnRlciBwbGwtdmlkZW8wIGhhcyBiZWVuIGNoYW5nZWQgYnkgaGRt aS4KCj4gV2hhdCB3b3VsZCBoYXBwZW4gaWYgd2UgZm9yY2UgcGxsLXZpZGVvMCB0byBhIGtub3du LCBmaXhlZCwgdmFsdWUgYW5kCj4gcmVtb3ZlIENMS19TRVRfUkFURV9QQVJFTlQgZnJvbSBib3Ro IHRoZSBwbGwtbWlwaSBhbmQgaGRtaSBjbG9ja3M/CgpBcHByb3hpbWF0ZWx5IHRocmVlIG1vbnRo cyBhZ28sIEkgd3JvdGUgIm9uZSBjb3VsZCBhcmd1ZSB0aGF0IHBsbC12aWRlbzAKc2hvdWxkIGJl IHNldCB0byAyOTdNSHogYXQgYm9vdCIsIHRvIHdoaWNoIEplcm5laiByZXNwb25kZWQ6ICJJZGVh bGx5LApjbG9jayByYXRlIHNldHRpbmcgY29kZSBzaG91bGQgYmUgaW1tdW5lIG9uICJpbml0aWFs IiB2YWx1ZXMsIHNldCBieQpib290bG9hZGVyIG9yIGRlZmF1bHQgdmFsdWVzLiBJZiBpdCdzIG5v dCwgdGhlbiBpdCBzaG91bGQgYmUgaW1wcm92ZWQgaW4KdGhlIHdheSB0aGF0IGl0IGlzLiI6Cmh0 dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWNsay80ODMxNzMxLjMxcjNlWVVRZ3hAamVybmVq LWxhcHRvcC8jdAoKVGhhdCdzIHdoYXQgZ290IG1lIHN0YXJ0ZXQgb24gdGhlIHdob2xlIHByb2Nl c3Mgb2YgYWxsb3dpbmcgcGxsLW1pcGkgdG8Kc2V0IGl0J3MgcGFyZW50IGluc3RlYWQgb2Ygc2lt cGx5IHNldHRpbmcgaXQgdG8gYSBrbm93biByYXRlIG9mIDI5NyBNSHouCgpTaG91bGQgSSByZXN1 bWUgdGhlIG90aGVyICgyOTdNSHopIGludmVzdGlnYXRpb24/CgpJIGhhZCBhbm90aGVyIGlkZWEs IGJ1dCBkb24ndCBrbm93IGhvdy9pZiB0aGF0J3MgcG9zc2libGU6IE1heWJlIHdlCmNvdWxkIHVz ZSBhIG5vdGlmaWVyIHRvIG5vdGlmeSBwbGwtbWlwaSAob3IgZXZlbiBiZXR0ZXI6CnRjb24tZGF0 YS1jbG9jaykgYW5kIHVzZSBzb21lIG1lY2hhbmlzbSB0byBkZWZlciBjYWxsaW5nIGNsa19zZXRf cmF0ZSgpCnRvIGEgcG9pbnQgaW4gdGltZSB3aGVuIHRoZSB3aG9sZSBwcm9jZXNzIG9mIHNldHRp bmcgdGhlIGNsb2NrcyBpcwpjb21wbGV0ZS4KCkJlc3QgcmVnYXJkcywKICBGcmFuawoKCj4KPiBN YXhpbWUKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxp bnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFk ZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4 LWFybS1rZXJuZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 822EEC83F12 for ; Mon, 28 Aug 2023 14:14:22 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 21D2710E2EA; Mon, 28 Aug 2023 14:14:20 +0000 (UTC) Received: from mout-p-102.mailbox.org (mout-p-102.mailbox.org [80.241.56.152]) by gabe.freedesktop.org (Postfix) with ESMTPS id 40CAC10E2EA for ; Mon, 28 Aug 2023 14:14:17 +0000 (UTC) Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4RZCGv27R1z9sr6; Mon, 28 Aug 2023 16:14:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oltmanns.dev; s=MBO0001; t=1693232055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+DO/l19fra2OrFIuBGZ5yUwxn75lwFHs9VDz/dQrv/o=; b=imCfI3Fl0CU6sxkurcZarAexLBD/sw/LLM0LSo6Esc/GeST0+FK1cfygO5a2HcEnwfkNEi hFwUwk0hHFU8cRjeVEUbZod44/7kwx3HpCnFxomHJr4m85AjXX/n6epx5B0tpt5BJ0xgn+ Nl8g6jnmVlh/+fv+SDKCHzlokI+qZ9h9C7dex3HUj2m6lNc44py+pyb83qvsXyxQXXimvd f+Nl52q9f0wtYeJodZ1dlzpuxG6QVt7T+8KmtC/tg2NORSKJVJb6okvtDbhwqDuxF6X8ak uztQy/lOw4qVyus8nZN/wV+MOcXO7Dk2Ls/Kf1IPjUUmj+9qoMM6vX99TPj1Uw== References: <20230825-pll-mipi_keep_rate-v1-0-35bc43570730@oltmanns.dev> <87ledzqhwx.fsf@oltmanns.dev> <878r9yb21b.fsf@oltmanns.dev> From: Frank Oltmanns To: Maxime Ripard Subject: Re: [PATCH 0/3] Make Allwinner A64's pll-mipi keep its rate when parent rate changes In-reply-to: Date: Mon, 28 Aug 2023 16:14:06 +0200 Message-ID: <87msybuudt.fsf@oltmanns.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4RZCGv27R1z9sr6 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dri-devel@lists.freedesktop.org, Icenowy Zheng , Samuel Holland , Stephen Boyd , Michael Turquette , linux-kernel@vger.kernel.org, Jernej Skrabec , linux-sunxi@lists.linux.dev, Chen-Yu Tsai , Ondrej Jirman , linux-clk@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Icenowy Zheng Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On 2023-08-28 at 10:25:01 +0200, Maxime Ripard wrote: > On Sat, Aug 26, 2023 at 11:12:16AM +0200, Frank Oltmanns wrote: >> >> On 2023-08-25 at 17:07:58 +0200, Frank Oltmanns wro= te: >> > Thank you for your feedback, Maxime! >> > >> > On 2023-08-25 at 10:13:53 +0200, Maxime Ripard wr= ote: >> >> [[PGP Signed Part:Undecided]] >> >> Hi, >> >> >> >> On Fri, Aug 25, 2023 at 07:36:36AM +0200, Frank Oltmanns wrote: >> >>> I would like to make the Allwinner A64's pll-mipi to keep its rate w= hen >> >>> its parent's (pll-video0) rate changes. Keeping pll-mipi's rate is >> >>> required, to let the A64 drive both an LCD and HDMI display at the s= ame >> >>> time, because both have pll-video0 as an ancestor. >> >>> >> >>> PATCH 1 adds this functionality as a feature into the clk framework = (new >> >>> flag: CLK_KEEP_RATE). >> >>> >> >>> Cores that use this flag, store a rate as req_rate when it or one of= its >> >>> descendants requests a new rate. >> >>> >> >>> That rate is then restored in the clk_change_rate recursion, which w= alks >> >>> through the tree. It will reach the flagged core (e.g. pll-mipi) aft= er >> >>> the parent's rate (e.g. pll-video0) has already been set to the new >> >>> rate. It will then call determine_rate (which requests the parent's >> >>> current, i.e. new, rate) to determine a rate that is close to the >> >>> flagged core's previous rate. Afterward it will re-calculate the rat= es >> >>> for the flagged core's subtree. >> >> >> >> I don't think it's the right way forward. It makes the core logic more >> >> complicated, for something that is redundant with the notifiers >> >> mechanism that has been the go-to for that kind of things so far. >> > >> > Yeah, that was my initial idea as well. But I couldn't get it to work. >> > See details below. >> > >> > Do you have an example of a clock that restores its previous rate after >> > the parent rate has changed? I've looked left and right, but to me it >> > seems that notifiers are mainly used for setting clocks into some kind >> > of "safe mode" prior to the rate change. Examples: >> > >> > sunxi-ng: >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/sunxi-ng/c= cu_mux.c#L273 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/sunxi-ng/c= cu_common.c#L60 >> > >> > but also others: >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/at91/clk-m= aster.c#L248 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/meson/meso= n8b.c#L3755 >> > https://elixir.bootlin.com/linux/v6.4.11/source/drivers/clk/qcom/clk-c= pu-8996.c#L546 >> > >> >> It's not really obvious to me why the notifiers don't work there. >> >> >> >>> This work is inspired by an out-of-tree patchset [1] [2] [3]. >> >>> Unfortunately, the patchset uses clk_set_rate() in a notifier callba= ck, >> >>> which the following comment on clk_notifier_register() forbids: "The >> >>> callbacks associated with the notifier must not re-enter into the clk >> >>> framework by calling any top-level clk APIs." [4] Furthermore, that >> >>> out-of-tree patchset no longer works with the current linux-next, >> >>> because setting pll-mipi is now also resetting pll-video0 [5]. >> >> >> >> Is it because of the "The callbacks associated with the notifier must >> >> not re-enter into the clk framework by calling any top-level clk APIs= ." >> >> comment? >> > >> > I don't think that's the reason. I'm fairly certain that the problem i= s, >> > that pll-mipi tries to set the parent rate. Maybe it should check if t= he >> > parent is locked, before determining a rate that requires the parent >> > rate to change. =F0=9F=A4=94 Currently, it only calls clk_hw_can_set_r= ate_parent() >> > which only checks the flag, but does not check if it is really possible >> > to change the parent's rate. >> > >> > Regardless, please don't prematurely dismiss my proposal. It has the >> > advantage that it is not specific for sunxi-ng, but could be used for >> > other drivers as well. Maybe there other instances of exclusive locks >> > today where the CLK_KEEP_RATE flag might work equally well. =F0=9F=A4= =B7 >> > >> >> If so, I think the thing we should emphasize is that it's about *any >> >> top-level clk API*, as in clk_set_rate() or clk_set_parent(). >> >> >> >> The issue is that any consumer-facing API is taking the clk_prepare l= ock >> >> and thus we would have reentrancy. But we're a provider there, and no= ne >> >> of the clk_hw_* functions are taking that lock. Neither do our own fu= nction. >> >> >> >> So we could call in that notifier our set_rate callback directly, or = we >> >> could create a clk_hw_set_rate() function. >> >> >> >> The first one will create cache issue between the actual rate that the >> >> common clock framework is running and the one we actually enforced, b= ut >> >> we could create a function to flush the CCF cache. >> >> >> >> The second one is probably simpler. >> > >> > I'm probably missing something, because I don't think this would work. >> > For reference, this is our tree: >> > >> > pll-video0 >> > hdmi-phy-clk >> > hdmi >> > tcon1 >> > pll-mipi >> > tcon0 >> > tcon-data-clock >> > >> > When pll-video0's rate is changed (e.g. because a HDMI monitor is >> > plugged in), the rates of the complete subtree for pll-video0 are >> > recalculated, including tcon0 and tcon-data-clock. The rate of tcon0 is >> > based on the rate that was recalculated for pll-mipi, which - in turn - >> > was of course recalculated based on the pll-video0's new rate. These >> > values are stored by the clk framework in a private struct. They are >> > calculated before actually performing any rate changes. >> > >> > So, if a notifier sets pll-mipi's rate to something else than was >> > previously recalculated, the clk framework would still try to set tcon0 >> > to the value that it previously calculated. >> > >> > So, we would have to recalculate pll-mipi's subtree after changing its >> > rate (that's what PATCH 1 is doing). >> >> Sorry, I forgot that this actually was possible by flagging pll-mipi >> with CLK_RECALC_NEW_RATES. But the real problem I was fighting with when >> trying to use the notifiers is something else. >> >> Initially, pll-video0 is set by the bootloader. In my case uboot sets it >> to 294 MHz. pll-mipi is set to 588 MHz. >> >> Afterward, there are actually two types of calls for setting pll-mipi in >> my scenario: >> 1. during boot when tcon-data-clock is set to drive the LCD panel >> 2. when the HDMI cable is plugged in > > Not really. Both of those clocks can change (or not) at any point in > time. What triggers the rate set is a modeset which might never happen > (if the display driver or output is disabled, if the fbdev emulation is > disabled or if there's never a compositor starting) or possibly happen > each frame on both output for all you know. Ok, thank you for the explanation and I apologize for not having the terminology straight. This would mean that in my description above there are two modesets that trigger the rate set 1. for the tcon-data-clock on boot when the internal display is activated and 2. for hdmi when the external monitor is activated. For reference: In my scenario I'm using a pinephone which has an internal LCD display and an HDMI connector (not supported in mainline). I understand that there could be more. Let's put a pin in that, because my understandig is, that this is not the relevant part here. >> In the first case, the rate for pll-mipi is based on the rate that >> tcon-data-clock requests. In that case, we do not want to restore the >> previous rate. >> >> In the second case, pll-mipi should try to remain running at the >> previous rate (the one that was requested by tcon-data-clock). That's >> the reason for setting core->req_rate in PATCH 1. >> >> Unfortunately, the notifier does not provide us with enough context to >> distinguish the two cases. > > I don't think any piece of code will be able to, really. In the first case, setting the pll-mipi clock starts from the bottom (tcon-data-clock) and uses clk_calc_new_rates() to get to the top-most clock that needs and can be changed. On it's way up to pll-video0 it passes pll-mipi. My proposal (PATCH 1) is to use that moment to store the rate in req_rate. In contrast, in the second case, setting the pll-mipi clock starts from the top. pll-video0's rate is changed and therefore a new rate is propagate for the whole subtree where, on its way down, it passes pll-mipi. My proposal (PATCH 1) is to use that moment to restore the rate from req_rate that was previously (see pragraph above) set. Since I did not see a way to achieve this using notifiers (and you seem to agree), I chose to propose a different path. > Your definition of CLK_KEEP_RATE is that it will "try to keep rate, if > parent rate changes" > > What happens if it fails, possibly because of rounding like you > mentioned already? Maybe "keep" is too strong of a word. I'm sorry for the confusion my poor choice of wording has caused. What I would like if for a clock to go back as closely as possible to the previous rate. And this is what PATCH 1 does by using the clocks determine_rate (or round_rate) operation. > Fundamentally, the problem is that you need different rates on two > subtrees, and we set both to have CLK_SET_RATE_PARENT and allow both to > change the parent rate if needed. This reads to me as if you are emphasizing the word "both" here. I'm aware that you know, but I state it here for the benefit of others: Up until kernel 6.5 only hdmi resets pll-video0. pll-mipi does not set pll-video0. This has changed in clk-next. Now also pll-mipi sets the parent rate. Icenowy's patches are aimed at (and work for) up to kernel 6.5. They restore pll-mipi's rate after pll-video0 has been changed by hdmi. > What would happen if we force pll-video0 to a known, fixed, value and > remove CLK_SET_RATE_PARENT from both the pll-mipi and hdmi clocks? Approximately three months ago, I wrote "one could argue that pll-video0 should be set to 297MHz at boot", to which Jernej responded: "Ideally, clock rate setting code should be immune on "initial" values, set by bootloader or default values. If it's not, then it should be improved in the way that it is.": https://lore.kernel.org/linux-clk/4831731.31r3eYUQgx@jernej-laptop/#t That's what got me startet on the whole process of allowing pll-mipi to set it's parent instead of simply setting it to a known rate of 297 MHz. Should I resume the other (297MHz) investigation? I had another idea, but don't know how/if that's possible: Maybe we could use a notifier to notify pll-mipi (or even better: tcon-data-clock) and use some mechanism to defer calling clk_set_rate() to a point in time when the whole process of setting the clocks is complete. Best regards, Frank > > Maxime