From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5F51C77B7A for ; Fri, 26 May 2023 14:45:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237052AbjEZOon (ORCPT ); Fri, 26 May 2023 10:44:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244062AbjEZOoi (ORCPT ); Fri, 26 May 2023 10:44:38 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 432EEE4F; Fri, 26 May 2023 07:44:09 -0700 (PDT) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 1960C21AC4; Fri, 26 May 2023 14:44:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685112247; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fcZ4ZRR0yos18sY7hIGTBFkGlmXiSRFOxBcjHXBe5hU=; b=J4h09ub7B+8cW6XM+zzyKpToCRFhpDx/CNcNoZrZzAVQoYGEBCP2s6MJYbAJEkhjqQwr6+ Ii2jyonq6j1wgE1F5cWTup3ltYGq2aZa7w8gMRcHYrnG3oXEpMwdEygPcGHtjblnr0n9bd xIplkFCoy2RpRD5gkrV4W7dOAktOra8= Received: from suse.cz (unknown [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 151282C141; Fri, 26 May 2023 14:44:04 +0000 (UTC) Date: Fri, 26 May 2023 16:44:03 +0200 From: Petr Mladek To: Doug Anderson Cc: Andrew Morton , Matthias Kaehlcke , kgdb-bugreport@lists.sourceforge.net, Stephane Eranian , mpe@ellerman.id.au, Tzung-Bi Shih , Daniel Thompson , Mark Rutland , linuxppc-dev@lists.ozlabs.org, Sumit Garg , npiggin@gmail.com, davem@davemloft.net, Marc Zyngier , Stephen Boyd , sparclinux@vger.kernel.org, christophe.leroy@csgroup.eu, Catalin Marinas , ravi.v.shankar@intel.com, Randy Dunlap , Pingfan Liu , Guenter Roeck , Lecopzer Chen , Ian Rogers , ito-yuichi@fujitsu.com, ricardo.neri@intel.com, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, Will Deacon , Chen-Yu Tsai , linux-kernel@vger.kernel.org, Masayoshi Mizuma , Andi Kleen Subject: Re: [PATCH v5 13/18] watchdog/hardlockup: Have the perf hardlockup use __weak functions more cleanly Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.13.I847d9ec852449350997ba00401d2462a9cb4302b@changeid> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-perf-users@vger.kernel.org On Wed 2023-05-24 12:38:49, Doug Anderson wrote: > Hi, > > On Wed, May 24, 2023 at 6:59 AM Petr Mladek wrote: > > > > On Fri 2023-05-19 10:18:37, Douglas Anderson wrote: > > > The fact that there watchdog_hardlockup_enable(), > > > watchdog_hardlockup_disable(), and watchdog_hardlockup_probe() are > > > declared __weak means that the configured hardlockup detector can > > > define non-weak versions of those functions if it needs to. Instead of > > > doing this, the perf hardlockup detector hooked itself into the > > > default __weak implementation, which was a bit awkward. Clean this up. > > > > > > >From comments, it looks as if the original design was done because the > > > __weak function were expected to implemented by the architecture and > > > not by the configured hardlockup detector. This got awkward when we > > > tried to add the buddy lockup detector which was not arch-specific but > > > wanted to hook into those same functions. > > > > > > This is not expected to have any functional impact. > > > > > > @@ -187,27 +187,33 @@ static inline void watchdog_hardlockup_kick(void) { } > > > #endif /* !CONFIG_HARDLOCKUP_DETECTOR_PERF */ > > > > > > /* > > > - * These functions can be overridden if an architecture implements its > > > - * own hardlockup detector. > > > + * These functions can be overridden based on the configured hardlockdup detector. > > > * > > > * watchdog_hardlockup_enable/disable can be implemented to start and stop when > > > - * softlockup watchdog start and stop. The arch must select the > > > + * softlockup watchdog start and stop. The detector must select the > > > * SOFTLOCKUP_DETECTOR Kconfig. > > > */ > > > -void __weak watchdog_hardlockup_enable(unsigned int cpu) > > > -{ > > > - hardlockup_detector_perf_enable(); > > > -} > > > +void __weak watchdog_hardlockup_enable(unsigned int cpu) { } > > > > > > -void __weak watchdog_hardlockup_disable(unsigned int cpu) > > > -{ > > > - hardlockup_detector_perf_disable(); > > > -} > > > +void __weak watchdog_hardlockup_disable(unsigned int cpu) { } > > > > > > /* Return 0, if a hardlockup watchdog is available. Error code otherwise */ > > > int __weak __init watchdog_hardlockup_probe(void) > > > { > > > - return hardlockup_detector_perf_init(); > > > + /* > > > + * If CONFIG_HAVE_NMI_WATCHDOG is defined then an architecture > > > + * is assumed to have the hard watchdog available and we return 0. > > > + */ > > > + if (IS_ENABLED(CONFIG_HAVE_NMI_WATCHDOG)) > > > + return 0; > > > + > > > + /* > > > + * Hardlockup detectors other than those using CONFIG_HAVE_NMI_WATCHDOG > > > + * are required to implement a non-weak version of this probe function > > > + * to tell whether they are available. If they don't override then > > > + * we'll return -ENODEV. > > > + */ > > > + return -ENODEV; > > > } > > > > When thinking more about it. It is weird that we need to handle > > CONFIG_HAVE_NMI_WATCHDOG in this default week function. > > > > It should be handled in watchdog_hardlockup_probe() implemented > > in kernel/watchdog_perf.c. > > > > IMHO, the default __weak function could always return -ENODEV; > > > > Would it make sense, please? > > I don't quite understand. I'd agree that the special case for > CONFIG_HAVE_NMI_WATCHDOG is ugly, but it was also ugly before. IMO > it's actually a little less ugly / easier to understand after my > patch. ...but let me walk through how I think this special case works > and maybe you can tell me where I'm confused. > > The first thing to understand is that CONFIG_HARDLOCKUP_DETECTOR_PERF > and CONFIG_HAVE_NMI_WATCHDOG are mutually exclusive from each other. > This was true before any of my patches and is still true after them. > Specifically, if CONFIG_HAVE_NMI_WATCHDOG is defined then an > architecture implements arch_touch_nmi_watchdog() (as documented in > the Kconfig docs for HAVE_NMI_WATCHDOG). Looking at the tree before my > series you can see that the perf hardlockup detector also implemented > arch_touch_nmi_watchdog(). This would have caused a conflict. The > mutual exclusion was presumably enforced by an architecture not > defining both HAVE_NMI_WATCHDOG and HAVE_HARDLOCKUP_DETECTOR_PERF. > > The second thing to understand is that an architecture that defines > CONFIG_HAVE_NMI_WATCHDOG is _not_ required to implement > watchdog_hardlockup_probe() (used to be called watchdog_nmi_probe()). > Maybe this should change, but at the very least it appears that > SPARC64 defines HAVE_NMI_WATCHDOG but doesn't define > watchdog_hardlockup_probe() AKA watchdog_nmi_probe(). Anyone who > defines CONFIG_HAVE_NMI_WATCHDOG and doesn't implement > watchdog_hardlockup_probe() is claiming that their watchdog needs no > probing and is always available. > > So with that context: > > 1. We can't handle any special cases for CONFIG_HAVE_NMI_WATCHDOG in > "kernel/watchdog_perf.c". The special cases that we need to handle are > all for the cases where CONFIG_HARDLOCKUP_DETECTOR_PERF isn't defined > and that means "kernel/watchdog_perf.c" isn't included. > > 2. We can't have the default __weak function return -ENODEV because > CONFIG_HAVE_NMI_WATCHDOG doesn't require an arch to implement > watchdog_hardlockup_probe(), but we want watchdog_hardlockup_probe() > to return "no error" in that case so that > "watchdog_hardlockup_available" gets set to true. > > Does that sound right? > > I'd agree that a future improvement saying that > CONFIG_HAVE_NMI_WATCHDOG means you _must_ implement > watchdog_hardlockup_probe() would make sense and that would allow us > to get rid of the special case. IMO, though, that's a separate patch. > I'd be happy to review that patch if you wanted to post it up. :-) > > If we want to add that requirement, I _think_ the only thing you'd > need to do is to add watchdog_hardlockup_probe() to sparc64 and have > it return 0 and put that definition in the same file containing > arch_touch_nmi_watchdog(). This is my understanding. IMHO, if we define watchdog_hardlockup_probe() in /arch/sparc/kernel/nmi.c then we could remove the CONFIG_HAVE_NMI_WATCHDOG check from the default watchdog_hardlockup_probe(). Honestly, I am afraid that nobody really thought about any rules. People were just adding their stuff any way that worked for them. And this is why we ended with this maze. It is not your fault. You just moved the ifdef from the header file into the function definition. But it showed very well how ugly it was. By ugly, I mean that powerpc, perf, and buddy hardlockup detectors make watchdog_hardlockup_probe() successful by implementing an alternative to the default weak implementation. Only, sparc64 reports success via a crazy check in the default weak configuration. The check is crazy because it makes the decision based on CONFIG_HAVE_NMI_WATCHDOG. But the related code is in arch/sparc/kernel/nmi.c which is compiled when CONFIG_SPARC64 is enabled. The connection between CONFIG_HAVE_NMI_WATCHDOG and CONFIG_SPARC64 is hidden in arch/sparc/Kconfig. It would be much more straightforward when the weak function is implemented in arch/sparc/kernel/nmi.c. It will be clear that probe() will succeed when the watchdog gets initialized. > powerpc also gets CONFIG_HAVE_NMI_WATCHDOG > as a side effect of selecting CONFIG_HAVE_HARDLOCKUP_DETECTOR_ARCH but > it looks like they implement watchdog_hardlockup_probe() already. Oh, > but maybe this will fix a preexisting (existed before my patches) > minor bug... Unless I'm missing something (entirely possible!) on > powerpc today I guess if you turn off CONFIG_PPC_WATCHDOG then > CONFIG_HAVE_HARDLOCKUP_DETECTOR_ARCH and CONFIG_HAVE_NMI_WATCHDOG > would still be defined and we'd end up returning 0 (no error) from > watchdog_hardlockup_probe(). That means that on powerpc today if you > turn off CONFIG_PPC_WATCHDOG that '/proc/sys/kernel/nmi_watchdog' will > still think the watchdog is enabled? Yeah, it seems that this bug was there. And it will get fixed when the default weak implementation of watchdog_hardlockup_probe() always returns false. Again, I'll let Andrew to decide whether this should get cleaned in this patchset or later. But it would be fine to fix this after we spent so much time understanding the mess. Best Regards, Petr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CD44C7EE23 for ; Fri, 26 May 2023 14:45:10 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QSSPw1ch2z3fH6 for ; Sat, 27 May 2023 00:45:08 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=J4h09ub7; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=suse.com (client-ip=2001:67c:2178:6::1c; helo=smtp-out1.suse.de; envelope-from=pmladek@suse.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=J4h09ub7; dkim-atps=neutral Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QSSNx069Pz3bmP for ; Sat, 27 May 2023 00:44:15 +1000 (AEST) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 1960C21AC4; Fri, 26 May 2023 14:44:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685112247; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fcZ4ZRR0yos18sY7hIGTBFkGlmXiSRFOxBcjHXBe5hU=; b=J4h09ub7B+8cW6XM+zzyKpToCRFhpDx/CNcNoZrZzAVQoYGEBCP2s6MJYbAJEkhjqQwr6+ Ii2jyonq6j1wgE1F5cWTup3ltYGq2aZa7w8gMRcHYrnG3oXEpMwdEygPcGHtjblnr0n9bd xIplkFCoy2RpRD5gkrV4W7dOAktOra8= Received: from suse.cz (unknown [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 151282C141; Fri, 26 May 2023 14:44:04 +0000 (UTC) Date: Fri, 26 May 2023 16:44:03 +0200 From: Petr Mladek To: Doug Anderson Subject: Re: [PATCH v5 13/18] watchdog/hardlockup: Have the perf hardlockup use __weak functions more cleanly Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.13.I847d9ec852449350997ba00401d2462a9cb4302b@changeid> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Ian Rogers , ito-yuichi@fujitsu.com, Lecopzer Chen , kgdb-bugreport@lists.sourceforge.net, ricardo.neri@intel.com, Stephane Eranian , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Daniel Thompson , Andi Kleen , Marc Zyngier , Chen-Yu Tsai , Matthias Kaehlcke , Catalin Marinas , Masayoshi Mizuma , ravi.v.shankar@intel.com, Tzung-Bi Shih , npiggin@gmail.com, Stephen Boyd , Pingfan Liu , linux-arm-kernel@lists.infradead.org, Sumit Garg , Randy Dunlap , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Andrew Morton , linuxppc-dev@lists.ozl abs.org, davem@davemloft.net Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed 2023-05-24 12:38:49, Doug Anderson wrote: > Hi, > > On Wed, May 24, 2023 at 6:59 AM Petr Mladek wrote: > > > > On Fri 2023-05-19 10:18:37, Douglas Anderson wrote: > > > The fact that there watchdog_hardlockup_enable(), > > > watchdog_hardlockup_disable(), and watchdog_hardlockup_probe() are > > > declared __weak means that the configured hardlockup detector can > > > define non-weak versions of those functions if it needs to. Instead of > > > doing this, the perf hardlockup detector hooked itself into the > > > default __weak implementation, which was a bit awkward. Clean this up. > > > > > > >From comments, it looks as if the original design was done because the > > > __weak function were expected to implemented by the architecture and > > > not by the configured hardlockup detector. This got awkward when we > > > tried to add the buddy lockup detector which was not arch-specific but > > > wanted to hook into those same functions. > > > > > > This is not expected to have any functional impact. > > > > > > @@ -187,27 +187,33 @@ static inline void watchdog_hardlockup_kick(void) { } > > > #endif /* !CONFIG_HARDLOCKUP_DETECTOR_PERF */ > > > > > > /* > > > - * These functions can be overridden if an architecture implements its > > > - * own hardlockup detector. > > > + * These functions can be overridden based on the configured hardlockdup detector. > > > * > > > * watchdog_hardlockup_enable/disable can be implemented to start and stop when > > > - * softlockup watchdog start and stop. The arch must select the > > > + * softlockup watchdog start and stop. The detector must select the > > > * SOFTLOCKUP_DETECTOR Kconfig. > > > */ > > > -void __weak watchdog_hardlockup_enable(unsigned int cpu) > > > -{ > > > - hardlockup_detector_perf_enable(); > > > -} > > > +void __weak watchdog_hardlockup_enable(unsigned int cpu) { } > > > > > > -void __weak watchdog_hardlockup_disable(unsigned int cpu) > > > -{ > > > - hardlockup_detector_perf_disable(); > > > -} > > > +void __weak watchdog_hardlockup_disable(unsigned int cpu) { } > > > > > > /* Return 0, if a hardlockup watchdog is available. Error code otherwise */ > > > int __weak __init watchdog_hardlockup_probe(void) > > > { > > > - return hardlockup_detector_perf_init(); > > > + /* > > > + * If CONFIG_HAVE_NMI_WATCHDOG is defined then an architecture > > > + * is assumed to have the hard watchdog available and we return 0. > > > + */ > > > + if (IS_ENABLED(CONFIG_HAVE_NMI_WATCHDOG)) > > > + return 0; > > > + > > > + /* > > > + * Hardlockup detectors other than those using CONFIG_HAVE_NMI_WATCHDOG > > > + * are required to implement a non-weak version of this probe function > > > + * to tell whether they are available. If they don't override then > > > + * we'll return -ENODEV. > > > + */ > > > + return -ENODEV; > > > } > > > > When thinking more about it. It is weird that we need to handle > > CONFIG_HAVE_NMI_WATCHDOG in this default week function. > > > > It should be handled in watchdog_hardlockup_probe() implemented > > in kernel/watchdog_perf.c. > > > > IMHO, the default __weak function could always return -ENODEV; > > > > Would it make sense, please? > > I don't quite understand. I'd agree that the special case for > CONFIG_HAVE_NMI_WATCHDOG is ugly, but it was also ugly before. IMO > it's actually a little less ugly / easier to understand after my > patch. ...but let me walk through how I think this special case works > and maybe you can tell me where I'm confused. > > The first thing to understand is that CONFIG_HARDLOCKUP_DETECTOR_PERF > and CONFIG_HAVE_NMI_WATCHDOG are mutually exclusive from each other. > This was true before any of my patches and is still true after them. > Specifically, if CONFIG_HAVE_NMI_WATCHDOG is defined then an > architecture implements arch_touch_nmi_watchdog() (as documented in > the Kconfig docs for HAVE_NMI_WATCHDOG). Looking at the tree before my > series you can see that the perf hardlockup detector also implemented > arch_touch_nmi_watchdog(). This would have caused a conflict. The > mutual exclusion was presumably enforced by an architecture not > defining both HAVE_NMI_WATCHDOG and HAVE_HARDLOCKUP_DETECTOR_PERF. > > The second thing to understand is that an architecture that defines > CONFIG_HAVE_NMI_WATCHDOG is _not_ required to implement > watchdog_hardlockup_probe() (used to be called watchdog_nmi_probe()). > Maybe this should change, but at the very least it appears that > SPARC64 defines HAVE_NMI_WATCHDOG but doesn't define > watchdog_hardlockup_probe() AKA watchdog_nmi_probe(). Anyone who > defines CONFIG_HAVE_NMI_WATCHDOG and doesn't implement > watchdog_hardlockup_probe() is claiming that their watchdog needs no > probing and is always available. > > So with that context: > > 1. We can't handle any special cases for CONFIG_HAVE_NMI_WATCHDOG in > "kernel/watchdog_perf.c". The special cases that we need to handle are > all for the cases where CONFIG_HARDLOCKUP_DETECTOR_PERF isn't defined > and that means "kernel/watchdog_perf.c" isn't included. > > 2. We can't have the default __weak function return -ENODEV because > CONFIG_HAVE_NMI_WATCHDOG doesn't require an arch to implement > watchdog_hardlockup_probe(), but we want watchdog_hardlockup_probe() > to return "no error" in that case so that > "watchdog_hardlockup_available" gets set to true. > > Does that sound right? > > I'd agree that a future improvement saying that > CONFIG_HAVE_NMI_WATCHDOG means you _must_ implement > watchdog_hardlockup_probe() would make sense and that would allow us > to get rid of the special case. IMO, though, that's a separate patch. > I'd be happy to review that patch if you wanted to post it up. :-) > > If we want to add that requirement, I _think_ the only thing you'd > need to do is to add watchdog_hardlockup_probe() to sparc64 and have > it return 0 and put that definition in the same file containing > arch_touch_nmi_watchdog(). This is my understanding. IMHO, if we define watchdog_hardlockup_probe() in /arch/sparc/kernel/nmi.c then we could remove the CONFIG_HAVE_NMI_WATCHDOG check from the default watchdog_hardlockup_probe(). Honestly, I am afraid that nobody really thought about any rules. People were just adding their stuff any way that worked for them. And this is why we ended with this maze. It is not your fault. You just moved the ifdef from the header file into the function definition. But it showed very well how ugly it was. By ugly, I mean that powerpc, perf, and buddy hardlockup detectors make watchdog_hardlockup_probe() successful by implementing an alternative to the default weak implementation. Only, sparc64 reports success via a crazy check in the default weak configuration. The check is crazy because it makes the decision based on CONFIG_HAVE_NMI_WATCHDOG. But the related code is in arch/sparc/kernel/nmi.c which is compiled when CONFIG_SPARC64 is enabled. The connection between CONFIG_HAVE_NMI_WATCHDOG and CONFIG_SPARC64 is hidden in arch/sparc/Kconfig. It would be much more straightforward when the weak function is implemented in arch/sparc/kernel/nmi.c. It will be clear that probe() will succeed when the watchdog gets initialized. > powerpc also gets CONFIG_HAVE_NMI_WATCHDOG > as a side effect of selecting CONFIG_HAVE_HARDLOCKUP_DETECTOR_ARCH but > it looks like they implement watchdog_hardlockup_probe() already. Oh, > but maybe this will fix a preexisting (existed before my patches) > minor bug... Unless I'm missing something (entirely possible!) on > powerpc today I guess if you turn off CONFIG_PPC_WATCHDOG then > CONFIG_HAVE_HARDLOCKUP_DETECTOR_ARCH and CONFIG_HAVE_NMI_WATCHDOG > would still be defined and we'd end up returning 0 (no error) from > watchdog_hardlockup_probe(). That means that on powerpc today if you > turn off CONFIG_PPC_WATCHDOG that '/proc/sys/kernel/nmi_watchdog' will > still think the watchdog is enabled? Yeah, it seems that this bug was there. And it will get fixed when the default weak implementation of watchdog_hardlockup_probe() always returns false. Again, I'll let Andrew to decide whether this should get cleaned in this patchset or later. But it would be fine to fix this after we spent so much time understanding the mess. Best Regards, Petr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4075C7EE23 for ; Fri, 26 May 2023 14:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=D5qXYr+tXHZlHwXnMNuftY6vagg+qNDKILVyiN8xS9o=; b=BFpMtYyUOl/OpX t6b22/mDUz5gngWDKF8YdTcZQPSQYL27k5wWsa19HYS7pzOQqWcMIgf80JuR0rIxn7C35/8A8M1iu zwMNGBeddiQyVF0naYxMUdywkIa9mr1b6Km+5dDxVP3mc0F/zMdmvRAtAxHpcFrLa+MLbdmUN2dwd RH8s6mwBQ3bNZXfV+NoDi151Wc/4D+HbeH6+bOEVI71FyD1vshkD6RzK5TpYiw8L+TrluFkoiR60Y z35pwbIyf3cxLBYqX3NTajw2wXfBE+vVzMOHShlcSpSE9PyLm5XDCkt5uAJCLoJv1PnEBdVOMT2z8 xc8KN9Kgcoyxn5s0zR0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q2Yfs-002qfT-1f; Fri, 26 May 2023 14:44:12 +0000 Received: from smtp-out1.suse.de ([2001:67c:2178:6::1c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q2Yfo-002qeL-2S for linux-arm-kernel@lists.infradead.org; Fri, 26 May 2023 14:44:10 +0000 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 1960C21AC4; Fri, 26 May 2023 14:44:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685112247; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fcZ4ZRR0yos18sY7hIGTBFkGlmXiSRFOxBcjHXBe5hU=; b=J4h09ub7B+8cW6XM+zzyKpToCRFhpDx/CNcNoZrZzAVQoYGEBCP2s6MJYbAJEkhjqQwr6+ Ii2jyonq6j1wgE1F5cWTup3ltYGq2aZa7w8gMRcHYrnG3oXEpMwdEygPcGHtjblnr0n9bd xIplkFCoy2RpRD5gkrV4W7dOAktOra8= Received: from suse.cz (unknown [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 151282C141; Fri, 26 May 2023 14:44:04 +0000 (UTC) Date: Fri, 26 May 2023 16:44:03 +0200 From: Petr Mladek To: Doug Anderson Cc: Andrew Morton , Matthias Kaehlcke , kgdb-bugreport@lists.sourceforge.net, Stephane Eranian , mpe@ellerman.id.au, Tzung-Bi Shih , Daniel Thompson , Mark Rutland , linuxppc-dev@lists.ozlabs.org, Sumit Garg , npiggin@gmail.com, davem@davemloft.net, Marc Zyngier , Stephen Boyd , sparclinux@vger.kernel.org, christophe.leroy@csgroup.eu, Catalin Marinas , ravi.v.shankar@intel.com, Randy Dunlap , Pingfan Liu , Guenter Roeck , Lecopzer Chen , Ian Rogers , ito-yuichi@fujitsu.com, ricardo.neri@intel.com, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, Will Deacon , Chen-Yu Tsai , linux-kernel@vger.kernel.org, Masayoshi Mizuma , Andi Kleen Subject: Re: [PATCH v5 13/18] watchdog/hardlockup: Have the perf hardlockup use __weak functions more cleanly Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.13.I847d9ec852449350997ba00401d2462a9cb4302b@changeid> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230526_074408_969843_3715501B X-CRM114-Status: GOOD ( 59.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkIDIwMjMtMDUtMjQgMTI6Mzg6NDksIERvdWcgQW5kZXJzb24gd3JvdGU6Cj4gSGksCj4g Cj4gT24gV2VkLCBNYXkgMjQsIDIwMjMgYXQgNjo1OeKAr0FNIFBldHIgTWxhZGVrIDxwbWxhZGVr QHN1c2UuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBGcmkgMjAyMy0wNS0xOSAxMDoxODozNywgRG91 Z2xhcyBBbmRlcnNvbiB3cm90ZToKPiA+ID4gVGhlIGZhY3QgdGhhdCB0aGVyZSB3YXRjaGRvZ19o YXJkbG9ja3VwX2VuYWJsZSgpLAo+ID4gPiB3YXRjaGRvZ19oYXJkbG9ja3VwX2Rpc2FibGUoKSwg YW5kIHdhdGNoZG9nX2hhcmRsb2NrdXBfcHJvYmUoKSBhcmUKPiA+ID4gZGVjbGFyZWQgX193ZWFr IG1lYW5zIHRoYXQgdGhlIGNvbmZpZ3VyZWQgaGFyZGxvY2t1cCBkZXRlY3RvciBjYW4KPiA+ID4g ZGVmaW5lIG5vbi13ZWFrIHZlcnNpb25zIG9mIHRob3NlIGZ1bmN0aW9ucyBpZiBpdCBuZWVkcyB0 by4gSW5zdGVhZCBvZgo+ID4gPiBkb2luZyB0aGlzLCB0aGUgcGVyZiBoYXJkbG9ja3VwIGRldGVj dG9yIGhvb2tlZCBpdHNlbGYgaW50byB0aGUKPiA+ID4gZGVmYXVsdCBfX3dlYWsgaW1wbGVtZW50 YXRpb24sIHdoaWNoIHdhcyBhIGJpdCBhd2t3YXJkLiBDbGVhbiB0aGlzIHVwLgo+ID4gPgo+ID4g PiA+RnJvbSBjb21tZW50cywgaXQgbG9va3MgYXMgaWYgdGhlIG9yaWdpbmFsIGRlc2lnbiB3YXMg ZG9uZSBiZWNhdXNlIHRoZQo+ID4gPiBfX3dlYWsgZnVuY3Rpb24gd2VyZSBleHBlY3RlZCB0byBp bXBsZW1lbnRlZCBieSB0aGUgYXJjaGl0ZWN0dXJlIGFuZAo+ID4gPiBub3QgYnkgdGhlIGNvbmZp Z3VyZWQgaGFyZGxvY2t1cCBkZXRlY3Rvci4gVGhpcyBnb3QgYXdrd2FyZCB3aGVuIHdlCj4gPiA+ IHRyaWVkIHRvIGFkZCB0aGUgYnVkZHkgbG9ja3VwIGRldGVjdG9yIHdoaWNoIHdhcyBub3QgYXJj aC1zcGVjaWZpYyBidXQKPiA+ID4gd2FudGVkIHRvIGhvb2sgaW50byB0aG9zZSBzYW1lIGZ1bmN0 aW9ucy4KPiA+ID4KPiA+ID4gVGhpcyBpcyBub3QgZXhwZWN0ZWQgdG8gaGF2ZSBhbnkgZnVuY3Rp b25hbCBpbXBhY3QuCj4gPiA+Cj4gPiA+IEBAIC0xODcsMjcgKzE4NywzMyBAQCBzdGF0aWMgaW5s aW5lIHZvaWQgd2F0Y2hkb2dfaGFyZGxvY2t1cF9raWNrKHZvaWQpIHsgfQo+ID4gPiAgI2VuZGlm IC8qICFDT05GSUdfSEFSRExPQ0tVUF9ERVRFQ1RPUl9QRVJGICovCj4gPiA+Cj4gPiA+ICAvKgo+ ID4gPiAtICogVGhlc2UgZnVuY3Rpb25zIGNhbiBiZSBvdmVycmlkZGVuIGlmIGFuIGFyY2hpdGVj dHVyZSBpbXBsZW1lbnRzIGl0cwo+ID4gPiAtICogb3duIGhhcmRsb2NrdXAgZGV0ZWN0b3IuCj4g PiA+ICsgKiBUaGVzZSBmdW5jdGlvbnMgY2FuIGJlIG92ZXJyaWRkZW4gYmFzZWQgb24gdGhlIGNv bmZpZ3VyZWQgaGFyZGxvY2tkdXAgZGV0ZWN0b3IuCj4gPiA+ICAgKgo+ID4gPiAgICogd2F0Y2hk b2dfaGFyZGxvY2t1cF9lbmFibGUvZGlzYWJsZSBjYW4gYmUgaW1wbGVtZW50ZWQgdG8gc3RhcnQg YW5kIHN0b3Agd2hlbgo+ID4gPiAtICogc29mdGxvY2t1cCB3YXRjaGRvZyBzdGFydCBhbmQgc3Rv cC4gVGhlIGFyY2ggbXVzdCBzZWxlY3QgdGhlCj4gPiA+ICsgKiBzb2Z0bG9ja3VwIHdhdGNoZG9n IHN0YXJ0IGFuZCBzdG9wLiBUaGUgZGV0ZWN0b3IgbXVzdCBzZWxlY3QgdGhlCj4gPiA+ICAgKiBT T0ZUTE9DS1VQX0RFVEVDVE9SIEtjb25maWcuCj4gPiA+ICAgKi8KPiA+ID4gLXZvaWQgX193ZWFr IHdhdGNoZG9nX2hhcmRsb2NrdXBfZW5hYmxlKHVuc2lnbmVkIGludCBjcHUpCj4gPiA+IC17Cj4g PiA+IC0gICAgIGhhcmRsb2NrdXBfZGV0ZWN0b3JfcGVyZl9lbmFibGUoKTsKPiA+ID4gLX0KPiA+ ID4gK3ZvaWQgX193ZWFrIHdhdGNoZG9nX2hhcmRsb2NrdXBfZW5hYmxlKHVuc2lnbmVkIGludCBj cHUpIHsgfQo+ID4gPgo+ID4gPiAtdm9pZCBfX3dlYWsgd2F0Y2hkb2dfaGFyZGxvY2t1cF9kaXNh YmxlKHVuc2lnbmVkIGludCBjcHUpCj4gPiA+IC17Cj4gPiA+IC0gICAgIGhhcmRsb2NrdXBfZGV0 ZWN0b3JfcGVyZl9kaXNhYmxlKCk7Cj4gPiA+IC19Cj4gPiA+ICt2b2lkIF9fd2VhayB3YXRjaGRv Z19oYXJkbG9ja3VwX2Rpc2FibGUodW5zaWduZWQgaW50IGNwdSkgeyB9Cj4gPiA+Cj4gPiA+ICAv KiBSZXR1cm4gMCwgaWYgYSBoYXJkbG9ja3VwIHdhdGNoZG9nIGlzIGF2YWlsYWJsZS4gRXJyb3Ig Y29kZSBvdGhlcndpc2UgKi8KPiA+ID4gIGludCBfX3dlYWsgX19pbml0IHdhdGNoZG9nX2hhcmRs b2NrdXBfcHJvYmUodm9pZCkKPiA+ID4gIHsKPiA+ID4gLSAgICAgcmV0dXJuIGhhcmRsb2NrdXBf ZGV0ZWN0b3JfcGVyZl9pbml0KCk7Cj4gPiA+ICsgICAgIC8qCj4gPiA+ICsgICAgICAqIElmIENP TkZJR19IQVZFX05NSV9XQVRDSERPRyBpcyBkZWZpbmVkIHRoZW4gYW4gYXJjaGl0ZWN0dXJlCj4g PiA+ICsgICAgICAqIGlzIGFzc3VtZWQgdG8gaGF2ZSB0aGUgaGFyZCB3YXRjaGRvZyBhdmFpbGFi bGUgYW5kIHdlIHJldHVybiAwLgo+ID4gPiArICAgICAgKi8KPiA+ID4gKyAgICAgaWYgKElTX0VO QUJMRUQoQ09ORklHX0hBVkVfTk1JX1dBVENIRE9HKSkKPiA+ID4gKyAgICAgICAgICAgICByZXR1 cm4gMDsKPiA+ID4gKwo+ID4gPiArICAgICAvKgo+ID4gPiArICAgICAgKiBIYXJkbG9ja3VwIGRl dGVjdG9ycyBvdGhlciB0aGFuIHRob3NlIHVzaW5nIENPTkZJR19IQVZFX05NSV9XQVRDSERPRwo+ ID4gPiArICAgICAgKiBhcmUgcmVxdWlyZWQgdG8gaW1wbGVtZW50IGEgbm9uLXdlYWsgdmVyc2lv biBvZiB0aGlzIHByb2JlIGZ1bmN0aW9uCj4gPiA+ICsgICAgICAqIHRvIHRlbGwgd2hldGhlciB0 aGV5IGFyZSBhdmFpbGFibGUuIElmIHRoZXkgZG9uJ3Qgb3ZlcnJpZGUgdGhlbgo+ID4gPiArICAg ICAgKiB3ZSdsbCByZXR1cm4gLUVOT0RFVi4KPiA+ID4gKyAgICAgICovCj4gPiA+ICsgICAgIHJl dHVybiAtRU5PREVWOwo+ID4gPiAgfQo+ID4KPiA+IFdoZW4gdGhpbmtpbmcgbW9yZSBhYm91dCBp dC4gSXQgaXMgd2VpcmQgdGhhdCB3ZSBuZWVkIHRvIGhhbmRsZQo+ID4gQ09ORklHX0hBVkVfTk1J X1dBVENIRE9HIGluIHRoaXMgZGVmYXVsdCB3ZWVrIGZ1bmN0aW9uLgo+ID4KPiA+IEl0IHNob3Vs ZCBiZSBoYW5kbGVkIGluIHdhdGNoZG9nX2hhcmRsb2NrdXBfcHJvYmUoKSBpbXBsZW1lbnRlZAo+ ID4gaW4ga2VybmVsL3dhdGNoZG9nX3BlcmYuYy4KPiA+Cj4gPiBJTUhPLCB0aGUgZGVmYXVsdCBf X3dlYWsgZnVuY3Rpb24gY291bGQgYWx3YXlzIHJldHVybiAtRU5PREVWOwo+ID4KPiA+IFdvdWxk IGl0IG1ha2Ugc2Vuc2UsIHBsZWFzZT8KPiAKPiBJIGRvbid0IHF1aXRlIHVuZGVyc3RhbmQuIEkn ZCBhZ3JlZSB0aGF0IHRoZSBzcGVjaWFsIGNhc2UgZm9yCj4gQ09ORklHX0hBVkVfTk1JX1dBVENI RE9HIGlzIHVnbHksIGJ1dCBpdCB3YXMgYWxzbyB1Z2x5IGJlZm9yZS4gSU1PCj4gaXQncyBhY3R1 YWxseSBhIGxpdHRsZSBsZXNzIHVnbHkgLyBlYXNpZXIgdG8gdW5kZXJzdGFuZCBhZnRlciBteQo+ IHBhdGNoLiAuLi5idXQgbGV0IG1lIHdhbGsgdGhyb3VnaCBob3cgSSB0aGluayB0aGlzIHNwZWNp YWwgY2FzZSB3b3Jrcwo+IGFuZCBtYXliZSB5b3UgY2FuIHRlbGwgbWUgd2hlcmUgSSdtIGNvbmZ1 c2VkLgo+IAo+IFRoZSBmaXJzdCB0aGluZyB0byB1bmRlcnN0YW5kIGlzIHRoYXQgQ09ORklHX0hB UkRMT0NLVVBfREVURUNUT1JfUEVSRgo+IGFuZCBDT05GSUdfSEFWRV9OTUlfV0FUQ0hET0cgYXJl IG11dHVhbGx5IGV4Y2x1c2l2ZSBmcm9tIGVhY2ggb3RoZXIuCj4gVGhpcyB3YXMgdHJ1ZSBiZWZv cmUgYW55IG9mIG15IHBhdGNoZXMgYW5kIGlzIHN0aWxsIHRydWUgYWZ0ZXIgdGhlbS4KPiBTcGVj aWZpY2FsbHksIGlmIENPTkZJR19IQVZFX05NSV9XQVRDSERPRyBpcyBkZWZpbmVkIHRoZW4gYW4K PiBhcmNoaXRlY3R1cmUgaW1wbGVtZW50cyBhcmNoX3RvdWNoX25taV93YXRjaGRvZygpIChhcyBk b2N1bWVudGVkIGluCj4gdGhlIEtjb25maWcgZG9jcyBmb3IgSEFWRV9OTUlfV0FUQ0hET0cpLiBM b29raW5nIGF0IHRoZSB0cmVlIGJlZm9yZSBteQo+IHNlcmllcyB5b3UgY2FuIHNlZSB0aGF0IHRo ZSBwZXJmIGhhcmRsb2NrdXAgZGV0ZWN0b3IgYWxzbyBpbXBsZW1lbnRlZAo+IGFyY2hfdG91Y2hf bm1pX3dhdGNoZG9nKCkuIFRoaXMgd291bGQgaGF2ZSBjYXVzZWQgYSBjb25mbGljdC4gVGhlCj4g bXV0dWFsIGV4Y2x1c2lvbiB3YXMgcHJlc3VtYWJseSBlbmZvcmNlZCBieSBhbiBhcmNoaXRlY3R1 cmUgbm90Cj4gZGVmaW5pbmcgYm90aCBIQVZFX05NSV9XQVRDSERPRyBhbmQgSEFWRV9IQVJETE9D S1VQX0RFVEVDVE9SX1BFUkYuCj4gCj4gVGhlIHNlY29uZCB0aGluZyB0byB1bmRlcnN0YW5kIGlz IHRoYXQgYW4gYXJjaGl0ZWN0dXJlIHRoYXQgZGVmaW5lcwo+IENPTkZJR19IQVZFX05NSV9XQVRD SERPRyBpcyBfbm90XyByZXF1aXJlZCB0byBpbXBsZW1lbnQKPiB3YXRjaGRvZ19oYXJkbG9ja3Vw X3Byb2JlKCkgKHVzZWQgdG8gYmUgY2FsbGVkIHdhdGNoZG9nX25taV9wcm9iZSgpKS4KPiBNYXli ZSB0aGlzIHNob3VsZCBjaGFuZ2UsIGJ1dCBhdCB0aGUgdmVyeSBsZWFzdCBpdCBhcHBlYXJzIHRo YXQKPiBTUEFSQzY0IGRlZmluZXMgSEFWRV9OTUlfV0FUQ0hET0cgYnV0IGRvZXNuJ3QgZGVmaW5l Cj4gd2F0Y2hkb2dfaGFyZGxvY2t1cF9wcm9iZSgpIEFLQSB3YXRjaGRvZ19ubWlfcHJvYmUoKS4g QW55b25lIHdobwo+IGRlZmluZXMgQ09ORklHX0hBVkVfTk1JX1dBVENIRE9HIGFuZCBkb2Vzbid0 IGltcGxlbWVudAo+IHdhdGNoZG9nX2hhcmRsb2NrdXBfcHJvYmUoKSBpcyBjbGFpbWluZyB0aGF0 IHRoZWlyIHdhdGNoZG9nIG5lZWRzIG5vCj4gcHJvYmluZyBhbmQgaXMgYWx3YXlzIGF2YWlsYWJs ZS4KPiAKPiBTbyB3aXRoIHRoYXQgY29udGV4dDoKPiAKPiAxLiBXZSBjYW4ndCBoYW5kbGUgYW55 IHNwZWNpYWwgY2FzZXMgZm9yIENPTkZJR19IQVZFX05NSV9XQVRDSERPRyBpbgo+ICJrZXJuZWwv d2F0Y2hkb2dfcGVyZi5jIi4gVGhlIHNwZWNpYWwgY2FzZXMgdGhhdCB3ZSBuZWVkIHRvIGhhbmRs ZSBhcmUKPiBhbGwgZm9yIHRoZSBjYXNlcyB3aGVyZSBDT05GSUdfSEFSRExPQ0tVUF9ERVRFQ1RP Ul9QRVJGIGlzbid0IGRlZmluZWQKPiBhbmQgdGhhdCBtZWFucyAia2VybmVsL3dhdGNoZG9nX3Bl cmYuYyIgaXNuJ3QgaW5jbHVkZWQuCj4gCj4gMi4gV2UgY2FuJ3QgaGF2ZSB0aGUgZGVmYXVsdCBf X3dlYWsgZnVuY3Rpb24gcmV0dXJuIC1FTk9ERVYgYmVjYXVzZQo+IENPTkZJR19IQVZFX05NSV9X QVRDSERPRyBkb2Vzbid0IHJlcXVpcmUgYW4gYXJjaCB0byBpbXBsZW1lbnQKPiB3YXRjaGRvZ19o YXJkbG9ja3VwX3Byb2JlKCksIGJ1dCB3ZSB3YW50IHdhdGNoZG9nX2hhcmRsb2NrdXBfcHJvYmUo KQo+IHRvIHJldHVybiAibm8gZXJyb3IiIGluIHRoYXQgY2FzZSBzbyB0aGF0Cj4gIndhdGNoZG9n X2hhcmRsb2NrdXBfYXZhaWxhYmxlIiBnZXRzIHNldCB0byB0cnVlLgo+IAo+IERvZXMgdGhhdCBz b3VuZCByaWdodD8KPiAKPiBJJ2QgYWdyZWUgdGhhdCBhIGZ1dHVyZSBpbXByb3ZlbWVudCBzYXlp bmcgdGhhdAo+IENPTkZJR19IQVZFX05NSV9XQVRDSERPRyBtZWFucyB5b3UgX211c3RfIGltcGxl bWVudAo+IHdhdGNoZG9nX2hhcmRsb2NrdXBfcHJvYmUoKSB3b3VsZCBtYWtlIHNlbnNlIGFuZCB0 aGF0IHdvdWxkIGFsbG93IHVzCj4gdG8gZ2V0IHJpZCBvZiB0aGUgc3BlY2lhbCBjYXNlLiBJTU8s IHRob3VnaCwgdGhhdCdzIGEgc2VwYXJhdGUgcGF0Y2guCj4gSSdkIGJlIGhhcHB5IHRvIHJldmll dyB0aGF0IHBhdGNoIGlmIHlvdSB3YW50ZWQgdG8gcG9zdCBpdCB1cC4gOi0pCj4gCj4gSWYgd2Ug d2FudCB0byBhZGQgdGhhdCByZXF1aXJlbWVudCwgSSBfdGhpbmtfIHRoZSBvbmx5IHRoaW5nIHlv dSdkCj4gbmVlZCB0byBkbyBpcyB0byBhZGQgd2F0Y2hkb2dfaGFyZGxvY2t1cF9wcm9iZSgpIHRv IHNwYXJjNjQgYW5kIGhhdmUKPiBpdCByZXR1cm4gMCBhbmQgcHV0IHRoYXQgZGVmaW5pdGlvbiBp biB0aGUgc2FtZSBmaWxlIGNvbnRhaW5pbmcKPiBhcmNoX3RvdWNoX25taV93YXRjaGRvZygpLgoK VGhpcyBpcyBteSB1bmRlcnN0YW5kaW5nLiBJTUhPLCBpZiB3ZSBkZWZpbmUKd2F0Y2hkb2dfaGFy ZGxvY2t1cF9wcm9iZSgpIGluIC9hcmNoL3NwYXJjL2tlcm5lbC9ubWkuYwp0aGVuIHdlIGNvdWxk IHJlbW92ZSB0aGUgQ09ORklHX0hBVkVfTk1JX1dBVENIRE9HIGNoZWNrIGZyb20KdGhlIGRlZmF1 bHQgd2F0Y2hkb2dfaGFyZGxvY2t1cF9wcm9iZSgpLgoKSG9uZXN0bHksIEkgYW0gYWZyYWlkIHRo YXQgbm9ib2R5IHJlYWxseSB0aG91Z2h0IGFib3V0IGFueSBydWxlcy4KUGVvcGxlIHdlcmUganVz dCBhZGRpbmcgdGhlaXIgc3R1ZmYgYW55IHdheSB0aGF0IHdvcmtlZCBmb3IgdGhlbS4KQW5kIHRo aXMgaXMgd2h5IHdlIGVuZGVkIHdpdGggdGhpcyBtYXplLgoKSXQgaXMgbm90IHlvdXIgZmF1bHQu IFlvdSBqdXN0IG1vdmVkIHRoZSBpZmRlZiBmcm9tIHRoZSBoZWFkZXIgZmlsZQppbnRvIHRoZSBm dW5jdGlvbiBkZWZpbml0aW9uLiBCdXQgaXQgc2hvd2VkIHZlcnkgd2VsbCBob3cgdWdseSBpdCB3 YXMuCgpCeSB1Z2x5LCBJIG1lYW4gdGhhdCBwb3dlcnBjLCBwZXJmLCBhbmQgYnVkZHkgaGFyZGxv Y2t1cCBkZXRlY3RvcnMKbWFrZSB3YXRjaGRvZ19oYXJkbG9ja3VwX3Byb2JlKCkgc3VjY2Vzc2Z1 bCBieSBpbXBsZW1lbnRpbmcKYW4gYWx0ZXJuYXRpdmUgdG8gdGhlIGRlZmF1bHQgd2VhayBpbXBs ZW1lbnRhdGlvbi4gT25seSwgc3BhcmM2NApyZXBvcnRzIHN1Y2Nlc3MgdmlhIGEgY3JhenkgY2hl Y2sgaW4gdGhlIGRlZmF1bHQgd2VhayBjb25maWd1cmF0aW9uLgoKVGhlIGNoZWNrIGlzIGNyYXp5 IGJlY2F1c2UgaXQgbWFrZXMgdGhlIGRlY2lzaW9uIGJhc2VkIG9uCkNPTkZJR19IQVZFX05NSV9X QVRDSERPRy4gQnV0IHRoZSByZWxhdGVkIGNvZGUgaXMgaW4KYXJjaC9zcGFyYy9rZXJuZWwvbm1p LmMgd2hpY2ggaXMgY29tcGlsZWQgd2hlbiBDT05GSUdfU1BBUkM2NAppcyBlbmFibGVkLgoKVGhl IGNvbm5lY3Rpb24gYmV0d2VlbiBDT05GSUdfSEFWRV9OTUlfV0FUQ0hET0cgYW5kCkNPTkZJR19T UEFSQzY0IGlzIGhpZGRlbiBpbiBhcmNoL3NwYXJjL0tjb25maWcuCgpJdCB3b3VsZCBiZSBtdWNo IG1vcmUgc3RyYWlnaHRmb3J3YXJkIHdoZW4gdGhlIHdlYWsgZnVuY3Rpb24KaXMgaW1wbGVtZW50 ZWQgaW4gYXJjaC9zcGFyYy9rZXJuZWwvbm1pLmMuIEl0IHdpbGwgYmUgY2xlYXIKdGhhdCBwcm9i ZSgpIHdpbGwgc3VjY2VlZCB3aGVuIHRoZSB3YXRjaGRvZyBnZXRzIGluaXRpYWxpemVkLgoKCj4g cG93ZXJwYyBhbHNvIGdldHMgQ09ORklHX0hBVkVfTk1JX1dBVENIRE9HCj4gYXMgYSBzaWRlIGVm ZmVjdCBvZiBzZWxlY3RpbmcgQ09ORklHX0hBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9BUkNIIGJ1 dAo+IGl0IGxvb2tzIGxpa2UgdGhleSBpbXBsZW1lbnQgd2F0Y2hkb2dfaGFyZGxvY2t1cF9wcm9i ZSgpIGFscmVhZHkuIE9oLAo+IGJ1dCBtYXliZSB0aGlzIHdpbGwgZml4IGEgcHJlZXhpc3Rpbmcg KGV4aXN0ZWQgYmVmb3JlIG15IHBhdGNoZXMpCj4gbWlub3IgYnVnLi4uIFVubGVzcyBJJ20gbWlz c2luZyBzb21ldGhpbmcgKGVudGlyZWx5IHBvc3NpYmxlISkgb24KPiBwb3dlcnBjIHRvZGF5IEkg Z3Vlc3MgaWYgeW91IHR1cm4gb2ZmIENPTkZJR19QUENfV0FUQ0hET0cgdGhlbgo+IENPTkZJR19I QVZFX0hBUkRMT0NLVVBfREVURUNUT1JfQVJDSCBhbmQgQ09ORklHX0hBVkVfTk1JX1dBVENIRE9H Cj4gd291bGQgc3RpbGwgYmUgZGVmaW5lZCBhbmQgd2UnZCBlbmQgdXAgcmV0dXJuaW5nIDAgKG5v IGVycm9yKSBmcm9tCj4gd2F0Y2hkb2dfaGFyZGxvY2t1cF9wcm9iZSgpLiBUaGF0IG1lYW5zIHRo YXQgb24gcG93ZXJwYyB0b2RheSBpZiB5b3UKPiB0dXJuIG9mZiBDT05GSUdfUFBDX1dBVENIRE9H IHRoYXQgJy9wcm9jL3N5cy9rZXJuZWwvbm1pX3dhdGNoZG9nJyB3aWxsCj4gc3RpbGwgdGhpbmsg dGhlIHdhdGNoZG9nIGlzIGVuYWJsZWQ/CgpZZWFoLCBpdCBzZWVtcyB0aGF0IHRoaXMgYnVnIHdh cyB0aGVyZS4gQW5kIGl0IHdpbGwgZ2V0IGZpeGVkIHdoZW4KdGhlIGRlZmF1bHQgd2VhayBpbXBs ZW1lbnRhdGlvbiBvZiB3YXRjaGRvZ19oYXJkbG9ja3VwX3Byb2JlKCkKYWx3YXlzIHJldHVybnMg ZmFsc2UuCgpBZ2FpbiwgSSdsbCBsZXQgQW5kcmV3IHRvIGRlY2lkZSB3aGV0aGVyIHRoaXMgc2hv dWxkIGdldCBjbGVhbmVkCmluIHRoaXMgcGF0Y2hzZXQgb3IgbGF0ZXIuIEJ1dCBpdCB3b3VsZCBi ZSBmaW5lIHRvIGZpeCB0aGlzCmFmdGVyIHdlIHNwZW50IHNvIG11Y2ggdGltZSB1bmRlcnN0YW5k aW5nIHRoZSBtZXNzLgoKQmVzdCBSZWdhcmRzLApQZXRyCgpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdAps aW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVh ZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg==